Home Office Data Loss

[beemer007]

Do we know if this is PND or PNC? I've seen both mentioned in different reports (by the same company FFS), and very few journalists seem to realise they're two very separate things (at least for now until NLEDs arrives.. how's that going BTW.. what's that, should have been in service two years ago?)

Apparently it's the PNC

Sent from my SM-G935F using Tapatalk

 

[Listy]

Apparently it's the PNC

Sent from my SM-G935F using Tapatalk

Ouch, that's going to hurt.

 

[Fiona_TG]

I’d loved to have been in the room when some poor sod of a Civil Servant walked into Priti’s office and said “Er, may I have a word?”

Yes, but I am sure he had his scapegoats lined up before knocking on the door.

 

[lokiuk]

Apparently it's the PNC

Sent from my SM-G935F using Tapatalk

Did they outsource it to Capita by any chance?

 

[Fiona_TG]

I’d loved to have been in the room when some poor sod of a Civil Servant walked into Priti’s office and said “Er, may I have a word?”

I wonder if it went anything like this.

(2) The Thick Of It - Massive irretrievable data loss - YouTube

 

[Yokel]

Or was it like this?



Why does Government keep outsourcing IT to Frank Spencer and Co, whilst Basil Fawlty continues to provide project management services? It would not surprise me if some organisations were still using reel to reel tapes.

 

[Tuffty]

How do you know if your one of the lucky/unlucky ones who has been deleted, asking for a friend

 

[anglo]

I’d loved to have been in the room when some poor sod of a Civil Servant walked into Priti’s office and said “Er, may I have a word?”

She wouldn't say much, or she'd get done for bullying

 

[Listy]

To those asking questions about outsourcing. Keep in mind the state and age of PNC, and its younger sister PND.

The PNC interface is archaic as hell! To get an idea what its like go to your BIOS and have a click round that (Similar to operating DOS), you don't, and can't use a mouse. You have to use arrow keys, and command prompts. The basic idea is younger than me by a couple of years, and the majority of it dates from the 1990's.

It desperately needs updating and that was the plan for NLEDS, which was to combine PND and PNC. But the replacement is such a huge task, keep in mind each of the forces has its own IT standards, and software*, That it's over shot its out of service date.
It's also an absolutely colossal amount of data to play with. Consider everyone who has been arrested or convicted has a PNC entry. Due to shitty data management they may have several entries for one person, as no one has yet twigged that they're the same.



*I was part of the dept that crashed TuServ a couple of years ago. Pile of arrsewater that it bloody program is!

 

[RangdoOfArg]

To those asking questions about outsourcing. Keep in mind the state and age of PNC, and its younger sister PND.

The PNC interface is archaic as hell! To get an idea what its like go to your BIOS and have a click round that (Similar to operating DOS), you don't, and can't use a mouse. You have to use arrow keys, and command prompts. The basic idea is younger than me by a couple of years, and the majority of it dates from the 1990's.

It desperately needs updating and that was the plan for NLEDS, which was to combine PND and PNC. But the replacement is such a huge task, keep in mind each of the forces has its own IT standards, and software*, That it's over shot its out of service date.
It's also an absolutely colossal amount of data to play with. Consider everyone who has been arrested or convicted has a PNC entry. Due to shitty data management they may have several entries for one person, as no one has yet twigged that they're the same.



*I was part of the dept that crashed TuServ a couple of years ago. Pile of arrsewater that it bloody program is!

The flip side to its elderly status was, with proper trained users, it was bombproof. Fields for making a nominal record often needed codes, for instance. So idiots could not type in nonsense. The downside was changes had to be pushed through slowly, and due to other pressures, may not occur at all.
Duplicate records are a nightmare.
That said, it is still probably the most comprehensive policing database in the world. If you consider it combines arrest and conviction data with live data on persons, including whether they are missing/wanted or intelligence is required and data on vehicles and stolen plant/equipment, it is one heck of a tool.
As said above, one of the issues is that any data on PNC is owned by the data controller. This is normally the Chief Constable of the force entering the data. There are also other users of PNC, also having a ‘data controller’ role. Unsurprisingly, they all have different ideas of what to enter and what to keep. Bit like herding cats.

 

[Listy]

The flip side to its elderly status was, with proper trained users, it was bombproof. Fields for making a nominal record often needed codes, for instance. So idiots could not type in nonsense. The downside was changes had to be pushed through slowly, and due to other pressures, may not occur at all.
Duplicate records are a nightmare.
That said, it is still probably the most comprehensive policing database in the world. If you consider it combines arrest and conviction data with live data on persons, including whether they are missing/wanted or intelligence is required and data on vehicles and stolen plant/equipment, it is one heck of a tool.
As said above, one of the issues is that any data on PNC is owned by the data controller. This is normally the Chief Constable of the force entering the data. There are also other users of PNC, also having a ‘data controller’ role. Unsurprisingly, they all have different ideas of what to enter and what to keep. Bit like herding cats.

I know. My main knowledge was of NFLMS, so I can only imagine how many times worse PNC is.

 

[RangdoOfArg]

I know. My main knowledge was of NFLMS, so I can only imagine how many times worse PNC is.

I had completely forgotten about NFLMS (and other limited additions). NFLMS in itself adds many hundreds of thousands of records to PNC.
There are lots of people with PNC records without an arrest to their name.
Luckily, in this balls up, they’re safe.

 

[Escape-from-PPRuNe]

Apparently they were "weeding" it when it happened, so this sounds more like either a user error (and a lack of measures to catch or prevent them), a management error, or a conflict of policies about what to keep, than anything to do with the hosting infrastructure.

 

[Listy]

Apparently they were "weeding" it when it happened, so this sounds more like either a user error (and a lack of measures to catch or prevent them), a management error, or a conflict of policies about what to keep, than anything to do with the hosting infrastructure.

Could it be an attempt to cut down on the number of duplications, so to reduce the amount of data that needs to be transferred to the new system?

 

[dogmeat]

Estimates detailing the loss of criminal evidence records by the UK Home Office and the police show the figure leaping from 150,000 to 400,000.

Thank you for creating an account purely to post that nonsense.

Oh and why aren't you using your normal account?

 

[Fiona_TG]

Do you get the impression recovery is not going well? BJ says the loss is unacceptable but at what point does he have to admit we will have to accept it? Will he do a Hirohito announcement, "recovery has not necessarily developed to HO advantage and the nation will have to endure the unendurable".

It's been about a week now, such an ongoing recovery window (especially considering around the clock effort) so perhaps those involved don't have a process or the means to make that happen and might not find one.

Johnson – Outrageous that police crime data has been lost (msn.com)

 

[re-stilly]

What amazes me is that the Opposition ire is not directed at the muppet that pressed OK at least 3 times to confirm the deletion but at Patel ...How in hell is she meant to know what some oik is doing in the IT Dept who has probably been rewarded with a cross posting or a promotion to stop them getting the sack

 

[crow_bag]

Knowing nothing about the system/s in question is it more likely that a data loss of this size is down to operator error I.e. people being told to delete stuff in error, or some sort of batch update that got rid of a lot of stuff in one go?

 

[Fiona_TG]

What amazes me is that the Opposition ire is not directed at the muppet that pressed OK at least 3 times to confirm the deletion but at Patel ...How in hell is she meant to know what some oik is doing in the IT Dept who has probably been rewarded with a cross posting or a promotion to stop them getting the sack

Without the fuller analysis of root cause it is hard to say what happened, but taking the wider view of managing data loss recovery for which processes and tested capability should exist, the question to me is why is it proving a challenge to do so?

We might be able to find some poor bod to blame in due course, but a natural disaster could well have had the same effect, no-one to blame there and that needs planning for. It always concerned me when senior management were more concerned with what might cause an outage than being ready to recover from it.

I would point out that once a disaster has happened the cause is almost irrelevant, it is the impact and related recovery management that should be the focus. If there are concerns about people then (broadly speaking) that is Business Continuity and not a technology matter, the latter seemingly the case in this case.

 

[WightMivvi]

Knowing nothing about the system/s in question is it more likely that a data loss of this size is down to operator error I.e. people being told to delete stuff in error, or some sort of batch update that got rid of a lot of stuff in one go?

As an ex-DBA (database administrator), there could be many causes.

PEBCAK (Problem Exists Between Chair And Keyboard). As others have suggested, a user may have thought there were deleting a few records but had actually selected the whole dataset.

Programming error. New code was not fully tested and went wibble.

Batch or processing error. Data updates not run in the correct order meaning records were overwritten or deleted rather than updated. (It depends on how data changes are applied - do they create a duplicate recordset then delete the original? Do they amend the existing dataset? I’ve seen both.

Migration error. Were they moving data to another server or mainframe and the migration went wrong? This can often happen with “planned maintenance“. For example, some electricity supplies need to be tested and certified every year, meaning anything using that supply has to be turned off. If that something is equipment hosting the data, and you must maintain access to that data for operational reasons, you have to move the data to equipment alternative equipment that will still be powered up.

Virus. As an experiment, I once wrote a sort of virus that put itself into databases then, everytime someone logged on, it picked a record at random and randomly changed one character within that record. It would easily look like a mis-type. However, if no one noticed then, over time, the whole database (and backups) would become garbage.

Backup problem. Backups don’t always work and the backup operators don’t always notice. If that happens, and you something happens that requires a data restore, you’ll suddenly find huge amounts of data are missing / corrupted / out-of-date.

Given that the data is still missing, and everyone is staying tight-lipped about it, I’m leaning towards an initial cause being made worse by a problem with the backups. If the backups (and schedule) were good, they could have restored the data, losing only one day’s data.

Is the Home Secretary at fault? I’d say no. This looks like an operating error by users and / or administrators / contractors. The only way the Home Secretary could be at fault is if she made a deliberate decision to sign off a policy or contract that specifically removed the requirement for regular backups, or which failed to include a service to restore data from backups.