Home Office Data Loss
- Thread starter Blogg
- Start date
MIA
Ouch, that's going to hurt.
Fiona_TG
LE
Yes, but I am sure he had his scapegoats lined up before knocking on the door.
lokiuk
LE
Did they outsource it to Capita by any chance?
Fiona_TG
LE
I wonder if it went anything like this.
(2) The Thick Of It - Massive irretrievable data loss - YouTube
anglo
LE
She wouldn't say much, or she'd get done for bullying
To those asking questions about outsourcing. Keep in mind the state and age of PNC, and its younger sister PND.
The PNC interface is archaic as hell! To get an idea what its like go to your BIOS and have a click round that (Similar to operating DOS), you don't, and can't use a mouse. You have to use arrow keys, and command prompts. The basic idea is younger than me by a couple of years, and the majority of it dates from the 1990's.
It desperately needs updating and that was the plan for NLEDS, which was to combine PND and PNC. But the replacement is such a huge task, keep in mind each of the forces has its own IT standards, and software*, That it's over shot its out of service date.
It's also an absolutely colossal amount of data to play with. Consider everyone who has been arrested or convicted has a PNC entry. Due to shitty data management they may have several entries for one person, as no one has yet twigged that they're the same.
*I was part of the dept that crashed TuServ a couple of years ago. Pile of arrsewater that it bloody program is!
The PNC interface is archaic as hell! To get an idea what its like go to your BIOS and have a click round that (Similar to operating DOS), you don't, and can't use a mouse. You have to use arrow keys, and command prompts. The basic idea is younger than me by a couple of years, and the majority of it dates from the 1990's.
It desperately needs updating and that was the plan for NLEDS, which was to combine PND and PNC. But the replacement is such a huge task, keep in mind each of the forces has its own IT standards, and software*, That it's over shot its out of service date.
It's also an absolutely colossal amount of data to play with. Consider everyone who has been arrested or convicted has a PNC entry. Due to shitty data management they may have several entries for one person, as no one has yet twigged that they're the same.
*I was part of the dept that crashed TuServ a couple of years ago. Pile of arrsewater that it bloody program is!
RangdoOfArg
LE
The flip side to its elderly status was, with proper trained users, it was bombproof. Fields for making a nominal record often needed codes, for instance. So idiots could not type in nonsense. The downside was changes had to be pushed through slowly, and due to other pressures, may not occur at all.
Duplicate records are a nightmare.
That said, it is still probably the most comprehensive policing database in the world. If you consider it combines arrest and conviction data with live data on persons, including whether they are missing/wanted or intelligence is required and data on vehicles and stolen plant/equipment, it is one heck of a tool.
As said above, one of the issues is that any data on PNC is owned by the data controller. This is normally the Chief Constable of the force entering the data. There are also other users of PNC, also having a ‘data controller’ role. Unsurprisingly, they all have different ideas of what to enter and what to keep. Bit like herding cats.
I know. My main knowledge was of NFLMS, so I can only imagine how many times worse PNC is.
RangdoOfArg
LE
I had completely forgotten about NFLMS (and other limited additions). NFLMS in itself adds many hundreds of thousands of records to PNC.
There are lots of people with PNC records without an arrest to their name.
Luckily, in this balls up, they’re safe.
Apparently they were "weeding" it when it happened, so this sounds more like either a user error (and a lack of measures to catch or prevent them), a management error, or a conflict of policies about what to keep, than anything to do with the hosting infrastructure.
Could it be an attempt to cut down on the number of duplications, so to reduce the amount of data that needs to be transferred to the new system?
dogmeat
LE
Thank you for creating an account purely to post that nonsense.
Oh and why aren't you using your normal account?
Fiona_TG
LE
Do you get the impression recovery is not going well? BJ says the loss is unacceptable but at what point does he have to admit we will have to accept it? Will he do a Hirohito announcement, "recovery has not necessarily developed to HO advantage and the nation will have to endure the unendurable".
It's been about a week now, such an ongoing recovery window (especially considering around the clock effort) so perhaps those involved don't have a process or the means to make that happen and might not find one.
Johnson – Outrageous that police crime data has been lost (msn.com)
It's been about a week now, such an ongoing recovery window (especially considering around the clock effort) so perhaps those involved don't have a process or the means to make that happen and might not find one.
Johnson – Outrageous that police crime data has been lost (msn.com)
re-stilly
LE
What amazes me is that the Opposition ire is not directed at the muppet that pressed OK at least 3 times to confirm the deletion but at Patel ...How in hell is she meant to know what some oik is doing in the IT Dept who has probably been rewarded with a cross posting or a promotion to stop them getting the sack
Fiona_TG
LE
Without the fuller analysis of root cause it is hard to say what happened, but taking the wider view of managing data loss recovery for which processes and tested capability should exist, the question to me is why is it proving a challenge to do so?
We might be able to find some poor bod to blame in due course, but a natural disaster could well have had the same effect, no-one to blame there and that needs planning for. It always concerned me when senior management were more concerned with what might cause an outage than being ready to recover from it.
I would point out that once a disaster has happened the cause is almost irrelevant, it is the impact and related recovery management that should be the focus. If there are concerns about people then (broadly speaking) that is Business Continuity and not a technology matter, the latter seemingly the case in this case.
WightMivvi
War Hero
As an ex-DBA (database administrator), there could be many causes.
PEBCAK (Problem Exists Between Chair And Keyboard). As others have suggested, a user may have thought there were deleting a few records but had actually selected the whole dataset.
Programming error. New code was not fully tested and went wibble.
Batch or processing error. Data updates not run in the correct order meaning records were overwritten or deleted rather than updated. (It depends on how data changes are applied - do they create a duplicate recordset then delete the original? Do they amend the existing dataset? I’ve seen both.
Migration error. Were they moving data to another server or mainframe and the migration went wrong? This can often happen with “planned maintenance“. For example, some electricity supplies need to be tested and certified every year, meaning anything using that supply has to be turned off. If that something is equipment hosting the data, and you must maintain access to that data for operational reasons, you have to move the data to equipment alternative equipment that will still be powered up.
Virus. As an experiment, I once wrote a sort of virus that put itself into databases then, everytime someone logged on, it picked a record at random and randomly changed one character within that record. It would easily look like a mis-type. However, if no one noticed then, over time, the whole database (and backups) would become garbage.
Backup problem. Backups don’t always work and the backup operators don’t always notice. If that happens, and you something happens that requires a data restore, you’ll suddenly find huge amounts of data are missing / corrupted / out-of-date.
Given that the data is still missing, and everyone is staying tight-lipped about it, I’m leaning towards an initial cause being made worse by a problem with the backups. If the backups (and schedule) were good, they could have restored the data, losing only one day’s data.
Is the Home Secretary at fault? I’d say no. This looks like an operating error by users and / or administrators / contractors. The only way the Home Secretary could be at fault is if she made a deliberate decision to sign off a policy or contract that specifically removed the requirement for regular backups, or which failed to include a service to restore data from backups.
