So how did we ever (truly) grade a source (of any sort) without comparing what had been provided with what transpired?
We didn't, truly. A finger in the wind was the best description. The "PJHQ Uncertainty Scale" (i.e. 95% = almost certain, 75%-95% = highly likely, etc) was a fiction, because there were never any percentages underpinning it, because there was never any quantitative analysis done. NATO grading (HUMINT style A-F and 1-6) was a fiction mostly based on the opinion of the handler, and barely understood even by them, which is why I saw idiotic grades like A5 (
this guy is always right but he's definitely wrong) or E1 (
this guy is totally unreliable but he's definitely right). Thus the traditional joke that HUMINT was only ever A1 (definitely reliable and right) or F6 (no idea and no idea).
All of these systems, and the individuals in them,
strongly resisted changing or evolving any of this, because they were (explicitly, in private conversations) afraid either of being held to their assessments or of having to prove their working when they could not.
Theoretically, all of these individual specialist sources should have gone to a generalist (OPINT / all-source) analysis cell and only
there would it have made sense to grade them or do quantitative analysis, but that process was the worst of all, the training was cursory (a few days), abysmal sub-GCSE analysis stuff, and so it basically never happened. Those were the cells I suggested should use forecasting techniques.
If you look at the touted growth in "intelligence capability" during TELIC/HERRICK, in either size or cost terms, it's actually no such thing. The growth was in
information collection (i.e. things that deliver reliable information, like images or signals). The Int Corps latched onto those, but never did it actually run the collection (that was OGD, RAF, etc) nor did it grow commiserate to the collection capability. Those capabilities which were predominantly Int Corps (HUMINT) didn't grow at anywhere like the same rate. This also accorded with the behaviour of commanders - what they demanded and used was hard information: FMV, IMINT, biometrics, SIGINT (unfortunately overestimating how "hard" the latter two were at times). The fact that a few Int Corps bods came attached with all of those to manage delivery and make powerpoints was incidental. Commanders accepted that the shark they wanted had a remora on it, but they still wanted the shark. Of course all this was spun by the Intelligence Corps to be an objective demonstration of their value, and suggest that what commanders had
really been asking for was a remora in a green hat.
I'll take your word for it. Intelligence support to the Green Army has historically been patchy, especially as the customer base remains - as far as I can tell - wedded to intuition and personal judgement and tends not to understand, never mind use, any form of predictive analysis. There's perhaps a vicious circle, where the customer base doesn't believe what it's told, or perhaps has unrealistic expectations and is thus constantly disappointed.
The problem is that today, "intelligence support to the Green Army" is
all there is. After they drained the old specialisms of independence, centralised all training to OPMI and formed 1 MI Bde / Bns, almost every Int Corps soldier came under that banner, including their training, standards, etc. A few escaped for a short period in some disciplines, but they either transferred and never came back (from RAF/IMINT or SIGINT), or were thrown back into the green Army Int Corps (from DSF etc) with all the problems above. The result has been to reinforce the lowest common denominator.
EW / SIGINT as a good example, is no longer the separate walled garden it once was (unless you're a linguist), and the officers and seniors have just as many incompetents. This was the case pre-2013, and structural changes mean it will have gotten worse since then. As an example, in 2012 I couldn't find a single Int Corps bod
on tour in operational jobs who could understand or perform a CRA, including the LCpl analysts responsible for that material. I had to learn and train my section to do it myself. The answer from the OISG (all-source / GS cell) was "that was the job of the geeks" elsewhere. They were just passive recipients of the answers. Which was all fine, except in the important minority of cases where the CRA was wrong, and they had no ability to check the working. But in many cases they effectively 100% relied on that material for targeting.
I prompted some angry conversations about that (targeting based on erroneous reporting that they couldn't understand was wrong because they didn't bother to learn the analysis technique), and put it in a POR. Everyone was totally blase about it in response. A shrug and "well, that's intelligence" was an actual reply from someone responsible for a wrong target (that they'd gone on). The police do this in the UK and it's an inquiry...
The whole edifice has been hollowed out, and once that's been the case for a couple of decades, it's very difficult to find anyone remaining who knows what good looks like to improve it again. Let alone able to push their way through the viscous layer of incompetent management who sense that any such reform will both reveal their historical incompetence, and put them out to pasture.