The Repeated Failure Of The US And UK Governments' 'Add More Hay' Approach To Surveillance
from the sipping-from-a-firehose dept
Recently we wrote about a UK Parliamentary report absolving the UK spy agencies of any responsibility for the failure to stop the killing of a British soldier last year. Significantly, one explanation given for the fact that the UK’s MI5 undervalued the threat, despite investigating the men responsible several times, was that it has several thousand suspects under surveillance at any one time, and so it was beyond its capabilities to follow all leads thoroughly.
Of course, that is a consequence of the “needle in a haystack” approach that the US and UK agencies have adopted: collect as much information as possible in the hope that somehow it will be possible to sift through all the irrelevant hay to find the needle. But as an important piece by Coleen Rowley in the Guardian points out, this is not the first time that a “failure to connect the dots” from information to hand resulted in missed opportunities to stop attacks:
as an FBI whistleblower and witness for several US official inquiries into 9/11 intelligence failures, I fear that terrorists will succeed in carrying out future attacks — not despite the massive collect-it-all, dragnet approach to intelligence implemented since 9/11, but because of it. This approach has made terrorist activity more difficult to spot and prevent.
She reminds us:
The common refrain back then was that, pre 9/11, intelligence had been flowing so fast and furiously, it was like a fire hose, “and you can?t get a sip from a fire hose”. Intelligence such as the Phoenix memo — which warned in July 2001 that terrorist suspects had been in flight schools and urgently requested further investigation — went unread.
She details other instances of recent intelligence failures where having too much information meant that key leads were buried, and opportunities to stop attacks lost. In other words, US and UK agencies have over ten years of bad experiences with the “collect it all” and “needle in a haystack” approach to intelligence gathering, and yet it remains their principal response to successive attacks and continuing failures. Rowley concludes:
After Edward Snowden described just how massive and irrelevant the US and UK monitoring had become, people started to grasp the significance of the saying: “If you?re looking for a needle in a haystack, how does it help to add hay?”
Of course, she’s not the first person to make this observation — for example, Techdirt pointed this out over a year ago. But as someone who has worked for the FBI, and been privy to many secret details of surveillance operations, Rowley speaks with a special authority. Her article in the Guardian is well-worth reading — especially by all those who continue to advocate the disproportionate and ineffective “add more hay” approach to keeping the public safe.
Follow me @glynmoody on Twitter or identi.ca, and +glynmoody on Google+
Filed Under: gchq, haystacks, information overload, needles, nsa, surveillance
Comments on “The Repeated Failure Of The US And UK Governments' 'Add More Hay' Approach To Surveillance”
It doesn’t have anything to do with keeping the public “safe”, only their jobs and all that money.
Another point of view
If your job is dependent on finding the needle in the haystack – but you don’t know what haystack it is in, then it makes perfect sense to combine every haystack in the world into a single haystack. By definition, you are 100% guaranteed that the needle you are looking for is now, in fact, in your haystack.
You can then spend the rest of your career looking through the haystack – as long as you are “working on it”, you are doing your job.
The odds of finding are something just less than zero divided by infinity – but that’s totally irrelevant. A high paying job has been saved, a massively expensive data center has been justified, and the Bureaucrats love you.
No matter what happens, you can find the data to prove that you knew it would.
uh.. wait..somethings wrong there…
Re: Another point of view
You have made one false assumption, that the needle is actually in a hay stack when it might be in a pile of leaves. The spy agencies are fixated on electronic communications, while any sensible terrorist will be using mail or courier, and cash.
So what they are doing isn’t working, and yet they keep doing it. Sounds as if the Russians and the Chinese are the ones to emulate now. Drop all of your ordinance on my position, war is hell, and we are not the good guys anymore.
Needle analogy
This is what always bothered me about the “needle in a haystack” analogy. On one hand, the typical claim went, “in order to find a needle in a hay stack, we need the whole stack.”
But really, what’s more difficult: finding a needle in a haystack or finding a needle in Nebraska?
Re: Clandestine isn't silent and black, only as quite as the background noise.
But really, what’s more difficult: finding a needle in a haystack or finding a needle in Nebraska?
To paraphrase Sherlock Holmes in The Sign of Four, finding a needle in a pile of identical needles.
Part of the problem is not just that there’s tons and tons and tons of intelligence, but that there’s bunches of background noise that is identical to that intelligence.
And given now that the people of the US are paranoid of its government and talking angrily of revolution, the background noise presents false positives even more.
But yeah, given how much has been invested into dragnet surveillance and given its poor record of actually providing useful data successfully filtered out, they’re going to have to continue to expand the purpose of the program in order to justify the expense, disregarding the damage it does to the state’s reputation, and disregarding the rights upon which the program infringes.
I am beginning to think that these agencies couldn’t find a needle in a needle stack. Are they incapable of understanding the consequences of false positives?
911 - failure to recognize hijacked planes
These people don’t even know why no government officials intercepted the hijacked 911 planes, even after terrorists attacked the World Trade Center using the first plane.
This information can be found at your local laundromat.
Re: 911 - failure to recognize hijacked planes
I thought the buzz was the military was ordered to stand down and let the planes keep going
Re: Re: The 911 planes and the military.
At the time, the generals only realized after the second attack that the hijacked planes were going to be used as bombs, themselves.
The problem with shooting down the planes is that doing so would take direct responsibility for ending the lives of hundreds of civilians, and even a general has difficulty making a decision like that, even when doing so would allegedly save the lives of magnitudes more.
We were more used to the plane hijackings of the 70s which involved landing and exchanging hostages. To be fair, due to a certain complacency and lack of imagination, we didn’t anticipate the planes being used as bombs, themselves, so we had a playbook that didn’t apply to the 9/11 attacks.
On the other hand, they’ve essentially ruined it for future attacks with airplanes. So long as 9/11 is fresh in our minds, passengers are willing to bum rush anyone who remotely looks like a terrorist, and generals now would shoot down a plane and label as “heroes” and “patriots” all the passengers sacrificed, rather than see another airplane fuel tank be used as a guided bomb.
Re: Re: Re: The 911 planes and the military.
“…passengers are willing to bum rush anyone who remotely looks like a terrorist…”
Meanwhile, those same passengers are having their nail trimmers confiscated to ensure they are as unprepared to do so as possible. After all, the less able the people are to protect themselves, the more they have to depend on bigger government to do it for them.
Re: Re: Re:2 Security Theater.
It’s been made clear long ago that the TSA actually makes airports less secure (and certainly less safe) than before 9/11. But we do love our tiger-repellant rocks.airports less secure (and certainly less safe) than before 9/11. (And that the TSA, by encouraging people to drive more and fly less has indirectly cuased more deaths by now than the original attacks)
But we do love our bear patrols and tiger-repellant rocks
Re: Re: Re: The 911 planes and the military.
“The problem with shooting down the planes is that doing so would take direct responsibility for ending the lives of hundreds of civilians, and even a general has difficulty making a decision like that, even when doing so would allegedly save the lives of magnitudes more.”
Perhaps I’m just too cynical, but based on on the military operations over the past couple of decades, I thought that generals couldn’t care less about civilian deaths except insofar as they create a PR problem.
“we didn’t anticipate the planes being used as bombs, themselves”
If by “we” you mean the military side of the federal government, then this was totally anticipated. It was even a wargame scenario.
Re: Re: Re:2 Hugeness
we didn’t anticipate the planes being used as bombs, themselves
If by “we” you mean the military side of the federal government, then this was totally anticipated. It was even a wargame scenario.
The military side of the federal government is huge, and totally does not engage in any kind of collective groupthink. So yeah, by we I meant our air-traffic control sector and the specific military subdivisions that connect with them to identify and respond to airborne threats.
I’m sure that we had theorists that imagined large-scale kamikaze attacks, but their notions never trickled down to the agents in charge of responding to realworld events.
Now that I think of it, I suspect that the current intelligence / surveillance sector suffers from the same problem now. It’s huge and it takes a ton of people to analyze a given bit of intel, determine whether it is legitimate and relevant and process its meaning.
Re: Re: Re:3 Hugeness
“The military side of the federal government is huge”
True. I should clarify: I meant in the upper echelons of the military structure.
Re: 911 - failure to recognize hijacked planes
There’s a lot of questions on this failure.
http://www.historycommons.org/context.jsp?item=a930langleylaunch
The alert fighters learn of 9/11 by phone call from fiance and not from a tower alert?
Yet before 9/11 occurred there were alerts on these terrorists, including one taking flight training stopped and ticketed for no drivers license in Florida. There is so much here that was missed even though they had ample warning. One agency not sharing info with another does not improve sorting though the haystack.
Back in the old days...
People used to sign Usenet posts with a line that had “Spook Fodder”. Keywords that might get tagged as “look at this” for automated systems scanning posts. With all the data being hoovered, they have more “look at this” than they can look at already. Any additional spook fodder will continue to make their data less relevant to its purpose.
Re: Back in the old days...
inb4 botnets flood the NSA with terroristy emails.
It is all about their image
Politicians: They don’t want to be the ones blamed if another attack gets through. The only thing they can do as a collective body is to throw money at the “problem” and make asinine speeches about something they know little to nothing about.
They happily spend taxpayer money like a drunken sailor on shore leave.
Thanks to them the various government agencies are spying on everything and everyone while proclaiming they have special secret laws that allow them to completely ignore the Constitution.
Politicians sworn to uphold and defend the Constitution stand by and do nothing.
Meanwhile everyone become less safe as the spy agencies drown in data they cannot possibly process.
Lobbyists: Their livelihood is tied to getting politicians to make laws and award contracts that favor their particular industry. They are under no obligation whatsoever to tell the truth… so they whisper terrifying lies into the ears of the politicians who in turn recoil in horror at the prospect of not appearing proactive in the fight against terror.
Thus more bad laws are created and wasteful contracts are awarded.
Corporations: The companies aligned towards information gathering and processing get huge contracts to help in the fight against the elusive terrorist. They have the ability to gather every form of electronic emission on the planet.
Of course there is no way to process all of that data, but never mind as the corporations are raking in billions of dollars as a result of their lobbyists appealing to the politicians to do something before another disaster strikes (not to mention the hefty campaign contributions that are handed out to those same politicians to “help” them reach the proper decisions).
American citizens are the only ones not benefiting from all of these idiotic actions by corporations, lobbyists, and the politicians.
We are far less safe and our Constitutional rights are becoming vague memories.
It seems as if there is no sane or honest person left in politics. If some lobbying group decided to pay politicians to create a law making Bozo the clown the next Czar against terrorists, you can bet it would get done.
Politicians don’t need any common sense, knowledge about the laws they are asked to create, or oversight of the intelligence agencies monsters they have created. They just need to appear to do something against terror (and to collect the campaign contributions they really care about).
Nothing else matters…
Re: The Cover-Your-Own-Ass Society
This is going to kill the United States, or at least turn it into a dark reflection of the republic it was for a while.
Re: It is all about their image
Sure they are! You’ve got massive spending on security theatre, patriotic whistleblowers running for their lives, seventeen overlapping anti-terror spy agencies, and you’re starting to look as bad as the Soviets and ChiComs (and Savak and Stasi) on their worst day. You’ve got cops harassing people walking down the street with their hands in their pockets. Your military has never been as potentially destructive, and you’re almost constantly at war with somebody. The war on drugs has never been as fever pitched. Prohibition was a drop in the bucket compared to what you have now.
American (USA) citizens are the only ones who can stop this too, as well being the ones to blame for it all. Apparently, they don’t care to. We get the gov’t we deserve.
If that’s not entertainment, I don’t know what is.
Re: Re: It is all about their image
American (USA) citizens are the only ones who can stop this too, as well being the ones to blame for it all. Apparently, they don’t care to. We get the gov’t we deserve.
You’re going to love one of the upcoming articles then, but to sum it up, despite overwhelming bi-partisan support, an amendment to a bill, designed to reign in the NSA was shelved by the House leadership and Congress is basically making it impossible to add it back in.
When a handful or less can override the majority like that when it comes to dealing with the NSA, the idea that it’s the public’s fault the government is out of control gets more than a little unbelievable.
The public told their reps, ‘Do something about the NSA’, the reps attempted to do something, only to have their efforts removed and blocked by a group of NSA supporters in the right/wrong(depending on which side you’re on) positions.
Re: Re: Re: It is all about their image
USA should take a look at what’s happening in Canada recently. The elected reps in parliament and provincial legislatures (a la state houses) are fighting their leadership, telling them their first duty is to represent those who elected them, not the party leaders.
Can parties in the US impeach their own caucus leaders? Wouldn’t that be fun to watch?
Re: Re: The government we deserve.
That’s like saying that a mouse eaten by a cat got the fate that it deserved.
A cat is faster than the mouse, deadlier than the mouse and can hear, see and smell better than the mouse. The mouse as an individual has no advantages on the cat. (Get as a species, the mouse wins because it by far outbreeds the cat. There is much more mouse-mass on earth than there is cat-mass.)
Democracy isn’t the government we deserve it’s the best form of government we could come up with so far, that distributes power the most widely without completely collapsing. American democracy is a democracy after it’s festered to corruption beyond reconciliation or reform (reform isn’t impossible, but an improvement rate that outpaces the rate of corruption isn’t likely) so ultimately the US is going to end in a lot of blood.
You can’t tell us this is the government we deserve anymore than you can tell the serfs that Joffrey is the king you deserve. (Or John Lackland, if you’re looking for a historical example).
We’ve suffered enough to know that centralized power sucks. And we’ve bled enough that I think we’ve earned our utopian regime many times over. But nature doesn’t care how much blood is shed. Good governance is not purchased, but rather it is invented and refined. And this takes time, and is hindered by corruption and government failure.
To say that we deserve the government we have is to condemn us as a species: occasionally we rise to the grace of rationalism, but most of the time we are victims of our own fear and greed, and for that crime, we are likely to die out having never colonized other worlds, let alone the stars so that our species could survive the long game.
Re: Re: Re: The government we deserve.
Tom & Jerry taught us that cats can’t fit into mouseholes. However, I agree, but the bolded part is what’s important. Mice by nature don’t value the individual. They choose a different path, the same one as fish in a school. Cats aren’t pack animals. They’re individual predators. Different strokes, for different folks, and it’s worked out fairly well for both. They value different outcomes based on their nature.
Not true. I’m sure we can do better than our present iteration of democracy. It doesn’t work. Perhaps given the chance to refine it, we could improve it and make democracy work. Or, if we try hard enough perhaps something out there as yet undiscovered would be better than democracy.
If we can keep from killing us all, or getting ourselves all killed, there’s still hope. We’re not all stupid, we’re not all lazy, and we’re not all excessively greedy, and if enough of us keep our eyes on the prize, we may yet come up with a solution that works for everyone.
What we’re doing now doesn’t work for anyone, rich and poor, powerful or weak. I think we are capable of figuring out what we’re doing wrong. We’ve plenty of indications in many areas. Discovering our errors, then successfully implementing the fixed solution won’t be easy, but that’s what needs to be done.
I’ll start it off. Authoritarianism doesn’t work. We’re too much like cats. Communism and socialism don’t work for the same reason. Elected representational government doesn’t work for the same reason. Free market economies don’t appear to work, or we as a species can’t figure out how to make them work, or we don’t want them to work.
Where do we go from there?
Re: Re: Re:2 The capable mouse.
I’m sure we can do better than our present iteration of democracy.
Some of us can consider better methods than our present iteration of democracy. I think upgrading our electorate system to something other than first-past-the-post might make for a valuable improvement.
But good luck getting that implemented on a national scale here in the US.
Imagine tweaks on the US system all you want, you’re not going to affect them without being a large corporation who can buy off both sides of the party line. Telling us that we get the government we deserve is telling us it’s our fault for not being that.
Let’s reverse the notion: Can we not, then, blame you for failing to manage the cats? Obviously, you are a capable mouse. Fix our government.
Perhaps they had the information, knew what was going to happen, and let it happen to justify more laws being passed that allow them to circumvent citizens rights so they can gather more information on people.
Collect ALL the haystacks!
Of course, she’s not the first person to make this observation — for example, Techdirt pointed this out over a year ago.
Even earlier 😉
Weak analogy but let's run with it
Needles and haystacks may be a good illustration for how tedious it can be to manually search for something. However, if you insist on using that model, the sheer computing power that has enabled modern communications may also be employed like a giant electromagnet and “hay flailer” to pull those needles out with, well, inhuman efficiency.
Of course, this is a highly unreliable process for fighting terrorism. Again, the analogy might be finding a particular style needle that is only made one or two times, then tossed into the haystack where there are huge numbers of other needles. Even if you can suck every needle out, the odd needles look a lot like the regular ones, and now you are back to manual labor to identify them among all the others. By the time you have a process for finding that odd needle, your process is useless. Probably. But you still have to keep checking, just in case.
However, the power to automatically sift through enormous amounts of information is a highly efficient and relatively reliable method for targeting large numbers of “enemies of the state”, since now we’re looking for large batches of similar needles and we can “tune” our magnet accordingly.
It is completely naive to think that any established power base, whether in Syria, China, or a “democratic” country can resist a deepening addiction to this drug of information abuse once it is available. The idea of a responsible surveillance state would be as ludicrous as a ‘Crystal meth, use it responsibly’ slogan in fighting drug abuse.
Re: In spookspeak there's a term called "Light Cover"
That’s the guy in the embassy who everyone knows is a CIA agent (or a KGB agent, respectively), even though he never admits it. He may even sport a shifty spooky demeanor and trench-coat so that he totally looks like a spy.
Such a guy serves two functions.
One is, as a magnet for those civilians who need to contact the CIA. They would get voluntary intel all the time. The CIA doesn’t put out its phone number, exactly, since it’s not really an investigative service in the way the FBI is. But in those cases that someone wants to help things along, he’s the guy you contact.
The other is to distract his rivals while the real operators do their work. He acts suspicious as hell, drives around, walks in the park, may even plant a false dead drop somewhere while picking up dry-cleaning and collecting mail from secondary postboxes. In the meantime, all the tails are following him impatiently looking for an activity for which they can arrest him.
“Light Cover” agents worked to throw off the other side. Both sides used them. They were very annoying to the tails, but you had to follow him with enough guys that you don’t lose him.
Adding more hay doesn’t help prevent terrorism, but it does help economic espionage, political blackmail, and mass surveillance in general. You know, the things mass surveillance is really about.
‘In other words, US and UK agencies have over ten years of bad experiences with the “collect it all” and “needle in a haystack” approach to intelligence gathering’
i think the answer is straight forward enough. these ‘intelligence’ agencies dont want to catch terrorists. what they want to do is blanket surveillance on citizens, more than anything, so as to know what the public is going to do in retaliation for the next bad law the government tries to bring in, or the next civil right it wants to remove (so as to keep the surveillance going!) and if, whilst doing all this surveillance stuff, they happen to come across, purely by luck, information relating to terrorists or terrorist plots, then they have bonuses that can be used when spinning the next crock of bullshit to the ever willing politicians who give the go ahead, after getting some ‘encouragement’ on the way!! with a setup like that, who needs terrorists anyway?
FOI request sent to a large variety of three letter agencies: What does hay look like?
Responses were varied, well actually two variables. One agency sent a 47 page document that describes what cows eat. The rest sent “No documents are responsive to your request” notices.
The fundamental mistake is believing: Information = Intelligence
As someone who once tossed the car keys in the trash while cleaning out the car at the car wash. I can assure you it’s a lot easier to find something, if ‘Intelligence’ is the thing you use to limit your search to only the places it’s likely to be found.