Skynet, But For Welfare: Automating Social Services Is Killing People

from the our-hands-are-clean-said-the-gov't-since-it's-just-robots-doing-the-work dept

We’ve talked before about the over-reliance on tech to do certain jobs that cannot be simplified to the sum of mathematical parts. The criminal justice system is starting to turn over sentencing to algorithms — something that seems like the smart thing to do but removes judicial and prosecutorial discretion from the mix, leaving defendants with the unpalatable option of challenging software they’re never allowed to examine.

Police departments are also moving towards predictive policing. Relying on historical data, cops are hoping to head off future crimes by allocating resources to areas where crime appears to be more likely to be committed. Sounds good on paper, but in reality, all it does is reinforce biases and push law enforcement to treat everyone in targeted areas as criminals. If the data being fed in reflects biased policing, crunching the numbers even harder isn’t going to erase that. It’s only going to reinforce it. And, again, suspected criminals aren’t able to access the data or software that puts them in law enforcement’s crosshairs.

A certain amount of automation is expected as government agencies seek to streamline public services. The problem isn’t necessarily the tech. It’s the removal of human interaction. As has been stated here frequently, moderation at scale is impossible. So is automated governing. Automated processes are as prone to failure as the people overseeing them. But when you decide software is going to do almost all of the work, those who need the assistance of other humans most are cut out of the loop.

Citizens looking for government assistance have grown accustomed to jumping through red tape hoops. Now, the hoops are inaccessible, but still must be jumped through. The most marginalized members of society are given URLs instead of contact names and numbers when many of them have no reliable access to the internet or a computer. A new series by The Guardian shows the human cost of going paperless. It’s happening all over the world, and it’s literally killing people.

The most disturbing story comes from Dumka in India. Here, we learn of the horrifying human impact that has befallen families as a result of Aadhaar, a 12-digit unique identification number that the Indian government has issued to all residents in the world’s largest biometric experiment.

Motka Manjhi paid the ultimate price when the computer glitched and his thumbprint – his key into Aadhaar – went unrecognised. His subsistence rations were stopped, he was forced to skip meals and he grew thin. On 22 May, he collapsed outside his home and died. His family is convinced it was starvation.

That’s the worst case scenario. But there is plenty of ugliness in between. Governments aren’t looking to defense budgets or law enforcement agencies to make cuts. Instead, they’re adding to their bottom lines by pursuing citizens they think have screwed the government. Another process being automated is governments’ attempts to collect alleged overpayments of social services funds. There’s a statute of limitations on most debt, but software is being used to resurrect ancient debt governments feel they’re owed. This is resulting in the destruction of people’s lives and finances as they find themselves unable to challenge these automated determinations.

In Illinois, the Guardian has found that state and federal governments have joined forces to demand that welfare recipients repay “overpayments” stretching back in some cases 30 years. This system of “zombie debt”, weaponized through technology, is invoking fear and hardship among society’s most vulnerable.

As one recipient described it: “You owe what you have eaten.”

It’s not just “zombie debt.” It’s also “robodebt.” The determinations of owed debt are made by automated processes. The collection process is also automated, separating those suddenly facing possibly undeserved clawbacks from the human assistance they need to determine whether or not the claim is valid. Bureaucracies have always been faceless. With the addition of cold calculations, they’ve weaponized this facelessness to deter citizens from pushing back against a decision-making process composed of 1s and 0s. This, too, is linked to a rising human cost.

Compare and contrast the public statements with reality. Here’s how the UK government is pitching its transition to automated governance:

“We are striking the right balance between having a compassionate safety net on which we spend £95bn, and creating a digital service that suits the way most people use technology,” said a DWP [Department for Work and Pensions] spokesperson. “Automation means we are improving accuracy, speeding up our service and freeing up colleagues’ time so they can support the people who need it most.”

If this sounds like a positive development, it’s only because you haven’t seen it in action.

[C]laimants have warned the existing automation in UC’s “digital by default” system has already driven some to hunger, breakdown and even attempted suicide. One described the online process as a “Kafka-like carousel”, another as “hostile” and yet another as a “form of torture”. Several said civil servants already appeared to be ruled by computer algorithms, unable to contradict their verdicts.

The same thing is going on in Australia. The government’s embrace of “austerity,” combined with its fondness for automating social services and debt collection processes has resulted in a world of hurt for the many Australians unable to challenge automated determinations or connect with actual humans willing to help them through the process. Asher Wolf’s Twitter feed is an invaluable resource for these issues, detailing the fallout of automated social assistance programs all around the world.

It’s not that all automation is bad. It’s that automation without strict oversight or human control isn’t making anything better for the most vulnerable members of society. When governments pay millions to let machines make decisions affecting humans, humans almost always seem to come out on the losing end. Legislators may proudly display charts showing incremental gains in efficiency, but few are willing to discuss the constituents they’ve sacrificed on the altar of automation. No system is perfect, but one that relies more on math than human discretion isn’t an improvement.

Filed Under: , , , , , , ,

Rate this comment as insightful
Rate this comment as funny
You have rated this comment as insightful
You have rated this comment as funny
Flag this comment as abusive/trolling/spam
You have flagged this comment
The first word has already been claimed
The last word has already been claimed
Insightful Lightbulb icon Funny Laughing icon Abusive/trolling/spam Flag icon Insightful badge Lightbulb icon Funny badge Laughing icon Comments icon

Comments on “Skynet, But For Welfare: Automating Social Services Is Killing People”

Subscribe: RSS Leave a comment
75 Comments
Anonymoussays:

Re:

Says a lot about you that you are constantly jumping to conclusions about people.

It’s called a metaphor, Stephen. The herd is society. The migration is social progress. Do I need to break it down any simpler for you? We’re all animals, or have you forgotten? Sucks to be a straggler, but the herd has got places to be and things to do, and it can’t stop and wait for everyone to catch up.

What is happening is a natural, and predictable, part of social progress. It’s not going to stop, it’s not going to slow down. Welcome to the real world, Stephen. You might find it a little uncomfortable.

Anonymoussays:

Re: Re:

"It’s called a metaphor"

  • Albeit, a bad one.

"The herd is society"

  • Mooooooo.

"We’re all animals"

  • Yes, by definition, but we do not have to act like it now do we?

"Sucks to be a straggler, but the herd has got places to be and things to do, and it can’t stop and wait for everyone to catch up."

  • and it is this type of shitty attitude in places of power and influence that result in the fucked mess that we now are in. Stragglers as you call them are not falling behind due to being lazy but that’s what you and others think – no? People are not a commodity, only a slaver thinks like that.

"What is happening is a natural, and predictable, part of social progress."

  • Progress .. yeah whatever wanna you call it. The real world would like to have a chat with you and those who think in a like manner. Those at the top, across the planet, are full of shit and you are enabling their excuses – way to go. Those who are less fortunate (not billionaires) are continually subjected to uncomfortable situations, by those who are fortunate – but they are not supposed to complain – because The Chosen One does not like it. Poor boy.
Anonymoussays:

Re: Re: Re:

Yes, we do have to act like animals. That’s what we are. Everything we do is the act of an animal. Don’t presume to attach meaning to my very simple metaphor to fit your own political narrative.

Animals become stragglers due to circumstances beyond their control, they don’t choose to separate from the herd. They don’t fall behind because they are lazy, where do you even get that from? The other animals also don’t leave them behind out of indifference or hatred, which seem to be the only motivators you understand.

How does that self-righteousness of yours manifest outside of the Internet? Do you do anything useful with it?

Anonymoussays:

Re:

Have you seen any Adam Curtis docs? The Trap talks about how performance targets became all important in an increasingly business-style management of social services.
HyperNormalisation also touches on how technology has done the opposite in many ways from the utopian vision that created it, so that instead of liberating people it actually trapped them.

Anonymoussays:

Really it isn’t the automation – it is a convenient excuse for the callousness and outright hatred deliberately embedded within all of the processes by design. The failure state could have defaulted to the old system and still saved resources or made the most generous assumptions – instead they outright chose evil like trying to bill 30 years back making a mockery of due process in bueracratic paralaw.

The reason I say this is not only to put the blame where it belongs but point out that things can actually be done right to aid at lower costs responding quicker.

Stephen T. Stonesays:

Any algorithm you could make will always have human biases within them, for they are designed by humans. Even an algorithm made with the intent to do good can be subverted by the inherent, unconscious biases of its creator(s). To wit: A medical algorithm designed with a seemingly race-blind metric resulted in racially disparate outcomes.

That Anonymous Cowardsays:

And yet they will manage to point out how much they spent on theses shitty systems as more evidence of how the poor just want even more.

Its trendy blaming the poor for being poor…
Why else would someone think that someone on welfare could clear up alleged 30 yrs of debt b/c the gov failed to check their math??
Your life doesn’t suck enough, we’re gonna add more debt & interest… we might even balance our budget.

Its easier to abuse the poor until you hit that point when there aren’t enough of them to blame or round up b/c your indifference finally killed them. They don’t vote enough to matter & definitely don’t make millions in contributions to keep law makers in office to keep people focused on the poor & not those who don’t want to pay their fair share.

When the rich pay less in taxes than people on welfare… perhaps the flaw isn’t the fault of the poor.

That One Guysays:

Re:

Its trendy blaming the poor for being poor…

Besides making for an easy target thanks to lacking the time and money to really fight back, part of that is likely due to the toxic flip side of the idea that ‘anyone can be successful if they just try hard enough‘, in that if someone isn’t successful then it’s not because of bad luck, a lack of opportunities or some other factor outside the person’s control it’s because they’re lazy and aren’t trying.

‘Work hard enough and anyone can be successful’ might provide incentive for some people to put that extra little effort needed into a project/job, but it comes with a nasty corollary that can all too easily be used to victimize people by the unscrupulous or naive.

Anonymoussays:

This sort of thing is happening almost everywhere. The aim isn’t so much to hit the most vulnerable members of society, in my opinion, it’s to hit everyone who isn’t a member of the various governments, the opposition politicians and everyone of their friends, plus all members of the elite, the famous and rich so they can continue to reap the benefits they have been and have gotten used to. The only wat that can happen is to target everyone outside the groups above. The other option is to get some idiot to start another world war or introduce a new superbug! Either way, the group mentioned will stay safe and keep getting rich while the rest of us perish. The big question then is what will happen to them when there’s no one left to keep giving them what they think they should have? Interesting.

Anonymoussays:

Re:

The other option is to get some idiot to start another world war or introduce a new superbug

Population reduction is never the goal of the elite. Franchises like "The Purge" have got it exactly backwards. Think about it: If the "unwashed masses" even come close to equaling the population among the elite, who will pay to keep the elite furnished with new yachts? No, the whole system works because the 99% pay the debts of the 1%. You touch on this later in your comment with "when there’s no one left to keep giving them what they think they should have". They know this already. That’s why population reduction is never going to be their goal.

Ninjasays:

Legislators may proudly display charts showing incremental gains in efficiency, but few are willing to discuss the constituents they’ve sacrificed on the altar of automation.

This is basically one of the main issues pretty much everywhere nowadays. Politicians and the elite pretty much ignore all the rest as humans. This complete lack of concern, empathy with the human aspects of all social interactions is pretty much driving everybody towards Chile. At some point people get fed up. And things go to hell.

Anonymoussays:

Zombie debt

It looks like the statute of limitations among the states is predominately single digit. How do they intend to defend their egregious overstep?
Then there is the disturbing news that some debtors are attempting to make the offspring pay their parents debts.
These lawless asses bellow law ‘n order but only for others not themselves.

Anonymoussays:

Re: Zombie debt

State and Federal governments as well as Municipalities have no statute of limitations for collection on debt contracts, they don’t even have to follow proper debt collection practices as they are exempt from that as well. Do not try to dismiss it in court, they will point to the expanded text and then you will lose after the judge throws his hands up and says "I am sorry, I have never seen this before but I have to rule in favor of the State in this case."

Anonymoussays:

Re: Automation is great...

Automation can’t make exceptions for unforeseen circumstances or adapt to a change in circumstance without significant oversight. That’s less important for email delivery than it is for food, water, housing and medical services.

Until automation can simulate a human level intelligence, it’s not appropriate for most life or death circumstances.

Anonymoussays:

The Trump Network!

You all seem to be missing the point. There is a network that is taking over the world right now. Trump’s Network! Yes, that’s right. THE NETWORKED ARMY OF TRUMP!

Count the people who come to his rallies. Each and every state will give him an audience that will fill arenas of ANY SIZE! MORE AND MORE PEOPLE WHO LOVE TRUMP! Do you get it? Do you understand? There is an ARMY behind Trump! Really! Guns and Everything!

Who is behind gay boy or Pocahontas? Who? Besides the leftist idiots who live at the expense of others in places like NYC, who? Masked idiots called Anti-Fa that gang up on pale white journalists? Is there an ARMY of those guys? PUSSIES!

I THINK NOT! Trump is the COMMANDER IN CHIEF! Of the TRUMP ARMY! ARMED AND DANGEROUS AND READY TO CALL BALLS AND STRIKES AS WE SEE THEM! AND LIVE WITH THE CONSEQUENCES!

PLAY BALL!

Skynet? That’s childhood fantasy, like you left-wingers cling to, like the children you are.

The Command In Chief COMMANDS ME! AND MY FRIENDS! YOU JUST WATCH!

Stephen T. Stonesays:

This is frightening, if only because it shows the mindset of the average Trump supporter: blindly following Trump, to the point of threatening or committing acts of violence, because he?s on ?the right side? (even though he?s really on his own side because he only gives a fuck about himself). The unyielding fealty to a man who all but claims the divine right of a king???or a god???makes me wonder how eager you are to start murdering people in his name.

Anonymoussays:

Re:

You are frightened? You should be. There is only so much patience that the American Public is willing to invest. After some time, we will hunt you down and tie you up with oratory that will make your head explode. Not literally, of course, Figuratively. Does that make sense to you? Oratory. Head Explode. Boom. Words can KILL! Words can KILL IDEAS! Especially STUPID IDEAS LIKE LEFTIST IDEAS! Words! YOU ARE FRIGHTENED OF WORDS. That’s why you hide comments, right? HIDE! FEAR! SHAKE! WORDS ARE COMING FOR YOU!

I’m not afraid of words, especially stupid words like yours. I’m a PROUD AMERICAN, WHICH AT LEAST I KNOW I’M FREE. TO TIE YOU UP AND SEND YOU WORDS THAT MAKE YOUR HEAD EXPLODE DEE DEE. LA LA LA.

Stephen T. Stonesays:

You are a sad person. I would give you my pity, but that well is tapped out for people like you.

And I?m not afraid of words. I?m afraid of people like you saying shit like this?

Trump is the COMMANDER IN CHIEF! Of the TRUMP ARMY! ARMED AND DANGEROUS AND READY TO CALL BALLS AND STRIKES AS WE SEE THEM! AND LIVE WITH THE CONSEQUENCES!

?because that tells me you?re more than willing to pick up a gun and shoot people???anyone, really???if Trump orders you to start indiscriminately killing other people. And you would do it with no questions asked. That is blind, violent loyalty to a man who would be king for no reason other than you want him to be the king of kings. That is what frightens me: the idea that people like you not only exist, but think you?d be doing God?s will.

Anonymoussays:

Re:

That’s RIGHT! God’s WILL! You are hilarious.

You are afraid of words because you have none to muster. You are a public idiot, and calling you out as a public idiot is GOD’s WORK! (Well, actually I get big money from the RNC to do so, but whatever, if the bible fits then speak in tongues).

Are you scared? Fearful? Frightened by your own imagination? What if I was your neighbor? Or your policeman, or postal worker, or Uber driver? Where are Trump supports? ALL AROUND US! ENVELOPING US WITH THE SMELL OF WALLMART!

Wait, are you Peter Stryck? You sound like him you whimpering little girl, you even sneer like him. Where’s DURHAM? HE’S COMING FOR ALL OF YOU!

HORRAY! KILL YOUR OPPONENTS WITH WORDS! Stephen is already laying in the corner curled up in a fetal position whimpering like the little baby he is.

Stephen T. Stonesays:

What if I was your neighbor? Or your policeman, or postal worker, or Uber driver?

If you were any of those things, I?d move somewhere that you aren?t.

Stephen is already laying in the corner curled up in a fetal position whimpering like the little baby he is.

You, uh?you got some sort of sexual paraphilia you?re not telling us about, champ?

Anonymoussays:

Re:

When you have nothing to say, you turn the discussion to either shit or sex, which may be the same thing to you. I never understood how you gay assholes stand the smell of shit while you fuck. Strange, no?

Why don’t you move now, Stephen. This is me, right next to you, watching you squirm and dance and try to change the subject.

You are SURROUNDED by Americans! Make good on your threat, Stephen! MOVE SOMEWHERE ELSE!

Not that you will, of course, you lying sack of shit. You lie about everything, just like all your socialist communist propaganda sleezeballs.

That One Guysays:

Re:

The unyielding fealty to a man who all but claims the divine right of a king???or a god???makes me wonder how eager you are to start murdering people in his name.

Compounding the pathetic nature of such people is something that you noted as an aside, in that said fealty would only ever be one way, as Trump has made crystal clear that the only person he cares about and is loyal to is himself, and he will happily throw anyone that he sees as a liability under a fleet of buses at a moment’s notice.

Bad enough to be mindlessly loyal for someone that actually cares about those below them, but to all but worship someone who sees anyone not themself as nothing more than tools to be used and discarded on a whim… that’s just beyond pathetic.

The Guysays:

Currently being fucked by this right now. Daughter has cancer, she is three and get’s SS payment (to me) I have a mortgage and bills and a job but only make $35,000. Does not cover the cost of living at all. They took every bit of documentation I had and gave us SNAP benefits for a year. Coded it wrong as a non disabled member HH and now they want the overpayment back. Had my "hearing" if you can call it that and she would not budge, said the system is accurate and that even if thet coded it wrong and even though I have supporting documentation and facts that I still OWE it. Fuck no I don’t according to Federal SNAP guidelines, showed them all the math but she didn’t want to even have the system reprocess all while admitting that there is nothing she can do and I am a "criminal stealing food from other people" all while she makes $72,395 per year from the State of Oregon (land of the fuck you state) to tell me that Oregon has different laws from Federal and they can manage how they want to based on OAR rules.

Anonymoussays:

Re:

If you don’t mind shamelessly parading your 3 year old on gofundme you may be able to get some contributions to pay the debt. If you really need it and it would actually help your 3 year old I would consider it.

I’m glad I’m not in that kind of situation because I would feel bad about the privacy of the 3 year old even if the money ultimately helps her.

Anonymoussays:

Re: Re:

Thought about that, one mother already does it but I don’t want my little girl hating me later in life for blasting her out all over the net. I just have to "work harder" somehow at my current dead end job. The market is out there for better jobs but I have no time to interview. If I miss one day I am fucked. Can’t even use FMLA for time off because of missed hours. Good thing my huge tech employer could give no fucks and offer shitty health coverage on top of it. But such is life. Keep on truckin’

Anonymoussays:

A great improvement would be for sites like this one to recognize the American People. Remember them? The ones that go to Trump Rallies? Ever see one of those? THE AMERICAN PEOPLE!

Wait, not all the American people. How about the American People that don’t have Biden Privilege? You know that privilege, the one that gets you FREE MONEY because of your NAME! That’s not American! That’s Corrupt!

The American People Will Prevail against your LEFTIST COMMUNIST SOCIALIST LEGAL BULLSHIT ARTISTS! Amen.

Anonymoussays:

Re:

Everybody knows that nothing will stand up to scrutiny. Just like telling the electoral college not to vote Trump into office. Just like the bimbos who took money to shut-up as "campaign violations". Just like the Russia witch hunt. Just like everything else you pathetic idiots have tried.

Don’t you get it? You did this to America, and the longer to try to keep it up, the longer Trump wlll stay in office. Mark my words – after a second term, if you keep it up, a third term will be inevitable. Proud American Heros will march in the streets if we have to. Without guns and our bibles and our history and the conviction of our American Hearts and Souls.

Obviously. You and your cause is what created Trump. Get over it already. Calm down. Find a candidate that has something American to say.

Or not, of course. We’re not tired of WINNING yet. We love WINNING, especially against you POWERLESS LOSERS!

Add Your Comment

Your email address will not be published. Required fields are marked *

Have a Techdirt Account? Sign in now. Want one? Register here

Comment Options:

Make this the or (get credits or sign in to see balance) what's this?

What's this?

Techdirt community members with Techdirt Credits can spotlight a comment as either the "First Word" or "Last Word" on a particular comment thread. Credits can be purchased at the Techdirt Insider Shop »

Follow Techdirt

Techdirt Daily Newsletter

Techdirt Insider Discord
The latest chatter on the Techdirt Insider Discord channel...
Loading...
Older Stuff
13:40 It's Great That Winnie The Pooh Is In The Public Domain; But He Should Have Been Free In 1982 (Or Earlier) (35)
12:06 Norton 360 Now Comes With Crypto Mining Capabilities And Sketchy Removal Process (28)
10:45 Chinese Government Dragnet Now Folding In American Social Media Platforms To Silence Dissent (14)
10:40 Daily Deal: The 2022 Ultimate Cybersecurity Analyst Preparation Bundle (0)
09:29 A Fight Between Facebook And The British Medical Journal Highlights The Difficulty Of Moderating 'Medical Misinformation' (9)
06:29 Court Ruling Paves The Way For Better, More Reliable Wi-Fi (4)
20:12 Eighth Circuit (Again) Says There's Nothing Wrong With Detaining Innocent Minors At Gunpoint (15)
15:48 China's Regulatory War On Its Gaming Industry Racks Up 14k Casualties (10)
13:31 Chinese Government Fines Local Car Dealerships For Surveilling While Not Being The Government (5)
12:08 Eric Clapton Pretends To Regret The Decision To Sue Random German Woman Who Listed A Bootleg Of One Of His CDs On Ebay (29)
10:44 ICE Is So Toxic That The DHS's Investigative Wing Is Asking To Be Completely Separated From It (29)
10:39 Daily Deal: The 2022 Complete Raspberry Pi And Arduino Developer Bundle (0)
09:31 Google Blocked An Article About Police From The Intercept... Because The Title Included A Phrase That Was Also A Movie Title (24)
06:22 Wireless Carriers Balk At FAA Demand For 5G Deployment Delays Amid Shaky Safety Concerns (16)
19:53 Tenth Circuit Denies Qualified Immunity To Social Worker Who Fabricated A Mother's Confession Of Child Abuse (35)
15:39 Sci-Hub's Creator Thinks Academic Publishers, Not Her Site, Are The Real Threat To Science, And Says: 'Any Law Against Knowledge Is Fundamentally Unjust' (34)
13:32 Federal Court Tells Proud Boys Defendants That Raiding The Capitol Building Isn't Covered By The First Amendment (25)
12:14 US Courts Realizing They Have A Judge Alan Albright Sized Problem In Waco (17)
10:44 Boston Police Department Used Forfeiture Funds To Hide Purchase Of Surveillance Tech From City Reps (16)
10:39 Daily Deal: The Ultimate Microsoft Excel Training Bundle (0)
09:20 NY Senator Proposes Ridiculously Unconstitutional Social Media Law That Is The Mirror Opposite Of Equally Unconstitutional Laws In Florida & Texas (25)
06:12 Telecom Monopolies Are Exploiting Crappy U.S. Broadband Maps To Block Community Broadband Grant Requests (7)
12:00 Funniest/Most Insightful Comments Of 2021 At Techdirt (17)
10:00 Gaming Like It's 1926: Join The Fourth Annual Public Domain Game Jam (6)
09:00 New Year's Message: The Arc Of The Moral Universe Is A Twisty Path (33)
19:39 DHS, ICE Begin Body Camera Pilot Program With Surprisingly Good Policies In Place (7)
15:29 Remembering Techdirt Contributors Sherwin And Elliot (1)
13:32 DC Metro PD's Powerful Review Panel Keeps Giving Bad Cops Their Jobs Back (6)
12:11 Missouri Governor Still Expects Journalists To Be Prosecuted For Showing How His Admin Leaked Teacher Social Security Numbers (39)
10:48 Oversight Board Overturning Instagram Takedown Of Ayahuasca Post Demonstrates The Impossibility Of Content Moderation (10)
More arrow
This site, like most other sites on the web, uses cookies. For more information, see our privacy policy. Got it