Issue #80: Trustworthiness In China


The RoboPsych Newsletter

Exploring The Psychology of 
Human-Robot Interaction
Issue 80: November 13, 2017

Trustworthiness In China
Quote of the Week:


“Imagine a world where many of your daily activities were constantly monitored and evaluated: what you buy at the shops and online; where you are at any given time; who your friends are and how you interact with them; how many hours you spend watching content or playing video games; and what bills and taxes you pay (or not). It's not hard to picture, because most of that already happens, thanks to all those data-collecting behemoths like Google, Facebook and Instagram or health-tracking apps such as Fitbit. But now imagine a system where all these behaviours are rated as either positive or negative and distilled into a single number, according to rules set by the government. That would create your Citizen Score and it would tell everyone whether or not you were trustworthy. Plus, your rating would be publicly ranked against that of the entire population and used to determine your eligibility for a mortgage or a job, where your children can go to school - or even just your chances of getting a date.

A futuristic vision of Big Brother out of control? No, it's already getting underway in China, where the government is developing the Social Credit System (SCS) to rate the trustworthiness of its 1.3 billion citizens. The Chinese government is pitching the system as a desirable way to measure and enhance "trust" nationwide and to build a culture of "sincerity". As the policy states, "It will forge a public opinion environment where keeping trust is glorious. It will strengthen sincerity in government affairs, commercial sincerity, social sincerity and the construction of judicial credibility."


Rachael Botsman, Wired Magazine
Big data meets Big Brother as China moves to rate its citizens

Trust Is Just a Number  

What if everything you do or say could be rated by everyone you encounter?

What if those ratings were aggregated into a single score; one number that depicted your trustworthiness, employability, social status, attractiveness?

If that sounds like familiar science fiction, it might be because you read Cory Doctorow's 2003 novel, Down and Out in the Magic Kingdom, or Gary Shteyngart's 2010 modern romantic tale, Super Sad True Love Story

Or, maybe you saw Nosedive, Season 3's premiere episode of Netflix's always provocative Black Mirror. 

Oh, and, of course, the granddaddy of them all, George Orwell's original classic, 1984, which forever turned the name “Big Brother” into a constant haunting presence. 

It's pretty clear that this kind of frightening prospect has been with us for decades. 

Now, what if that score was created and monitored by your nation's government and used to determine everything about your life: what kind of job you could get, what medical treatment you could receive, even where you could get dinner reservations? 

And, what if that government was that of the People's Republic of China?

That's what's about to happen if the current (voluntary) pilot program for the country's “Social Credit System” (SCS) goes into (mandatory) effect for all 1.3 billion Chinese citizens in 2020. If the current version of the SCS is implemented, not only would your own behavior be distilled into a single score ranging from 350 to 950, but everything anyone in your social circles does or says would effect your score as well. (From Wired: "People will have an incentive to say to their friends and family, Don't post that. I don't want you to hurt your score but I also don't want you to hurt mine.")

Black Mirror's Nosedive depicts some of the possibilities as lead character Lacie strives to land a new apartment in a gated community restricted to members whose average rating is 4.5 or above on a 5 point scale. She's only a 4.2 when she applies, so getting some 5 star encounter ratings from influencers (4.7s or 4.8s!) is just the ticket for landing her dream home. No can probably guess how that goes.

The psychological implications of living in this mega-gamified world of relentless encounter scoring, massive data collection, and the constant computation of points, badges, and leaderboards are stark. So far, though, we're told that:
Higher scores have already become a status symbol, with almost 100,000 people bragging about their scores on Weibo (the Chinese equivalent of Twitter) within months of launch.
This is what happens when a country that lived through decades of state-enforced economic deprivation is plunged into 21st century-style capitalism. That is, normal interpersonal competition for rewards and the status they bring will drive a significant fraction of the population into full-fledged (but, state-managed) competitiveness.

What else can we predict? The Wired article speculates that there will likely be a market for “reputation consultants” who will help people (many of them desperate) to change their lives by raising their singularly significant SCS.

Black markets? Of course. 

Hacking? It goes without saying.

But, here's what the government has to say about the policy:
The Chinese government is pitching the system as a desirable way to measure and enhance "trust" nationwide and to build a culture of "sincerity". As the policy states, "It will forge a public opinion environment where keeping trust is glorious. It will strengthen sincerity in government affairs, commercial sincerity, social sincerity and the construction of judicial credibility."
Think about that: the government believes that “glorious” trust and social sincerity will be fostered and strengthened through constant anonymous evaluation of every aspect of one's life by people whose own fate lies in the hands of every person (rater) with whom one comes in contact. 

Let's be clear: this ubiquitous social management system will have intended and unintended consequences far beyond “social sincerity.” Its unprecedented pervasiveness and complexity guarantees it. 

And, while we in the democratic West may shake our heads and bemoan this horrifying breach of civil liberties in China, our own lives are themselves subject to increasingly invasive scrutiny.

Technology makes scrutiny possible; government policy makes it acceptable; individual compliance makes it normative and oppressive.

As closed circuit TV cameras blanket our environment, as facial recognition capability becomes commonplace, as ingestible drug monitoring microbots become standard procedure for assuring patient medication compliance, our sensitivity to our steady loss of privacy fades.

To say that these systems will enable the growth of “glorious trust” is the kind of doublespeak that Orwell wrote about in 1949. The question now is, how can we use the power of technological monitoring capabilities to ensure trustworthiness without resorting to the paralyzingly totalitarian ethos of China's new Social Credit System? Going too far puts us at risk for a future of fear, suspicion, and subterfuge that could catastrophically set back social cohesion.

If you're interested in more thoughts about China's proposed system, check out Dr. Julie Carpenter and me talking about the issues on the latest episode of the RoboPsych Podcast
Tom Guarriello, Ph.D. 

Thanks for subscribing to the RoboPsych Newsletter

If you're not subscribed, please sign up here.  
Be sure to subscribe to the RoboPsych Podcast!
Our mailing address is:

unsubscribe from this list    update subscription preferences 

This email was sent to *|EMAIL|*
why did I get this?    unsubscribe from this list    update subscription preferences