China and the AI Edge

 In “The Big Nine: How the Tech Titans & Their Thinking Machines Could Warp Humanity,” published March 5 by PublicAffairs, Amy Webb examines how artificial intelligence (AI) is evolving in China and the U.S. and explores three potential futures for AI, ranging from the optimistic to the catastrophic. In this excerpt, Webb, a professor at NYU’s Stern School of Business and founder of the Future Today Institute, looks at how the three Chinese companies—Baidu, Alibaba, and Tencent—that are part of the Big Nine are working with the Chinese government:

China’s economy has been growing at a fast pace, and the rapid development of AI is only going to speed China’s ascent.

In late 2017, modeling and analysis my Future Today Institute team and I did showed that AI has the potential to boost China’s economy 28% by 2035. AI—fueled by the sheer number of Chinese people and their data, widespread automation, machine learning and self-correction at scale, and improvements in capital efficiencies—will stimulate growth across Chinese manufacturing, agriculture, retail, fin-tech and financial services, transportation, utilities, health care, and entertainment media (including platforms). Right now, there is no other country on Earth with as much data as China, as many people as China, and as many electronics per capita. No other country is positioned to have a bigger economy than America’s within our lifetimes. No other country has more potential to influence our planet’s ecosystem, climate, and weather patterns—leading to survival or catastrophe—than China. No other country bridges both the developed and developing world like China does. As a Communist power and economic powerhouse, China is a partner that’s now too big to ignore, a political adversary that has radically different viewpoints on human rights, and a conduit for global alliances. With increased wealth comes power. China is positioning itself to influence the global supply of money and international trade. This necessarily unseats other countries from those positions of power and influence, and it also weakens democratic ideals worldwide.

Second, China will leverage its advancements in AI and economic stimulus to modernize its military, giving it an advantage over Western nations. That transition has already begun as part of an airborne domestic surveillance program, code-named Dove. More than 30 military and government agencies have deployed “spy birds” that look like white birds, mimicking the flapping actions of biological wings. The drones are part of a biologically inspired drone program intended to subvert radar and evade human detection. The drones capture footage, and an AI system looks for patterns, recognizes faces, and identifies anomalies. But spy birds, while scary sounding, are the least of your worries.

Late in 2017, an unreleased Pentagon report obtained by Reuters reporters warned that Chinese firms were skirting US oversight and gaining access to sensitive US AI technology with potential military applications by buying stakes in American firms. China’s People’s Liberation Army is investing heavily in a range of AI-related projects and technologies, while PLA research institutes are partnering with the Chinese defense industry.

China hasn’t waged physical war on any country since the 1979 Sino-Vietnamese War. And it wouldn’t appear as though China has any serious military adversaries—it hasn’t suffered terrorist attacks, it doesn’t have antagonistic relationships with the usual suspects (e.g., Russia, North Korea), and it hasn’t made enemies of other nations. So why the military push?

Because in the future, wars will be fought by code. Not hand-to-hand combat. Using AI techniques, a military can “win” by destabilizing an economy rather than demolishing countrysides and city centers. From that perspective, and given China’s unified march advancing artificial intelligence, China is dangerously far ahead of the West.

In my view, we’ve come to this realization too late. In my own meetings at the Pentagon with Department of Defense officials, an alternative view on the future of warfare (code vs. combat) has taken a long time to find widespread alignment. For example, in 2017, the DoD established an Algorithmic Warfare Cross-Functional Team to work on something called Project Maven—a computer vision and deep-learning system that autonomously recognizes objects from still images and videos. The team didn’t have the necessary AI capabilities, so the DoD contracted with Google for help training AI systems to analyze drone footage. But no one told the Google employees assigned to the project that they’d actually been working on a military project, and that resulted in high-profile backlash. Four thousand Google employees signed a petition objecting to Project Maven. They took a full-page ad out in the New York Times, and ultimately dozens of employees resigned. Eventually, Google said that it wouldn’t renew its contract with the DoD.

Amazon, too, came under fire because of a Pentagon contract worth $10 billion. In October 2018, House Appropriations committee members Tom Cole, a Republican from Oklahoma, and Steve Womack, a Republican from Arkansas, accused the DoD of conspiring with Amazon to tailor the contract so that no other tech giant would qualify. But that wasn’t the only complaint. There was a small wave of dissent at Amazon. Some Amazon workers were outraged that the company would do any work at all with the US military, while others didn’t like that Amazon’s facial recognition technology was being used by law enforcement. In response, Jeff Bezos told a conference audience, “If big tech companies are going to turn their back on the US Department of Defense, this country is going to be in trouble.”

While in the US, our tech giants are navigating a tricky path between national security and full transparency, the relationships Chinese companies Baidu, Alibababa, and Tencent (BAT) have with China’s government are exactly opposite. But here’s a chilling example: the current posture of the US military is that a human must be kept in the loop, regardless of how advanced AI, unmanned systems, and robots become. This will ensure that we don’t someday cede lethal authority to software. That is not the case in China. People’s Liberation Army (PLA) Lieutenant General Liu Guozhi, who directs the Chinese military’s Science and Technology Commission, is quoted as warning, “(We) must … seize the opportunity to change paradigms.” It was an indirect way of announcing China’s intent to rebuild its military might.

Third, if economic and military advantages aren’t concerning, China’s views on privacy will be. Reporters Without Borders ranks China 176^th out of 180 nations. It warns that President Xi has “been trying to export this oppressive model by promoting a new international media order under China’s influence.” China’s state and privately-owned media work in lockstep with the Communist Party. Foreign journalists are monitored continuously and must apply for special visas to do their reporting. Once on the ground, they are monitored continuously. There are more than 50 journalists detained in jails throughout China. Dissident blogger Yang Tongyan and Nobel peace laureate Liu Xiaobo both died of untreated cancers while incarcerated. Even members of the general public can be detained for reposting content on social media that runs counter to the official party line.

In what will later be viewed as one of the most pervasive and insidious social experiments on humankind, China is using AI in an effort to create an obedient populace. The State Council’s AI 2030 plan explains that AI will “significantly elevate the capability and level of social governance” and will be relied on to play “an irreplaceable role in effectively maintaining social stability.” This is being accomplished through China’s national Social Credit Score system, which according to the State Council’s founding charter will “allow the trustworthy to roam everywhere under heaven while making it hard for the discredited to take a single step.” It’s an idea that goes back to 1949, when the Communist Party first took power and began experimenting with various social control schemes. During Mao Zedong’s rule in the 1950s, social policing became the norm: workers were forced into communal farm groups, and they were assigned rankings based on their output. Individuals policed each other as members of farm groups, and that ranking determined how much access someone had to pubic goods. The system broke down under Mao, and it collapsed again in the 1980s because, as it turns out, humans aren’t accurate judges of each other— they’re motivated by their own individual needs, insecurities, and biases.

In 1995, then President Jiang Zemin envisioned a social policing system that leveraged technology—and by the mid-2000s, the Chinese government was working to build and implement a scoring system that functioned automatically. It partnered with Peking University to establish the China Credit Research Center to research how to build and implement an AI-powered national social credit score system. This partially explains the current president’s insistence on AI. It promises not only to make good on that idea proposed at the dawn of the Communist Party; importantly, it promises to keep the Communist Party in power.

In the city of Rongcheng, an algorithmic social credit scoring system has already proven that AI works. Its 740,000 adult citizens are each assigned 1,000 points to start, and depending on behavior, points are added and deducted. Performing a “heroic act” might earn a resident 30 points, while blowing through a traffic light would automatically deduct 5 points. Citizens are labeled and sorted into different brackets, ranging from A+++ to D, and their choices and ability to move around freely are dictated by their grade. The C bracket might discover that they must first pay a deposit to rent a public bike, while the A group gets to rent them for free for 90 minutes. It isn’t just individuals getting scored. In Rongcheng, companies are also scored for behavior—and their ability to do business depends very much on their bracket standings.

AI-powered directional microphones and smart cameras now dot the highways and streets of Shanghai. Drivers who honk excessively are automatically issued a ticket via Tencent’s WeChat, while their names, photographs, and national identity card numbers are displayed on nearby LED billboards. If a driver pulls over on the side of the road for more than seven minutes, they will trigger another instant traffic ticket. It isn’t just the ticket and the fine—points are deducted in the driver’s social credit score. When enough points are deducted, they will find it hard to book airline tickets or land a new job. There was a popular episode of Black Mirror portending a dystopian future like this. In Shanghai, that future has already arrived.

State-level surveillance is enabled by the BAT, who are in turn emboldened through China’s various institutional and industrial policies. Alibaba’s Zhima Credit service hasn’t publicly disclosed that it is part of the national credit system; however, it is calculating a person’s available credit line based on things like what that person is buying and who their friends are on Alipay’s social network. In 2015, Zhima Credit’s technology director publicly said that buying diapers would be considered “responsible behavior,” while playing video games for too long would be counted as a demerit.

China’s Police Cloud is one tangible example of AI that’s already deployed: it  was built to monitor and track people with mental health issues, who have publicly criticized the government, and who are ethnic minorities. The Integrated Joint Operations Program uses AI to detect pattern deviations, such as jaywalking. China’s social credit scores rate and rank citizens based on their behavior; decision-making AI systems use those scores to determine who’s allowed to secure a loan, who can travel, and even where their children are allowed to go to school.

Robin Li, one of Baidu’s founders, argued that to the Chinese, privacy isn’t a core value as it is in the West. “The Chinese people are more open or less sensitive about the privacy issue,” Li told an audience at the China Development Forum in Beijing. “If they are able to trade privacy for convenience, safety, and efficiency, in a lot of cases, they are willing to do that.” Or maybe it has more to do with repercussions.

I’d argue that China’s national social credit score isn’t about strengthening the Communist Party or a complicated way of achieving strategic advantage over those working on AI in the West. Rather, it’s about exerting total control to shape our global economy. Early in 2018, President Xi told the state news agency Xinhua that “by tightening our belts and gritting our teeth, we built ‘two bombs and one satellite,’” which was a reference to a military weapons program developed under Mao. “This was because we made best use of the socialist system. We concentrated our efforts to get great things done. The next step is to do the same with science and technology. We must cast away false hopes and rely on ourselves.”

Xi rejects the notions of market economies, a free internet, and a diverse ecosystem of competing and complementary ideas. China’s tightly controlled domestic economy walls itself off from competition. It is enabling “splinternets” in which the rules of the internet depend on a user’s physical location. It is centralizing cyberpolicy, clamping down on free speech, and asserting itself into every aspect of the third era of computing via regulatory control: the internet’s infrastructure, the global flow of data, and the hardware are increasingly subject to Beijing’s approval. Speaking at an event in 2016, Xi said that the government would henceforth have total discretion to determine how it would protect networks, devices, and data.

It will exert this considerable control by enticing its Belt and Road Initiative partners with infrastructure and tech pilots. Tanzania was selected as an early pilot partner—and, perhaps not coincidentally, the country has now adopted many of China’s data and cyberpolicies. Tanzania’s government was given technical assistance by Chinese counterparts, and a senior Tanzanian official said that “our Chinese friends have managed to block such media in their country and replaced them with their homegrown sites that are safe, constructive, and popular.” The same is happening elsewhere in Africa. Vietnam has now adopted China’s stringent cybersecurity laws. As of June 2018, India was considering legislation that would mirror China’s requirements for housing domestic data and sourcing domestic cybersecurity technologies.

What if China starts influencing its Belt and Road Initiative partners such that one of its primary exports is its national social credit score system? It’s easy to see how the world’s autocracies, like Turkey and Rwanda, could become a buyer of China’s surveillance technology. But what about in other countries, such as Brazil and Austria, that have given in to populism and as of this writing have nationalistic leaders? What if a government agency in your country is inspired or strong-armed into adopting a social credit score system, one that begins monitoring you without your explicit consent? Would you ever know that you had a score, perhaps one that landed you on a watch list?

What if foreign companies are assessed brackets and either given preferential treatment or prevented from doing business with China—or even with each other? As China’s economy grows, what happens if this power and influence propagates throughout the internet, our gadgets and devices, and AI itself?

What if China builds a social credit score for people outside its borders, using data it mines on the free and open web and the West’s social networks? What if it’s scraping all the ambient data you’re leaving behind after your trips visiting the Great Wall and Forbidden City? What about all the hacking operations we hear about periodically, where big data breaches appear to be coming from networks based in China?