Jul. 20th, 2018

Earlier this month it came out that among Facebook’s myriad algorithmically induced advertising categories was an entry for users whom the platform’s data mining systems believed might be interested in treason against their government. The label had been applied to more than 65,000 Russian citizens, placing them at grave risk should their government discover the label. Similarly, the platform’s algorithms silently observe its two billion users’ actions and words, estimating which users it believes may be homosexual and quietly placing a label on their account recording that estimate. What happens when governments begin using these labels to surveil, harass, detain and even execute their citizens based on the labels produced by an American company’s black box algorithms?

One of the challenges with the vast automated machine that is Facebook’s advertising engine is that its sheer scale and scope means it could never possibly be completely subject to human oversight. Instead, it hums along in silence, quietly watching the platform’s two billion users as Big Brother, silently assigning labels to them indicating its estimates of everything from their routine commercial interests to the most sensitive and intimate elements of their personality, beliefs and medical conditions that could be used by their governments to manipulate, arrest or execute them.

Such concerns are unfortunately far from hypothetical. I can personally attest that there are many governments across the world that very much aware of the potential of Facebook’s advertising tools for surveillance and indeed use them actively to track specific demographics and interests, using the company’s built-in reporting tools to identify geographic areas and demographics to target for further surveillance.

Today much of the governmental use of Facebook’s ad targeting tools revolves around using its publicly accessible targeting and reporting tools to understand things like which neighborhoods have the highest density of persons in a particular demographic that also have a particular interest of concern to the government. By running large numbers of parallel campaigns covering all of the permutations of a set of demographics and interests, governments can even learn which demographics are most associated with particular interests and which interests are most strongly correlated with particular demographics. Geographic reporting tools allow neighborhood-level identification of where those demographics and interests coincide, allowing surveillance resources to be increased in those areas.

The public availability of Facebook’s targeting tools means intelligence agencies need no court orders to leverage them, foreign intelligence services can use them to track and surveil on foreign soil and even local law enforcement agencies can use them with few restrictions. The global availability of Facebook’s advertising platform offers a particularly powerful and inviting tool for intelligence agencies attempting to map out adversarial nations, allowing them to better understand demographic and interest correlations and geographic affinities and guide the allocation of their own ground based resources.

In spite of their incredible power and public availability, Facebook’s ad tools are still a relatively blunt instrument compared to traditional individual-level surveillance tools.

As I alluded to earlier this week, what happens when countries in which homosexuality is a criminal offense that can potentially bring the death penalty use Facebook’s tools to target those communities? Using only Facebook’s public advertising tools, they can estimate popular neighborhoods and hangouts, correlated interests in those areas and so on, but they can’t readily compile a list of real names and addresses of everyone in their country that Facebook believes may be homosexual.

Given that homosexuality in some countries is classified as a crime under their formal legal code, could those countries use a court order to force Facebook to provide a list of all names of individuals in their country that its algorithms believe may be homosexual? The laws of many countries would make it difficult for Facebook to attempt to shield its users from a lawful request for a list of individuals suspected of committing what is in that country a serious crime.

Compounding matters, those individuals may have no idea that Facebook has identified them as potentially homosexual. They may take great care in all of their communications, friendship connections, likes, statements, status updates and all other online actions in an attempt to prevent the government from suspecting them. Yet, Facebook’s unyielding all-watching Eye of Sauron is not easily fooled and will likely eventually assign them a marketing label indicating its belief of their sexual preference based on the most nuanced patterns invisible to the human eye.

While Facebook agreed to remove its treason category due to its illegality in all countries (left unspoken was its limited marketing use which mean it likely was generating little revenue), a company spokesperson stated that the company would not be removing other categories that could place individuals at grave risk of arrest or death. Noting that homosexuality categories can be used by LGBTQ advocacy groups to reach people interested in those topics, the company said that they would not be restricting their use of homosexuality categories or any of its other sensitive topics categories even in countries where they are illegal.

When asked whether the company would at least consider limiting its application of sensitive categories in countries where they are illegal, such as not automatically labeling homosexual users against their will in countries where they could face the death penalty, the company offered that since marketers wanted to target those sensitive categories even when they placed users at grave risk of physical harm or death, “we’ll be keeping [them].”

It is remarkable that the company would not even consider placing the life safety of their users ahead of its marketing interests and that revenue generation is prioritized even when it has a very real possibility of leading to the death of those users. Such are the ethical and moral standards of today’s Silicon Valley.

This raises the question then of what Facebook would do when confronted with a formal legal request, such as public court order or a more secretive National Security Letter or similar, that ordered the company to hand over the names and IP addresses of all users that its algorithms believed were interested in certain topics or belonged to certain demographic groups.

When asked “has Facebook ever received a request from any government agency worldwide that asked it to provide a list of user names of accounts that had specific advertising interest labels associated with them” a company spokesperson replied that the company would provide that information to any government “In response to a legal request (like a search warrant, court order or subpoena) if we have a good faith belief that the law requires us to do so. This may include responding to legal requests from jurisdictions outside of the United States when we have a good-faith belief that the response is required by law in that jurisdiction, affects users in that jurisdiction, and is consistent with internationally recognized standards” and pointed to its data policy.

When asked whether the company had indeed received such requests and actually provided a list of names in response that its algorithms believed were interested in those categories, the company let its answer above stand.

Such a response is truly frightening, as it demonstrates just how clearly the central role Facebook is increasingly playing as a tool for law enforcement, intelligence agencies and repressive regimes to crack down on legitimate dissent or internationally recognized human rights. It also raises important questions about the company’s legal exposure if it knowingly assists a repressive regime track down and execute citizens based on internationally protected statuses.

Putting this all together, instead of bringing the world together, social media is increasingly helping to elevate the voices tearing it apart, while its international reach, massive centralized data warehouse and algorithms that can divine the most sensitive and intimate elements of our lives are likely to increasingly become a go-to one-stop shop for the world’s intelligence agencies to spy on and influence the world while governments themselves increasingly leverage their legal powers to force Facebook to help them hunt down dissent and those different from themselves. Welcome to a world even Orwell could not have imagined.