{"id":6876,"date":"2017-10-04T21:37:04","date_gmt":"2017-10-05T01:37:04","guid":{"rendered":"http:\/\/www.zephoria.org\/thoughts\/?p=6876"},"modified":"2018-07-21T21:39:19","modified_gmt":"2018-07-22T01:39:19","slug":"your-data-is-being-manipulated","status":"publish","type":"post","link":"https:\/\/www.zephoria.org\/thoughts\/archives\/2017\/10\/04\/your-data-is-being-manipulated.html","title":{"rendered":"Your Data is Being Manipulated"},"content":{"rendered":"
\n
\n
\n
<\/div>\n
\n
\n
\n
<\/div>\n<\/div>
Excerpt from \u201cThe Anatomy of a Large-Scale Hypertextual Web Search Engine,\u201d Sergey Brin and Larry Page (April\u00a01998)<\/figcaption><\/figure>\n<\/div>\n
\n

What follows is the crib from my keynote at the 2017\u00a0<\/em>Strata Data\u00a0<\/em><\/strong><\/a><\/span>Conference<\/em><\/strong><\/span>\u00a0in New York City. Full\u00a0<\/em>video<\/em>\u00a0can be found here<\/a>.\u200a<\/em><\/span><\/p>\n<\/div>\n<\/div>\n<\/section>\n

\n
\n
\n<\/div>\n
\n
\n

In 1998, two graduate students at Stanford decided to try to \u201cfix\u201d the problems with major search engines. Sergey Brin and Larry Page wrote\u00a0a paper<\/a>\u00a0describing how their PageRank algorithm could eliminate the plethora of \u201cjunk results.\u201d Their idea, which we all now know as the foundation of Google, was critical.\u00a0But it didn\u2019t stop people from trying to mess with their system.<\/span>\u00a0In fact, the rise of Google only increased the\u00a0sophistication<\/span>\u00a0of those invested in search engine optimization.<\/p>\n<\/div>\n<\/div>\n<\/section>\n

\n
\n
\n<\/div>\n
\n
\n
\n
\n
<\/div>\n
<\/canvas><\/div>\n<\/div>
\u201cgoogle bombing\u201d\u200a\u2014\u200adiverting search engine rankings to subversive commentary about public\u00a0figure<\/figcaption><\/figure>\n<\/div>\n
\n

Fast forward to 2003, when the sitting Pennsylvania senator Rick Santorum publicly compared homosexuality to\u00a0bestiality<\/span>\u00a0and pedophilia. Needless to say, the LGBT community was outraged. Journalist\u00a0Dan Savage called on his readers<\/a>\u00a0to find a way to \u201cmemorialize the\u00a0scandal<\/span>.\u201d One of his fans created\u00a0a website<\/a>\u00a0to associate Santorum\u2019s name with anal sex. To the senator\u2019s horror, countless members of the public jumped in to link to that website in an effort to influence search engines.\u00a0This form of\u00a0<\/span>crowdsourced<\/span>\u00a0SEO is commonly referred to as \u201c<\/span>Google bombing,\u201d and it\u2019s a form of media manipul<\/span>ation intended to mess with data and the information landscape.<\/span><\/p>\n<\/div>\n<\/div>\n<\/section>\n

\n
\n
\n<\/div>\n
\n
\n
\n
\n
<\/div>\n
<\/canvas><\/div>\n<\/div>
Media Manipulation and Disinformation Online (cover), March 2017. Illustration by Jim\u00a0Cooke<\/figcaption><\/figure>\n

Media manipulation is not new.<\/span>\u00a0<\/span>As many adversarial actors know, the boundaries between\u00a0<\/span>propaganda<\/span>\u00a0and social media marketing are often fuzzy<\/span>.<\/span>Furthermore,\u00a0<\/span>any company that uses public signals to inform aspects of its product\u200a\u2014\u200afrom Likes to Comments to Reviews\u200a\u2014\u200aknows full well that\u00a0<\/span>any system you create will be gamed for fun, profit, politics, ideology, and power<\/span>.<\/span>Even Congress is now grappling with that reality.<\/span>\u00a0But I\u2019m not here to tell you what has always been happening or even\u00a0what is currently happening<\/a>\u200a\u2014\u200aI\u2019m here to help you understand what\u2019s about to happen.<\/p>\n<\/div>\n<\/div>\n<\/section>\n

\n
\n
\n<\/div>\n
\n
\n

At this moment, AI is at the center of every business conversation.<\/span>\u00a0<\/span>Companies, governments, and researchers are obsessed with data.<\/span>\u00a0Not surprisingly, so are adversarial actors.\u00a0<\/span>We are currently seeing an evolution in how data is being manipulated.<\/span>\u00a0<\/span>I<\/span>f we believe that data can and should be used to inform people and fuel technology,\u00a0<\/span>we need to start building the infrastructure necessary to limit the corruption and\u00a0<\/span>abuse<\/span>\u00a0of that data<\/span>\u200a\u2014\u200aand grapple with how biased and problematic data might work its way into technology and, through that, into the foundations of our society.<\/span><\/p>\n

In short,\u00a0<\/span>I think\u00a0<\/span>we need to reconsider what security looks like in a data-driven world<\/span>.<\/span><\/h4>\n
\n
\n
<\/div>\n
<\/canvas><\/div>\n<\/div>
Shutterstock by\u00a0goir<\/figcaption><\/figure>\n

Part 1: Gaming the\u00a0System<\/strong><\/h4>\n

Like search engines,\u00a0<\/span>social media introduced a whole new target for manipulation<\/span>. This attracted all sorts of people, from social media marketers to state actors. Messing with Twitter\u2019s trending topics or Facebook\u2019s news feed became a hobby for many.\u00a0<\/span>For $5, anyone could easily buy followers, likes, and comments on almost every major site.<\/span>\u00a0The economic and political incentives are obvious, but alongside these powerful actors, there are also a whole host of people with less-than-obvious intentions coordinating attacks on these systems<\/span>.<\/span><\/p>\n<\/div>\n<\/div>\n<\/section>\n

\n
\n
\n<\/div>\n
\n
\n
\n
\n
<\/div>\n
<\/canvas><\/div>\n<\/div>
Piechart example of Rick-Rolling<\/figcaption><\/figure>\n

For example, when a distributed network of people decided to help\u00a0propel Rick Astley to the top of the charts<\/a>\u00a020 years after his song \u201cNever Gonna Give You Up\u201d first came out, they weren\u2019t trying to help him make a profit (although they did). Like other memes created through networks on sites like 4chan, rickrolling was for kicks.\u00a0But<\/span>through this<\/span>\u00a0<\/span>practice, lots of people learned how to make content \u201cgo viral\u201d or otherwise mess with systems.\u00a0<\/span>In other words, they learned to\u00a0hack the attention economy<\/a>.<\/span>\u00a0And, in doing so,\u00a0<\/span>they\u2019ve developed strategic practices of manipulation that can and do have serious consequences<\/span>.<\/span><\/p>\n<\/div>\n<\/div>\n<\/section>\n

\n
\n
\n<\/div>\n
\n
\n

A story like \u201c#Pizzagate\u201d doesn\u2019t happen accidentally\u200a\u2014\u200ait was produced by a wide network of folks looking to toy with the information ecosystem.<\/strong><\/span>\u00a0<\/strong>They created a\u00a0<\/span>cross-platform network of fake accounts known as<\/span>\u201csock puppets\u201d<\/span>\u00a0which they use to subtly influence journalists and other powerful actors to pay attention to strategically produced questions, blog posts, and YouTube videos.\u00a0<\/span>The goal with a story like that isn\u2019t to convince journalists that it\u2019s true,\u00a0<\/span>but to get them to foolishly use their amplification channels to negate it.<\/span>\u00a0<\/span>This produces a\u00a0<\/strong><\/span>\u201cBoomerang effect<\/strong>,\u201d<\/span>\u00a0whereby those who don\u2019t trust the media believe that there must be merit to the conspiracy, prompting some to \u201cself-investigate<\/span>.\u201d<\/span><\/p>\n<\/div>\n<\/div>\n<\/section>\n

\n
\n
\n<\/div>\n
\n
\n
\n
\n
<\/div>\n
<\/canvas><\/div>\n<\/div>
Hydrargyrum<\/a>\u00a0CC BY-SA\u00a02.0<\/a><\/figcaption><\/figure>\n

Then there\u2019s the universe of content designed to\u00a0\u201copen the Overton window\u201d\u200a<\/strong>\u2014\u200aor increase the range of topics that are acceptable to discuss in public.<\/span>\u00a0<\/span>Journalists are tricked into spreading problematic frames.<\/span>\u00a0<\/span>Moreover,<\/span>recommendation engines can be used to encourage those who are open to problematic frames to go deeper.<\/span>\u00a0Researcher Joan Donovan studies white supremacy; after work, she can\u2019t open Amazon, Netflix, or YouTube without being recommended to consume neo-Nazi music, videos, and branded objects<\/span>.\u00a0<\/span>Rad<\/span>ical trolls also know how to leverage this infrastructure to cause trouble.\u00a0<\/span>Without tripping any of Twitter\u2019s protectiv<\/span>e mechanisms, the well-known troll weev managed to use the company\u2019s ad infrastructure to amplify white supremacist ideas to those focused on social justice, causing outrage and anger.<\/span><\/p>\n

By and large, these games have been fairly manual attacks of algorithmic systems, but as we all know, that\u2019s been changing. And it\u2019s about to change\u00a0again.<\/h4>\n<\/div>\n<\/div>\n<\/section>\n
\n
\n
\n<\/div>\n
\n
\n

Part 2: Vulnerable Training\u00a0Sets<\/h4>\n

Training a machine learning system requires data. Lots of it. While there are some standard corpuses, computer science researchers, startups, and big companies are increasingly hungry for new\u200a\u2014\u200aand different\u200a\u2014\u200adata.<\/p>\n

\n
\n
<\/div>\n
<\/canvas><\/div>\n<\/div>
Cognitive Psychology for Deep Neural Networks: A Shape Bias Case Study, June 29,\u00a02017<\/figcaption><\/figure>\n

The first problem is that\u00a0<\/span>all data is biased<\/span>, most notably and recognizably by reflecting the biases of humans and of society in general<\/span>.<\/span>\u00a0<\/span>Take, for example, the popular ImageNet dataset.\u00a0Because humans categorize by shape faster than they categorize by color, you end up with some weird artifacts in that data.<\/a><\/span><\/p>\n<\/div>\n<\/div>\n<\/section>\n

\n
\n
\n<\/div>\n
\n
\n
\n
\n
<\/div>\n
<\/canvas><\/div>\n<\/div>
(a) and (c) demonstrate ads for two indvidual\u2019s names, (b) and (d) demonstrate that the advertising was suggesting criminal histories based on name type, not actual\u00a0records<\/figcaption><\/figure>\n

Things get even messier when you\u2019re dealing with social prejudices.\u00a0<\/span>WhenLatanya Sweeney searched for her name on Google<\/a>, she was surprised to be given ads inviting her to find out if she had a criminal record.\u00a0<\/span>As a curious computer scientist, she decided to run a range of common black and white names through the system to see which ads popped up. Unsurprisingly, only<\/span>black names<\/span>\u00a0produced ads for criminal justice products<\/span>.<\/span>\u00a0<\/span>This isn\u2019t because Google knowingly treated the names differently, but because searchers were more likely to click on criminal justice ads when searching for black names.<\/span>Google learned\u00a0<\/span>American<\/span>\u00a0racism and amplified it back at\u00a0all<\/em>\u00a0of its users<\/span>.<\/span><\/p>\n

Addressing implicit and explicit cultural biases in data is going to be a huge challenge for everyone who is trying to build a system dependent on data classified by or about\u00a0humans.<\/span><\/h4>\n<\/div>\n<\/div>\n<\/section>\n
\n
\n
\n<\/div>\n
\n
\n

But there\u2019s also a new challenge emerging.\u00a0The same decentralized networks of people\u200a\u2014\u200aand state actors\u200a\u2014\u200awho have been messing with social media and search engines are increasingly eyeing the data that various companies use to train and improve their systems.<\/span><\/p>\n

Consider, for example, the role of reddit and Twitter data as training data. Computer scientists have long pulled from the very generous APIs of these companies to train all sorts of models, trying to understand natural language, develop metadata around links, and track social patterns.\u00a0They\u2019ve trained models to detect depression, rank news, and engage in conversation.\u00a0<\/span>Ignoring the fact that\u00a0<\/span>this data is not representative in the first place<\/span>, most engineers who use these APIs believe that it\u2019s possible to clean the data and remove all problematic content.<\/span>\u00a0I can promise you it\u2019s not.<\/span><\/p>\n

No amount of excluding certain subreddits, removing of categories of tweets, or ignoring content with problematic words will prepare you for those who are hellbent on messing with\u00a0you.<\/span><\/h4>\n

I\u2019m watching countless actors experimenting with ways to mess with public data with an eye on major companies\u2019 systems.\u00a0They are trying to fly below the radar.\u00a0<\/span>If you don\u2019t have a structure in place for strategically grappling with how those with an agenda might try to route around your best laid plans, you\u2019re vulnerable.<\/span>\u00a0This isn\u2019t about accidental or natural content. It\u2019s not even about\u00a0<\/span>culturally biased data<\/span>.\u00a0<\/span>T<\/span>his is about\u00a0<\/span>strategically gamified content injected into systems by people who are trying to guess what you\u2019ll do<\/span>.<\/span><\/p>\n<\/div>\n<\/div>\n<\/section>\n

\n
\n
\n<\/div>\n
\n
\n

If you want to grasp what that means, consider\u00a0the experiment Nicolas Papernot and his colleagues published last year<\/a>.<\/span>\u00a0In order to understand the vulnerabilities of computer vision algorithms, they decided to alter images of stop signs so that they still resembled a stop sign to a human viewer even as the underlying neural network interpreted them as a yield sign. Think about what this means for autonomous vehicles. Will this technology be widely adopted if the classifier can be manipulated so easily?<\/p>\n

\n
\n
<\/div>\n
<\/canvas><\/div>\n<\/div>
Practical Black-Box Attacks against Machine, March 19, 2017. The images in the top row are altered to disrupt the neural network leading to the misinterpretation on the bottom row. The alterations are not visible to the human\u00a0eye.<\/figcaption><\/figure>\n

Right now,\u00a0<\/span>most successful\u00a0<\/span>data-injection attacks on machine learning models<\/span>are happening in the world of research, but more and more, we are seeing people try to mess with mainstream systems.<\/span>\u00a0Just because they haven\u2019t been particularly successful yet doesn\u2019t mean that they aren\u2019t learning and evolving their attempts.<\/p>\n<\/div>\n<\/div>\n<\/section>\n

\n
\n
\n<\/div>\n
\n
\n

Part 3: Building Technical Antibodies<\/h4>\n

Many companies spent decades not taking security vulnerabilities seriously, until breach after breach hit the news.<\/span>\u00a0Do we need to go through the same pain before we start building the tools to address this new vulnerability?<\/p>\n

If you are building data-driven systems, you need to start thinking about how that data can be corrupted, by whom, and for what\u00a0purpose<\/span>.<\/span><\/h4>\n<\/div>\n<\/div>\n<\/section>\n
\n
\n
\n<\/div>\n
\n
\n

In the tech industry, we have lost the culture of Test<\/span>.<\/span>\u00a0Part of the blame rests on the shoulders of social media.\u00a0<\/span>Fifteen years ago, we got the bright idea to shift to a culture of the \u201cperpetual beta.\u201d<\/span>\u00a0<\/span>We invited the public to be our quality assurance engineers.<\/span>\u00a0<\/span>But\u00a0<\/span>internal QA wasn\u2019t simply about finding bugs. It was about integrating adversarial thinking into the design and development process<\/span>.<\/span>\u00a0<\/span>And\u00a0<\/span>asking the public to find bugs in our systems doesn\u2019t work well when some of those same people are trying to mess with our systems<\/span>.<\/span>Furthermore, there is currently no incentive\u200a\u2014\u200aor path\u200a\u2014\u200afor anyone to privately tell us where things go wrong.<\/span>\u00a0<\/span>Only when journalists shame us by finding ways to trick our systems into advertising to neo-Nazis do we pay attention.<\/span>\u00a0Yet, far more maliciously intended actors are starting to play the long game in messing with our data. Why aren\u2019t we trying to get ahead of this?<\/span><\/p>\n<\/div>\n<\/div>\n<\/section>\n

\n
\n
\n<\/div>\n
\n
\n

On the bright side, there\u2019s an emergent world of researchers building adversarial thinking into the advanced development of machine learning\u00a0systems.<\/span><\/h4>\n

Consider, for example, the research into\u00a0generative adversarial networks<\/a>\u00a0(or GANs).\u00a0<\/span>For those unfamiliar with this line of work, the idea is that you have two unsupervised ML algorithms\u200a\u2014\u200aone is trying to generate content for the other to evaluate. The first is trying to trick the second into accepting \u201cwrong\u201d information. This work is all about trying to find the boundaries of your model and the latent space of your data.<\/span>\u00a0We need to see a lot more R&D work like this\u200a\u2014\u200athis is the research end of a culture of Test, with true adversarial thinking baked directly into the process of building models<\/span>.<\/p>\n<\/div>\n<\/div>\n<\/section>\n

\n
\n
\n<\/div>\n
\n
\n
\n
\n
<\/div>\n
<\/canvas><\/div>\n<\/div>
White Hat Hackers\u200a\u2014\u200athose who hack for \u201cthe right reasons.\u201d For instance, testing the security or vulnerabilities of a system (Image: CC\u00a0Magicon<\/a>,\u00a0HU)<\/figcaption><\/figure>\n

But these research efforts are not enough.\u00a0<\/span>We need to actively and intentionally build a culture of adversarial testing, auditing, and learning into our development practice.<\/span>\u00a0<\/span>We need to build analytic approaches to assess the biases of any dataset we use.<\/span>\u00a0And we need to build tools to monitor how our systems evolve<\/span>\u00a0with as much effort as we build our models in the first place<\/span>.<\/span>My colleague\u00a0Matt Goerzen argues<\/a>\u00a0that\u00a0<\/span>we also need to strategically invite white hat trolls to mess with our systems and help us understand our vulnerabili<\/span>ties.<\/span><\/p>\n<\/div>\n<\/div>\n<\/section>\n

\n
\n
\n<\/div>\n
\n
\n

The tech industry is no longer the passion play of a bunch of geeks trying to do cool shit in\u00a0<\/span>the<\/span>\u00a0world. It\u2019s\u00a0<\/span>now the foundation of our democracy, economy, and information landscape<\/span>.<\/span><\/h4>\n

We no longer have the luxury of only thinking about the world we want to build.\u00a0<\/span>We must also strategically think about how others want to manipulate our systems to do harm and cause chaos.<\/span><\/p>\n<\/div>\n<\/div>\n<\/section>\n<\/div>\n","protected":false},"excerpt":{"rendered":"

Excerpt from \u201cThe Anatomy of a Large-Scale Hypertextual Web Search Engine,\u201d Sergey Brin and Larry Page (April\u00a01998) What follows is the crib from my keynote at the 2017\u00a0Strata Data\u00a0Conference\u00a0in New York City. Full\u00a0video\u00a0can be found here.\u200a In 1998, two graduate students at Stanford decided to try to \u201cfix\u201d the problems with major search engines. Sergey […]<\/p>\n","protected":false},"author":6,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_s2mail":"yes","footnotes":""},"categories":[1],"tags":[],"class_list":["post-6876","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/www.zephoria.org\/thoughts\/wp-json\/wp\/v2\/posts\/6876","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.zephoria.org\/thoughts\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.zephoria.org\/thoughts\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.zephoria.org\/thoughts\/wp-json\/wp\/v2\/users\/6"}],"replies":[{"embeddable":true,"href":"https:\/\/www.zephoria.org\/thoughts\/wp-json\/wp\/v2\/comments?post=6876"}],"version-history":[{"count":1,"href":"https:\/\/www.zephoria.org\/thoughts\/wp-json\/wp\/v2\/posts\/6876\/revisions"}],"predecessor-version":[{"id":6877,"href":"https:\/\/www.zephoria.org\/thoughts\/wp-json\/wp\/v2\/posts\/6876\/revisions\/6877"}],"wp:attachment":[{"href":"https:\/\/www.zephoria.org\/thoughts\/wp-json\/wp\/v2\/media?parent=6876"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.zephoria.org\/thoughts\/wp-json\/wp\/v2\/categories?post=6876"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.zephoria.org\/thoughts\/wp-json\/wp\/v2\/tags?post=6876"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}