{"id":6781,"date":"2016-06-14T15:07:11","date_gmt":"2016-06-14T19:07:11","guid":{"rendered":"http:\/\/www.zephoria.org\/thoughts\/?p=6781"},"modified":"2016-10-07T15:18:34","modified_gmt":"2016-10-07T19:18:34","slug":"be-careful-what-you-code-for","status":"publish","type":"post","link":"https:\/\/www.zephoria.org\/thoughts\/archives\/2016\/06\/14\/be-careful-what-you-code-for.html","title":{"rendered":"Be Careful What You Code For"},"content":{"rendered":"

Most people who don\u2019t code don\u2019t appreciate how hard it is to do right.<\/span>Plenty of developers are perfectly functional, but <\/span>to watch a master weave code into silken beauty is utterly inspiring<\/span>.<\/span> Unfortunately, most of the code that underpins the tools that we use on a daily basis isn\u2019t<\/span> so pretty. T<\/span>here isa lot<\/em> of digital duct tape.<\/span><\/p>\n

<\/p>\n

CC BY-NC 2.0<\/a>-licensed photo<\/a> by Dino\u00a0Latoga<\/a>.<\/p>\n

I\u2019m a terrible <\/span>programmer. Don\u2019t get me wrong\u200a\u2014\u200aI\u2019m perfectly capable of mashing together code to get<\/span> a sorta-kinda-somewhat reasonable outcome.<\/span>But the product is inevitably a Frankensteinesque monstrosity<\/span>. I\u2019m not alone. This is why I\u2019m concerned about the code that is being built. Not all code is created equally.<\/span><\/p>\n

If you want to understand what we\u2019re facing, consider what this would mean if we were <\/span>constructing cities<\/span>. <\/span>In the digital world, we are simultaneously building bridges, sewage systems, and skyscrapers.<\/span> Some of the bridge builders have civil engineering degrees, some of our sewage contractors have been plumbers in past lives, but <\/span>most of the people building skyscrapers have previously only built tree houses and taken a few math classes<\/span>. Oh, and there aren\u2019t any inspectors to assess whether or not it\u2019s all going to fall apart.<\/span><\/p>\n

Code is key to civic life<\/span>, but <\/span>we need to start looking under the hood and thinking about th<\/span>e externalities of our coding practices<\/span>, especially as we\u2019re building code as fast as possible with few checks and balances.<\/span><\/p><\/blockquote>\n

Area One: Environmental Consequences<\/h4>\n

Let\u2019s play a game of math. Almost 1 billion people use Gmail<\/a>. More than that are active on Facebook each month<\/a>. Over 300 million are active on Twitter each month<\/a>. All social media\u200a\u2014\u200aincluding Facebook and Twitter\u200a\u2014\u200asend out notifications to tell you that you have new friend requests, likes, updates, etc. Each one of those notifications is roughly 50KB. If you\u2019re relatively active, you might get 1MB of notifications a day. That doesn\u2019t seem to be that much. But if a quarter of Gmail users get that, this means that Google hosts over 90 petabytes of notifications per year. All of that is sitting live on server so that any user can search their email and find past emails, including the new followers they received in 2007. Is this really a good use of resources? Is this really what we want when we talk about keeping data around?<\/p>\n

The tech industry uses crazy metaphors. Artificial intelligence. Files and folders. They often have really funny roots that make any good geek giggle. (UNIX geeks, did you know that the finger command<\/a> is named as such because that word meant someone is a \u201csnitch\u201d in the 1970s? You probably had a dirtier idea in mind.<\/p>\n

\n
\n
<\/canvas><\/div>\n<\/div>
CC BY 2.0<\/a>-licensed photo<\/a> by Pattys-photos<\/a>.<\/figcaption><\/figure>\n

We don\u2019t know who started calling the cloud the cloud, <\/span>but he (<\/span>and it\u2019s inevitably a he<\/span>) did<\/span>us all a disservice<\/span>.<\/span> When the public hears about the cloud, they think about the fluffy white things in the sky.<\/span> \u201c<\/span>What were the skies like when you were young? They went on forever\u2026And the skies always had little fluffy clouds<\/a>.\u201d<\/span> Those clouds giveth. They offer rain, which gives us water, which is the source of life.<\/span><\/p>\n

But what about the clouds we techies make<\/a>? <\/span>T<\/span>hose clouds take. <\/span>They require rare earth metals and soak up land, power, and water.<\/span> Many big companies are working hard to think about the environmental impact of data centers, to think about the carbon implications.<\/span> (I\u2019m proud to work forone of them<\/a>.) Big companies still have a long way to go, but at least they\u2019re trying. <\/span>But <\/span>how many developers out there are trying to write green code<\/a>?<\/span>At best, folks are thinking about the cost-per-computation, but most developers are pretty sloppy with code and data. <\/span>And there\u2019s no LEED-certified code. <\/span>Who is going to start certifying LEED code!?<\/span><\/p>\n

In the same sense, how many product designers are thinking about the environmental impact of every product design decision they make? <\/span>Product folks are talking about how notifications might annoy or engage users but not the environmental impact of them<\/span>. And for all those open data zealots, is the world really better off having petabytes of data sitting on live servers just to make sure it\u2019s open and accessible just in case? It\u2019s painful to think about how many terabytes of data are sitting in open data repositories that have never been accessed.<\/span><\/p>\n

And don\u2019t get me started about the blockchain or 3D printing or the Internet of Things. <\/span>At least bitcoin got one thing right: this really is about mining<\/a>.<\/span><\/p>\n

Area Two: Social Consequences<\/h4>\n

In the early 2000s, Google thought that I was a truck driver. I got the best<\/em>advertisements. I didn\u2019t even know how many variations of trucker speed there were! All because I did fieldwork in parts of the country that only truckers visit. Consider how many people have received online advertisements that clearly got them wrong.<\/span> Funny, huh?<\/p>\n

Now\u2026Have you ever been arrested? Have you ever been incarcerated?<\/p>\n

Take a moment to think about the accuracy of our advertising ecosystem\u200a\u2014\u200athe amount of money and data that goes into making ads right.<\/span> Now think about what it means that the same techniques that advertisers are using to \u201cpredict\u201d what you want to buy are also being used to predict the criminality of a neighborhood or a person<\/a>.<\/span> And those that work in law enforcement and criminal justice have less money, oversight mechanisms, and technical skills.<\/span><\/p>\n

Inaccuracy and bias are often a given in advertising<\/span>. <\/span>But is it OK that we\u2019re using extraordinarily biased data about previous arrests to predict future arrests<\/a><\/span> and determine where police are stationed?<\/span> Is it OK that we assess someone\u2019s risk at the point of arrest and give judges recommendations<\/a> for bail, probation, and sentencing? Is it OK that local law enforcement agencies are asking tech vendors to predict which children are going to commit a crime before they\u2019re 21?<\/span> Who is deciding, and who is holding them accountable?<\/span><\/p>\n

We might have different political commitments when it comes to policing and <\/span>criminal<\/span> justice. But when it comes to tech and data analysis, I hope that we can all agree that <\/span>accuracy matters.<\/span> Yet, <\/span>we\u2019re turning a blind eye to all of the biases that are baked into the data and, thus, the models that we build.<\/span><\/p>\n

\n
\n
<\/div>\n
<\/canvas><\/div>\n<\/div>
CC BY-NC 2.0<\/a>-licensed photo<\/a> by Thomas\u00a0Hawk<\/a>.<\/figcaption><\/figure>\n

Take a moment to consider that 96% of cases are plead out<\/a>. Those defendants never see a jury of their peers. At a minimum, 10%\u200a\u2014\u200abut most likely much more\u200a\u2014\u200aof those who take a plea are innocent. Why?<\/a> Last I saw, the average inmate at Riker\u2019s waits ~600 days<\/a> for their trial to begin. Average. And who is more likely to end up not making bail? Certainly not rich white folks.<\/span><\/p>\n

Researchers have long known that whites are more likely to use and<\/em> sell drugs.<\/span> And yet, who is arrested for drugs?<\/a> Blacks. 13% of the US population is black, but over 60% of those in prison are black. Mostly for drug crimes.<\/p>\n

Because blacks are more likely to be arrested\u200a\u2014\u200aand more likely to be prosecuted and serve time, guess what our algorithms tell us about who is most likely to commit a drug crime?<\/span> About where drug crimes occur? Police aren\u2019t sent by predictive policing tools to college campuses. They\u2019re sent to the hood.<\/span><\/p>\n

Engineers argue that judges and police officers should know the limits of the data they use. Some do\u200a\u2014\u200athey\u2019re simply ignoring these expensive, tax-payer-costing civic technologies. But in a world of public accountability, where police are punished for not knowing someone was a risk before they shoot up a church, many feel obliged to follow the recommendations for fear of reprisal. <\/span>This is how racism gets built into the structures of our systems.<\/span> And civic tech is implicated in this.<\/span><\/p>\n

I don\u2019t care what your politics are. <\/span>If you\u2019re building a data-driven system and you\u2019re not actively seeking to combat prejudice, you\u2019re building a discriminatory\u00a0system<\/strong><\/span>.<\/strong><\/span><\/p><\/blockquote>\n

Solution: Audits and Inspection<\/span><\/h4>\n

Decisions made involving tech can have serious ramifications that are outside of the mind\u2019s eye of development. We need to wake up<\/span>. <\/span>Our technology is powerful, and we need to be aware of the consequences of our code.<\/span><\/p>\n

Before our industry went all perpetual beta<\/span>, we used to live in a world where <\/span>Test or Quality Assurance meant something<\/span>.<\/span> Rooted in those domains is a practice that can be understood as an internal technical audit.<\/span> We need to get back to this.<\/span> We need to be able to answer simple questions like:<\/p>\n

    \n
  • Does the system that we built produce the right output given the known constraints<\/span>?<\/span><\/li>\n
  • Do we understand the biases and limitations of the system and the output<\/span>?<\/span><\/li>\n
  • Are those clear to the user so that our tool cannot enable poor decision-making or inaccurate impressions?<\/span><\/li>\n
  • What are the true social and environmental costs of the service?<\/span><\/li>\n<\/ul>\n

    We need to start making more meaningful trade-offs. And that requires asking hard questions.<\/span><\/p>\n

    Audits don\u2019t have to be adversarial.<\/span> They can be a way of honestly assessing the limitations of a system and benchmarking for improvement<\/span>. <\/span>This approach is not without problems and limitations, but, <\/span>if you cannot understand whether a model is helping or hurting, discriminating or resulting in false positives, <\/span>then you should not be implementing that technology<\/span> in a high stakes area where freedom and liberty are at stake.<\/span>Stick to advertising<\/span>.<\/span><\/p>\n

    Technology can be amazingly empowering. But only when it is implemented in a responsible manner.<\/span> Code doesn\u2019t create magic. Without the right checks and balances, it can easily be misused.<\/span> I<\/span>n the world of civic tech, we need to conscientiously think about the social and environmental costs, just as urban planners do.<\/span><\/p>\n<\/div>\n<\/div>\n<\/section>\n

    \n
    <\/div>\n<\/section>\n