I find it deeply disturbing that the tech industry represents 9% of the U.S. GDP and only five Big Tech companies account for 25% of the S&P 500. Prior to Covid, most of the growth in stock market came from Big Tech (not the Trump Administration…). Now, as the U.S. economy is all sorts of wacky, Big Tech is what is keeping the stock market’s chin above water. In the process, Big Tech is accounting for more and more of the stock market. ::gulp::

If capitalism and stock markets aren’t your thing, it’s easy to shrug your shoulders at this. But the stock market is infrastructural in profound (and disturbing) ways to American life. Professors: university endowments depend on the stock market staying strong. So do the few remaining pension plans (hiiii government workers!). The S&P 500 is also important for nearly all retirement plans and, much to our collective chagrin, the stability of the banking world itself. Economists tend to scare the heck out of me whenever they talk about how many things are connected to the “overall health of the economy” which is increasingly dependent on a small number of Big Tech companies. And oh boy do they feel the heat to keep the economy chugging along.

Inside the tech industry, there’s another strange calculation. High status employee compensation in tech is also tethered to the stock market. Because of talent wars, tech companies panic when their stocks fall because their employees have no incentive to stay since that’s so much of their compensation. The talent wars have all sorts of other perverse incentives. For example, companies have little incentive to invest in training people for fear that they will go elsewhere. And it shouldn’t be surprising that tech companies conspired to wage-fix in an effort to cap the salaries of certain classes of workers so as to not be in a perennial talent war with each other.

Given how intense the talent wars have been in recent years, I can’t help but be fascinated by the mass layoffs happening now in tech. Over the last few months, companies are coming forward with their tails between their legs saying that they over-hired which is why they needed to lay people off. But did they all really do the exact same thing? Or is there more going on here?

In a classic text in sociology, Paul DiMaggio and Woody Powell mapped out an idea called “institutional isomorphism” where they highlighted how corporations and other large institutional arrangements move in alignment with one another. They describe practices of coordination, mimicry, and normative pressures. In other words, there are structural reasons why companies in entire sectors tend to do the same darn thing.

This might explain the collective over-hiring, but I can’t help but wonder if we’re also watching an inversion of the wage-fixing dynamic. By collectively moving at once, the tech companies are also putting a big pause on the talent wars (outside of the tiny number of very specific roles). Right now, there is widespread fear of job loss across the tech industry. In response, tech workers are staying put. They’re not going anywhere. Unless they’ve been forced out. (Random aside: will we see a massive influx in startups in a few years due to layoffs?)

I wonder if tech leaders think that this hovering threat of more and more layoffs will prompt workers into working harder, faster, more in-line with the company’s goals. Fear is a motivator. Do tech leaders believe that’s effective? Moreover, is it? Are remaining workers helping build the value of these companies at faster rates that benefits the economy? Or is fear creating all sorts of externalities within these companies? I honestly don’t know. I’m waiting for the b-school research!

Amidst the chaos inside the tech industry, we have AI. AI is often described as the cause of the chaos, but I can’t help but wonder if it’s just the hook. AI offers all sorts of imaginaries. And imaginaries are necessary to keeping stock markets going up up up. People want to imagine that this new technology will transform society. They want to imagine that this new technology will strengthen the economy as a whole (even if a few companies have to die).

Many social scientists and historians are critics of AI for reasons that make total sense to me. Technologies have historically reified existing structural inequities, for example. However, the fear-mongering that intrigues me is that coming from within the technical AI community itself. The existential threat conversation is a topic of a different rant, but one aspect of it is relevant here.

Many in the AI tech community believe that self-coding AIs will code humans out of existence and make humans subordinate to AIs. This is fascinating on soooo many levels. The a-historic failure to recognize how humans have repeatedly made other humans subordinate is obviously my first groan. Yet, more specifically to this situation is the failure of extraordinarily high status, high net-worth individuals to reckon with how the tech industry has made people subordinate in a capitalistic context already.

Poke around a bit and these folks will talk about how programmers are doomed. And I can’t help but be fascinated by their angst. At the center of this existential threat is a threat to their own status, power, and domination. They’re afraid that they will become subordinate to the machine (?or other political arrangements?). But they’re projecting this onto all of humanity without appreciating the ways in which so many people already feel subordinate to a machine, namely a particular arrangement of capital and power that is extraordinarily oppressive.

So I keep coming back to this question: How much of the computer science panic of an AI robot takeover is actually coming from an anxiety that their status, power, and wealth is under threat? (And, as such, their agency… But that’s a topic for another rant.)

I keep trying to turn over rocks and make sense of the hype-fear continuum of AI that’s unfolding and what is really standing out to me are the layers and layers of anxiety. Anxiety from tech workers about job precarity and existential risk. Anxiety from tech leaders about the competitiveness of their organizations. Anxieties from national security experts about geopolitical arrangements. Anxieties from climate scientists about the cost of the GPU fights surpassing the crypto mining. Anxieties from economists and politicians about the fundamentals of the economy.

So I keep wondering… what are going to be the outcomes of an anxiety-driven social order at the cornerstone of the economy, the social fabric, and the (geo)political arrangements? History is not comforting here. So help me out… How else should I be thinking of this arrangement? And what else in tangled up in this mess? Cuz more and more, I’m thinking that obsessing over AI is a strategic distraction more than an effective way of grappling with our sociotechnical reality.

When it comes to AI’s potential future impact on jobs, Camp Automation tends to jump to the conclusion that most jobs will be automated away into oblivion. The progressive arm of Camp Automation then argues for the need for versions of universal basic income and other social services to ensure survival in a job-less world. Of course, this being the US… most in Camp Automation tend to panic and refuse to engage with how their views might intersect with late-stage capitalism, structural inequality, xenophobia, and political polarization.

The counterweight to Camp Automation is Camp Augmentation, of which I am far more analytically aligned. Some come to Camp Augmentation because they think that Camp Automation is absolutely nutsoid. But there are also plenty of folks who have studied enough history to have watched how fantasies of automation repeatedly turn into an augmented reality sans ugly headwear.

Mixed into Camp Automation and Camp Augmentation is a cultural panic about what it means to be human anyways. I find this existential angst-ing exhausting for its failure to understand how this is at the core of philosophy. It’s also a bit worrying given how most attempts throughout history of resolving this have involved inventing new religions. Oh, the ripples of history.

While getting into what it means to be human is likely to be a topic of a later blog post, I want to take a moment to think about the future of work. Camp Automation sees the sky as falling. Camp Augmentation is more focused on how things will just change. If we take Camp Augmentation’s stance, the next question is: what changes should we interrogate more deeply? The first instinct is to focus on how changes can lead to an increase in inequality. This is indeed the most important kinds of analysis to be done. But I want to noodle around for a moment with a different issue: deskilling.

Moral Crumple Zones

Years ago, Madeleine Elish decided to make sense of the history of automation in flying. In the 1970s, technical experts had built a tool that made flying safer, a tool that we now know as autopilot. The question on the table for the Federal Aviation Administration and Congress was: should we allow self-flying planes? In short, folks decided that a navigator didn’t need to be in the cockpit, but that all planes should be flown by a pilot and copilot who should be equipped to step in and take over from the machine if all went wrong. Humans in the loop.

Think about that for a second. It sounds reasonable. We trust humans to be more thoughtful. But what human is capable of taking over and helping a machine in a fail mode during a high-stakes situation? In practice, most humans took over and couldn’t help the plane recover. The planes crashed and the humans got blamed for not picking up the pieces left behind by the machine. This is what Madeleine calls the “moral crumple zone.” Humans were placed into the loop in the worst possible ways.

This position for the pilots and copilots gets even dicier when we think about their skilling. Pilots train extensively to fly a plane. And then they get those jobs, where their “real” job is to babysit a machine. What does that mean in practice? It means that they’re deskilled on the job. It means that those pilots who are at the front of every commercial plane are less skilled, less capable of taking over from the machine as the years go by. We depend structurally on autopilot more and more. Boeing took this to the next level by overriding the pilots with their 737 MAX, to their detriment.

To appreciate this in full force, consider what happened when Charles “Sully” Sullenberger III landed a plane in the Hudson River in 2009. Sully wasn’t just any pilot. In his off-time, he retrained commercial pilots how to fly if their equipment failed. Sully was perhaps the best positioned pilot out there to take over from a failing system. But he didn’t just have to override his equipment — he had to override the air traffic controllers. They wanted him to go to Teterboro. Their models suggested he could make it. He concluded he couldn’t. He chose to land the plane in the Hudson instead.

Had Sully died, he would’ve been blamed for insubordination and “pilot error.” But he lived. And so he became an American hero. He also became a case study because his decision to override air traffic control turned out to be justified. He wouldn’t have made it. Moreover, computer systems that he couldn’t override prevented him from a softer impact.

Sully is an anomaly. He’s a pilot who hasn’t been deskilled on the job. Not even a little bit. But that’s not the case for most pilots.

And so here’s my question for our AI futures: How are we going to prepare for deskilling on the job?

How are Skills Developed?

My grandfather was a pilot for the Royal Air Force. When he signed up for the job, he didn’t know how to fly. Of course not. He was taught on the job. And throughout his career, he was taught a whole slew of things on the job. Training was an integral part of professional development in his career trajectory. He was shipped off for extended periods to learn management training.

Today, you are expected to come to most jobs with skills because employers don’t see the point of training you on the job. This helps explain a lot of places where we have serious gaps in talent and opportunity. No one can imagine a nurse trained on the job. But sadly, we don’t even build many structures to create software engineers on the job.

However, there are plenty of places where you are socialized into a profession through menial labor. Consider the legal profession. The work that young lawyers do is junk labor. It is dreadfully boring and doesn’t require a law degree. Moreover, a lot of it is automate-able in ways that would reduce the need for young lawyers. But what does it do to the legal field to not have that training? What do new training pipelines look like? We may be fine with deskilling junior lawyers now, but how do we generate future legal professionals who do the work that machines can’t do?

This is also a challenge in education. Congratulations, students: you now have tools at your disposal that can help you cut corners in new ways (or outright cheat). But what if we deskill young people through technology? How do we help them make the leap into professions that require more advanced skills?

There’s also a delicate balance regarding skills here. I remember a surgeon telling me that you wanted to get scheduled surgery on a Tuesday. Why? Because on Monday, a surgeon is refreshed but a tad bit rusty. By Tuesday, they’re back in the groove but not exhausted. Moreover, there was a fine line between practice and exhaustion — the more that surgeons are expected to do each week, the higher the number of jobs that they’ll do badly at. (Where that holds up to evidence-based scrutiny, I don’t know, but it seems like a sensible myth of the profession.)

Seeing Beyond Efficiency

Efficiency isn’t simply about maximizing throughput. It’s about finding the optimum balance between quality and quantity. I’m super intrigued by professions that use junk work as a buffer here. Filling out documentation is junk work. Doctors might not have to do that in a future scenario. But is the answer to schedule more surgeries? Or is the answer to let doctors have more downtime? Much to my chagrin, we tend to optimize towards more intense work schedules whenever we introduce new technologies while downgrading the status of the highly skilled person. Why? And at what cost?

The flipside of it is also true. When highly trained professionals now babysit machines, they lose their skills. Retaining skills requires practice. How do we ensure that those skills are not lost? If we expect humans to be able to take over from machines during crucial moments, those humans must retain strong skills. Loss of knowledge has serious consequences locally and systemically. (See: loss of manufacturing knowledge in the US right now…)

There are many questions to be asking about the future of work with new technologies on the horizon, many of which are floating around right now. Asking questions about structural inequity is undoubtedly top priority, but I also want us to ask questions about what it means to skill — and deskill — on the job going forward.

Whether you are in Camp Augmentation or Camp Automation, it’s really important to look holistically about how skills and jobs fit into society. Even if you dream of automating away all of the jobs, consider what happens on the other side. How do you ensure a future with highly skilled people? This is a lesson that too many war-torn countries have learned the hard way. I’m not worried about the coming dawn of the Terminator, but I am worried that we will use AI to wage war on our own labor forces in pursuit of efficiency. As with all wars, it’s the unintended consequences that will matter most. Who is thinking about the ripple effects of those choices?

(Some thoughts on the efforts to regulate children’s use of social media)

Have you noticed how many people ages 65+ watch television every day? According to the Bureau of Labor Statistics, almost 90% of those in this age bracket watch TV Every.Single.Day!!!! ::gasp:: And that data was collected before the pandemic! By <hand-waving logics of a moral panic>, it must be so much worse now!

And they don’t just watch a little bit. According to Nielsen, before the pandemic, these elders were watching over 7 hours a day of television! Our elders are glued to their boob tubes.

Television is a serious problem! Their brains are wasting away. Their health is suffering. Their ability to maintain friends is declining. They’re unable to recognize disinformation. Elders’ brains are not fully baked anymore; they can’t handle television. This foolish medium is making fools out of our elders, making them unable to participate responsibly in a democratic society. We must put a stop to this. We must stop TV! And if we can’t stop TV, we must prevent them from watching it!

It is time that we protect our elders by unplugging them. Clearly they won’t do it themselves. And clearly we can’t figure out how to regulate television. So we must regulate our elders’ use of television. For their own good!

Going forward: Only those under 65 should be allowed to watch television.

Now, I know that our elders won’t see how important it is that we do this to them for their own good so we need to develop newfangled surveillance technology to ensure that no one over 65 can turn on their television set. Sure, that technology might be a little creepy, but how bad can it be? It’s not like age verification face scanning technologies could be racist, right!?! And sure, some of those sneaky elders might think that they can trick the system by getting plastic surgery or wearing makeup, but we can put a stop to that too, right? We just need to collect more data from them. I mean, what could go wrong if we collected their name, date of birth, and social security number? That way we’ll know that they’re really who they say they are. Those sneaky elders.

Le sigh.

What is New is Old

For over a decade, I studied how teenagers use social media. I had a front row seat to multiple moral panics. I even wrote an entire book dedicated to unpacking said moral panics: It’s Complicated: The Social Life of Networked Teens. I was also a co-lead on a task force on internet safety where we were asked to identify the dangers that youth were facing and identify interventions that would help. With the help of an amazing advisory board, Andrew Schrock and I scoured the research space trying to map out the different risks teens faced vis-a-vis solicitation, harassment, and problematic content. Little did I understand at the time that my real job was to “prove” that age verification technologies were the “solution” to all online safety problems. I learned that lesson the hard way when our research led us to a different set of recommended interventions. This lesson was pounded into me when a state Attorney General yelled at me to “go find different data” and when a Frontline reporter told me that she was encouraged to investigate my efforts to show that I was falsifying data to defend tech companies. (She concluded I was not falsifying data and the story never happened.)

But here we are again. A new moral panic is unfolding around teenagers’ use of social media. And once again, the “solution” seems to be to restrict access and use age verification technologies to enforce this approach. A few weeks ago, Utah took the first stab with a law that prohibits minors from accessing social media “without parental permission.” At first blush, this looks like an extension of the federal Children’s Online Privacy Protection Act (COPPA) that requires for-profit websites that collect data about children under 13 to get permission from parents.

COPPA seemed like a good idea at the time it was passed back in the 1990s. In practice, COPPA is the reason why all sites require you to be 13+ to join. Of course, every social media company knows children under the age of 13 are lying about their age to get access to their sites. A decade ago, Eszter Hargittai, Jason Schultz, John Palfrey, and I decided to figure out what parents thought about this. We quickly discovered that parents teach their kids to lie to get access to social media. So much for that being effective. (Full disclosure: I created dozens of accounts for my kids for different sites during the pandemic. Over and over, I’ve been stymied by the processes of parental approval and just given up and given them a false birthdate.)

Utah’s law goes beyond COPPA because it’s not just worried about data privacy and advertisement. It’s centered on a purported mental health crisis that kids are facing, supposedly because of social media.

All of this seems to connect back to a dubious interpretation of a Centers for Disease Control report on “Youth Risk Behavior.” The report is super interesting. It turns out that teenagers are having less sex (although those who are might be engaged in more risky sex). It also turns out that bullying at school declined during the pandemic (duh) but that bullying online didn’t go up or down even then.

But the thing that caught the eye of regulators was that mental health seems to have skyrocketed in recent years. I looked at this data and shook my head. My head swirled thinking about the pandemic, the rise in financial instability and food scarcity in some communities, the rising costs of college, the rise in visible hate speech, anti-trans and anti-abortion legislation, the fear kids have of a mass shooter at school, and a slew of other trends that I hear young people angst about. But apparently regulators preferred a different interpretation. They looked at this and went: “blame social media!!!”

Jessica Grose took many of these interpretations to task in her op-ed “Stop Treating Adolescent Girls as Emotionally Abnormal.” I want to call particular attention to her colleague’s remark: “The most predictable thing in the world is for people to respond to this article with their own reasons for why this is taking place based entirely on their own specific hills on which they have decided to die.”

Still, most news coverage of these stories were full of sheer panic. WashPo responded to this study with a story titled “The crisis of student mental health is much vaster than we realize.” Their editorial board followed up with a note that “America’s teens are in crisis. States are racing to respond.” My immediate thought was: are they? They’re looking to ban social media as though that’s the cause of this crisis. As though if social media goes away, the problem will go away. A Financial Times reporter took it to the next level, conflating correlation and causation with the headline that “Smartphones and social media are destroying children’s mental health” (note: the story itself is full of hedge language like “may”). And then a writer at The New Yorker penned a piece entitled “The case for banning children from social media” that hinges on his own experiences as a parent.

I cringed. One basic rule of research is never to take one’s personal experiences as extrapolate based on them. And one thing I learned as a researcher of young people is that parents will always look for something to blame in a way that minimizes their own agency. And I get it. Parenting is haaaard. And emotionally exhausting. And guilt-inducing. It’s soooo much safer to justify the situation that’s frustrating you by blaming structural conditions that you can’t do anything about. But it’s not honest. And it doesn’t hold up empirically.

The CDC survey offers sound empirical evidence that young people are currently reporting higher levels of duress. There’s also a lot of other empirical signal that mental health struggles are on the rise. Those who follow these trends over decades aren’t be surprised. Adults are also more anxious and more depressed right now. It turns out that tends to impact kids. Financial instability, political polarization, food scarcity, geopolitical conflict, and many other factors tend to correlate with anxiety and depression, even if causality is messier. Lots of trend lines are all over the place right now on lots of different measures.

Two “new” factors are harder to evaluate. One is the pandemic. Researchers generally expect this to have negative repercussions within community but it’ll take a lot of work to tease out what is the pandemic directly and what are ripple effects (e.g., financial instability). The other new one that has become the modern day boogie many is whatever the new technology is. Social media (and, more recently, mobile phones) have been favorites for the last decade.

My research consistently found that teens turn to these technologies to connect with others, especially when they were struggling. Surprise surprise, when kids were stuck at home during the pandemic, they wanted to talk with their friends via phone, social media, and in video games. When young people feel isolated, they look for others like them in various online fora. And so, yes, there will be a correlation between certain kinds of online behaviors and mental health states.

Where things get dicier concerns causality. Chicken and egg. Does social media cause mental health problems? Or is it where mental health problems become visible? I can guarantee you that there are examples of both. But here’s the thing…. Going to school and church are often a “cause” of mental health duress. Parents and siblings are often a source of mental health duress. No one in their right mind would argue that we need to prevent all youth from attending school or church or living with their parents or siblings. We take a more tempered approach because there are also very real situations in which we need to remove some children from some environments (namely abusive ones).

So why do we want to remove ALL children from social media?

This is a story of control, not a story of protecting the well-being of children.

A century ago, we forced teenagers into compulsory high school to prevent them from being able to fraternize with older adults because we were afraid that 16yos would compete with adults for jobs as the Great Depression was unfolding. Fifty years ago, moral panics around comic books normalized a world in which we restricted children’s access to content. Can we admit that much of this content was political in nature and those who restricted it opposed those politics? Now we’re back to book banning and “don’t say gay” frames. This is not about children’s mental health. This is about preventing children from being active members of our contemporary political polis. This is about using rhetoric around children’s “innocence” to ensure that they don’t encounter views that politicians don’t want them to have. This isn’t new. This is as old of a strategy as it gets.

I care deeply about children’s mental health. And there’s a lot that can and should be done. Let’s start with giving every child access to mental healthcare. Let’s make talking to a counselor free. Let’s ensure that children can talk to a trained therapist without being surveilled by their parents (or even needing parental permission).

I am deeeeeeply worried about social and structural conditions that increase mental health crises. Let’s eradicate food scarcity. Let’s make it possible for parents to stay home with newborns and sick children without being docked pay or losing their jobs. Let’s build a social safety net.

I also fully know how frustrating it is to see your own child struggling and escaping into a zombie state in front of a screen. But parents, please take a deep breath and look at the situation more holistically. Why is this giving them pleasure? What are they escaping? What social itch are they scratching? And are you able to create other paths for pleasure, escape, and socialization?

Revisiting Our Collective Habits

I began this post satirically by focusing on elders and television. But let’s also be real. Many elders do have a seriously unhealthy relationship with television at this point. We know that the answer is not to ban elders from accessing TV (even if some of us might really really want that). But what we can see in this unhealthy dynamic is an important lesson about habits, a lesson that applies to all of us.

Many elders got into the habit of watching TV years ago. It may have started out with the nightly news or prime time TV, an opportunity to escape after an exhausting day of work. And it expanded from there. For many, the pandemic made it much worse. And as they watched more TV, it got harder to do other things. Other things were exhausting physically. Or mentally.

This is not the only bad habit we’ve seen adults develop over time. We have a better framework for talking about what happens when a glass of wine after work turns into a bottle of wine a day habit.

What we do know is that breaking habits is HARD. And it’s hard for everyone. This is why, as parents, we don’t want to see our kids develop bad habits. And, especially after the acute phase of the pandemic, many of us recognize that we — and our kids — have gotten into bad habits around technology. We used technology as a babysitter while we were trying to work from phone. And we haven’t broken that habit at all. But block our kids from accessing social media through regulation will not produce a healthy response to technology overnight. If we want to change our habits in relationship to technology because we don’t like them, we need to be thoughtful about them.

When I was spending lots of time with teenagers, one of the things that they always told me was that parents were the real addicts. They couldn’t let go of their phone (or Twitter or … ). I looked around and realized how true this is. Go to a kids’ sports game or playground and you’ll see a bunch of parents staring into their devices. So, parents, here’s a thing you can do. Every time you pick up your device in front of your kids, verbalize what you’re doing. “I’m looking up directions” will be easy to say out loud. “You’re annoying me so I’m going to look at TikTok” will be far more uncomfortable. Set a new habit. Be visible about why you are using technology and ask your kids to do the same. Talk with them about your bad habits and ask them to hold you accountable. Then you can build trust and ask the same of them.

These bills aren’t tools to empower parents or address a very real mental health crisis. They’re a mechanism to control youth, enrich age verification vendors, and turn our kids into political pawns.

These laws sound good because we are worried about our kids and because there is deep and reasonable animosity towards Big Tech. (The geopolitical fight over TikTok is adding to the chaos.) Let’s pass data privacy laws that protect all of us (including our elders who are an identity theft nightmare!). Let’s build mental health infrastructure. Let’s increase our social safety net. But please please, let us not ban children from social media “for their own safety.” Cuz that’s just not what this is about.

AI is here and it will change everything! OMG the sky is falling! Programmers are now obsolete. No, they are needed more than ever before. Large language models will destroy journalism, democracy, society. No, they will free us from drudgery and we will all be happier. Cancer will be solved because of AI. AI’s hallucinations will usher in a new era of disinformation.

I just returned from a three month sabbatical spent mostly offline diving through history and I feel like I’ve returned to an alien planet full of serious utopian and dystopian thinking swirling simultaneously. I find myself nodding along because both the best case and worst case scenarios could happen. But also cringing because the passion behind these declarations has no room for nuance. Everything feels extreme and fully of binaries. I am truly astonished by the the deeply entrenched deterministic thinking that feels pervasive in these conversations.

Deterministic thinking is a blinkering force, the very opposite of rationality even though many people who espouse deterministic thinking believe themselves to be hyper rational. Deterministic thinking tends to seed polarization and distrust as people become entrenched in their vision of the future. Returning to the modern world, I’m finding myself frantically waving my hands in a desperate attempt to get those around me to take a deep breath (or maybe 100 of them). Given that few people can see my hand movements from my home office, I’m going to pretend like it’s 2004 and blog my thoughts in the hopes that this post might calm at least one person down. Or, maybe, if I’m lucky, it’ll be fed into an AI model and add a tiny bit of nuance.

What is deterministic thinking?

Simply put, determinism is “if x, then y.” It is the notion that if we do something (x), a particular outcome (y) is inevitable. Determinisms are not inherently positive or negative. It is just as deterministic to say “if we build social media, the world will be a more connected place” as it is to say “social media will destroy democracy.”

It is extraordinarily common for people who are excited about a particular technology to revert to deterministic thinking. And they’re often egged on to do so. Venture capitalists want to hear deterministic thinking in sales pitches. Same with the National Science Foundation. In many social science fields, these futuristic speech acts gets labeled in a pejorative manner as “technological determinism.” Inventors and corporations are often accused of being deterministic, which is a shorthand intended to dismiss their rhetoric as ahistoric and socially oblivious. (Academics who study technological determinism are often more nuanced in this scholarship than their blog posts and op-eds.)

Meanwhile, however, tech critics often fall into the same trap. Many professional critics (including academics, journalists, advocates) are incentivized to do so because such rhetoric appeals to funders and makes for fantastic opinion pieces. Their rhetoric is rarely labeled deterministic because that’s not the language of futurists. Rather, they are typically dismissed as being clueless about technology and anti-progress. They’re regularly not invited to “the party” (where the technology is being debated by those involved in creating it) because they’re seen as depressing.

Determinism’s ugly step-sibling is “solutionism.” Solutionism is the belief that x will be a solution not just to achieve y but to achieve all possibly y’s. Solutionists tend to be so enamored with x that they cannot engage with any criticism of x.

In a world where technologies are given power, authority, and funding, those with positive deterministic views (and solutionistic mindsets) often have more resources and power, leading to a mutually self-destructive polar vortex rich with righteousness that is anything but rational. This is often visible through shouting matches. Right now, the cacophony is overwhelming. And it’s breaking my brain.

Embracing Probabilistic Futures

The counter to determinism is not indeterminism. It is unsatisfying to throw our hands up in the air and say “any future is possible” because, well, that’s kinda bullshit. Some futures are more likely to occur than others. Some outcomes are more likely because of a particular technical intervention than others.

The key to understanding how technologies shape futures is to grapple holistically with how a disruption rearranges the landscape. One tool is probabilistic thinking. Given the initial context, the human fabric, and the arrangement of people and institutions, a disruption shifts the probabilities of different possible futures in different ways. Some futures become easier to obtain (the greasing of wheels) while some become harder (the addition of friction). This is what makes new technologies fascinating. They help open up and close off different possible futures.

To be clear, this way of thinking isn’t unique to technology. You can treat laws and regulations the same way, for example. Policies are often implemented with a vision of a new future; the reality of a new law is that things change but it’s not always as predictable as politicians might hope.

At both the macro and micro level, a wide range of interventions into a system rearrange possible futures. Bridges rearrange the social fabric of a city while chemotherapy rearranges the health futures of a patient. Neither determine the future. They just make some futures more likely and other futures less likely.

Context matters here. A bridge to nowhere (oh, thank you pork barrels) doesn’t have as much impact on futures as a bridge connecting two metropolises for the first time. Chemotherapy might be a general all-purpose intervention for cancer patients but the trade-offs between its potential upsides and downsides vary tremendously based on the particular cancer and the particular patient, and thus the probabilistic futures made possible are not consistent or always constructive.

When we’re dealing with medical interventions, we’re always living and breathing probabilistic futures. We hope that an intervention has a desired outcome but no responsible doctor is willing to be deterministic when treating a patient. Wise doctors think holistically about the patient and enroll them into probabilistic thinking to decide how to intervene in order to move towards desired futures. They take context into account and make probabilistically driven recommendations. (Shitty doctors can also be solutionistic in mindset, which sucks.)

Strangely, however, when it comes to a lot of other technologies, probabilistic thinking goes out the window. I find that especially odd when it comes to discussing artificial intelligence given that most AI models are the sheer manifestation of probabilistic thinking. There is no one future of LLMs (and many of those closest to these developments get this). And yet, hot damn is the rhetoric nutsoid.

The Blessings and Curses of Projectories

Even though deterministic thinking can be extraordinarily problematic, it does have value. Studying the scientists and engineers at NASA, Lisa Messeri an Janet Vertesi describe how those who embark on space missions regularly manifest what they call “projectories.” In other words, they project what they’re doing now and what they’re working on into the future in order to create for themselves a deterministic-inflected roadplan. Within scientific communities, Messeri and Vertesi argue that projectories serve a very important function. They help teams come together collaboratively to achieve majestic accomplishments. At the same time, this serves as a cognitive buffer to mitigate against uncertainty and resource instability. Those of us on the outside might reinterpret this as the power of dreaming and hoping mixed with outright naiveté.

We can dismiss these scientists’ projectories (and the projectories of the institution) as delusional, but a lot of creativity comes from delusional thinking. This is why I often have a larger tolerance for projectories and deterministic fantasy-making that many social critics.

Where things get dicy is where delusional thinking is left unchecked. Guard rails are important. NASA has a lot of guardrails, starting with resource constraints and political pressure. But one of the reasons why the projectories of major AI companies is prompting intense backlash is because there are fewer other types of checks within these systems. (And it’s utterly fascinating to watch those deeply involved in these systems beg for regulation from a seemingly sincere place.)

Right now, the check that is most broadly accessible is a reputational check. And so, in a fascinating twist of fate, those who are trying to push back against the development of AI systems have reverted to the same act of generating projectories that are daaaaaark and dystopic. (And to be clear, many of those who are most engaged in these alternative projectories are themselves AI researchers and developers.) The scramble for thought leadership and control of the narrative is overwhelming.

Projectories have power. Power for those who are trying to invent new futures. Power for those who are trying to mobilize action to prevent certain futures. And power for those who are trying to position themselves as brokers, thought leaders, controllers of future narratives in this moment of destabilization. But the downside to these projectories is that they can also veer way off the railroad tracks into the absurd. And when the political, social, and economic stakes are high, they can produce a frenzy that has externalities that go well beyond the technology itself. That is precisely what we’re seeing right now.

A Different Path Forward

Rather than doubling down on deterministic thinking by creating projectories as guiding lights (or demons), I find it far more personally satisfying to see projected futures as something to interrogate. That shouldn’t be surprising since I’m a researcher and there’s nothing more enticing to a social scientist than asking questions about how a particular intervention might rearrange the social order.

But I also get the urge to grab the moment by the bull’s horns and try desperately to shape it. This is the fascinating thing about a disruption like what we’re seeing with AI technologies. It rearranges power, networks, and institutions. And we’re watching a scramble by individuals and organizations to obtain power in this insanity. In these moments, there is little space for deeply reflexive thinking, for nuanced analysis, which is especially unfortunate because that’s precisely what we need right now. Not to predict futures (or to prevent them) but to build the frameworks of resilience.

Consider the best case scenario of a doctor and patient navigating a medical intervention. In such a situation, there are mechanisms for data collection (ranging from bio-specimens to verbal reflections) and an iterative process for choosing how to proceed. Each medical treatment is viewed as an intervention that needs to be considered with future steps taken based on information gleaned in the process.

How do we do this same activity at scale? How do create significant structures to understand and evaluate the transformations that unfold as they unfold and feed those back into the development cycle? How do we build assessment protocols for evaluating new AI models?

Consider the rhetoric that surrounds how AI will disrupt XYZ industry. Some researchers, executives, and workers who know those industries are shifting their energies to ask these questions. But we don’t have mechanisms in place to really “see” let alone evaluate the disruptions. Why not? If pundits are predicting such disruptions, shouldn’t we be building the mechanisms to see if those futures are unfolding the way that determinists think they will?

Those who are building new AI systems are talking extensively about the potential for disruption (with both enthusiasm and fear) but I see very little scaffolding outside of the sciences to even reflect on how these disruptions will unfold. I know that researchers are scrambling to jump in (often with few dedicated resources) and organizational leaders are convening meetings to discuss their postures vis-a-vis these new systems, but I’m surprised by how little scaffolding there is to ensure that there are evaluations along the way.

And that brings me back to a question about all of these disruptions. Is the goal to disrupt and then let a free-for-all happen because that’s how change should occur? Or is the goal to disrupt to actually drive towards futures that people imagine? Cuz right now, even with the rhetoric of the latter, the former seems more at play.

Deterministic rhetoric lacks nuance. It also lacks an appreciation for human agency or a recognition that disruptions are situated within complex ecosystems that will shapeshift along the way. But what bothers me most about the deterministic framing that is hanging over all things AI right now is that it’s closing out opportunities for deeper situated thinking about the transformations that might unfold over the next few years.

How can we push past our current love affair with determinism so that we can have a more nuanced, thoughtful, reflexive account of technology and society?

I, for one have no clue what’s coming down the pike. But rather than taking an optimistic or a pessimistic stance, I want to start with curiosity. I’m hoping that others will too.

This update is to let you know about a new essay that’s now online in in-press form: “Differential Perspectives: Epistemic Disconnects Surrounding the US Census Bureau’s Use of Differential Privacy.” Click here to read the full essay.

When the U.S. Census Bureau announced its intention to modernize its disclosure avoidance procedures for the 2020 Census, it sparked a controversy that is still underway. The move to differential privacy introduced technical and procedural uncertainties, leaving stakeholders unable to evaluate the quality of the data. More importantly, this transformation exposed the statistical illusions and limitations of census data, weakening stakeholders’ trust in the data and in the Census Bureau itself.

Jayshree Sarathy and I have been trying to make sense of the epistemic currents of this controversy. In other words, how do divergent ways of sense-making shape people’s understanding of census data – and what does that tell us about how people deal with census data controversies.

We wrote an essay for an upcoming special issue of Harvard Data Science Review that will focus on differential privacy and the 2020 Census. While the special issue is not yet out, we were given permission to post our in-press essay online. And so I thought I’d share it here for those of you who relish geeky writings about census, privacy, politics, and controversies. This paper draws heavily on Science and Technology Studies (STS) theories and is based on ethnographic fieldwork. In it, we analyze the current controversy over differential privacy as a battle over uncertainty, trust, and legitimacy of the Census. We argue that rebuilding trust will require more than technical repairs or improved communication; it will require reconstructing what we
identify as a ‘statistical imaginary.’ Check out our full argument here.

For those who prefer the tl;dr video version, I sketched out some of these ideas at the Microsoft Research Summit in the fall.

We are still continuing to work through these ideas so by all means, feel free to share feedback or critiques; we relish them.

Like everyone who cares about Crisis Text Line and the people we serve, I have spent the last few days reflecting on recent critiques about the organization’s practices. Having spent my career thinking about and grappling with tech ethics and privacy issues, I knew that – had I not been privy to the details and context that I know – I would be outraged by what folks heard this weekend. I would be doing what many of my friends and colleagues are doing, voicing anger and disgust. But as a founding board member of Crisis Text Line, who served as board chair from June 2020 until the beginning of January 2021, I also have additional information that shaped how I thought about these matters and informed my actions and votes over the last eight years. 

As a director, I am currently working with others on the board and in the organization to chart a path forward. As was just announced, we have concluded that we were wrong to share texter data with Loris.ai and have ended our data-sharing agreement, effective immediately. We had not shared data since we changed leadership; the board had chosen to prioritize other organizational changes to support our staff, but this call-to-action was heard loud and clear and shifted our priorities. But that doesn’t mean that the broader questions being raised are resolved. 

Texters come to us in their darkest moments. What it means to govern the traces they leave behind looks different than what it means to govern other types of data. We are always asking ourselves when, how, and should we leverage individual conversations borne out of crisis to better help that individual, our counselors, and others who are suffering. These are challenging ethical questions with no easy answer. 

What follows is how I personally thought through, balanced, and made decisions related to the trade-offs around data that we face every day at Crisis Text Line. This has been a journey for me and everyone else involved in this organization, precisely because we care so deeply. I owe it to the people we serve, the workers of Crisis Text Line, and the broader community who are challenging me to come forward to own my decisions and role in this conversation. This is my attempt to share both the role that I played and the framework that shaped my thinking. Since my peers are asking for this to be a case study in tech ethics, I am going into significant detail. For those not seeking such detail, I apologize for the length of this. 

Most of the current conversation is focused on the ethics of private-sector access to messages from texters in crisis. These are important issues that I will address, but I want to walk through how earlier decisions influenced that decision. I also want to share how the ethical struggles we face are not as simple as a binary around private-sector access. There are ethical questions all the way down.

What follows here is, I want to emphasize, my personal perspective, not the perspective of the organization or the board. As a director of Crisis Text Line, I have spent the last 8 years trying to put what I know about tech ethics into practice. I am grateful that those who care about tech ethics are passionate about us doing right by our texters. We have made changes based on what we have heard from folks this weekend. But those changes are not enough. We need to keep developing and honing guiding principles to govern our work. My goal has been and continues to be ensuring ethical practices while navigating the challenges of governing both an organization and data. Putting theory into practice continues to be more challenging than I ever imagined. Given what has unfolded, I would also love advice from those who care as I do about both mental health and tech ethics.

First: Why data?

Even before we launched the CTL service, I knew that data would play a significant role in the future of the organization. My experience with tech and youth culture was why I was asked to join the board. Delivering a service that involved asynchronous interactions via text would invariably result in the storage of data. Storing data would be needed to deliver the service; the entire system was necessarily designed to enable handoffs between counselors and to allow texters to pick up conversations hours (or days) later.

Storing data immediately prompted three key questions:

1. How long would we store the data that users provided to us?
2. Could we create a secure system?
3. Under what conditions would we delete data?

As a board, we realized the operational necessity of stored data, which meant an investment in the creation of a secure system and deep debate over our data retention policies. We decided that anyone should have the right to remove their data at any point, a value I strongly agreed with. The implementation of this policy relied on training all crisis counselors how to share this info with texters if they asked for it; we chose to implement the procedure by introducing a codeword that users could share to trigger a deletion of their data. (This was also documented as part of the terms of service, which texters were pointed to when they first contacted us. I know that no one in crisis reads lawyer-speak to learn this, which is why I was more interested in ensuring that our counselors knew this.)

Conducting the service would require storing data, but addressing the needs of those in crises required grappling with how data would be used more generally. Some examples of how data are used in the service: 

• When our counselors want to offer recommendations for external services, they pull on outside data to bring into the conversation; this involves using geographic information texters provide to us.
• Our supervisors review conversations both to support counselors real-time and give feedback later with an eye towards always improving the quality of conversations.

Our initial training program was designed based on what we could learn from other services, academic literature, and guidance from those who had been trained in social work and psychology. Early on, we began to wonder how the conversations that took place on our platform could and should inform the training itself. We knew that counselors gained knowledge through experience, and that they regularly mentored new counselors on the platform. But could we construct our training so that all counselors got to learn from the knowledge developed by those who came before them? 

This would mean using texter data for a purpose that went beyond the care and support of that individual. Yes, the Terms of Service allowed this, but this is not just a legal question; it’s an ethical question. Given the trade-offs, I made a judgment call early on that not only was using texter data to strengthen training of counselors without their explicit consent ethical, but that to not do this would be unethical. Our mission is clear: help people in crisis. To do this, we need to help our counselors better serve texters. We needed to help counselors learn and grow and develop skills with which they can help others. I supported the decision to use our data in this way.

A next critical turning point concerned scale. My mantra at Crisis Text Line has always been to focus on responsible scaling, not just scaling for scaling sake. But we provide a service that requires a delicate balance of available counselors to meet the needs of incoming texters. This meant that we had to think about how to predict the need and how to incentivize counselors to help out at spike moments. And still, there were often spikes where the need exceeded the availability of counselors. This led us to think about our ethical responsibilities in these moments. And this led to another use of data: 

• When there are spikes in the service without enough counselors, we triage incoming requests to ensure that those most at physical risk get served fastest; this requires analyzing the incoming texts even before a conversation starts.

This may not seem like a huge deal, but it’s an ethical decision that I’ve struggled with for years. How do you know who is in most need from just intake messages? Yes, there are patterns, but we’ve also learned over the years that these are not always predictable. More harrowingly, we know retrospectively that these signals can be biased. Needless to say, I would simply prefer for us to serve everyone, immediately. But when that’s not possible, what’s our moral and ethical responsibility? Responding to incoming requests in order might meet some people’s definition of “fair,” but is that ethical? Especially when we know that when people are in the throes of a suicide attempt, time is of the essence? I came to the conclusion that we have an ethical responsibility to use our data to work to constantly improve the triage algorithm, to do the best we can to identify those for whom immediate responses can save a life. This means using people’s data without their direct consent, to leverage one person’s data to help another. 

Responsible scaling has introduced a series of questions over the years. I’ve reflected in my head on one for years that we’ve never implemented: Should we attempt to match need to expertise? In other words, should our counselors specialize? To date, we haven’t, but it’s something I think a lot about. But there are also questions that have been raised that we have intentionally abandoned. For example, there was once a board meeting where the question of automation came up. We already use some automation tools in training and for intake; should some conversations be automated? This was one of those board meetings where I put my foot down. Absolutely not. Data could be used to give our counselors superpowers, but centering this service on humans was essential. In this context, my mantra has always been augmentation not automation. The board and organization embraced this mantra, and I’m glad for it.

Next: Data for Research

From early on, researchers came to Crisis Text Line asking for access to data. This prompted even more reflection. We had significant data and we were seeing trends that had significant implications for far more than our service. We started reporting out key trends, highlighting patterns that we then published on our website. I supported this effort because others in the ecosystem told us it helped them to learn from the patterns that we were seeing. This then led to the more complicated issue of whether or not to allow external researchers to study our data with an eye towards scholarship. 

I’m a scholar. I know how important research is and can be. I knew how little data exists in the mental health space, how much we had tried to learn from others, how beneficial knowledge could be to others working in the mental health ecosystem. I also knew that people who came to us in crisis were not consenting to be studied. Yes, there was a terms of service that could contractually permit such use, but I knew darn straight that no one would read it, and advised everyone involved to proceed as such. 

I have also tracked the use of corporate data for research for decades, speaking up against some of Facebook’s experiments. Academic researchers often want to advance knowledge by leveraging corporate data, but they do not necessary grapple with the consequences of using data beyond IRB requirements. There have been heated debates in my field about whether or not it is ethical to use corporate trace data without the consent of users to advance scientific knowledge. I have had a range of mixed feelings about this, but have generally come out in opposition to private trace data being used for research. 

So when faced with a similar question at Crisis Text Line, I had to do a lot of soul searching. Our mission is to help people. Our texters come to us in their darkest hours. Our data was opening up internal questions right and left about how to best support them. We don’t have the internal resources to analyze the data to answer all of our questions, to improve our knowledge base in ways that can help texters. I knew that having additional help from researchers could help us learn in ways that would improve training of counselors and help people down the line. I also knew that what we were learning internally might be useful to other service providers in the mental health space and I felt queasy that we were not sharing what we had learned to help others.

Our organization does not exist for researchers to research. Our texters do not come to us to be research subjects. But our texters do come to us for help. And we do help them by leveraging what we learn helping others, including researchers. Texters may not come to us to pay it forward for the next person in need, but in effect, that’s what their engagement with us was enabling. I see that as an ethical use of data, one predicated on helping counselors and texters through experience mediated by data. The question in my mind then was: what is the relationship of research to this equation?

I elected to be the board member overseeing the research efforts. We have explored – and continue to explore – the right way to engage researchers in our work. We know that they are seeking data for their own interests, but our interest is clear: can their learnings benefit our texters and counselors, in addition to other service providers and the public health and mental health ecosystem. To this end, we have always vetted research proposals and focused on research that could help our mission, not just satisfy researcher curiosity. 

Needless to say, privacy was a major concern from day one. Privacy was a concern even before we talked about research; we built privacy processes even for internal analyses of data. But when research is involved, privacy concerns are next-level. Lots of folks have accused us of being naive about reidentification over the last few days, which I must admit has been painful to hear given how much time I spend thinking about and dealing with reidentification in other contexts. I know that reidentification is possible and that was at the heart and soul of our protocols. Researchers have constrained access to scrubbed data under contract precisely because there’s a possibility that, even with our scrubbing procedures, reidentification might be possible. But, we limited data to minimize reidentification risks and added contractual procedures to explicitly prevent reidentification.

When designing these protocols, my goal was to create the conditions where we could learn from people in crisis to help others in crisis without ever, in any way, adding to someone’s crisis. And this means privacy-first.

More generally though, the research question opened up a broader set of issues in my mind. Our service can directly help individuals. What can and should we do to advance mental health more generally? What can and should we be providing to the field? What is our responsibility to society outside our organization?

Next: Training as a Service

Our system is based on volunteers who we train to give counsel. As is true in any volunteer-heavy contexts, volunteers come and go. Training is resource intensive, but essential for the service. Repeatedly, volunteers approached us as a board to tell us about the secondary benefits of the training. Yes, the training was designed to empower a counselor to communicate with a person who was in crisis, but these same skills were beneficial at work and in personal relationships. Our counselors kept telling us that crisis management training has value in the world outside our doors. This prompted us to reflect on the potential benefit of training far more people to manage crises, even if they did not want to volunteer for our service.

The founder of Crisis Text Line saw an opportunity and came to the board. We did not have the resources to simply train anyone who was interested. But HR teams at companies had both the need for, and the resources for, larger training systems. The founder proposed building a service that could provide us with a needed revenue stream. I don’t remember every one of the options we discussed, but I do know that we talked about building a separate unit in the organization to conduct training for a fee. This raised the worry that this would be a distraction to our core focus. We did all see training as mission-aligned, but we needed to focus on the core service CTL was providing. 

We were also struggling, as all non-profits do, with how to be sustainable. Non-profit fundraising is excruciating and fraught. We were grateful for all of the philanthropic organizations who made starting the organization possible, but sustaining philanthropic funding is challenging and has significant burdens. Program officers always want grantees to find other sources of money. There are traditional sources: foundations, individual donors, corporate social responsibility donations. In some contexts, there’s government funding, though at that time, government was slashing funding not increasing it. Funding in the mental health space is always scarce. And yet, as a board, we always had a fiduciary responsibility to think about sustainability.  

Many of the options in front of us concerned me deeply. We could pursue money by billing insurance companies, but this had a lot of obvious downsides to it. Many of the people we serve do not have access to insurance. Moreover, what insurers really want is our data, which we were strongly against. They weren’t alone – many groups wanted to buy our data outright. We were strongly against those opportunities as well. No selling of data, period. 

Big tech companies and other players were increasingly relying on CTL as their first response for people in crisis, without committing commensurate (or sometimes, any) resources to help offset that burden. This was especially frustrating because they had the resources to support those in crisis but had chosen not to, preferring to outsource the work but not support it. They believed that traffic was a good enough gift.

This was why we, as a board, were reflecting on whether or not we could build a revenue stream out of training people based on what we learned from training counselors. In the end, we opted not to run such an effort from within Crisis Text Line, to reduce the likelihood of distracting from our mission. Instead, we gave the founder of Crisis Text Line permission to start a new organization, with us retaining a significant share in the company; we also retained the right to a board seat. This new entity was structured as a for-profit company designed to provide a service to businesses, leveraging what we had learned helping people. This company is called Loris.ai.

Loris.ai planned on learning from us to build training tools for people who were not going to serve as volunteers for our service. Yet, the company was a separate entity and the board rejected any plan that involved full access to our systems. Instead, we opted to create a data-sharing agreement that paralleled the agreement we had created with researchers: controlled access to scrubbed data solely to build models for training that would improve mental health more broadly. We knew that it did not make sense for them to directly import our training modules; they would be training people in a different context. Yet, both they and we believed that there were lessons to be learned from our experiences, both qualitatively and quantitatively.

I struggled with this decision at the time and ever since. I could see both benefits and risks in sharing our data with another organization, regardless of how mission-aligned we were. We debated this in the boardroom; I pushed back around certain proposals. In the end, some of the board members at the time saw this decision through the lens of a potential financial risk reduction. If the for-profit company did well, we could receive dividends or sell our stake in order to fund the crisis work we were doing. I voted in favor of creating Loris.ai for a different reason.  If another entity could train more people to develop the skills our crisis counselors were developing, perhaps the need for a crisis line would be reduced. After all, I didn’t want our service to be needed; the fact that it is stems from a system that is deeply flawed. If we could build tools that combat the cycles of pain and suffering, we could pay forward what we were learning from those we served. I wanted to help others develop and leverage empathy. 

This decision weighed heavily on me, but I did vote in favor of it. Knowing what I know now, I would not have. But hindsight is always clearer.

Existential Crisis

In June of 2020, our employees came to us with grave concerns about the state of the organization. This triggered many changes to the organization and a reckoning as a board. I stepped in as board chair. As we focused on addressing the issues raised by employees, I felt as though we needed to prioritize what they were telling us. My priority was to listen to our staff, center the needs of our workers and texters, learn from them, and focus on our team, core business, and organizational processes. We also needed to hire a permanent CEO. The concerns we received were varied and diverse, requiring us to prioritize what to focus on when. 

Data practices were not among the dominant concerns, but they were among the issues raised. The most significant data concern raised to us was whether our data practices were as strong as the board believed them to be. This prompted three separate, interlocking audits. We had already conducted a privacy and security audit, but we revisited it in greater depth. We also hired two additional independent teams to conduct audits around 1) data governance and 2) ethical use of and bias in data. I was the board member overseeing this work, pushing each of these efforts to probe more deeply, engaging a range of stakeholders along the way (including counselors, staff, partners, and domain experts).

I quickly learned that as much as scholars talk about the need to do audits of ethics/biases, there is not a good roadmap out there for doing this work, especially in the context of a fairly large-scale organization. As someone who cares deeply about this, I was glad to be pushing the edges and interrogating every process, but I also wanted us to have guidance on how to strengthen our efforts even further. There is always room to improve, and there isn’t yet a community of practice for people doing this in real-time while people are depending on an organization’s work. Still, we got great feedback from the audits and set about to prioritize the changes that needed to be implemented.

Aside from the data audits, most of our changes over the last 18 months have been organizational and infrastructural, focused on strengthening our team, processes, and tools. As the board chair, I deliberately chose not to prioritize any changes to our contractual relationship with Loris.ai, in favor of prioritizing the human concerns raised by our staff. We focused our energies internally and on our core mission. When Loris asked the Crisis Text Line founder to leave the board, we chose not to offer up a replacement. Our most proactive stance over the last 18 months was to freeze the agreement with Loris, with an explicit commitment to reconsider the relationship in 2022 once a new CEO was in place. As a result of these decisions, we have not shared any data since the change in leadership. 

Governance 

The practice of non-profit governance requires collectively grappling with trade-off after trade-off. I have been a volunteer director of the board of Crisis Text Line for 8 years both because I believe in the mission and because I have been grateful to govern alongside amazing directors from whom I constantly learn. This doesn’t mean it’s been easy and it definitely doesn’t mean we always agree. But we do push each other and I learn a lot in the process. We strived to govern ethically, but that doesn’t mean others would see our decisions as such. We also make decisions that do not pan out as expected, requiring us to own our mistakes even as we change course. Sometimes, we can be fully transparent about our decisions; in other situations – especially when personnel matters are involved – we simply can’t. That is the hardest part of governance, both for our people and for myself personally. 

I want to own my decisions as a director of Crisis Text Line. I voted in favor of our internal uses of data, our collaborations with researchers, and our decision to contribute to the founding of Loris.ai. I did so based on a calculation of ethical trade-offs informed by my research and experiences. I want to share some aspects of the rubric in my mind: 

1. Consent. Consent in my mind exists in a more complex context than the simpler view I had before I began volunteering at CTL. I believe in the ideal of informed consent, which has informed my research. (A ToS is not consent.) But I have also learned from our clinical team about the limits of consent and when consent undermines ethical action. I have also come to believe that there are times when other ethical values must be prioritized against an ideal of consent. For example, I support Crisis Text Line’s decision to activate Public Safety Answering Points (PSAPs) when a texter presents an imminent life-or-death risk to themselves or to someone else, even when they have not consented to such an activation. Many members of our staff and volunteers are mandatory reporters who have the legal as well as ethical obligation to report. At the same time, I also support our ongoing work to reduce reliance on PSAPs and our policy efforts to have PSAPs center mental health more.  

2. Present and future. Our mission is to help individuals who come to us in need and to improve the state of mental health for people more generally. I would like to create a world in which we are not needed. To that end, I am always thinking about what benefits individuals and the collective. I’m also thinking about future individuals. What can we learn now that will help the next person who comes to us? And what can we do now so that fewer people need us? I believe in a moral imperative of paying it forward and I approach data ethics with this in mind. There is undeniably a tension between the obligation to the individual and the obligation to the collective, one that I regularly reflect on.

3. The field matters. We are a non-profit and part of a broader ecosystem of mental health services. We cannot serve everyone; even for those whom we do serve in crisis, we cannot be their primary mental health provider. We want there to be an entire ecosystem of support for people in crisis, of which we play just one part. We have a responsibility to the individual in the moment of crisis and we have a responsibility to learn from and strengthen the field to help individuals downstream. To this end, I think we have an ethical responsibility to give back to the ecosystem, not just to the individual in the moment. But we need to balance this imperative with respect for the individuals during their darkest moments.

4. Improve over time. Much of our data begins as conversations, involving data from both texters and counselors. As you might imagine, when our counselors’ attempts to help someone need improvement, it weighs deeply on our entire staff. Both counselors and texters benefit when counselors learn from reviewing their conversations, from reviewing what worked or didn’t work in others’ conversations, and from lessons learned being fed back into training. My eye is always on what will improve those conversations. (This is why an obsession at the board level is quality over quantity.)

The responsibility of CTL is a heavy one, in ways that may not be obvious to those who haven’t worked in this field or seen the sometimes-counterintuitive challenges of serving people in crisis. I use the needs and prioritizations of our texters and team as my first and most important filter when judging what decisions to make. I see helping counselors and staff succeed as key to helping serve people in need. This sometimes requires thinking about how texter data can help strengthen our counselors; this sometimes requires asking if conducting research will help them grow; and this sometimes requires asking what is needed to strengthen the broader ecosystem.

When it comes to thinking about texters, I’m focused on the quality of the conversation and the safety of the texter. When it comes to safety, I’m often confronted with non-knowledge, which is harrowing. (Did someone who was attempting suicide survive the night? Emergency responders don’t necessarily tell us, so we rely on hearing back from the texter, but what’s the healthiest way to followup with a texter?) I still don’t know the best way to measure quality; I have scoured the literature and sought advice from many to guide my thinking, but I am still struggling there and in conversation with others to try to crack this nut. I’m also thankful that there’s an entire team at Crisis Text Line dedicated to thinking about, evaluating, and improving conversation quality.

I regularly hear from both texters and counselors, whose experiences shape my thinking, but I also know that these are but a few perspectives. I read the feedback from our surveys, trying to grapple with the limitations and biases of those responses. There is no universal texter or counselor experience, which means that I have to constantly remind myself about the diversity of perspectives among texters and counselors. I cannot govern by focusing on the average; I must strive to think holistically about the diversity of viewpoints. When it comes to governance, I am always making trade-offs – often with partial information – which is hard. I also know that I sometimes get it wrong and I try to learn from those mistakes. 

These are some of the factors that go through my head when I’m thinking about our data practices. And of course, I’m also thinking about our legal and fiduciary responsibilities. But the decisions I make regarding our data start from thinking through the ethics and then I factor in financial or legal considerations. 

As I listen and learn from how people are responding to this conversation and from the decisions that I contributed to, it is clear to me that we have not done enough to share what we are doing with data and why. It’s also clear to me that I made mistakes and change is necessary. I know that after the challenges of the last year, I have erred on the side of doing the work inside the organization rather than grappling with the questions raised our arrangement with Loris.ai. 

In order to continue serving Crisis Text Line, I need to figure out what we – and I – can do better. I am fascinated by my peers calling to make this a case study in tech ethics. I think that’s quite interesting, and I hope that my detailing this thinking can contribute to that effort. I hope to learn from whatever case study emerges.

To that end, to my peers and colleagues, I also have some honest questions for all of you who are frustrated, angry, disappointed, or simply unsure about us: 

• What is the best way to balance the implicit consent of users in crisis with other potentially beneficial uses of data which they likely will not have intentionally consented to but which can help them or others? 
• Given that people come to us in their darkest moments, can/should we enable research on the traces that they produce? If so, how should this be structured? 
• Is there any structure in which lessons learned from a non-profit service provider can be transferred to a for-profit entity? Also, how might this work with partner organizations, foundations, government agencies, sponsors, or subsidiaries, and are the answers different?
• Given the data we have, how do we best serve our responsibility to others in the mental health ecosystem?
• What can better community engagement and participatory decision-making in this context look like? How do we engage people to think holistically about the risks to life that we are balancing and that are shaping our decisions?  (And how do we not absolve our governance responsibilities to perform ethics, as we’ve seen play out in other contexts?)

There are also countless other questions that I struggle with that go beyond the data issues, but also shape them. For example, as always, I will continue to push up against the persistent and endemic question that plagues all non-profits: How can we build a financially sustainable service organization that is able to scale to meet people’s needs? I also struggle every day with broader dynamics in which tech, data, ethics, and mental health are entangled. For example, how do we collectively respond to mental health crises that are amplified by decisions made by for-profit entities? What is our collective responsibility in a society where mental health access is so painfully limited? 

These questions aren’t just important for a case study. These are questions I struggle with every day in practice and I would be grateful to learn from others’ journeys. I know I will make mistakes, but I hope that I can learn from them and, with your guidance, make fewer.

I’m grateful to everyone who cares enough about the texters we serve to engage in this conversation. I’m particularly grateful to be in a community that will call in anyone whom they feel isn’t exercising proper care with people’s data and privacy. And most of all I am thankful for the counsel, guidance, and clarity of our workers at Crisis Text Line, who do the hard work of caring for texters every day, while also providing clear feedback to help drive the future of the organization. I can only help that my decisions help them succeed at the hard work they do.

I warmly welcome any advice from all of you who’ve been watching the conversation and who care about seeing CTL succeed in its mission.

Earlier this week, the CDC paused the roll-out of the Johnson & Johnson COVID-19 vaccination after 6 women experienced serious blood clots. Their caution has merit, given that the FDA has been approving vaccinations in advance of the typical large-scale evaluations because speed is seen as so crucial. Reasonably, there is a desire to know more about these blood clots before more might appear. Yet, there was also sheer frustration from many in the medical community because the choice to pause the roll-out suggested that there was a serious issue, that the vaccine was dangerous. In a context in which vaccine hesitancy is likely to undermine herd immunity, any suggestion that the vaccine might have consequences can be twisted and contorted. 

Across many mailing lists and Twitter streams, I kept seeing data points trying to ground the seriousness of the blood clots in the J&J vaccine. Most referenced the frequency of blood clots that women experience while taking the birth control pill, roughly 1/1000. People also highlighted how common blood clots are for those who are in the throes of COVID-19. These were meant to highlight just how rare and statistically insignificant blood clots are when taking the J&J vaccine. 

Yet, as these attempts to ground the conversation unfolded, a different kind of outrage formed. A handful of people highlighted women they knew who had died of blood clots most likely related to birth control. Many more women who took hormonal birth control expressed frustration that they had no idea that they were at increased risk of a blood clot. Sure, it’s part of the fine print of that printout you get from CVS when picking up your pill, but this wasn’t something doctors emphasized. Unlike the J&J vaccine situation, the relationship between birth control and blood clots – or even COVID-19 and blood clots – hasn’t been front page news.

As I was processing the back-and-forth about statistical risk and who was responsible for sharing what with whom, and at what level of amplitude, I couldn’t help but think about all of the scholarship into the politics of numbers. We’re living at a time when politicians are simultaneously espousing the need for “evidence-based policymaking” and working to diligently undermine, contort, or weaponize evidence. This is what scholars of “agnotology” mean when they talk about the manufacturing of ignorance through the seeding of doubt. Or what other scholars highlight as the “weaponization of transparency.” 

I couldn’t help but feel empathy for the scientists at J&J and the FDA who have been working around the clock trying to make a vaccine available to the public, trying to be responsible stewards of information and statistical risk in a context where their desire for caution can be turned on its head to undermine the legitimacy of their work. I also found myself feeling empathy for journalists who recognize the importance of reporting on this development, even as they know that their reporting is easily evolving into misinformation that’s undermining the vaccine roll-out. Working with numbers is itself political.

To work in the world of medicine and science, statistics and probabilities is to grapple with trade-offs at a macro level, which present ethical conundrums even in the best of times. After all, that one terrible death from a blood clot could perhaps have been prevented by not taking the vaccine. But this is where we enter into the world of trade-offs, of unknowns, of morality. Without a vaccine rollout, many more people will die of blood clots from COVID-19. Had that woman been infected with COVID-19, she might have still succumbed to a blood clot. Medicine alters the dimensionality of risk. So how do ethics get negotiated? And by whom? This is the story of public health. 

Those complexities underpinning the advancement of science are complicated further by a politicized context such as that which surrounds the COVID-19 vaccine. Each act of communication can be twisted and contorted to convey different agendas, different values, different goals. Amplified transparency of risk is itself a political act. Sprinkle in the expectation in our current society that individuals are expected to make informed decisions for themselves, their families, and their communities, and we have a recipe for disaster. This is what the production of ignorance – aka misinformation, information disorder, agnotology, etc… – looks like in practice. The very acts of scientific transparency, which are intended to help inform decision-making, are twisted on their head, serving to undermining the legitimacy of scientific work and the coordination of a public that must work together to address a deadly disease. 

I keep wondering what it will take for the public to trust scientific information. But, perhaps, a better question might be: What kind of information is needed to help a fragmented public work together to solve societal-level challenges?

Note to the reader: These are questions that I’m struggling with. If you have thoughts, ideas (or even reading recommendations!), don’t hesitate to reach out: zephoria [at] zephoria [dot] org.

If you look at the roster of the Biden-Harris transition team, it’s quickly apparent that the incoming administration is tech-forward. Given the systematic dismantlement of the federal government over the last four years, and the significant logistical and scientific needs underpinning a large-scale vaccine roll-out, it is unsurprising to hear that the new team is looking to bring in tech talent. Under the Obama Administration, the White House invested significantly in shoring up the Office of Science and Technology Policy, an office that has for all intents and purposes laid dormant for four years under the current Administration. The Obama Administration also hired the first Chief Technology Officer (CTO) to help envision what a tech-forward US government might look like. As the Biden-Harris transition builds its plans for January 20th, many people in my networks are abuzz, wondering who might be the next CTO.

My advice to the transition team is this: You need a VP of Engineering even more than you need a CTO.

To the non-geeks of the world, these two titles might be meaningless or perhaps even interchangeable. The roles and responsibilities associated with each are often co-mingled, especially in start-ups. But in more mature tech companies, they signal distinct qualifications and responsibilities. Moreover, they signal different ideas for what is top priority. In their ideal incarnation, a CTO is a visionary, a thought leader, a big picture thinker. The right CTO sees how tech can fit into the big picture of a complex organization, sits in the C-suite to integrate tech into the strategy. A tech-forward White House would want such a person precisely to help envision a technocratic government structure that could do great things. Yet, a CTO is nothing more than a figurehead if the organizational infrastructure is dysfunctional. This can prompts organizations to want to build new tech separately inside an “office of the CTO” rather than doing the hard work of fixing the core organizational infrastructure to ensure that larger visions can work. When it comes to government, we’ve learned the hard way how easily a tech-forward effort located exclusively inside the White House can be swept away.

Inside tech companies, there is often a more important but less visible role when it comes to getting things done. To those on the outside, a VP title appears far less powerful, far less important than a C-Suite title. If you’re not a tech geek, a VP of Engineering might appear less important than a CTO. But in my experience, finding the right VP of Engineering is more essential than getting a high profile CTO when a system is broken. A VP-Eng is a fixer, someone who looks at broken infrastructure with a debugger’s eye and recognizes that the key to success is ensuring that the organizational and technical systems function hand-in-hand. While CTOs are often public figures in industry, a VP-Eng tends to shy away from public attention, focusing most of their effort on empowering their team to do great things. VP-Engs have technical chops, but their superpower comes from their ability to manage large technical teams, to really understand the forest and see what’s getting in the way of achieving a goal so that they can unblock that and ensure that their team thrives. A VP-Eng also understands that finding and nurturing the right talent is key to success, which is why they tend to spend an extraordinary amount of time recruiting, hiring, training, and mentoring.

When structured well, the CTO faces outwards while the VP-Eng faces inwards. They can and should be extraordinarily complementary roles. Yet, even though the Obama Administration invested in a CTO and built numerous programs to bring tech talent into the White House and sprinkle tech workers throughout all of the agencies, that tech-forward team never invested in a VP-Eng. They never invested in people whose job it was to truly debug the underlying problems that prevent government agencies from successfully building and deploying technical systems.

…

As I listen to friends and peers in Silicon Valley talk about all of the ways in which tech people are going to go east to “fix government,” I must admit that I’m cringing. Government functions very differently than industry, by design. In industry, our job is to serve customers. Yes, our companies might want more customers, but we have the luxury of focusing on those who have money and those who want to use our tools. Government must serve everyone. Much to the chagrin of capitalists, the vast majority of government resources goes to the hardest problems, to ensuring that whatever the government implements can serve everyone.

I have spent 20 years calling bullshit on “the pipeline problem” as industry’s excuse for its under-investment in hiring and retaining BIPOC and non-male talent. Even as tech workers are slowly starting to wake up to the realization that justice, equity, diversity, and inclusion are essential to the long-term health of tech, I’m watching the flawed logics that underpin the narrative about pipeline problems infuse the conversation about why government tech is broken. Government tech isn’t broken because government lacks talent. Government tech is broken because there are a range of stakeholders who are actively invested in ensuring that the federal government cannot execute, who are actively working to ensure that when the government is required to execute, it does so through upholding capitalist interests. Moreover, there are a range of stakeholders who would rather systematically undermine and hurt the extraordinarily diverse federal talent than invest in them.

If Silicon Valley waltzes into the federal government in January with its “I’ve got a submarine for that” mindset thinking that it can sprinkle tech fairy dust all over the agencies, we’re screwed. The undermining of the federal government’s tech infrastructure began decades ago. What has happened in the last four years has just sped up a trend that was well underway before this administration. And it’s getting worse by the day. The issue at play isn’t the lack of tech-forward vision. It’s the lack of organizational, human capital, and communications infrastructure that’s necessary for a complex “must-reach-everyone” organization to transform. Rather than coming in with hubris and focusing on grand vision, we need a new administration who is willing to dive deep and understand the cracks in the infrastructure that make a tech-forward agenda impossible. And this is why we need a federal VP-Eng whose job it is to engage in deep debugging. Cuz the bugs aren’t in the newest layer of code; they’re down deep in the libraries that no one has examined for years.

…

If the new administration is willing to invest in infrastructural repair, my ethnographic work in and around government has led me to three core areas that I would prioritize first. Two are esoteric structural barriers that prevent basic functioning. The third is a political weakness.

1. Procurement. Government outsourcing to industry is modern-day patronage. You don’t need Tammany Hall when you have a swarm of governmental contractors buzzing about. When politicians talk about about “small government,” what they really mean is “no federal employees.” Don’t let talk of “efficiency” fool you either. The cost of greasing the hands of Big Business through procurement procedures theoretically designed for efficiency is extraordinarily expensive. Not only is the financial cost of outsourcing to industry mind-boggling and bloated, but there are additional cost to morale, institutional memory, and mission that are not captured in the economic models. Government procurement infrastructure is also designed for failure, to ensure that government agencies are unable to deliver which, in turn, prompts Congress (regardless of who is in power) to reduce funding and increase scrutiny, tightening the screws on a tightly coupled system to increase the scale and speed of failure. It is a vicious cycle. Government procurement infrastructure is filled with strategically designed inefficiencies, frictions, and insanely corrupt incentives that undermine every aspect of government. They key here is not to replicate industry; the structures of contracting, outsourcing, and supply chains within a capitalist system do not make sense in government — and for good reason. A VP-Eng and a tech-forward government should begin by understanding the damage and ripple effects caused by OMB Directive A-76, which fundamentally shapes tech procurement.

2. Human Resources. Too many people in the tech industry think that HR is a waste of space…. that is, until they find that recruiter who makes everything easier. As such, in industry, we often talk about “people operations” or “talent management” instead of HR. We recognize the importance of investing in talent over the long-term, even if we reject HR. In government, HR is the lifeblood of how work happens in government and it was redesigned by progressives in the 20th century to ensure a more equitable approach to hiring and talent development. For decades, government created opportunities for women and Black communities when industry did not. Unfortunately, this aspect of the “deep state” was not at all appreciated by those invested in maintaining America’s caste system. Those invested in racist hierarchies didn’t need to be explicit about their agendas; they could rely on the language of capitalism to systematically undermine the talent of the federal government. Just as outsourcing in government has statistically taken jobs from Black federal workers and given them to white contractors, a range of HR policies have been designed to make working in government hellacious. Those who have stuck around — out of duty, out of necessity — have become enrolled in an existentially broken system. Some have chosen to sit back and not do their jobs, waiting to be fired. Others took the opposite approach, masochistically throwing themselves at the problem. People come from the outside and complain that government workers are lazy, stupid, incompetent. But it is the system that has produced these conditions. The system has been starved, the policies and protocols are corroded. It is through the purposeful torturing of HR that an executive branch hellbent on destroying federal government can wage the greatest damage; this has been underway for 40 years but the proverbial frogs are now sitting in boiling water. HR will require a lot of repair-work, not quick-fix policy changes. An untended HR system in government becomes a bottleneck unimaginable to those in industry and that’s where we are. Existing talent will require nurturing, and this investment is crucial because their institutional knowledge is profound. Any administration who wants to build a government that can respond to crises as grand as a pandemic or climate change will need to create the conditions for government to be a healthy workplace not just for the next four years but for decades to come. They will need a “people ops” mindset to HR. A VP-Eng should start with a listening tour of those who work on tech projects in agencies.

3. Communications. It never ceases to amaze me that the top communications professional in every federal agency is a political appointee. And every incoming administration — regardless of partisan affiliation — tends to fill these positions with campaign comms people who helped them win the election. Unfortunately, the type of comms that’s needed to win an election (which requires appealing to only some people) is not the same as the type of comms that’s needed to be accountable to the public as a whole for 365 days per year. Over and over, the comms people that White Houses install focus on speaking to their political base and to Congress. This is all fine and well if the only comms need is to negotiate policy outcomes. But the partisan perversion of comms within agencies has another outcome — it delegitimizes the agency among members of the public who are not affiliated with that political party, not to mention the wide majority of the public who is outright disgusted by all partisan tomfoolery. If your political interest is to eliminate the federal government, undermining the legitimacy of federal agencies benefits you. If that’s not your goal, you need to rethink your approach to communications. Right now, every agency needs a crisis comms expert at the helm to regain control over the agency’s narrative. When things are more stable, they need strategic comms professionals who can build a plan for re-legitimization. Each agency also needs an org comms expert whose job, like a VP-Eng, is to repair internal communications infrastructure so that information can effectively flow. Most politicians and government watchdogs think that the key to greater transparency is to increase oversight, just as progressives did after Nixon. But given how broken comms is in all of these agencies, turning up the heat through FOIA, GAO, and Congressional hearings will not increase accountability right now; it will increase breakage. Inside tech companies, comms is often seen as soft, squishy, irrational work, an afterthought that should not be prioritized. But comms, like HR, is the infrastructure that makes other things possible. A VP-Eng needs a comms counterpart working alongside them to achieve any organizational transformation.

…

Addressing these three seemingly non-tech issues would do more to enable a tech-forward government than any new-fangled shiny tech object. There is so much repair work to be done inside government. Yet, as I listen to those I know in Silicon Valley talk about all of the ways they wish to “fix government,” I fear that we will see a significant flood of solutionism when what’s needed most is humility and curiosity. Humility to understand that the structure of governmental agencies exists in response to the never-ending flow of solutionist interventions. And curiosity to understand how and why road blocks and barriers exist — and which ones to strategically eradicate to empower civil servants who are devoted to ensuring that government functions for the long-term, regardless of who is in power. Grand visioning has its role, but when infrastructure is breaking all around us, we need debuggers and maintenance people first and foremost. We need people who find joy in the invisible work of just making a system function, of recognizing that technical systems require the right organizational structures to thrive. This is the mindset a VP-Eng brings to the table.

In conclusion… If you are working on the transition or planning to jump into government in January, please spend some time understanding why the system is the way it is. If you are a tech person, do not presume you know based on your experience with other broken systems or based on what you read in the news; take the time to learn. If you are not a tech person, do not assume that tech can fix what politics can’t; this is a classic mistake with a long history. If the goal is truly to “build back better,” it requires starting with repairing the infrastructure. Without this, you are building on quicksand.