Demon in the Mind

I will suppose...that some malignant demon, who is at once exceedingly potent and deceitful, has employed all his artifice to deceive me;

I will suppose that the sky, the air, the earth, colors, figures, sounds, and all external things, are nothing better than the illusions of dreams

– Descartes

A new bogeyman now haunts our dreams, our courts, our civic discourse. At your every turn, its threat compels you to look over your shoulder, checking and checking again whether your decision was made under its nefarious influence. Words which would have before passed innocently by now demand critical scrutiny, to be tested for any trace of the impurity. Worse, even when no trace can be found there is no relief – it only means the menace has burrowed deep beneath your ability to detect it.

The phantom of "unconscious bias" is here, inside your mind.

"Bias": The Error That Never Came

In 1995, two psychologists set forward a hypothesis of "implicit associations" (sometimes referred to as "unconscious biases") – that stereotypes ("associations") exist which are inherently beyond conscious inspection (i.e. they are "implicit" or "unconscious") and which drive unfair behavior (i.e. "biased" behavior). It was a monumentally influential hypothesis, which quickly drew hundreds of empirical studies over the next two decades to validate its claims.

After those hundreds of studies, the dust settled, showing the Implicit Association Test not predictive of any behaviors of bias or discrimination across individuals,1 and the insignificant effects that it did show were at odds with what the underlying theory would suggest.2 Even when experimental procedures were able to influence IAT scores of individuals, changes in those implicit associations had no effect on behavior.3

This empirical refutation might seem counter-intuitive – "if one step in a mental process seems skewed, shouldn't that skew the overall result?" one might ask. In fact, human cognition is built on a host of subsystems, each of which, on their own, are naive in some way relative to the whole. These somehow-naive steps along a path of thought are called heuristics, and though each heuristic is by itself often wrong, when layered and tempered with non-heuristic modulation, the final result can correct for errors along the way, as we see with implicit associations.4 We use heuristics because layering these simple tools is faster and more efficient than wielding a tool far more complex, and because errors in individual steps can be caught and not propagate to the whole.5 To fixate on one incomplete part of the overall process risks blowing any misstep out of proportion, exaggerating a non-issue to a mythic scale and creating a bogeyman out of nothing.

"Unconscious": Through a Mirror Dimly

Consciousness is a wondrously complex phenomenon, and teasing apart its mysteries is one of the hardest problems of our time.6 It is a light of awareness inside your mind, illuminating mysteries wherever you direct it. Within the vast halls of your mind, you are, at any point of time, not conscious of almost any of it; the light of your consciousness can reveal only a small fraction at once. A process which you are not currently aware of would be a place in your mind where your light of consciousness is not currently shining. "Unconscious", in this sense, is an ephemeral label, applying to the vast majority of your mind at any point in time.

On the other hand, the assertion of an "unconscious process" suggests some place in your mind where your light of consciousness cannot possibly shine. This would stand against an ever-growing heap of evidence that the light of our consciousness can shine almost anywhere inside of ourselves. Claims that any mental process is beyond the reach of conscious inspection are thus, by their nature, highly speculative, and so far there has been no clear evidence of anything like the unconscious processing which "unconscious bias" would require. In our minds, there is almost nothing which is conscious by default, and there is almost nothing which is not somehow inspectable by consciousness. The label of "unconscious" applied as an inherent property of an process in the mind is almost surely a mistake.7

In fact, when this question is directly studied in the context of IAT, researchers found that implicit associations were not beyond conscious inspection and that participants were able to accurately predict their scores. Even though implicit associations are measured in a way which does not require conscious involvement, so that the light of your awareness might be elsewhere during the test, these stereotypes are still inspectable by our consciousness.8

"Unconscious Bias": A Game of Cups and Balls

If "unconscious bias" is not in fact a bias, as it has no behavioral effects, and if it is not unconscious, as it is inspectable by our awareness, then how has this strange idea spread so far and with such religious vigor? The fault is, ultimately, a fault of our minds.

In the twentieth century, in reaction to deep models of human nature which seemed beyond the reach of empirical verification, there arose an epistemic approach called "Behaviorism". Instead of hypothesizing about the fundamental structure of the mind (Structuralism), Behaviorism instead focused on the data which was readily available, the behavior which mind generated.

This school of thought started in psychology, most famously explored by B. F. Skinner, as a way to circumvent the metaphysical quagmire of depth psychology. This style of "behavior as actually observed", instead of relying on deep claims made by a model, made its way into economics as a reaction to the classical model of the "rational man". Throughout the latter half of the century, Behavioral Economics started cataloguing repeatable situations in which experimentees would reliably departure from some expectation of "rational behavior" – these reliable departures are called biases. Naturally, these biases form some pattern, and so low-resolution stories are often told to help communicate their findings.

As humans are wont to do, we were not satisfied with mere "classes of unexpected behavior". We wanted stories. We wanted deep truths. We wanted to use this knowledge to peer into the depths of the soul.

But the approaches we have to understand "bias" fundamentally ignores the inner workings of the mind. Any stories we tell about the results of such behaviorist methodology are merely that — stories. Informed by all the presuppositions found in culture, they grow impervious to falsification, unchecked by invalidation of a methodology which cannot see them. Any cogent juxtaposition of "unconscious" and "bias" would require some deep epistemic innovation to allow to link human behavior with human nature.

Beyond the general challenges of linking "unconscious" and "bias" together, the story which this pair now tells draws its particular virality from a deep source in the mind: the idea that some people have fundamental sympathy or antipathy for generalized classes of things is itself a well-catalogued feature of irrational thought in mythological anthropology.9 Though in reality our thought is highly contextualized, our minds prefer to simplify the behavior of others into vague, generic preferences of attraction or repulsion. What a great twist of irony — that a careless pursuit of rationality should itself fall prey to such a great error of irrationality.

This predilection for gross oversimplification is combined with the fact that the weight of persistent inequality hangs unexplained over our society. The hypothesis of implicit associations could have explained part of our condition, could have suggested some path towards relief if changes to implicit associations did in fact change behavior, and much of the popularity of this idea is owed to its supposed usefulness towards that goal.10 The utter simplicity of its messages, compared to far more complex models (of intergenerational transfer mechanisms, or of inherent centralization of resources in hierarchical systems, or of corruption in intergenerational institutions) offered an easy way out, graspable without prerequisites of any deep understanding of the world.

The hope of the intellectually lazy becomes the error of the foolish, becomes the obstacle of real progress.

From Behaviorism it sang to me, making Structuralist claims
Abandoning its cogency, switching epistemic frames
Against the mounting evidence, still now I find
The Unconscious Bias is there
Inside my mind

No principle of charity in our civic duet
Our penchant for Bulverism grows stronger yet
This stain of original sin infects all mankind
The Unconscious Bias is there
Inside our minds

  1. Oswald, Mitchell, Blanton, Tetlock. "Predicting Ethnic and Racial Discrimination: A Meta-Analysis of IAT Criterion Studies", Journal of Personality and Social Psychology, 2013. link:

    IATs...were typically poor predictors of the types of behavior, judgments, or decisions that have been studied as instances of discrimination, regardless of how subtle, spontaneous, controlled, or deliberate they were.
    ...the IAT provides little insight into who will discriminate against whom, and provides no more insight than explicit measures of bias... Overall, simple explicit measures of bias yielded predictions no worse than the IATs.

  2. Ibid.:

    One might argue that, although the predictive utility of the IAT was low in most criterion domains, the existence of some weak, reliable effects might nonetheless be of interest to science if they advance basic theory (e.g., Mook, 1983; Prentice & Miller, 1992; Rosenthal & Rubin, 1979). However, it was not just the magnitude but also the pattern of effect sizes in the current analysis that are hard to reconcile with current theory. ...This finding raises questions about the proper conception of implicit bias (i.e., as more state-like or trait-like; cf. Smith & Conrey, 2007) and suggests that situational conditions can powerfully sway even the relationship between implicit bias and spontaneous behaviors.

  3. Forscher, Devine, Nosek. "A Meta-Analysis of Procedures to Change Implicit Measures", Journal of Personality and Social Psychology, 2019. link:

    Many procedures changed explicit bias, but to a smaller degree than they changed implicit bias. We found no evidence of change in behavior. Finally, changes in implicit bias did not mediate changes in explicit bias or behavior. Our findings suggest that changes in measured implicit bias are possible, but those changes do not necessarily translate into changes in explicit bias or behavior.

  4. Oswald et al. "Predicting Ethnic and Racial Discrimination":

    ...implicitly measured intergroup biases are much less of a behavioral concern than many have worried— precisely because explicit attitudes often diverge from implicit attitudes. Such an oppositional process, in which explicit attitudes often win out in charged domains, is consistent with Petty and colleagues' metacognitive model of attitudes (e.g., Petty, Briñol, & DeMarree, 2007).

  5. Kahneman. Thinking, Fast and Slow. 2011.

  6. Chalmers. "Facing up to the problem of consciousness". Journal of Consciousness Studies, 1995. link.

  7. Newell, Shanks. "Unconscious Influences on Decision Making: A Critical Review", Behavioral and Brain Sciences, 2012. link:

    When we look at the individual’s use of low-level brain "decisions," do we see clear evidence of unconscious processing? This is far from obvious. Evidence that such decisions are cognitively impenetrable (in other words, immune from top-down attentional control and conscious knowledge) is very controversial. ...The evidence suggests that very few influences or processes are truly uncontrollable.
    Again, the simple point is that in neither situation do we need to posit “magical” unconscious processes producing answers from thin air (cf. Hogarth 2010; Kahneman & Klein 2009). As we have seen, when one undertakes a critical examination of the empirical evidence for genuine unconscious influences on decision making, the evidence is remarkably weak.

  8. Hahn, Judd, Hirsh, Blair. "Awareness of Implicit Attitudes", American Psychological Association, 2013. link:

    We took a different approach and directly asked participants to predict their results on upcoming Implicit Association Test (IAT) measures of implicit attitudes toward 5 social groups. We found that participants were surprisingly accurate in their predictions. Across 4 studies, predictions were accurate...
    Altogether, the research findings cast doubt on the belief that attitudes or evaluations measured by the IAT necessarily reflect unconscious attitudes.

  9. Frazer. The Golden Bough. 1889.

  10. Oswald et al. "Predicting Ethnic and Racial Discrimination":

    Explicitly endorsed ethnic and racial biases have become less common, yet societal inequalities persist. In response, psychologists have theorized that implicit biases must be a key sustainer of these inequalities, and IAT research has become the primary exhibit in support of this theory.