Allan Lichtman's Keys to the White House is Pseudoscience
And how to make your own pseudoscientific prediction model about anything, told as a parable
Historian Allan Lichtman’s Keys to the White House model has been getting lots of play due to his numerous media appearances in the wake of President Biden’s debate performance and exit from the 2024 Presidential Race. His work was even considered by Biden’s inner circle in deciding whether he would stay in or not. The model has successfully ‘predicted’ the winner of every Presidential Election (at least in the popular vote) since 1860 except one.
It is also pseudoscientific nonsense and people should stop letting Lichtman on TV.
The Parable of Allan and the Women
A man -- let's call him Allan Lichtman -- is in a room with two short, dark-haired women.
"All women have dark hair," declares Allan.
Into the room walks another short, dark-haired woman.
"See?" says Allan.
Into the room walks a tall, light-haired woman.
Allan scratches his head and thinks for a moment.
"All short women have dark hair," he declares.
Into the room walks a short, light-haired woman with blue eyes.
Allan scratches his head again, then looks at the eyes of all the dark-haired women already in the room. None of them are blue.
"All short, non-blue-eyed women have dark hair," Allan says.
A short, non-blue-eyed, light-haired woman wearing a red dress enters the room.
Allan looks at the dresses of all the dark-haired women already in the room. They are of various colors, but some are red.
Allan stops looking at the women's dresses.
Allan looks at the shoes of the woman who has just entered. She is wearing flats.
Allan looks at the shoes of all the dark-haired women already in the room. They are all wearing heels.
"All short, non-blue-eyed, heel-wearing women have dark hair," says Allan.
Then, he gives all the dark-haired women, and only the dark-haired women, a prize.
There’s trouble a-brewing
If another dark-haired woman walks into the room and she also shares all those same 'winning' qualities, she too gets a prize. This presents no problem for Allan. This is the equivalent of Keys to the White House correctly predicting the incumbent will win.1
If a light-haired woman enters the room and she lacks at least one of the winning attributes, this also presents no problem for Allan. This is the equivalent of Keys to the White House correctly predicting the incumbent will lose.2
Two other situations do present a problem for Allan, however:
A light-haired woman enters the room who also possesses all the (heretofore) identified winning attributes.
A dark-haired woman enters the room, but she does not posses all the winning attributes.3
(1) is the equivalent of Keys to the White House predicting that the incumbent will win but he goes on to lose. (2) is the equivalent of predicting that the incumbent will lose but he goes on to win.
These problematic scenarios would obviously be embarrassing for Lichtman. Lucky for him, he has two techniques he can use if these ever come up.
Minting Keys
The first one is preferred before he declares his 'model' finalized and we've already seen examples of it. It's the technique Nate Silver refers to as 'minting new keys,' and it's fundamental to the construction of the 'model.'4 Whenever someone who he predicts will win actually loses (or appears likely to), Allan looks for another attribute that all the previous winners had but the recent loser lacks. We can see Allan do this successfully on the first try above with eye color, and we can see him try to do it unsuccessfully with dress color before rejecting dress color as a criterion and settling on heel height.
The key (pun intended) thing to recognize is that Allan starts from already knowing who the winners are and then looks for criteria that differentiate them from the losers. Seems reasonable until you realize that, in Lichtman’s version of doing this, any old differentiating criteria will do: they do not have to have any causal influence on the winner winning, or, indeed, have to have anything to do with subject at hand at all. High heels do not cause dark hair, for example. It's works equally well if the keys are based on plausibly causal factors or random correlations.
The only somewhat clever thing real Lichtman does that fake Allan in the parable doesn't is that real Lichtman does a good job of selecting criteria that sound like plausible things that might cause a candidate to win or lose an election. But, again, he doesn't have to do that. He could just as easily be selecting the sort of arbitrary, clearly unrelated criteria that fake Allan is and say, e.g., "my model has accurately predicted the outcome of the last n elections based on the candidates’ shoe sizes, underwear preference and second letter of their mothers' maiden names." And his model could be just as accurate.5
The entire exercise is backward-looking, arbitrary and rationalistic; transparently so once you remove the veneer of the winning criteria at least superficially having something to do with electoral success. This is what I was trying to capture when, paraphrasing Christopher Hitchens, I tweeted this:
Wiggling
The second technique Lichtman uses to disguise his snake oil is less interesting, more transparent and has been used by false prophets since time immemorial. Because it's kind of embarrassing to be caught minting new keys after you've published your 'model,’ it's the one he prefers.
Licthman uses the fuzziness of certain keys to give himself wiggle room. You'll notice that in the example, most of the criteria are non-objective. Fake Allan doesn't say "women under five-foot-five," he says "short women." Where exactly do you draw the line between blue and blue-green eyes? I intentionally chose criteria that are largely subjective in the parable to highlight this effect.
In fairness to Lichtman, some of his Keys are objective. And that also lends his approach a veneer of respectability. But as long as there are a few subjective ones, he leaves himself space. And he's got enough: 'Strong Short-Term Economy' (Key 5) is defined as the economy either being 'in recession' (for which a generally accepted, objective economic definition exists) or 'widely perceived to be', which is the equivalent of 'Allan can credibly point to at least a meaningful minority of people who report dissatisfaction with the economy.'6 Keys 12 and 13 have to do with a candidate being 'charismatic' or 'a national hero,' which, as far as I can tell, are defined as 'Allan Lichtman declares you to be charismatic or a national hero.'7
You can see Lichtman begin to wiggle in real time once Harris relaced Biden in this tweet:
Anticipating that Harris will beat Trump after telling anyone who would listen for three weeks that the Democrats were going to get shellacked if they replaced Biden, he attempts to redefine his 'Contest Key' (Key 2) by memory-holing the three weeks of post-debate chaos and asserting there was no strife around who the Democratic nominee would be. By mentioning the excitement generated by the initial phases of her candidacy, I suspect he's also laying the groundwork to award her the aforementioned ‘Charisma Key’, or possibly declare her a national hero for providing an alternative to the widely-unpopular-and-perceived-as-unfit Trump and Biden. I think there's also a chance he'll redefine his ‘Incumbency Key’ to grant it to Harris (assuming Biden finishes his term as expected at the time of this writing).
Lichtman’s method, summarized
Returning to the parable, if a light-haired woman of intermediate height walks into the room, fake Allan is hoping that she is disqualified on one of the other criteria (e.g. she's got blue eyes). If she possesses all the winning attributes, however, he looks to see exactly where she falls in the height order. If he can plausibly draw the line (or move the goalpost, if you prefer) so that the new woman falls on the tall side of it and all the dark-haired women fall on the short side of it, he wiggles. If he can't, he looks for a different subjective criterion he might be able to do that with ('how high does it have to be to be a high heel?') and wiggles if he can. If he can't plausibly do that with any of the existing criteria, he is forced to mint a new key.
What’s the deal?
Lichtman is a professional historian. As such, I expect his personal perspective on who will win the next election is well-informed by historical knowledge. Given his background, I’d have no objection to him presenting his opinions on the subject as expert ones.
But at the end of the day, they are opinions. His opinions may be better in quality than yours or mine, but they are not different in kind. By encapsulating what amount to opinions into what he presents as a model, I suspect what Lichtman is trying to do is elevate his opinions to the status of perceived fact. After all, if the model has worked so often before, it’s better than opinion, right? Consider how different the media appearances linked at the top of this article would have been if Lichtman were merely stating his opinions about what the Democrats should do (as myriad commentators and ordinary people were doing at the time), rather than presenting his conclusions as scientific. But personal ‘reckons,’ however well-informed are still ‘reckons,’ not science. And presenting them as scientific doesn’t make them scientific.
It reminds me of a brief conversation in the Seinfeld episode “The Engagement” in which Kramer explains that he doesn’t wear a watch because he has a system for telling time by the sun that is accurate “to within an hour,” to which Elaine retorts that she can tell time accurately within an hour by the sun without a system. When she asks what Kramer does at night, he replies “Night is harder, but it’s only a couple of hours.” Casting Lichtman as Kramer here, I think the 2024 race is night: a time where Lichtman’s ‘system’ isn’t of much use, and that, like Kramer, he is underestimating how long the night is (i.e. how ill-suited his ‘system’ is to the case at hand).
Which raises the most depressing part of Lichtman’s dilemma. Like Kramer, Lichtman has used his system to create a sort of intellectual prison for himself. If you or I want to tell time at night, we can simply buy a watch. If I’ve declined to wear a watch before (say because I don’t like how they feel on my wrist), but I suddenly have a need to tell time at night, there’s little to stop me from deciding to tolerate the mild annoyance that’s previously been holding me back and just wear one. But Kramer, can’t. His eccentricity and identity as the ‘tell time by the sun system guy’ would make the reputational switching cost too high if he’s built his personal brand as The Human Sundial.
Lichtman’s brand is The Oracle of the Keys. As Oracle, it is his job to interpret what the Keys say, and the Keys are always speaking.8 In the wake of Biden’s exit from the 2024 race, political commentators or ordinary people are free to suspend their personal reckoning and say “I don’t know what’s going to happen next,” or to express less than 100% certainty about their reckons (“It’s hard to say what’s going to happen next, but my guess is…”). Responsible pollsters point out the limitations of their polls in absorbing rapidly-changing events. Aggregators of polls like Silver suspend publishing their models, citing a need to wait for the polls to settle down following a drastic change like replacing a candidate. These are the actions of intellectually honest people.
The equivalent for Lichtman would be to say something like “Since my approach is based entirely on an analysis of historical precedent, we should probably take what it says about an historically unprecedented election with a grain of salt.” This is, of course, the opposite of what Lichtman did during the post-debate period in 2024, when he made the above-cited media appearances claiming that Biden would be a fool to exit the race when the Keys suggested he do the opposite. And after Biden’s exit, I suspect why, seeing the massive (and predictable, I would add) popular and fundraising wave Harris rode into her candidacy, he felt instantly compelled to start wiggling so as not to set himself up to appear foolish should Harris win later. These are not the acts of an intellectually honest person. Nor is attacking the intellectually honest people for being intellectually honest.
Lichtman’s systemic study of past elections could provide a valuable contribution to the election discourse alongside polls, political commentary and the historical perspective of others.9 I personally consider all of those things in formulating my own opinions. But however good those opinions are, they aren’t scientific: not even in the sense that proper, ‘scientific polling’ is scientific.10 But by presenting something as scientific that isn’t, and by engaging in contortions to make reality appear to conform to a ‘model’ not built to accommodate it, Lichtman is engaging in pseudoscience. One who draws attention to oneself by unabashedly engaging in pseudoscience publicly and repeatedly is either the acme of cringe or, if doing it knowingly, a huckster. In both cases, such a person should not be taken seriously.
Technically it's the 'incumbent party' but I'm using 'incumbent' here as shorthand.
The fact that the incumbent need not posses all the keys but only enough of the keys for Keys to the White House to predict he will win is irrelevant to the analysis, which is one reason this simplification makes it easier to spot the flaws.
Or, in the case of the real Keys to the White House, enough of the winning attributes.
Part of my motivation in writing this article is that I think Silver’s critique of the Keys, while correct, is needlessly statistical. Lichtman’s model doesn’t rely on statistical concepts. It’s essentially a sorting algorithm relying on logical concepts, as the parable makes clear. I suspect this is due in equal parts to Silver’s ‘brand’ being stats and the fact that his critique was written at a time when Silver’s employer required him to adhere to editorial guidelines that required him to treat Licthman and the Keys civilly. Recently, Silver has been more direct and less genteel in his criticism.
To be clear, some of them are quite plausible things to think would be a factor in election outcomes, and in many historical cases undoubtedly have been and will be again. I’m not implying that the real Lichtman is pulling random correlations out of his hat and then asking ‘is this plausible?’ I’m merely pointing out that someone who engaged in the same exercise by doing that about this or any other subject would be able to find a set of 13 random or loose correlations, mint some keys, and arrive at a model with similar predictive power to Lichtman’s.
The technique seen here with Key 5, and several others, is particularly effective at disguising the subjectivity of the key. Key 5 combines an objective standard for a strong economy (the absence of a recession, which, as noted above, has a widely-accepted, rigorous definition as ‘two or more consecutive quarters of negative GDP growth’) with people’s perceptions about the strength of the economy where those perceptions can, and famously often do, contradict reality. It’s a naked attempt to have it both ways.
It’s similar with the ‘Charisma Keys,’ which combine an at least somewhat clearer standard of the candidate being a ‘national hero’ (drawing to mind grand-scale wartime leaders who went on to be President like Grant and Eisenhower) with ‘charisma,’ e.g. a candidate who seems like the kind of fun person it might be cool to have a beer with.
Fake Allan would have done well to have included a few more highly subjective criteria in his ‘model,’, rather than more relative ones. Borderline cases sometimes provide useful wiggle room, but some women are objectively taller than others. ‘Is this woman charismatic?’ on the other hand…
Though it’s by no means definitive, another thing that should raise our suspicion that Lichtman’s approach is unscientific is that interpreting the Keys requires an oracle.
And he could surely find work. The news channels frequently have historians as guests. Presidential historian Doris Kearns Goodwin is a regular guest on CNN, for example.
One of the ways in which scientific polling is scientific is that it acknowledges its own limitations.