Annie Duke - decision strategist, bestselling author and former professional poker player - joins me for a conversation about why smart people so often make bad decisions.
I always have a blast chatting with Annie. Today, we explore why explanations can feel so satisfying, even if they later turn out to be wrong. We also discuss why misinterpretation is more dangerous than misinformation, why data is often true but misleading, and how our brains are wired for certainty in a probabilistic world.
I hope you enjoy this conversation as much as I did. We’ve shared some highlights below, together with links & a full transcript. As always, if you like what you hear/read, please leave a comment or drop us a review on your provider of choice.
— Jim
Books Mentioned
Thinking in Bets; by Annie Duke
Quit; by Annie Duke
What Works on Wall Street; by Jim O’Shaughnessy
Outlive; by Peter Attia
The Millionaire Next Door; by Thomas J. Stanley and William D. Danko
Good to Great; by Jim Collins
Links
Highlights
Data Describes; It Doesn’t Explain
Annie Duke: When we see data […] it’s just a description of something that’s been observed. So we can take a simple bar chart. It’s a description. There were this many at this time, this many at this time, it’s just a description. But when we see it’s small this year, and then it’s bigger this year and it’s bigger the next year, we think it’s actually giving us why. We sort of jump to that why. And the why that we’re going to jump to is certainly under the hood, going to be driven by bias. But I don’t think that people are purposely lying to you. I don’t think they’re purposely trying to mislead you.
And I think that when you’re looking at data that you might have generated, say in your own investment firm or whatever, that you’re certainly not trying to lie to yourself there. But we do end up in some sense lying in the sense that the conclusion or the explanation that we’re jumping to is inaccurate, and it’s because we don’t know how to interrogate the data. So this is what I’m trying to address.
The Investment Strategy That Worked… Until It Didn’t
Jim O’Shaughnessy: I had an experience where I put it in my book What Works on Wall Street. For a five year period, there was an investment strategy which was really simple. It was one line: buy the 50 stocks with the highest gain in sales year over year. Makes a lot of intuitive sense to people. Wow, their sales are hockey sticking. And then you show them the actual results of that strategy over a five year period and it kills the market. It triples the S&P 500. The problem was it was for five years and we were looking at a data set of nearly 50 years. When you run that strategy over the entire data set, it’s worse than T bills. And yet I could have, and if I’d been duplicitous and not honest, I could have gone out and raised a shit ton of money on that.
Annie Duke: That’s exactly right.
Jim O’Shaughnessy: And people just fall in love with that idea.
Annie Duke: One of my favorite examples that I have always used is when someone says to you that this player has had a hit six out of their seven last at bats, that you always know that on the eighth one they did not get a hit. And that’s that problem. Why are you showing me these five years? Because I think sometimes that can be disingenuous. Sometimes someone can literally be trying to sell you on something that isn’t true. Again, even in that case, it’s up to you. Because if you look at those five years and you fact check, it’s going to be right. So it’s up to you to say, well, wait, what else should I be asking about this? Is this the right sample to be looking at, for example? Or why are they showing me these exact five years? Or whatever it might be. It’s up to you to say, hey, hold on a second.
But then if you’re, and this is a very strong tendency within business, you have to understand that you’re actually doing that to yourself all the time.
Jim O’Shaughnessy: All the time.
Annie Duke: Because what people do a lot is they kind of massage. They don’t massage the data, they just data mine. They’re just looking for which comparison is going to tell me that this thing that I did is better. And I think that most people don’t understand that there’s something wrong with that. They think that they actually find the truth when they find out that X is better than Y. And they don’t understand, well, if you try a hundred different ways, you’re just randomly going to find out that X is better than Y. And it doesn’t necessarily mean that’s going to hold going forward.
🤖 Machine-Generated Transcript
Jim O’Shaughnessy
Annie Duke, welcome back to Infinite Loops. It’s so good to see you in person.
Annie Duke
In person this time, which is so amazing.
Jim O’Shaughnessy
Last time I was on Zoom. I was reminiscing about, do you remember when we were at Friend of a Farmer and we were talking about your last book?
Annie Duke
Yeah.
Jim O’Shaughnessy
And that turned out to be so great. But before we started recording, you started telling me about your new book, which absolutely fascinates me. We’re not going to ask you the whole thing, but first off, tell our listeners and viewers a little bit about what it’s about because I think, boy, is the time right for this book.
Annie Duke
Oh, well, thank you. Yeah. So a lot of what I think about, obviously I’m trying to help people to make more effective decisions. And part of the way that I think about that is that at the core of every single decision is a forecast. So if you think about any decision you make, right, you’re considering different options. Should I take this way to work or this way to work?
Jim O’Shaughnessy
Everything is a bet.
Annie Duke
Right. What you have to do for any of those options that you’re considering is make a forecast of how much that option that you select is going to on average help you gain ground toward your goals in alignment with whatever your values are. So obviously when we get into the value thing, we get into utility. But those forecasts, as you’re thinking about that, right, I’m thinking about this option, here’s the set of possible outcomes. There’s a payoff associated with each of those outcomes. There’s probabilities of those occurring. That’s how we calculate the expected value or ground gained toward your goals. Those forecasts can only be as good as the inputs.
How do we think about those inputs? Well, we have experiences that we might learn from. A lot of what my past writing has been about is how the way that we interpret those experiences will be biased in a way to cause us to come to inaccurate conclusions or lessons learned about the experiences that we have. So for example, if you think about the centerpiece of Thinking in Bets, it’s really this resulting problem that makes it so that when we have an experience, things go poorly, that we draw the wrong conclusions about what the decision quality was and then that causes us to make worse decisions going forward. If you think about that, it’s really a forecasting problem, that it’s changing the way you think about the probability of a bad outcome occurring under the same decision circumstances. So that’s how we’re learning from our own experience.
But there’s this other source of input, there’s this other input into the decisions that we make, which is the information that we come across. And we’re going to come across that information or data, either because it exists out in the wild, we find that data or that information somewhere else. A user on social media might offer it to us, or we might self generate. So we could be doing analyses of our portfolio performance, for example, or we might be looking over all the founders we’ve ever invested in and trying to signal detect what founders are good or what founders are bad, or we’re tracking how our sales are doing given maybe different marketing strategies that we’re trying.
And this is all data generation and we might be creating data visualizations and things like that, and then we’re trying to come to conclusions about what is causing us to observe those things that will then help us to make decisions about how to get better outcomes for ourselves in the future. So we might be looking at, can we come to the conclusion, is there an explanation for what we’re seeing in the data that says that the marketing strategies that we’re trying are actually improving sales? And if we think that’s true, then we may invest more money in those marketing strategies. As an example, we might see data about the efficacy of vaccines and that might drive decisions for ourselves about whether we ought to get a vaccine or not, depending on what we’re looking for in terms of our own health outcomes.
As an example, that’s this other source of input, separate and apart from our own experience that is driving the types of decisions that we make. Anyway, I was thinking about that particular problem and I said it’s interesting. Everybody really seems to be very worried about misinformation in this world. But I come across some work from Duncan Watts, who’s at Penn, and he had actually been looking at let’s divide it into two worlds, misinformation, let’s call it someone just making something up and lying to you.
Jim O’Shaughnessy
Right. Propaganda, motivated reasoning.
Annie Duke
Bots, that kind of stuff. Versus information that is either offered in a way that’s been misinterpreted by the person who’s offering it. So they’re not lying. So the core fact that they’re offering you is true. So that’s the difference. So in misinformation, the fact that’s being offered is not true. Someone says 17,000 people died in New York City last year, you can go look that up. It’s not true. Versus I offer a fact that’s true. Let’s say it were true that 340 people were murdered in New York City last year. Let’s just say that’s true. And then I draw a conclusion from that number that is misinterpreted. So it ends up being misleading. And it could be that the person offering it is actually offering that interpretation that’s misleading or misinterpreted. Or you could just misinterpret it yourself.
So let’s sort of divide it in those two worlds. And what Duncan Watts found is that it’s 41 to 1. The misleading thing, the misinterpretation, is actually the bigger problem. So when I saw that, this actually had aligned with something that had made me really mad, which was an article that I read in the Washington Post in 2022, where I really felt this problem of, look, if you think about it, I don’t think that the majority of people are trying to lie to you. I really don’t. Yes, do we all have our biases? Sure. But I think what the biases are doing is under the hood, driving the types of explanations that we’re jumping to.
When we see data, so you have some sort of piece of data, and that data is merely, and I think it’s hard for people to wrap their heads around it, but it’s just a description. It’s just a description of something that’s been observed. So we can take a simple bar chart. It’s a description. There were this many at this time, this many at this time, it’s just a description. But when we see it’s small this year, and then it’s bigger this year and it’s bigger the next year, we think it’s actually giving us why. We sort of jump to that why. And the why that we’re going to jump to is certainly under the hood, going to be driven by bias. But I don’t think that people are purposely lying to you. I don’t think they’re purposely trying to mislead you.
And I think that when you’re looking at data that you might have generated, say in your own investment firm or whatever, that you’re certainly not trying to lie to yourself there. But we do end up in some sense lying in the sense that the conclusion or the explanation that we’re jumping to is inaccurate, and it’s because we don’t know how to interrogate the data. So this is what I’m trying to address, and it aligned with this work that I had seen from Duncan Watts, because I was already really itching in this space to write this book because of this Washington Post article.
So here’s what happened in 2022. It was October 2022, I read this article in the Washington Post. And at that time, the Washington Post would have been considered liberal bias for sure. And at that time, 2022, certainly pro vaccine bias. So the headline of this article caught my eye because of that, because it was basically in friction with what I knew their bias was. And the title of the article was “Covid is no Longer a Pandemic of the Unvaccinated.” So I really perked up. I was like, whoa, what kind of data is in this? Because the Washington Post is certainly not, we’re not talking about a paper that’s anti vax where I would have all sorts of reasons to be skeptical of that. In this case, I’m like, wow, something must have happened.
So I read the article and the main data, and again, this is a description. This is what we have to remember. It’s just a description. The main data that they’re citing is that in August of that year, 58% of the people who had died of COVID were vaccinated, 42% were not. Okay, so the reporter, having seen this, and go fact check, it’s true. You can fact check for August of 2022. You’ll see that’s true. And this is the problem.
Jim O’Shaughnessy
But the old quantity is just red flags.
Annie Duke
Your head is exploding, right? So this is what happened. It’s like, this is bigger than this. 58% is more than 42%. So that means more vaccinated people died of COVID than not vaccinated. So vaccines must not work. Clearly, the editor had not seen that there might be an issue with this explanation. So I said, well, okay, wait, I’m going to keep reading because they’re going to tell me what percentage of the population are vaccinated. Obviously. No, they did not.
So I was like, all right, let me go look that up. So I went and looked it up because it wasn’t in the article, which made me very upset. So I went and looked it up and they had a definition in there. And according to their definition, at that time, 80% of the population was vaccinated. So I was like, okay, well, wait, just right. Let’s just start here. 80% of the population is vaccinated. 58% of the people who died of COVID are vaccinated. Let’s look at it from the other frame. 20% of the population is unvaccinated, and they accounted for 42% of the deaths. So I was like, okay, wait. I certainly think that this explanation, which is the headline, is not only unwarranted, but dangerous.
Jim O’Shaughnessy
Yeah.
Annie Duke
So then I was like, well, okay, if I really wanted to take a step further, I ought to age match, because obviously older people are both more likely to be vaccinated, more likely to die of COVID than younger people. There was a statistician who had … Somebody else had gotten upset about the article, and they did do the age matching, and it turned out that you were five times more likely to die if you were unvaccinated of COVID in the month that the Washington Post declared that Covid was no longer a pandemic of the unvaccinated.
And I just said, I don’t think this reporter’s trying to lie. I just think they don’t know how to interrogate data. So they see this data and they don’t know the questions you ask. The most simple of which would be out of how many? Which was the first question I asked. Well, wait, how many people are vaccinated versus unvaccinated? I need to know that thing before I can even interpret this.
On another stream, I’ve been seeing my clients making these type of errors too. So a very simple example would be, I had a client who was trying to increase their inbound top of funnel. This was an investment client trying to increase their inbound top of funnel investment opportunities. And they showed me a histogram and it was just here’s the size of inbound top of funnel in 2016, 2017, 2018, 2019, so on and so forth. And all of a sudden around 2020, this thing hockey sticks and there’s a little arrow there. There’s an arrow that says “implemented experimental marketing strategies.”
So they said, well, we just want you to look at this because, this is part of my job, we want you to look at this because what we really want to do is invest heavily in these marketing strategies because you can see that they caused this to go up. So notice that again, we’re talking about this big jump. So I sort of call it crossing the chasm from description. Okay, you’ve shown me a description: this is how many in each year, to explanation: these marketing strategies cause this thing to occur. That’s an explanation. But there’s this big thing in between that you’ve just crossed the chasm. Interrogation. How would we interrogate that data in order to understand that?
Now, I’m guessing to a lot of people that when I sort of describe this situation, they’re like, well, isn’t that obvious? But I can show you why it’s not obvious pretty easily. I could say, what if I had an arrow there and it said “Russia invaded Ukraine”? Would you fall for this? And the answer, of course, is no. Because there’s this great concept in cognitive psychology. Tania Lombrozo talks about this called explanatory satisfaction. And it’s when an explanation, it’s exactly what the name is, feels satisfying. And that could be because it confirms a bias that you have. So if I have some sort of bias and the explanation confirms that bias or beliefs that I already have, I’m probably going to feel pretty satisfied by it. At which point, just like when you’re satisfied by your meal, you don’t go look for dessert. You’re not going to go and try to do other things with the data.
Another way that it can be satisfying is that, and this happens, I think, a lot in the investment world, it feels insightful and contrarian because then you feel like you discovered something that other people don’t know. And so that could be another way. Or sometimes it’s just simply that it makes sense.
Jim O’Shaughnessy
Yeah.
Annie Duke
That it kind of...
Jim O’Shaughnessy
Or that your reasoning is being motivated by some other external factor.
Annie Duke
Yeah, it could be that. I think the “makes sense” thing, though, is a good thing to sort of just linger on, which is we don’t like randomness. We want to know why things are occurring. So when we kind of land on an explanation where it sort of solves that discomfort, that we don’t know why something happened, we’ll often just stop just because, oh, that makes sense. So that actually can create explanatory satisfaction. It’s just that feeling of, yeah, that makes sense. So that would be when you don’t necessarily have a dog in the race. It’s like, okay, I’ve got no dog in this fight. Now I don’t have to feel uncomfortable that I don’t know why things are happening in the world.
Okay, so we can take this example and say, well, why are they jumping to that conclusion? Well, doesn’t it feel good that you experimented with some marketing strategies and then your inbound top of funnel hockey sticked? So anyway, I looked at this and I said, oh, okay, that’s interesting. Before you do that, could you just make me literally an identical histogram that’s just total opportunities in each year. That’s all I want. And so they did that. And then marketing strategies never got implemented because it just turned out that there were just a whole bunch of extra opportunities that had been created partly because of COVID.
So it could have been that the opportunities were flat and it really worked. I didn’t have an opinion, and I want to make this clear, I didn’t have an opinion about whether those strategies did work or didn’t work. There’d be no reason for me to have the opinion. My opinion was whether you could draw that conclusion from the data that I had been shown. That was all that I cared about at that time. And I said, well, I don’t know until I know how many total opportunities there were. And if you can show me that, then you could say, well, what proportion of the total opportunities did you actually capture? And if it’s true that you were capturing a larger proportion of the total opportunities, then I’m going to support your conclusion.
So I just want to make it clear it’s not just, oh, the media is telling you things and not giving you context or whatever, which is a problem if you don’t know how to take that into your own hands and actually ask the type of questions that I did of that article. But it’s also within your own business, particularly in this world of data insights and AI is generating data for you, and how do you even know how to interpret it? Because the AI isn’t going to interpret it for you. You have to know how to ask the right questions.
Whether you’re asking the questions of an AI or your own data or your insights team or whatever it might be, you have to take that into your own hands because that’s going to determine choices you make about your health, where you live. It’s going to help you decide if you’re trying to compare the performance of one person versus another, it’s going to tell you what you think is causing the outcomes that you observe in the world. And then sometimes that means that you’re going to say this is signal for something to occur. Or sometimes it’s, if I do this, I can cause a greater likelihood of X occurring or a lower likelihood of X occurring. It really matters.
So the book is really just trying to get people to say I have to approach data with enough skepticism that I’m always going back to the… I’m never skipping the interrogation step. And then I need to know what the things are that I need to interrogate. And they’re super simple things like out of how many, what am I comparing it to? Is the sample any good? And so I’m just building that out through fun narratives and the COVID story which is in there, obviously there’s a lot of sports stories in there and then there’s stories from my own clients and trying to get people to understand you need to not allow information to happen to you. You have to look at the information and say, if I were to think about how I explain what I am observing, because again, data is just a description of an observation, something you’ve observed in the world. If I’m going to explain it, what are the questions that I need to ask of this in order to actually get to the closest to the right explanation that I could.
Jim O’Shaughnessy
Yeah. And as you were talking about it, some data is just so consistent with people’s priors that they just, even if you tell them. I had an experience where I put it in my book What Works on Wall Street. For a five year period, there was an investment strategy which was really simple. It was one line: buy the 50 stocks with the highest gain in sales year over year. Makes a lot of intuitive sense to people. Wow, their sales are hockey sticking. And then you show them the actual results of that strategy over a five year period and it kills the market. It triples the S&P 500. The problem was it was for five years and we were looking at a data set of nearly 50 years. When you run that strategy over the entire data set, it’s worse than T bills. And yet I could have, and if I’d been duplicitous and not honest, I could have gone out and raised a shit ton of money on that.
Annie Duke
That’s exactly right.
Jim O’Shaughnessy
And people just fall in love with that idea.
Annie Duke
One of my favorite examples that I have always used is when someone says to you that this player has had a hit six out of their seven last at bats, that you always know that on the eighth one they did not get a hit. And that’s that problem. Why are you showing me these five years? Because I think sometimes that can be disingenuous. Sometimes someone can literally be trying to sell you on something that isn’t true. Again, even in that case, it’s up to you. Because if you look at those five years and you fact check, it’s going to be right. So it’s up to you to say, well, wait, what else should I be asking about this? Is this the right sample to be looking at, for example? Or why are they showing me these exact five years? Or whatever it might be. It’s up to you to say, hey, hold on a second.
But then if you’re, and this is a very strong tendency within business, you have to understand that you’re actually doing that to yourself all the time.
Jim O’Shaughnessy
All the time.
Annie Duke
Because what people do a lot is they kind of massage. They don’t massage the data, they just data mine. They’re just looking for which comparison is going to tell me that this thing that I did is better. And I think that most people don’t understand that there’s something wrong with that. They think that they actually find the truth when they find out that X is better than Y. And they don’t understand, well, if you try a hundred different ways, you’re just randomly going to find out that X is better than Y. And it doesn’t necessarily mean that’s going to hold going forward.
Jim O’Shaughnessy
Yeah.
Annie Duke
So I don’t really address so much of that in this book of things like just hygiene around data. You have to segregate some of the data so you can do all the massaging you want on 60% of the data, but then you have to see if it’s predictive of the other 40%, for example. I don’t care if you massage 60%. You just have to make sure it predicts the other 40% that got quarantined.
Jim O’Shaughnessy
Yeah, you have to have an out of sample holdout.
Annie Duke
Exactly. You can obviously, within the social sciences now and cognitive science, you have to preregister your hypotheses. You have to preregister the analyses that you’re going to do. That helps you with that type of hygiene. I don’t, that’s not, the everyday person doesn’t necessarily need to know that. What the everyday person needs to know is, well, am I seeing the whole sample? That’s a good question. What’s the denominator? That’s my favorite one. Just out of how many?
There’s a famous example on hormone replacement therapy that now has just become quite well known, partly through Peter Attia’s book Outlive. They looked at a large sample of people who were taking hormone therapy versus not. These are women who are perimenopausal, menopausal, or postmenopausal. And in the group of people who were taking hormone therapy, five ended up with cancers, reproductive cancers. And in the group that weren’t, four did. Now, the way that number was reported in the media was that you had a 25% increase in your risk of cancer. That’s true if you’re looking at 4 versus 5. But you have to say, well, wait a minute, but out of how many people total? Because I need to know what the absolute risk is before I compare the relative risk.
So it turned out it was out of 1,000. So it was four out of a thousand versus five out of a thousand. Now you’re smiling because immediately you’re like, well, then that’s not even a difference. I think it ends up, I can’t remember exactly. I have it in the book.
Jim O’Shaughnessy
I think it’s tiny, it’s minuscule, it’s under 1%.
Annie Duke
I think it’s 0.06%. I mean, it’s really small.
Jim O’Shaughnessy
Yeah.
Annie Duke
It’s not a 25% increase. But because it was reported that way, a generation of women were not given hormone replacement therapy and suffered through what are, for some people, really debilitating symptoms of menopause. So these questions that you have to ask are really high stakes. I’m not just being a nitpicky, I know the scientific method and you need to know how to interpret data. You, as an everyday person in your life are making decisions about things like what medical treatments are going to work for me, what are the things that I’m going to do, what type of exercise am I going to do, what type of marketing strategies am I going to invest in?
What type of signals do I think that I’m detecting in terms of what makes a good investment versus a not so good investment, so on and so forth, where it could be as simple as which baseball player is better than the other one, which I have an example of in the book. So these things matter. That’s the thing. They matter for government policy, they matter for your own health issues, they matter for your business. And we have to start paying attention to how do we get good at this thing that feels in some ways like eggheading and “don’t tell me about your math.” But you don’t need to think… These questions are actually really not very, as you know, all of my books are about mathematical concepts. They’re not what I would call mathy.
Jim O’Shaughnessy
No. Not at all.
Annie Duke
It’s just more of these general concepts of, well, why don’t you just ask out of how many. And if that was the only question you ever learned, your life would be a lot better.
Jim O’Shaughnessy
Totally agree. And there’s a couple of things. So I did this series on the Great Reshuffle, and one of the things that I was thinking about as I was getting ready to chat with you was the world we came from was kind of a clockwork world. A implies B implies C implies D. You climb the ladder, you get your gold watch, you get a pension. Very simple, very deterministic. And what’s a deterministic type game? Chess. If you know the rules, you can get pretty good at chess. The world we’re going into is a probabilistic world. And what game is best for that? High stakes poker. You have to make decisions on incomplete information. You have to understand that people might be lying to you. That is bluffing. You have to be able to update your priors immediately when the new card turns over.
So I just don’t think your average person is built for that. It’s something I’ve banged on and on about. We are deterministic thinkers living in a probabilistic world and hilarity or tragedy often ensue. And so my question to you is one who is deeply in this domain. Is it just part of our human nature that we don’t grok this stuff?
Annie Duke
So I think, here’s the thing. Well, first of all, let me just, a slight quibble. There has been no time in our history where we weren’t living in a probabilistic world. The question is, how big an error was it to act like it was deterministic?
Jim O’Shaughnessy
Deterministic. Right, that’s a great…
Annie Duke
So the slower the cycle of change, the less of an error it is to act like the world is deterministic. But I can show you that even back then, the world was probabilistic because we did have big sea changes. You lived your life thinking that you were going to be in ag. And then you shifted to everything all of a sudden shifted to manufacturing.
Jim O’Shaughnessy
Yep.
Annie Duke
And it’s like, whoa, my whole life just got turned upside down. And that’s where you got, for example, the Luddite movement, where they’re saying, whoa, we don’t like that this changed. We thought the world was going to be this way. We thought that was a good prediction to make. And now, ooh, technology. And stop that. So I think that now the rapidity with which it’s shifting makes it so that it’s much more obvious that it’s an error to behave as if the world is deterministic. You’re going to get punished much more quickly for thinking that. And I think that’s particularly true over, say, the last 25 years, that cycle has sped up.
Because even back in 1985 or whatever, I would say even in 1990, the early 90s, you still could believe that if I go do this, then these things will happen and these opportunities will be available to me. So let me just kind of start there. But let’s think about an evolving human who lives their life not believing that things will be the same. So you have to figure out your migration pattern. If you’re a nomad, if you’re not a nomad, you have to figure out when are you planting, when are you harvesting, when are you doing? Don’t you have to believe the world is deterministic in that case? I mean, that’s why we have rituals. That’s where Thanksgiving or harvest comes from. We’re planting at a certain time during the year.
Now, obviously, they understood there were ups and downs in terms of rain. And when that happened, that was very uncomfortable. And so they would do rain dances, or in the worst cases, they might sacrifice a virgin or two. Which is, you know, but where they’re trying to sort of overcome some of the randomness that occurs. But of course, they must believe that because they have to plant at a certain time, they have to harvest at a certain time. And that they have to believe that’s going to be the same over time.
As the world becomes more complex because we’re more global, our social groups are bigger, and as technology advances at a faster pace, then what happens is that our minds that were built to be deterministic, I think for relatively good reason, to keep structure in society and structure in our day and structure in our year and structure in our lives, now all of a sudden is butting up against the complexity and the speed of change. Where it then becomes a much bigger error.
I can take that to a broader view of just kind of bias in general. Which is when we think about a lot of the heuristic side, these shortcuts that we take, they mostly work.
Jim O’Shaughnessy
Yeah.
Annie Duke
Which is why we have them. But then when they don’t work, it’s really bad. And the more sort of complex the world is, the worse it is. So you can think about something like the availability bias, for example. Well, if we’re living in a very small territory in a social group that isn’t more than 300, if we see things more, they probably do occur more in the world. It’s actually a pretty reasonable shortcut under those circumstances. But when the world becomes global and it’s not just what we’re seeing in our area, and we’re looking at the news and they show fires but not drownings, or I live in America, so I think John is a more popular name when it’s actually globally not that popular, it becomes a problem. That’s a way to add complexity into the problem.
So most of the ways in which we think about cognitive error, a lot of them develop because they were shortcuts that worked most of the time in the world in which we evolved. And I think that’s true of this thinking about things deterministically. In the world that we evolved, I think that was probably mostly okay, and you mostly weren’t punished for it unless you were the person who got sacrificed because you didn’t have enough rain that year. Then you got super punished for that.
Jim O’Shaughnessy
But I think another thing at play, and by the way, I agree with you, the world was always probabilistic, but in the current environment, it’s really much more important to understand that very high uncertainty. And yet another thing is this illusion of certainty we seek. We prefer the candidate who is certain in his or her pronouncements. And one of the things I was thinking about, you’ve seen the news on the prediction markets getting, doing deals with CNN, CNBC, and I was thinking how cool to be able to use prediction markets to spin them up on and have a ticker on. The CEO says we’re going to, he says with great certainty we are going to double sales next year. Wouldn’t it be great if you had a decently liquid, because you were good to call out liquidity, a decently liquid prediction market where the bets could go right underneath the statement.
Annie Duke
Well, I think that would be great as long as at the same time we were educating people about how to think probabilistically. Obviously, this is part of the reason why I co founded the Alliance for Decision Education, because I think trigonometry is such a silly thing to teach the average person. I don’t think we’re sailing by sextant anymore. So I don’t exactly know why people need to know trigonometry. If you’re raising a barn, I don’t know. But I think that’s something that if you’re actually going to be a structural engineer, you can probably take as an elective later. And it would probably be better to be teaching people statistics and probability as kind of a core decision skill. You can teach a lot of stuff a lot earlier than at which point that would be appropriate.
But the reason why I mention that is because we all know that you can go back to the very famous 2016 election where I think Hillary Clinton was 60% to win, according to, I mean, Nate Silver was essentially sort of running something that would have looked like a prediction market. I think that had you, if you had done a prediction market, it would have been 60, 65, probably wouldn’t have been, relatively in line with that. And when Trump won, people were like, you were wrong. So there we go.
So, yes, I think it’d be wonderful to have that just as sort of for people in the know. Certainly. And I think that if it went below 50%, it might be helpful if someone was declaring it to be 100%. Would it be helpful if it was the market was saying it was 78% and the person was declaring it to be 100%? I don’t think it would because I think that people would sort of interpret that as, okay, well, that’s definitely a sure thing.
Jim O’Shaughnessy
Yeah, yeah.
Annie Duke
As opposed to, okay, that’s three to one-ish. So I’m going to put one bullet in a four chamber gun. Russian roulette, anybody? And I think that most people would be like, whoa, no, if you put it that way, no. But I think you have to get it into those concrete terms for people to actually get it. And likewise, I think that there are things that people can say where they could even add some nuance to it. And the prediction market would say this is 35%. But actually given what the payoff for that would be, it’d be like, you would be willing to say, okay, yeah, I would take the 35% because the payoff’s gonna be 5 to 1 or something like that. And I think that these concepts are actually just very hard for people to grasp.
Obviously, as a poker player, that’s the whole gig. It’s, okay, I remember once I was actually live streaming way back. This was in the early days of it. I was live streaming, playing in a poker tournament, and I had a hand and I was getting 11 to 1 from the pot. And this was on a final bet, so I wasn’t going to have to invest any more money. There was nothing implied that was going to happen in the future. It was just a straight call on the end and I had a terrible hand. I was getting 11 to 1 from the pot and I said, well, I’m profitable here at 9%. And better than that, actually. It’s 8%. And I was like, well, I’ve never been that sure about anything. I guess I should just call.
Because I haven’t. I said I have to be, well, I have to be over 90% sure that this is not the best hand. And I just went, okay. I’m not sure about anything. I mean, I’m sure about my own name, but about that I’m not. And I called and I won the pot. And it was literally I was trying to sort of take the magic out of it. It’s like I don’t have a dead read on this person. I do not. I would not. I’m not sure that they’re bluffing. I’m not declaring anything with certainty. I’m just saying, I don’t know, I’ve never been over 90% on anything. I’m calling. And I won the pot. And I think that type of thinking, it’s, you have to get to people early to start to get them to be able to view the world through that lens. So I think it would be great as, ooh, what’s this? And why should I care? And then if we back that up with really good education, I think that would be the most amazing thing ever.
Jim O’Shaughnessy
Yeah. So talk to me about education because that’s another hobby horse of mine. We are using an antiquated educational system for a world that no longer exists. It was designed for the industrial era where essentially you wanted to be able to have people sit in a room for eight hours and take instruction. If you were put in charge, if you were the new secretary, do we still have an education department? I don’t know whether they accidentally...
Annie Duke
I don’t know if they’re doing anything at the moment, but I don’t think it’s been canceled yet.
Jim O’Shaughnessy
But tell me what it would look like, because I agree with you, by the way, that early education especially would be able to mold the way that the younger people are thinking on their way up. But what would you change about the curriculum?
Annie Duke
I think that there’s a lot of people who are working in ed reform and a lot of the work is, we’re going to create this special type of school where people are doing something completely different. And I think that’s fine and I think it’s great to have those kinds of things, but I think that we do need to live in the reality that whatever you’re doing, 80% of kids are going to public school. And I’m including charter schools in that. 80% of kids are in public schools or charter schools. And so you need to be thinking about how do we change education within that system. I just think that’s, it’s very important to acknowledge that because I could say all sorts of things about here’s the kind of school that I would create. But that’s great. But you’re not getting to most of the population.
Jim O’Shaughnessy
Thus is why I made you the head of the Department of Education.
Annie Duke
Well, thank you.
Jim O’Shaughnessy
So keeping that as stipulated, keeping that in mind for public school, since 80% of kids go to those.
Annie Duke
Right. So basically what I would be doing is changing the curriculum out of something which is very fact based into something which is very decision based. This shouldn’t surprise you. So there’s some interesting things that you can do, actually. So at the Alliance, we’ve actually developed a tool and it’s AI. It’s an AI tool where you as a teacher can input a lesson plan. So let’s imagine that you’re teaching Macbeth or you’re teaching about the water cycle or pick your topic. You can input it into this tool. And all that the tool will do is embed decision education, decision skills into the lesson plan.
So we could think about something like Macbeth, for example, or Eisenhower. And let’s think about a skill that we can teach people. Let’s explore the counterfactuals. What if Macbeth had done this? Oh, that’s cool. How do you think that this would have turned out? In the water table, we could say, okay, here’s the average rainfall in this area per year. What if we put a mountain here? Because really, what are you teaching people in a fact base? You’re like, here’s the rainfall in an area where this mountain is here. And mountains affect things. But you’re not actually asking people to put that into an actual forecast of what would happen to the rain if you stuck a mountain in a place, where they’re having to take into account all the other things about the climate in that area. As an example.
Going back to Macbeth, it’s like, well, what were all the options? Why do you think they chose this option? How did that affect Lady Macbeth? How was Lady Macbeth’s decisions taking into account Macbeth’s desires versus her desires versus so and so forth? What’s interesting about it is I think it actually makes the material stickier anyway because you’re not just memorizing facts. They’re now thinking for themselves. So whatever decision skill you want to think about, teaching agency or optionality or forecasting or bias, what were their different biases? You can start to teach the cognitive biases through that lens. For example, this idea of counterfactual thinking. You can start to embed that into any lesson. And I think it just becomes much more interesting than fact based.
On top of that, I think that you have to start skilling kids up on how are you thinking about getting good inputs from AI into your decisions. So just a concrete example of what we’re doing. We have a forecasting competition. Super fun. So you can think about Good Judgment Project, which is Phil Tetlock, Superforecasters and Barb Mellers. They’ve got Good Judgment. People go on there. You as an adult can go on there and you can enter a forecasting competition. It’s very fun. So we said, well, why aren’t we doing this with kids?
We have a forecasting competition. We actually are able to give some kids training and some not. Preliminary results are showing that the training is quite effective. The training is actually based on the work from my dissertation which showed that training novices on some simple concepts in forecasting is very effective. Things like what’s the base rate is a really good thing to teach people. So we do that with middle schoolers and high schoolers and we encourage them to use AI because that’s the world we’re living in.
Jim O’Shaughnessy
Totally. Could not agree more.
Annie Duke
But what we’re trying to teach them is AI is a big fat liar unless you know how to prompt it correctly. So what we ask them to do is show your work. If you’re using AI to help you try to figure out how you’re going to forecast, and we have kids forecast things they care about. In the first week of the Taylor Swift new album drops, how many copies do you think it will sell? So they care about that kind of thing. If you’re going to use AI to help you with that, show your work and show the prompts. And we’re going to also give you training on how to prompt AI, which is a decision skill in and of itself. How do you prompt it to get you to give you an answer that’s more accurate? How do you check your sources? How do you get it to be skeptical? How do you get it to give you counter arguments? How do you get the right what ifs?
And what they learn is that the better you are at prompting the AI, the better the forecast that you’re going to be able to make because your inputs are going to be better. And then you’re more likely to win the prizes that we’re offering along with that. So I think that there’s just, starting in kindergarten, you can start to teach this stuff. You can start to teach very simple probability. And we used to, by the way, it’s gotten dropped out of the education system until about eighth grade now. We used to start teaching probability in kindergarten. So let’s bring that back in.
A kindergartner could learn the concept of agency. They could learn the concept of options. Well, what are the different choices that you have? For those different choices, what do you think would happen if you chose this thing? Why do you think that would happen? Why is that a conversation that we’re not having with kindergarteners? Instead we’re teaching trigonometry. It’s so weird to me.
Jim O’Shaughnessy
Again, you had me at hello there. Because I agree we’re living in a world where I think when I was coming up, your memory was super important.
Annie Duke
Oh, my gosh.
Jim O’Shaughnessy
Your ability to recall facts. And when you’re talking about Macbeth, I immediately went, “the lady doth protest too much.”
Annie Duke
There was a soliloquy I had to learn from King Lear. Cordelia. It was one from Cordelia. Because we all had to memorize something. I can still say it. Why is this living rent free in my head?
Jim O’Shaughnessy
Exactly. But that isn’t the case anymore. We have ready...
Annie Duke
Exactly. We had a set of encyclopedias on our...
Jim O’Shaughnessy
Same with me.
Annie Duke
I mean, I’m sure that at the time they were 25 years old. I mean, I don’t think any of the information was particularly correct anymore.
Jim O’Shaughnessy
No.
Annie Duke
If I wanted to know something, I had to look it up there.
Jim O’Shaughnessy
Yeah, exactly. And so it’s not like the world we were coming from was some bizarro world. No, it’s, that’s what we had available.
Annie Duke
That’s right.
Jim O’Shaughnessy
To us. That’s right. And so you had, if you had a good memory, good for you. You’re going to go a lot further. If you were a high agency person, sort of by your natural disposition, good for you. You’re going to go a lot further. But I don’t think that these things are necessarily, well, the memory might be innate, but I don’t think agency is innate. I think that you can learn agency.
Annie Duke
Exactly. That’s exactly right. You can see, actually there are socioeconomic differences in the way that people approach agency. So when you see people of higher socioeconomic status, which generally means that you sort of just live in a world where you have more agency, and you see how they’re interacting with a doctor, for example, they’re asking, they’re acting like, hey, I’m your customer. And they’re asking all sorts of questions and they’re just not accepting what they say. But when you see people of lower socioeconomic status and generally, what obviously correlates with that is the parents tend to be not as well educated. For example, again, not always, but, and they’re living in a world where it’s a little bit more happening to you, or at least it feels like that, when they go to the doctor, the doctor says that, that’s it. They don’t even ask any questions and they’re told not to. Don’t be rude to that doctor. They’re the authority in the room.
Whereas when you’ve been living a life where you understand, you know a lot of doctors, you’re like, well, you’re an idiot. I mean, you’re the dumbass at the party. Oh, my God, would you shut up? You’re just more likely to be asking questions and skeptical and getting second opinions and things like that, which, of course, is an act of agency. So I think it’s incredibly important to teach very early on this idea that you’re an agent of your own decisions.
And we can get back to this core idea that’s really throughout all of my writing, which is, look, you got two things. Only two. That’s it. That determine how your life turns out. One is luck, which, sorry, it’s exogenous force that acts upon you. You have no control. You can say, “I make my own luck” all day. It’s like, that literally is not a sentence of English that makes sense. I mean, it’s grammatically correct, but that’s about it. It has words in it that have meanings, but that’s it. It’s not like you’re reading a Lewis Carroll poem or something like that. It’s actual words of English, but as a string of English, it doesn’t make any sense because you can’t make your own luck. Definitionally.
Jim O’Shaughnessy
I actually want to push back. I don’t believe you make your own luck, but I believe that if you have your reticular activating system, your aperture set very wide, you might be able to always be scanning for opportunities that are lucky.
Annie Duke
I’m 100% with you on that.
Jim O’Shaughnessy
Okay.
Annie Duke
I think that when people say, “I make my own luck,” what they mean is I make decisions that increase the probability that I’m going to...
Jim O’Shaughnessy
I’m totally with you.
Annie Duke
That doesn’t mean that you’ve made any luck.
Jim O’Shaughnessy
No…
Annie Duke
And by the way, nobody who’s had bad things happen to them has ever said, “I make my own luck.” I just want to point that out, which is why, you know, it’s dumb.
Jim O’Shaughnessy
But also, the person who is forever going on and on about how everything in the world is due to luck is probably one of the most unlucky people in the world.
Annie Duke
Well, of course. I mean, that person obviously didn’t have good outcomes. That is for sure. And that’s the point. We want to take credit. So it’s always the most successful people who are like, “I make my own luck.”
Jim O’Shaughnessy
Right.
Annie Duke
And it’s like, no. I mean, you may make better than average decisions, but you do not make your own luck. And obviously, at sort of the start of anybody’s life is, what if I had been born in 1600?
Jim O’Shaughnessy
What my life would, you know, actually, I love playing that game.
Annie Duke
Yeah.
Jim O’Shaughnessy
Because it totally immediately informs you of how, I say this often. I am so lucky that I was born in America.
Annie Duke
Right.
Jim O’Shaughnessy
In 1960. Because if it was 1870...
Annie Duke
Exactly.
Jim O’Shaughnessy
My opportunity set is gonna be a lot smaller.
Annie Duke
Exactly. And the thing is, people will be like, oh, Jim, you’re being so humble. It’s like, no, it’s just true.
Jim O’Shaughnessy
Yes.
Annie Duke
If Bill Gates had been born 30 years later, no Microsoft, no billions, because the computer already would have happened.
Jim O’Shaughnessy
I make this point all the time about my own book, What Works on Wall Street. The only reason I got to write that book was because chance luck made me born in 1960. If Ben Graham had access to the computers that I had and the databases that I had, he would have written that book. It would not have been an opportunity for me.
Annie Duke
That’s exactly right. And the thing is that doesn’t take away from the accomplishment. You saw the opportunity, you put in the hard work, you did the research, you wrote the great book. That is, that’s the thing. It’s like a blend. And it doesn’t discount your accomplishment to say, hey, there was a lot of luck in what happened. I mean, LeBron James is tall. I’m pretty sure he didn’t have any control over that. He’s a great basketball player, but he’s tall.
Jim O’Shaughnessy
That’s right.
Annie Duke
What if he were five four? I don’t think he ends up being LeBron James.
Jim O’Shaughnessy
I had a statistician on the podcast who wrote a book about the NBA, and he said it may not surprise you that your chances of being selected for the NBA go drastically higher if you’re over 7 feet tall.
Annie Duke
That’s exactly right. And a little coordinated, and then you’re pretty good. But you don’t even need to be that coordinated because you just go like this. I’ve got a basket. So you have to be smart enough to know the rules. So that’s thing number one is luck, which you do not make. It’s just the thing. It happens. You have no control. The other thing is decision quality.
When we think about it, I’m born into the world, and when I’m born into the world to whatever parents I’m born to, with whatever talents or lack of talent that I might have, physical characteristics, so on and so forth, there’s some distribution of outcomes that’s available to me. That’s true. Now the question is, what’s the probability that I end up at the right tail? And that’s going to be determined by the decisions that I make. The better decisions I make, the more that I can get to that right tail of the distribution on average. And obviously, that doesn’t mean...
Jim O’Shaughnessy
Mean everybody.
Annie Duke
I can go through a green light and a car can hit me.
Jim O’Shaughnessy
Yep.
Annie Duke
What am I going to do? But if I’m consistently going through green lights and not going through red lights, the probability that I have better outcomes is going to be higher than someone who’s consistently going through red lights. And that’s all we’re trying to do. The person going through the red lights could just literally have that for their whole life, be unscathed, but it’s a much lower probability that will occur for them.
Jim O’Shaughnessy
And many times people will confuse in their minds probability versus possibility. They’re two very different things. Could happen. Yeah, exactly. It’s how they market the New York Lotto. Hey, you never know.
Annie Duke
Yeah, it could happen.
Jim O’Shaughnessy
Well, you do kind of know.
Annie Duke
You do kind of know. Look, every once in a while, you’re going to get more than a dollar back for the dollar you spend.
Jim O’Shaughnessy
Yeah.
Annie Duke
It’s rare, but it occasionally will happen. And even when you calculate that, it would be really good if you calculated in the fact that the more people play, the more likely you’re sharing, which then makes it really hard.
Jim O’Shaughnessy
Fran Lebowitz had a great line which was, I think my chances of winning the lottery are the same whether I buy a ticket or not.
Annie Duke
Close to it.
Jim O’Shaughnessy
But it is one of those. I love this framing, because you’re absolutely right. The quality of your decision making, on average, it does not mean you could make all the best decisions in the world. And just sadly, you are in the cohort that bad shit happens to. But the two together, why is it so hard for people?
Annie Duke
Well, I mean, let’s go back to explanatory satisfaction. People are so uncomfortable with randomness.
Jim O’Shaughnessy
They really hate it.
Annie Duke
They really hate it. And obviously, in its worst form, it becomes a conspiracy theory, of course. And this concept of, we’re sampling a lot of stuff all the time, and sometimes things will happen at the same time together. There was a study where it’s basically just you’re given sort of a toy example. And it’s like, hey Jim, you know, there’s this museum and we just found out, again, data, which is a description. We just found out that people who visit the portrait room, the portrait gallery, are more likely to give a donation on their way out. You’re sort of asked why? And people are kind of a little bit uncomfortable. I don’t know why, and they’re sort of making things up, but they don’t necessarily have a lot of confidence in what they make up.
But then when you show them a scientific paper, and this is true, that when people are being observed, they’re more pro social, then all of a sudden they’re like, oh, it’s because there are eyes in the room and that’s why they’re doing it. And they’re 100% confident because it’s taken away this, I don’t know why these two things are occurring together, which even on that small little bit and that thing that has nothing to do with you still makes you uncomfortable.
Now how do you get them out of that? You’d say to them, well, here are a whole bunch of other explanations. Then all of a sudden they’ll sort of get off that, this is 100%. If you give them this one piece of scientific data, which is true, and you let them make the connection, they actually don’t self generate other explanations. But when you’re like, oh, did you know that the portrait room is shabbier than the other galleries as an example? So you start to generate other things that could be true. People who visit the portrait gallery are of higher net worth. So now I can start to give you a lot of other things and then you go back to not being, you go back to that, but you’re sort of back to being uncomfortable and you kind of want to know, but if I give you a pathway to certainty, you’re going to take it.
And I would go back to, you’re living in a small tribe in this small area and how are you supposed to survive if you think the world is just random? You have to sort of...
Jim O’Shaughnessy
Well, that’s the illusion of certainty.
Annie Duke
Right.
Jim O’Shaughnessy
It, I think is very much an evolutionary aspect of our behavior. And it’s one of the, I had an example that I put in a couple of my books, which is this professor did this study where it was Mr. Smith and Mr. Jones and he told them that the object of the study was to see how good, by getting feedback, they could get at judging whether a picture of a cell was cancerous or not. What he didn’t tell them was that Mr. Smith was getting true feedback. So in other words, if he guessed right, he got a green light. If he guessed wrong, he got a red light. Mr. Jones was getting feedback based on Mr. Smith’s guesses. Okay, so if Mr. Smith guessed wrong, but Jones guessed right, he’d still get a red light.
What’s really interesting, and they did this with multiple participants, and what is really interesting is the guy getting the true feedback is pretty soon improving his batting average to about, he’s getting seven out of 10. This poor guy over here is not. But what’s really interesting is when they come and they ask them, well, how do you make your prediction? The guy who’s getting real feedback offers concise, straightforward examples. Well, if the cell seems to have a broken edge, or if it’s fuzzy over here, I guess that it’s sick. And if it doesn’t have those things, I guess that it’s healthy.
Mr. Jones, who’s getting the bad feedback, launches. He looks at this guy and he’s like, that’s the most pedestrian example I’ve ever heard in my life. No, no. Sometimes it’s this and sometimes it’s that. And he just weaves this tapestry, very complicated tapestry. What’s really interesting, all of the guys getting it wrong are in awe of the guy who is getting it wrong.
Annie Duke
Oh, my God, you’re so smart.
Jim O’Shaughnessy
And when they go and do it again, his performance materially declines.
Annie Duke
That’s interesting.
Jim O’Shaughnessy
And so it’s almost like we over index on people who wave their hands a lot.
Annie Duke
I really try to teach people that when you’re communicating information, this is so hard because of the point that you just made, that your job is to communicate at the exact same time what you know and what you don’t know, literally at the exact same time. Now, what we tend to want to do is be like, I know this thing for sure, which is a very bad representation of what you know and what you don’t know.
Jim O’Shaughnessy
Very bad.
Annie Duke
It’s a very bad representation. So what I try to tell people to do is, let’s imagine that I’m working with somebody and their job is they’re making sales forecasts. Okay, that’s fine. Give me a point estimate. That’s great, because there’s some useful information I can derive from your point estimate. But then what I’d like you to do is give me a lower bound and an upper bound, and I can set 90% confidence interval, 80% confidence interval. And just for those who don’t know, a 90% confidence interval just means that the true answer, if I were omniscient, would land within that boundary 90% of the time. Okay, so give me a lower bound and an upper bound. Now, what you’ve actually done is told me what you know and what you don’t know.
So there’s a very simple example that I do in my classes, which is, so I have a dog. The people in my class have not seen a picture of my dog. So I always start with, what’s the weight of my dog? Very often they’ll be like, I don’t know. And I’m like, okay, that’s fine. But give me your best guess. Okay? So I’m just trying to get them to give me their best guess. And usually they’ll say somewhere between 25 and 45, maybe 50 pounds somewhere in there. So let’s imagine they say 35 pounds. Okay? So I’m like, okay, that’s your best guess. What do you think the smallest amount my dog weighs is? And usually they’ll say 10 pounds. Because I didn’t say puppy. So let’s say 10 pounds. I’ll go, what do you think the biggest amount my dog weighs is? And they’ll often say something like around 120, sometimes as high as 150. I say, okay, that’s interesting.
And then I start to ask them some questions. So I can ask you a question. Why do you think they don’t say 500 pounds?
Jim O’Shaughnessy
Why do I think, yeah. Because there would be a very rare dog that weighs 500 pounds.
Annie Duke
I think it would be dead, actually.
Jim O’Shaughnessy
It could be a prehistoric dog that...
Annie Duke
Was gonna be a dog that I own today. Because dogs don’t weigh that much.
Jim O’Shaughnessy
That’s right.
Annie Duke
So when I say to them, your initial instinct was, well, I don’t know what your dog weighs, but actually, you know a lot about what my dog weighs, because things can weigh 10,000 pounds or a million pounds. And you just gave me a very small range in the context of what the possibilities are.
Jim O’Shaughnessy
And what you know about the dog.
Annie Duke
But you gave me 10 to 120 pounds. So, you know, they don’t come in one pound sizes. If I’m calling it a dog. And, you know, it doesn’t come in 500 pound sizes. That’s amazing. So now I ask them another question, which I’ll ask you. You can pretend to be the person. All right, so it’s interesting because you gave me 10 pounds to 120 pounds, but your point estimate was 35 pounds, which is a lot closer to 10 than it is to 120. Why is that?
Jim O’Shaughnessy
I wouldn’t make that estimate, though.
Annie Duke
Well, so what they say, and I would argue they’re correct to do this, is that larger dogs are rarer, so there’s skew in the distribution. The majority of dogs that people have are on the small end of the scale. Think about people in New York City who have apartments. They’re going to tend to have smaller dogs.
Jim O’Shaughnessy
Well, sorry to interrupt, but that was where my mind immediately went.
Annie Duke
Oh, yeah, that you would have...
Jim O’Shaughnessy
Where, I would say, where are you living?
Annie Duke
Yes. Okay. So that’s my next thing that I point out. Okay, so this is, I’m refusing to give you any information except it’s a dog. Okay. So I say, oh, that’s interesting. So the fact that the point estimate is closer to the lower end than the higher end actually tells me something about the shape of the distribution. How common are small dogs compared to, wow, you really know a lot about dogs. And then I say, okay, so let’s imagine that I’m asking you to do this. I’m saying, give me a point estimate, lower bound and upper bound. Tell me, what are the things that you want to ask me? And they’ll say things like, where do you live? What’s the breed? And I say, oh, okay. So that’s interesting. So if I actually allow you to build, the uncertainty in it causes you to start to generate questions that will help you because it opens you.
But here’s the even better thing. If all I do is declare to you 40 pounds, if you say something different to me, you now are at risk of me thinking you are wrong. But if instead I say, well, I think Annie’s dog is 35 pounds, maybe lowest 10, highest 120 pounds, included in that is a request for information. Included in that is, hey, if you know anything, can you help me? So now maybe you’ll be like, you know what? Annie has a toy poodle. I actually think you should lop off that top. I think it’s probably going to be lower. But the point is that it invites you into the conversation to help me.
And now when we’re thinking about, I want to express both what I know and what I don’t know, now all of a sudden you find out, oh, but that doesn’t mean I don’t know anything.
Jim O’Shaughnessy
Right? That’s what I love. Yeah. Actually, that’s what I love about the framing of this.
Annie Duke
And I’m more likely to end up knowing more at the end of the process because first of all, I’m going to be more inquisitive, but also other people are going to more freely offer me information. So if I could change the world, that earnings estimate would be lower bound this, upper bound this, with some context, here’s the circumstances under which we would see the lower bound. Simple example with sales estimates. Let’s imagine that someone calls a number like a million net new ARR at the end of Q4, lower bound 800, upper bound, 2 million. You immediately know that there’s a big account that might slip to Q1. It’s so easy to figure that out. And you can just say to the person, tell me why. Now, as someone who’s trying to budget for the organization, isn’t that way better?
Jim O’Shaughnessy
It absolutely is. And yet I’m also thinking about another example that you’re, what you’re, by the way, I love that exercise because what I love about it is you are showing people that they know a lot more intuitively than they think they do. And they’re learning that through asking questions, which is great. But there is an example of another professor. I love what these guys get up to, but essentially they give you one base rate. They say that there is a town of 100,000 people, 70,000 of them are lawyers, 30,000 are engineers. And then your job is to guess from random sampling that we take out of a big bag full of names, how many of the 10 that I pull out are lawyers and engineers. So I pull out 10 names and they’re just a name. Annie, Jim, Jean-Marc, Richard. Just the name. What do you think most people say when asked how many of these are lawyers and how many are engineers?
Annie Duke
I don’t know. I mean, I would hope they just go, well, you gave me the base rate, so I guess I’m going to say seven of them.
Jim O’Shaughnessy
Right. They either do that, and this is actually, again, through iterative trials of this, they use one of two strategies.
Annie Duke
Okay.
Jim O’Shaughnessy
They either go with the base rate and say that seven are lawyers and three are engineers.
Annie Duke
Yeah. Given that I literally don’t know anything else.
Jim O’Shaughnessy
That’s right. That’s all you know. And the other one they go with is they say they’re all lawyers. They just keep saying they’re all lawyers to try because they know they got to do it multiple times. But then they add a little twist. The next round with a different group, of course, the next round adds descriptive but meaningless information. Annie is 44 years old and wears glasses. Jim is 65 years old and does this, this and this. When they add descriptive information, people begin to largely deviate from the base rate.
Annie Duke
Well, I mean, obviously the guy who wears glasses is an engineer.
Jim O’Shaughnessy
Right. Well, but then they add stereotypical.
Annie Duke
Yeah.
Jim O’Shaughnessy
And when they add stereotypical, people completely ignore the base rate. Frank is shy, likes mathematical puzzles and wears glasses. You could jack up the number of lawyers to 99.
Annie Duke
It doesn’t matter.
Jim O’Shaughnessy
It doesn’t matter.
Annie Duke
That one’s the...
Jim O’Shaughnessy
And so how would you solve for that? Because that’s such a natural tendency.
Annie Duke
As a mathematician, you ought to just be ignoring that information and you should just go with the base rate as your first best guess. Unless there’s something causal. So obviously wearing glasses doesn’t cause you to be anything.
Jim O’Shaughnessy
Right?
Annie Duke
If you’re going to deviate from the base rate as your forecast, you have to believe that there’s some sort of dislocation that’s occurred. So there has to be some causal difference. So the example I give is, don’t, if you have to predict the probability that a Cat 3 or higher hurricane is going to make landfall in the US in a given year, don’t use statistics from the 60s because something has changed. So that I’m really trying to sort of pound that home that your best guess should always be the base rate unless you have a really good reason to deviate from it. That is actually part of the training from my dissertation work. And we do get people, we can get people to do that. It’s part of the training that we’re giving to the high schoolers and the middle schoolers.
Now I’m going to put on my linguistics hat. There’s a part of linguistics which is called pragmatics. And it just has to do with, well, we know what pragmatics are. What is pragmatically? So, okay, so let’s imagine that there’s only one pencil on the table. There’s only one, and it’s red. There’s a very big difference in how you think about the redness of the pencil. If I say, can you hand me that pencil versus can you hand me that red pencil? So you can intuit this immediately. Because there’s only one pencil on the table. So it must be a red pencil. This must be what it’s called. The redness is a thing about it.
Jim O’Shaughnessy
Like...
Annie Duke
So it’s a correcting pencil or something.
Jim O’Shaughnessy
Yeah, yeah.
Annie Duke
Okay. This is just a thing about language, is that when we offer information to people that is outside of what needs to be offered in the situation, we automatically assume that information is incredibly important. So there’s an old study that they did which was on similarity, and it was basically, how do you interpret “North America is similar to China” versus “China is similar to North America.” And it turns out that people interpret those two things very differently because there are rules about what is being compared to what. The son is like the father, not the father is like the son as an example. So there’s things that have to do with social status, size, whatever.
So let’s imagine that I said to you, the bicycle is next to the house. I’m sure that what you’re imagining is a normal bicycle and kind of a normal house. But what if I said to you, the house is next to the bicycle. It’s like, you’re like, what’s the deal with that bicycle? Is it a statue?
Jim O’Shaughnessy
A big statue.
Annie Duke
Right. So you could think about this. I don’t know.
Jim O’Shaughnessy
Or is it a little model house?
Annie Duke
Right.
Jim O’Shaughnessy
That you could build?
Annie Duke
You’re like, what’s going on? So a real life example of this is in Philadelphia. There’s a very big clothespin.
Jim O’Shaughnessy
Yeah, I’ve seen it.
Annie Duke
Right. Which is a really big statue. So if you’re saying, I’m standing next to the clothespin.
Jim O’Shaughnessy
Clothespin.
Annie Duke
It makes sense in the context of that. But it doesn’t make any sense if it’s a legit clothespin. Because you’re bigger.
Jim O’Shaughnessy
Yes.
Annie Duke
Okay. So I do think that when we’re doing those types of things, we have to take into account pragmatics. So there’s the classic base rate neglect. Linda is mousy and she’s quiet and she has whatever. And what’s the chances she’s a librarian? And it has to do with this, what they call the conjunction fallacy.
Jim O’Shaughnessy
Yeah, that’s what I was actually thinking of. Yep.
Annie Duke
And but as a linguist, what I say is, but why are you offering me all this information about Linda? So, yes, we don’t want to fall for it because it is a way of trickery. And you can see that trickery occurring in people who are actually trying to sell you or fool you or something like that. So we would like not to fall for it, but it’s not that surprising that you are. Because when you’re saying, what are the chances that Linda is both a D&D fan and a librarian or whatever it is, that when you’re giving all of this extra detail, as someone who speaks the language, what’s going through your head is, why are you giving me all this detail? It must be meaningful. In the same way that you told me that the pencil was red.
Jim O’Shaughnessy
My mind sort of automatically goes to, if I’m being fed a ton of additional information that they’re trying to sell me on something.
Annie Duke
That is exactly right.
Jim O’Shaughnessy
So I immediately get a little more suspicious.
Annie Duke
Yes. So you intuitively know it.
Jim O’Shaughnessy
Yeah.
Annie Duke
See, that’s the thing. Because you’re like, wait, why are you telling me all these completely unnecessary details?
Jim O’Shaughnessy
I love the overlaying linguistics and the different rules of language that I had never really actually thought about that, how they can make things that seem stupid, statistically they are. I love that. I’ve never actually thought about...
Annie Duke
You know what I think is interesting about it? I bet you didn’t think the conversation was going to go here.
Jim O’Shaughnessy
I love it.
Annie Duke
What I think is really interesting about it is that if you think about the framing effect, it’s in the same vibe world, right?
Jim O’Shaughnessy
Yep.
Annie Duke
And they do recognize the framing effect. So we should understand that there’s just a grammatical effect also. Or a pragmatic effect of, I’m offering you information. Why am I doing that?
Jim O’Shaughnessy
Right.
Annie Duke
Where people, yes, in reality, you should just say 70%. So 70% chance it’s a lawyer. Except why are you giving me all this information? Weird. So maybe I should be paying attention to it. So even they understood it when it was, okay, 1,000 people will die of this cancer. I’ve got a new experimental treatment. If we give it to these thousand people, 600 of them will live. Everybody’s like, yes. If you say 400 will die, they’re like, oh, I don’t know. So they’re understanding that. That, okay, there’s these issues of just sort of how things are framed, but then we also have to take into account the grammatical context in which you’re saying them. And why are you being inefficient? Because that’s what you’re doing when you’re giving the extra information is that you’re adding inefficiency into the piece.
Jim O’Shaughnessy
It’s being very inefficient. I love that. This is absolutely...
Annie Duke
The strange twists and turns of the conversation that nobody was expecting.
Jim O’Shaughnessy
I think that’s actually great, though, because I think immediately about Claude Shannon and information theory. And what he said was pretty straightforward. Information is unexpected. And so built into our processing of that is this natural, why is Annie giving me this additional information? I love that.
Annie Duke
Right. So should it change my probability? Well, in most normal communications, yes. That is what I would say is because you are assuming that I am an honest broker who is trying to help you to understand the world in some way. So I don’t think it’s unreasonable for you to say we’re just having a conversation. I just told you a whole bunch of extra stuff. And it should probably change your forecast. Now, when a politician is, don’t fall for it, please. Or a salesperson is adding in all of this extra context...
Jim O’Shaughnessy
Well, you know, Shannon himself said the amount of information in a stump political speech is probably zero, whereas the amount of information in a short poem could be off the charts. Because of the honest, his notion of the unexpected. But I love the way that feeds into, why are you giving me this information, linguistically? Why are you adding red pencil? There must be something about that particular type of pencil that is meaningful or you would not have been inefficient in a normal exchange.
Annie Duke
I mean, again, going back to evolution, how could we as humans have survived in social groups that we needed in order to be able to overcome the fact that we’re physical weaklings if we did not believe that people were generally telling us the truth?
Jim O’Shaughnessy
Right.
Annie Duke
You have to. For social cohesion.
Jim O’Shaughnessy
Unless you’re a psychopath or a sociopath, you default to truth.
Annie Duke
Exactly. So if we’re just normal people, I’m not a politician, I’m not a salesperson, I’m not whatever. We assume we’re just normal people. And I say, can you hand me that red pencil when it’s the only pencil on there? You’re going to assume that I’ve added that in for a reason because I’m trying to give you some sort of, otherwise I wouldn’t do it.
Jim O’Shaughnessy
Right.
Annie Duke
The reason why I tell people in any training that I ever do, you have to start with the base rate. And then you better have a damn good reason that you’re deviating from it. It better be super, I mean, and obviously, in the market, this is huge. People think dislocations are happening all the time. And it’s like, why are you assuming there’s a dislocation? They’re rare. They’re really rare. And you better have a really good reason for why you think that things are so different now. Words to live by, man.
Jim O’Shaughnessy
But I’m thinking of the cartoon from the New Yorker with the guy at the payphone. So that dates the cartoon.
Annie Duke
Sure.
Jim O’Shaughnessy
And he’s shouting into the payphone, “I don’t care what your earnings estimates are, sell.” And in the background is a burning factory.
Annie Duke
Right, right, right, right.
Jim O’Shaughnessy
Exactly.
Annie Duke
So you better have a good reason.
Jim O’Shaughnessy
He had a good reason.
Annie Duke
He had a good reason. He was looking at the factory legit burning down. So I think that you don’t want to live in one world or the other. The one world is things will always stay the same. And that goes back to the beginning of the conversation. There’s a pace of change. And we don’t want to be one of those people who’s just, it’s all as it has been, it always will be. That’s going to be a really bad place to live. But you also don’t want to be like, it’s different now, it’s different now, it’s different now. So that’s why it’s always, start with the base rate, assume equilibrium, which just means that base rate is going to hold. Obviously there’s variation around it, but that should be your best guess.
And this is one of the things that I always caution people about, particularly when we’re thinking about, for example, valuations. Okay, you’re looking at a company that’s valued three times higher in terms of multiple, which is a lot higher, three times higher in terms of multiple to earnings than any company that’s been like it before. Just because people are like, yo, we’re so excited about this company, you have to understand that there’s just a lot of contagion and social proof that could be happening there. You better be able to tell me why. Why is this company different than all these other companies? Why does it deserve a different valuation?
And the example that I give to try to illustrate that is if you were looking at the price to earnings ratio for Amazon, and you were saying, well, okay, it’s a retail store. Walmart is 4 to 6 times earnings. And here’s Amazon, which is 25 times earnings or whatever. Then you may say, well, that’s just wrong. But if you’re like, well, maybe there’s something different about this company, but you have to understand that, yes, in that particular case, I’m giving you an example where there was something different. It’s not bound by physical space and distribution is different. It’s online, it can do all sorts of other stuff, so on and so forth, that it may be actually deserving of that higher ratio. In that case, you’d be correct.
You also have to understand that under those circumstances, you have a bias toward thinking that it’s different. So whatever explanations you’re coming up with for why it’s different, you should be suspicious of. And what I would really recommend, if you really want to get a good answer about it, is to ask six very smart people to answer that question independently. Don’t all be in a room together, just have them do it independently. Why do you think this is different? And then you should actually add some context onto it. Imagine it’s a year from now and the stock is cratered. Why do you think that is? Why do you think people got it wrong? Imagine it’s a year from now and that price seemed to be reasonable. Why do you think that is? What do you think people got right about the price?
And have six really smart people independently answer that question and you’re more likely to get to whether that’s really a dislocation or not.
Jim O’Shaughnessy
That’s actually a good AI prompt too, by the way. Rather than tell the AI, no matter which model you’re using, say, what type of people should I ask about this problem? And then it generates all the different characters, attaches different probabilities to them. That’s another little trick that is very helpful. Please attach a probability to this and you get the different actors, you get cognitive diversity and you get much closer to an interesting answer from the AI because it defaults to the dead middle and hallucinates anything that is inconvenient.
Annie Duke
Yeah, I would actually add a little color to that, depending on the model that you’re using, is that it defaults to whatever is going to get you to use it more. To be fair, that’s not true of all of them, but for many of them, for the majority of them, I think. Is it Perplexity that doesn’t do that?
Jim O’Shaughnessy
Perplexity generally does not. We have our own AI lab and we have them all in there. And Perplexity generally doesn’t do it.
Annie Duke
So I think that tends to be better on that.
Jim O’Shaughnessy
However, you can prompt even the most sycophantish...
Annie Duke
But that’s why we want to teach about prompts to kids. Because that’s the thing. It’s like you have to take control over this.
Jim O’Shaughnessy
Oh, absolutely.
Annie Duke
What people don’t understand, and this is true of any social media use, is you have to think about AI the same way as social media. It wants you to engage with it.
Jim O’Shaughnessy
It’s maximizing the objective function for you keeping doing it.
Annie Duke
You keeping doing it, which means that it’s gonna tell you whatever’s gonna keep you doing it. So in the same way that if I see some random, you see little clips of things all the time on social media where they’re clipping out 20 seconds of an interview. If it’s not particularly high stakes, whatever. But if you’re gonna change your behavior or your opinion or something based on that 20 second clip, I would really highly recommend you go look at the whole interview. Please go look at the whole interview because there is no context around this thing. And I think that you have to view social media the same, I mean, you have to view LLMs the same way.
When you ask it something and it tells you something, you better try to get the whole interview. Okay? Only cite scientific journals. Give me the citations with the links so that I can go look at them. Show me exactly where you’re drawing this conclusion from. What if this conclusion that you’re giving, what if what you’re telling me is completely wrong? Why do you think that could happen?
I’ve actually just done this with the Google AI where I’ve prompted it twice differently. I actually did it recently. I was trying to find out if a celebrity was still married to another celebrity. And I asked it in a certain way and it was like, yes, they’re still married. And then I asked it in a different way and it was like they got divorced two years ago. The actual answer was they were still married. Which I actually knew, but it was amazing. It was within a second of each other that it gave me a completely different answer. And so you should just act like if I asked in a different way, it would give me a different answer. So what would be the way that I would ask it that might get me a different answer? Because I want to see the whole clip.
Jim O’Shaughnessy
Yeah, I love that. The last thing that I want to talk about is another little bugaboo of mine and that is self selected samples. When that book came out, The Millionaire Next Door, people, “this is the greatest book.” And so I read it. I’m like, well, his sample was completely self selected.
Annie Duke
Of course. There’s, you know, it’s like how many books are like the five habits?
Jim O’Shaughnessy
Exactly. From Good to Great.
Annie Duke
Oh my God. Literally, I’m not for burning books. I’m just, please don’t read them.
Jim O’Shaughnessy
But why does it work so well? Why did they become bestsellers?
Annie Duke
Can I tell you something? This is why I’m writing this book. Because this problem is in the book and it’s not just when people are trying to sell you. So it’s a natural proclivity of us as human beings to say, if I want to have a good outcome, I should look at things that have the good outcomes and try to figure out why. It goes to that explanation. So I’m going to look at all these successful people. I’m going to figure out the things that they’re doing and then if I do those things, I’ll be successful too.
Now, separate and apart from, well, you have to understand there was a lot of luck in there. Bill Gates could do all the things he does today. And if he was born 30 years later, I’m sorry, bud. I’m sure you’re a very smart and wonderful person, but you’re not having that result. And if I ran Bill Gates, if I Monte Carlo’d Bill Gates, what percentage of the time...
Jim O’Shaughnessy
Yeah, I love doing...
Annie Duke
Right. Let’s Monte Carlo Bill Gates when he’s born at the exact same time. So it’s really hard for people to wrap their head around them again because we’re determinists. So we think you will always have the result that you have and that the things that you do will create that result. Again, looping right back to the beginning of the conversation. So it’s this natural thing. It’s this natural thing to look at the survivors and what people, again, compared to what? What about people who do those exact same things?
Jim O’Shaughnessy
And die.
Annie Duke
Who didn’t succeed. We have to look at the whole thing.
Jim O’Shaughnessy
No, no, no. I completely agree.
Annie Duke
I want to look at 50 years, not five.
Jim O’Shaughnessy
Right.
Annie Duke
So working with a client and they were like, well, okay, we generally don’t like to hire people that we would call just difficult to work with. Okay. So let’s think, you don’t really like their OCEAN score, for example. So they were like, but you know what? We looked at our most successful engineers and 80% of those engineers were difficult. We were looking at our top 20% performers for engineers, and 80% of them were difficult. So they were like, we want to actually start changing sort of signal detection for hiring engineers. And we want to add this to sort of say, if they’re difficult, we want to be more likely to hire them. Right?
Okay, so this is what I’m saying. It’s not just someone who’s trying to sell a book. Okay. So anyway, so they say this to me because I was working on the hiring process with them and I said, oh, that’s interesting. Did you ask the same question about how difficult your bottom 20% engineering hires were? And they were like, what? And I said, well, look, I just don’t know whether this is a quality of successful engineers or just...
Jim O’Shaughnessy
Or engineers.
Annie Duke
So anyway, they went and did that and of course it was literally the same. Just 80% had this quality. If we actually get over our bias, remember I said that one of the things that brings explanatory satisfaction is you think, oh, I’ve learned something that other people don’t know. If we get over our bias, that someone who tends to be disagreeable actually does better in this job and we should overcome our bias. And they’re trying really hard and they don’t understand that they just wrote Good to Great. Like, they don’t get it. Oh, that’s The Millionaire Next Door.
So, you know, I think that this is part of the problem is we want to believe that we can come up with an answer. And the natural way to do that is to say, well, let’s look at the successful ones. And you have to again say, well, I don’t want to just look at these five years. I want to look at all 50. And that’s actually going to help me because God forbid I start waking up at 4 in the morning because I think that that’s going to make me a billionaire.
Jim O’Shaughnessy
It is one of my hottest buttons. I was sitting here on that couch and I saw one of those listicles. This is like 2018, 2019, and I just lost it.
Annie Duke
Yeah, your head explodes.
Jim O’Shaughnessy
And I did this big thread on Twitter. Don’t, please just stop. Don’t do this.
Annie Duke
Yeah. There’s a famous example that is in this world about vitamin E. So in the 90s, it was like, oh, people who take vitamin E are super healthy, so you should start taking vitamin E. And then that result was published. And more people were taking vitamin E. Anyway, when you do a randomized controlled trial, vitamin E, taking vitamin E supplements is net bad. So the question is why? And it has to do with the sampling error problem. Well, the people who are taking vitamin E are self selected. So if they’re taking a vitamin supplement, it means that on average they’re healthier because they are more interested in health. So even if you reduce their health a tiny bit without knowing it, they’re still going to appear healthier than the average person who’s not taking vitamin E.
Jim O’Shaughnessy
Because of the self selection bias.
Annie Duke
Exactly. So you have to do a randomized controlled trial where you’re like, you’re taking vitamin E, you’re not, and then let’s see if it’s net good or bad. And that gets you to the appropriate comparison. And then it turns out vitamin E supplementation is actually net bad. But we only know that if we do a randomized control trial. And this is how we end up doing things like taking hydroxychloroquine because there’s some random small sample, whatever, without a control group. And then people are like, I’m going for it.
And here’s the thing is that I’m fine if it’s a free roll. I don’t care if you’re going off a small sample that is suggestive data if there are no negative side effects to the thing that you’re doing. I’m really not. But we know that that’s not true of hydroxychloroquine. Hydroxychloroquine has very bad adverse effects. And so the people who do take hydroxychloroquine are basically saying, I’m willing to bear those adverse effects in order for the benefit that this drug is going to offer me.
Now there are some things that you can do that really just don’t have any downside. One of them that is very close to no downside are statins. Although I just want to say for the record, I’m not a medical doctor, so please don’t go take statins because I told you to. Go talk to your doctor about it. But my understanding as a non medical professional is that hydroxychloroquine has very bad side effects and statins can have some muscle weaknesses, some issues that are pretty rare. It tends to be in people who are sicker. So if you’re a healthy individual, it’s a relatively free...
Jim O’Shaughnessy
And I’ve read recently actually just on statins, the newer class of statins has a lower incidence of the muscle weakening.
Annie Duke
Yes. And you can take CoQ10 and things like that in order to resolve it. So the point is that if you’re on the fence and you’re like, oh my cholesterol’s high, should I take this thing or not? And I would give this advice for any drug is say, well, what are the side effects? What are the things that can go wrong? And when the doctor says not too much, you can be like, okay, whatever. But if the doctor says, ooh, a whole hell of a lot, you should be like, well, can you show me the randomized control trial, please?
Jim O’Shaughnessy
Exactly.
Annie Duke
Because I would actually like to understand if this is effective and then I would like to understand if it is effective, how effective is it? What’s the probability of the adverse side effect? What’s the probability that I’m going to get the benefits that I see? And now I can actually say, am I lengthening my life or shortening it? So you better go through all of that. But when the answer is meh, not too much, then I don’t really care.
It’s actually a similar problem to what you’re asking about The Millionaire Next Door. Well, what about all the people who weren’t millionaires? Did they have the same habits? Did they do the same things? What’s the probability, given that you’re doing those things, that you, is it better than average? There’s all these questions that you have to ask of that and you should have been asking that of that drug as well, given that there were bad things that could occur from it.
So if somebody says if you work out, let’s say if you take a brisk 30 minute walk five times a week, you’re more likely to become a millionaire, then go do it. Because actually, independent of any millionaireness, taking walks is good for you. But I would really like it if you didn’t believe it was going to make you a millionaire. That’s the only thing. But go and walk. Maybe it’ll help, I don’t know, but it will help you in other ways.
Jim O’Shaughnessy
I love both the theme and when is the book coming out?
Annie Duke
Well, so the manuscript isn’t even due until June, so it’s going to be out in Q4 of 2026 or maybe Q1 of 2027 would be my guess. So you have a little bit to go.
Jim O’Shaughnessy
I really want… will you send me the manuscript?
Annie Duke
I will.
Jim O’Shaughnessy
I would love to read. Sounds like… I’ve loved all your books as you know.
Annie Duke
Well, thank you.
Jim O’Shaughnessy
But I love this book because I think it’s incredibly necessary.
Annie Duke
Well, particularly in the information environment we live in.
Jim O’Shaughnessy
Yes, exactly. And I just think anything that can be helpful to people who are drowning, that’s a good thing with very little downside.
Annie Duke
Yeah. I just want to be clear, there are no equations in the book. It’s literally just, hey, ask this question. And it’s just a checklist of questions that you can ask that will allow you to avoid what I always say, impaling yourself on the data. Please don’t impale yourself.
Jim O’Shaughnessy
Well, back to evolution though, we evolved to make stories click. It is, you know, when I was selling quant investing, what did I do? I told stories about it. And I used to joke, I’m going to tell you stories about why you shouldn’t listen to stories when making investment decisions. But I never put numbers like, no, because people just totally glaze over.
Annie Duke
Right. And I think that the important thing is that this is the mind that we’re born with and the idea that you can overcome the types of errors that we’re going to make by sheer, well, I know about it now and so I’m fine or whatever. I mean it’s hilarious. And so you have to start to think about how do I build structure into the way that I’m thinking so that I can mitigate the errors that I might be making. That’s always been my approach in my writing. There is nowhere in the book Quit that I say, oh, you know about the sunk cost fallacy, so you’re fine. In fact, I say quite the opposite. Instead I say, look, you have to start making pre commitment contracts. You have to set kill criteria which are just stopping rules.
You have to follow them. You need to write them down on a piece of paper. You’ve got to say, if I observe this signal, I will stop doing the thing that I’m doing. And that’s going to really help you to get to these decisions to stop things faster. Because you can’t just, the one that was always hilarious to me was investors who would say, oh yeah, I know about the sunk cost fallacy. So every morning I just imagine, what if I didn’t own these things, would I buy them again? And I’m like, well that’s nice, but you do own them, so that doesn’t work.
So if there were no transaction costs, then I would say, if there were no transaction costs, I would just tell you, well, actually sell it. Just sell everything and then decide whether you want to rebuy it. Now, obviously, unfortunately, there are transaction costs, so that becomes impractical. But you can set rules around when would you sell and when would you not that are going to be much more helpful than your nice little thought experiment, which is cute, but it’s not actually going to help you overcome anything.
And I think that with this idea of this jumping to explanation that we do and then getting settled into that explanatory satisfaction, I can give you this conceptually all I want. I can tell you that these are errors that we’re making left and right. I can tell you really fun stories about baseball players and orthopedists and vitamin E and education policy. I actually have one about school size and things like that. I can tell you this all the time. It’s going to be obvious when I tell you this, that these errors are being made and you’re going to be like, cool, done. I’m fixed now. And no, the book is actually saying you need to literally go through this checklist and these are the set of questions that you have to ask of any piece of information that you see. And until you’ve asked all of these questions, you cannot draw a conclusion from it.
Jim O’Shaughnessy
I love it. I love it. And can you program your personal AI to go through everything for you?
Annie Duke
Well, you should be able to. And I think that’s actually something that AI ought to be helpful with.
Jim O’Shaughnessy
Yeah.
Annie Duke
You can input a checklist into the AI and then the AI can actually tick it off for you and make sure that you’re actually going through that. Now, I wouldn’t necessarily trust the AI to answer the question that you’re asking unless you’re a very good prompter of AI and you do... But it can at least get you to ask the question.
Jim O’Shaughnessy
Exactly right. So which is kind of a forcing function. It’s like, why is it saying that? Because when you were going earlier about getting people about learning about how much they actually do know about dogs, it kind of the same thing. If you have that on your AI, I just kind of think, how cool would it be if I had that and I’m reading an article and it’s running in the background, that would be really cool.
Annie Duke
So that’s actually that type of sort of decision copilot kind of thing is something that we’re actually working on at the Alliance.
Jim O’Shaughnessy
Oh, very cool.
Annie Duke
We’re very lucky to have had Eric Horvitz just join the board from Microsoft. And he actually is working on a, he’s been working on a tool like that. That we’re hoping to be able to translate for younger people as well.
Jim O’Shaughnessy
Fantastic. Annie. I always have so much fun when I’m talking to you because you know so much about so many things. I love the linguistic thing.
Annie Duke
That was just, that was...
Jim O’Shaughnessy
I’m going to be just thinking about for the rest of the night.
Annie Duke
I like to surprise. I like to surprise.
Jim O’Shaughnessy
Well, you get one last chance. I don’t know if you remember from the last time you were on because it was a long time ago, but we ask, we’re going to wave a wand. We’re going to live in magic world. We’re going to make you empress of the world. You can’t kill anyone. You can’t put anyone in a re education camp. But what you can do is speak into a magic microphone and you can incept the entire population of the world. Whenever their morning is, they’re going to wake up with the two things you’re about to incept for me and our audience that they’re going to wake up and they’re going to say, you know, I’ve never really acted on these ideas, these morning ideas that I get, but this time I’m going to be different. These two ideas that you’ve incepted in them, they’re going to think are their own and they’re going to act on them. What two things are you going to incept into the...
Annie Duke
Oh my gosh, this is so hard. I don’t know what I answered last time. If I could actually get people to think probabilistically, please, that would be so great. Imagine how much better it would be to watch the news. Oh my gosh, that’d be so much better. Just view the world as it is. It’s probabilistic. First of all, you won’t be so freaking surprised and probably not as angry all the time. I think that would be good.
And I think the first explanation that comes to your mind for what you see is not necessarily the right one. You have to be much more skeptical. And I think that would, I think that’s a really big one because all this information is floating around and you just think you know what it means. It’s like unless you actually know how to ask a question of that data, you just don’t. Think about how many political arguments are based on the exact same piece of information. And you’re interpreting it one way and I’m interpreting it another. And we never stop to think, well, maybe we’re just not interpreting it right.
Jim O’Shaughnessy
Yeah, maybe we’re wrong.
Annie Duke
Maybe we’re wrong. I’m just gonna, I just wanna add a third one in. Just, but this is not me. I’m gonna attribute this to somebody else.
Jim O’Shaughnessy
Okay.
Annie Duke
I had a conversation with Laurie Santos, who’s wonderful. She does all, you know, she’s the happiness person over at Yale. I asked her, sort of, what’s the best thing you think that somebody can do to be happier? And her answer was, when you’re with other people, don’t even have the phones anywhere near where anybody can see them, because just having it here makes the conversations worse. Which I’ve actually implemented in my life. But the one that really changed my life was get out in nature. I’ll go out with my dog for two hours, and I literally don’t touch my phone. I don’t think anything has had such a profound impact on my happiness more than that.
Jim O’Shaughnessy
Now you got to tell me, what kind of dog do you have?
Annie Duke
Oh, I’m trying to hide it because, maybe people will take my class on Maven, no it’s fine. I have a mini Bernedoodle named Otis who is one of the loves of my life. I think that my children and husband might say the love of my life. I’m not sure, but...
Jim O’Shaughnessy
Annie, thank you so much. So great to see you. And this is so much fun to do in person.
Annie Duke
Yeah, it’s so nice to do it not on Zoom. I was so happy that we could make this work.
Jim O’Shaughnessy
Yeah. Thanks for coming.
Annie Duke
Well, thank you for having me. It’s always a joy.
