Reducing Type 1 Errors Significantly Improves Your Investment Performance

During their latest episode of the VALUE: After Hours Podcast, Taylor, Carlisle, and special guest Juan Torres discussed Reducing Type 1 Errors Significantly Improves Your Investment Performance. Here’s an excerpt from the episode:

Tobias: JT, have you got some veggies for us today?

Jake: I do. Before I get into those, I wanted to do a quick shoutout to my guys here that I’ve been staying with in Florida. These two guys, Paul and Rick, who originally, they met and worked together running the main guys at Notre Dame’s Endowment. They spun out of there now and they have their own thing going on, but they still teach a class at Notre Dame called The Art of Investing. They have great people coming through, it’s like, Todd Combs and the CEO of Booking. These guys know everybody. Don’t be surprised if you see these guys all over the place here soon, because they’re going to have a podcast coming out that’s associated with the interviews that they’re doing for this class. I think it’s going to be really good. So, Paul and Rick will be on your radar, I imagine. Don’t be surprised you heard it here first.

So, anyway, yeah, let’s get into some vegetables now. I should probably apologize a little bit, because we’re going to be talking about type 1 and type 2 errors, and we’re going to have a little scenario in here that has some numbers in it. I know sometimes numbers are hard to translate in an audio context. So, I’ll do my best to go slow and probably repeat numbers. Maybe writing them down might help if you really want to learn from this. But I’m basically stealing this little segment out of a book called What I Learned About Investing from Darwin by this India fund manager named Pulak Prasad. So, without further preamble, let’s get into it.

So, type 1 errors are those are where you’d be making a bad investment when you think that you’re making a good one, right? So, it’s like sins of commission. You buy something, you lose money on it. A type 2 error is when you reject a good investment idea, because you erroneously think it’s going to be bad, right? And so, these are the sins of O mission. We’re missing out on something that would have been good for us. Okay. So, it’s unlikely that you’re going to have this perfect calibration between type 1 and type 2 errors, because they’re like opposite ends of a teeter totter. So, I think it’s an important question to ask yourself is, which direction if you had to lean? And chances are you have to lean one way or the other, which one should you try to lean toward, type 1 or type 2?

So, what we’re going to do is run through a little bit of some scenario to show mathematically which way you should probably lean. Okay. So, let’s assume that your investable universe is 4,000 possible companies that you could buy. And to make the math easy, we’re going to also assume that 25% of these companies will provide us with a reasonable return over, let’s say, the next several years. They’re decent enough businesses, management is okay, acceptable growth rates, not crazy leverage, reasonably priced, whatever your version of what makes for a good investment. What we’re saying then to simplify is that 1,000 of the 4,000 companies are good investments, and 3,000 of the 4,000 are actually going to lead to bad results, okay? So, these are our two pools that we’re fishing in, basically.

Next, we’ll assume that you’re actually a pretty good investor, and you’re able to identify with 80% accuracy, which companies to put your money in and which to avoid at an 80% clip? So, you’re actually pretty good at picking and avoiding. Set another way your error rate is 20% for, both type 1 and type 2 errors, okay? So, what do you think is the probability then of making good investment, if it’s 80% hit rate?

Tobias: One in three.

Jake: One? Care to hazard, I guess?

Juan: I’m going to go with 15%.

Jake: Five-zero?

Juan: One-five.

Tobias: One-five.

Jake: One-five. Okay. You guys are both way too pessimistic.

[laughter]

Jake: So, it’s actually 57%. I’ll walk you through the math of this real quick. So, remember, there are 1,000 good investments, and you commit a type 2 error at a 20% rate. So, you’re mistakenly rejecting 20% of these. You’re only correct in selecting 800 of those 1,000 companies, right? Is that making sense? Then there are also 3,000 bad investments, and you commit a type 1 error at a 20% rate where you mistakenly select 600 of those companies thinking that they’ll be good investments. So, the universe is then actually like 1,400 that you’re making, so it’s 800 plus 600, these two pools. And of those 1,400, only 800 actually are. So, 800 divided by 1,400 gives you a 57% hit rate, okay?

Tobias: Good one.

Juan: [chuckles] Interesting.

Jake: So, you have 80% accuracy and yet, you end up somehow closer to a coin flip. All right. Now, what if, and this is to get at, which way should we lean, what if you could magically reduce your error rate one side or the other, like, type 1 or type 2? So, let’s imagine first that you’re going to reduce your type 2 errors, and we’re going to take it down from 20% down to 10%. So, you want to focus on not missing out on those good opportunities, the big winners, okay? Of the 1,000 good investments, then you’ll select 900 of them, which is a 90% hit rate. Of the 3,000 bad investments, you’re still at that 20% rate, right? So, you’re at 600 that you thought were good are going to end up not being good of the 3,000.

So, you’ll have selected 1,500 investments, 600 plus 900, but only 900 of them are going to be good. So, you’re going to go 900 divided by 1,500, that gives you a 60% chance, okay? So, you’ve improved your probabilities then from 57% up to 60%, just a little marginal increase. Not that great. Now let’s imagine that you can reduce your type 1 errors from 20% down to 10%. So, you’re working hard to get better at rejecting bad investments, okay? Out of the 3,000 bad investments in that market, you’ll select 300 of them, right? This is that 10%. Your type 2 error stays at 20% because that’s not what we were focusing on, and so you’ll still erroneously reject 200 of the 1,000 good investments by making your 20% error rate, but you then therefore select 800 correctly.

All right. So, you’ll have selected then 1,100 businesses, 300 plus 800, but only 800 of them are actually good. So, 800 divided by 1,100 is actually 73%. So, we made a huge jump. We improved by 16% or 16 points basically instead of just 3%, by focusing on type 1 error reduction. So, I think this math shows, and obviously, this is a little bit of a hypothetical example, but I think the math is pretty clear that improving your rate of rejection of bad investments instead of focusing on not missing out on the good ones is the key, like the ticket to doing better in this game. So, the fear of losing money should trump the fear of missing out, if you have to choose between those two on the spectrum, which again, we always end up back at buffet, rule number one, don’t lose money. And that’s again, what we’re talking about. So, there’s a little hypothetical math for you on why you should probably lean one way or the other.

Tobias: That’s a good one. That’s very like Buffett says, “Lots more mistakes of a mission than commission.”

Juan: That’s really interesting. For someone who runs a podcast that talks a lot about probabilities, probabilities are really difficult to understand, and calculate, and all of that, but they are very powerful when you can actually understand what the math implies.

Jake: When hopefully, this was reasonably not too difficult to follow along with the numbers since they were such round numbers. But again, it is always hard to just say numbers and then picture them.

Tobias: That’s good. Thanks, JT.

Juan: The other thing is that even the improvement from 57% to 60% compounded, those three points would make a huge difference in the way that you will compound over the period of your career as an investor.

Jake: Yeah, but I’ll take the 16% improvement over the 3%, if I had my choice.

You can find out more about the VALUE: After Hours Podcast here – VALUE: After Hours Podcast. You can also listen to the podcast on your favorite podcast platforms here:

Apple Podcasts