Algorithms To Live By #2: Optimal Stopping - How Long To Look For And When To Stop To Find The Best (The 37% Rule)

The 37% rule came to exist as a result of the study of optimal's stopping most famous puzzle, the secretary problem. The setup of the problem is as follows. Imagine you are interviewing candidates for a position as a secretary, and your goal is to maximize the chances of hiring the best candidate in the pool. Even though you cannot assign scores to the candidates, between any two, you have a preference. Therefore, you know their relative ranking once you interview each one. The order in which they come in is random, and they come one at a time. You can offer the job to an applicant at any point and they cannot refuse it (or in other words, are guaranteed to say yes). Once you offer the position and the candidate accepts it, the whole process is terminated. So, if you make an offer, you don't get to interview the remaining. Furthermore, once you pass over an applicant, he/she is gone forever.

Pixabay Image Source.

The secretary problem asks the question: for how long should I look for the ideal candidate, and when should I stop and make an offer? The secretary problem became very famous in the mathematical circles of the 1950s. It went to several different geographic places purely by word of mouth. The true origins of the problem are unknown. Somehow, this problem got to the ears of Merrill Flood, a renown figure in the mathematical world. In 1958, it was him that made the discovery of the 37% rule. He also claimed that he had been considering the problem since 1949.

To understand the 37% percent rule, first we have to address different likelihoods of finding the best candidate for different pool sizes. If you have only one candidate, hiring him/her guarantees having hired the best candidate! Congratulations... It is when the pool size starts growing that the problem becomes more interesting. How often is it that you come across a candidate that is the "best-yet"? The first candidate will always be the "best-yet", since he/she is the first. But as we go on, on one hand, "best-yet" candidates will be better and better. But on the other hand, we will come across these "best-yet" candidates less often as the list of previous candidates to compare to becomes larger and larger. Therefore, a needed "trial" period is needed in order to develop a benchmark. Afterwards, you need to make a decision.

The Look-Then-Leap Rule states that you have to set up a predetermined amount of time for looking and exploring your options to gather data in order to develop a benchmark. Afterwards, you need to enter the leap stage. In the leap stage you are to commit to any option that passes the benchmark. Then, what portion of your options should you go through during the looking phase, and what would be the chances of choosing the best one in the whole pool? As the pool grows, this number changes, but it stabilizes at 37%! You should go through 37% of the pool. Funny enough, with this method, your chances of finding the best one in the pool are also 37%. At first, you might think that 37% chance of finding the best candidate is not good enough, but think about the following. As the pool size grows, our chances of choosing the best candidate at random are 1/n with n being the total number of candidates. Nevertheless, if you mix the Look-Then-Leap Rule with the 37% Rule, the pool size doesn't matter. You will always have a 37% chance of finding the best.

Image Source.

The secretary problem is very similar to some real world scenarios. Think about the housing market in New York City or San Francisco. The demand is so big that rent advertisements go up and down within hours. Sometimes, they do not even go up! Therefore, when you go check out an apartment, if you want it, you need to make an immediate offer. Even though the problems are so similar, they cannot be solved in the same manner. While it makes sense to calculate the exact numbers for optimal solutions, it is unrealistic to ask a person to check 37% of the whole market. Nevertheless, this algorithm provides you with a general approach. It suggests that you scan a portion of the market without compromises, and then commit to something that passes your developed benchmark. Maybe if you reduce your housing search to a county or a delimited area, you could even apply the 37% rule exactly!

Sources:

• Algorithms To Live By

If you want to check out other thoughts that this awesome book has evoked, click on these past posts:

• Algorithms To Live By #1: Expectations, Authors, And Algorithms

Best,

@capatazche