Algorithm Aversion: Compare the Mistakes You Can See

The failed lunch rush needs a comparison

Imagine a café owner, Maya, looking at sandwiches nobody bought. Her forecasting software predicted a busy lunch. Yesterday she made a poor prediction herself, but that felt understandable: a regular customer cancelled. Today’s mistake feels like evidence that the software cannot be trusted.

The café is fictional. It illustrates a distinction worth keeping: evidence that a forecaster makes mistakes is different from evidence that the available alternative makes fewer mistakes. Both can matter. A fair comparison needs a record of both.

In five forecasting studies published in 2015, Berkeley Dietvorst, Joseph Simmons and Cade Massey varied whether participants saw a statistical model perform, a human perform, both or neither. Participants then chose whose forecasts would determine their bonus. Seeing the imperfect model perform reduced confidence in it and willingness to choose it over the worse human forecaster, including among participants who had seen the model outperform the human.

That is the particular form of algorithm aversion at issue here. These were statistical forecasting tasks, not tests of today’s conversational assistants. They do not show that everyone abandons a system after one error, or that every reluctance to automate is mistaken.

Put the embarrassing day back into the record

Here is an invented five-day example. Each forecast is written down before lunch, and demand is fully observed: nobody is turned away because food has run out. “Error” means the absolute distance between the forecast and actual demand, so misses above and below the outcome do not cancel.

On Thursday, demand was 30 sandwiches. The model predicted 50 and Maya predicted 35, giving errors of 20 and five. Maya has a legitimate complaint about that day. Across the whole example, however, the model’s errors total 36 and hers total 55. Divide by five days and the average absolute errors are 7.2 and 11 sandwiches, respectively.

A decision based only on Thursday would choose the weaker overall forecaster in this constructed record. A decision based only on the average would miss the need to investigate Thursday. The table makes both facts visible.

Five invented days cannot establish which real forecasting system deserves trust. This is a calculation showing what a comparison must contain. In practice, Maya would need enough relevant observations to judge whether the apparent advantage holds, including days like the one she is planning for.

There is also a measurement trap. If she prepared 40 sandwiches and sold all 40, sales alone would not reveal whether demand was 40 or 60. Recording stockouts matters before treating sales as a complete answer key.

Small adjustments can change willingness to use advice

In a follow-up published in 2018, the same researchers studied whether people would use an imperfect model if they could adjust its forecasts. Participants predicted students’ percentiles on a standardized mathematics test. In one study, adjustment limits were two, five or ten percentile points.

Even the smallest allowance increased willingness to use the model relative to having to accept it unchanged. The differences in uptake between those three adjustment limits were not statistically significant. Participants with adjustment options also forecast more accurately overall than those without them, because they relied more on the useful model.

This separates two questions: does an override encourage someone to use a strong starting point, and does the override itself improve that starting point? The study supports the first benefit without making every individual correction wise.

For Maya, a useful record would therefore keep three numbers: her independent forecast, the model’s original forecast and her final adjusted forecast. If she saves only the final order, she cannot later tell whether her intervention helped. The same principle of preserving an independent judgment appears in our article on how AI feedback can amplify human bias, which examines a different problem: learning a distortion through repeated exposure.

Sometimes the algorithm label earns extra trust

The opposite pattern also exists. In research published in 2019, Jennifer Logg, Julia Minson and Don Moore found that people often moved further toward identical advice when it was described as algorithmic rather than human. Their tasks included numerical estimation and forecasting. They called this algorithm appreciation.

The experiments examined advice before performance feedback. Appreciation weakened when people compared the algorithm with their own judgment rather than another person’s. Choosing an adviser without a performance record and reacting after seeing its errors are different situations. “People trust algorithms” is too broad to describe either result well.

Give a correction a reason you can check

Suppose Maya knows the office across the street will be closed tomorrow, while the model’s inputs do not include that closure. She has specific new information to consider. She can record the reason before lunch and later compare the original forecast with her adjustment.

“Too high” provides much less to investigate. If she repeatedly lowers forecasts because they feel uncomfortable, saving those changes will show whether they improve results or simply make the decision feel more familiar.

She also needs to choose what success means. Average absolute error treats a ten-sandwich overestimate and a ten-sandwich underestimate equally. Her business may attach different costs to waste and missed sales. A model can win one comparison while losing another, so the chosen measure should reflect the decision being made.

For the next lunch, Maya can keep a short record: the two original predictions, the final order, any new information, observed demand and any stockout. When a forecast fails, that record gives her something more useful than a verdict about machines or people: a way to identify which part of the decision needs changing.