For Accurate Forecasts, Limit the Human Factor
Posted by admin
Everyone’s Doing It
Adjusting forecasts was a popular activity: on average over our sample, three-quarters were changed in some way. Forecasters at the food manufacturer modified 91 percent of theirs, for example. While the retailer’s four forecasters adjusted only 8 percent of theirs, they had over 26,000 to produce each week, so there probably wasn’t time to make many more changes. The pharmaceutical firm held 17 forecast review meetings — tying up about 80 hours of management time — every month.
Many of the changes were small, as if the forecasters sometimes simply wanted to put their calling card on forecasts by tweaking them slightly to show that they were still doing their jobs. We received anecdotal evidence indicating that managers would alter more of those forecasts that were presented earlier in a review meeting. Later forecasts were simply waved through as people got tired and felt that they’d already done enough to justify calling the meeting.
Of course, people didn’t make changes purely to show how hard they were working. They usually felt that they had good reasons and, in some cases, we found that this was so. But the problem is that people have a tendency to find an explanation for every movement in their sales graphs, including swings that really are random. This makes them overconfident that their adjustments will increase accuracy, so they tend to adjust forecasts even when it isn’t appropriate. We’re brilliant at inventing theories for everything we observe. At 1:01 p.m. one December day in 2003, just after Saddam Hussein had been captured, the price of U.S. treasuries rose. Half an hour later the price fell. The Bloomberg news channel used his capture to explain both movements. The dull statistical forecast is no competition for these colorful but often groundless tales, so it gets adjusted.
The Illusion of Control
All this adjustment behavior can have some odd consequences, according to psychologists. When we perform a task that demands our skill, we normally think we can control the results. For example, if you learn to play a musical instrument, you expect to make fewer mistakes as you practice more. Many swings on a sales graph result from random, unpredictable events. Yet, because forecasters see making adjustments as skillful, they can start believing that they have some control over what they’re forecasting and, therefore, that they can predict movements on the graph. This is known as the “illusion of control.” It’s likely to cause forecasters to make even more changes. After all, the more you adjust, the more control you think you have.
Despite these concerns, judgmental adjustments to statistical forecasts can still play a useful role in improving accuracy. We found that, on average, they reduced the average absolute percentage error of the forecasts by 3.6 percentage points for all companies except the retailer. But this modest improvement masked considerable variations in the effectiveness of the changes.
Is it possible to filter out the types of adjustment that are unlikely to help? Larger changes are more likely to increase accuracy, although it takes some nerve to make them and risk being badly wrong. Bigger alterations are, therefore, likely to be made for very good reasons. For example, a forecaster might have reliable information on an important future event, such as a sales promotion, that will cause the statistical forecast to have a big error. In contrast, the smaller changes may be the type of tweak that we mentioned earlier, or they may be the result of a forecaster hedging their bets because the information they have on a future event is unreliable. The lesson is clear: although small alterations, by definition, do relatively little harm to accuracy, they’re generally a waste of time.
Bias Toward Optimism
Positive adjustments are more likely to be wrong than the negative ones. Psychologists tell us that people have an innate bias towards optimism. For example, construction projects usually take longer to finish and cost more than predicted. Some of this may be a result of deliberate misrepresentation to win contracts, but there is evidence that optimism bias still plays a role in these poor estimates. It seems, therefore, that when forecasters are asked to estimate the effects of, say, an advertising campaign they cannot resist being optimistic. And, of course, this is often exacerbated by the infectious enthusiasm of their colleagues in sales and marketing.
A particularly damaging intervention is the so-called wrong-sided adjustment. An example of this is when you adjust the forecast upwards but should have made a negative revision. Suppose that the statistical forecast was for 600 units and you pushed it up to 650. If actual sales turn out to be 580 units, you’ll have made a wrong-sided adjustment. Any such alteration is bound to reduce accuracy, yet they were common in our sample, particularly when the change was positive. More than a third of the positive changes made by the non-retailers were in the wrong direction. If we could remove a representative sample of only half of these wrong-sided positive adjustments, it would improve accuracy by seven percentage points.
The first stage in eliminating wrong-sided adjustments is to catalog the reasons behind every change. Survey evidence from U.S. forecasters has shown that 69 percent of firms claim to do this, but our experience is that reasons are often recorded in a shorthand form that makes them indecipherable later even to the forecasters themselves. When big errors have occurred, an analysis of the causes can be helpful. This should be done as part of a quality improvement program rather than in an atmosphere of blame. A good forecasting support system can help by encouraging the compilation of records to make it easier to review events — sales promotions, say — and reflect on how today’s circumstances match those of the past.
Newer Not Always Better
Many of the forecasters we spoke to shared Henry Ford’s philosophy that “history is bunk.” In review meetings, the most recent movements in sales graphs were examined closely, while earlier data was often ignored. In one firm the forecasters said that they never fit their statistical methods to data that was over three years old because “the trends were different back then.” So great was the bias towards recency that statistical methods were sometimes fitted only to the past six months’ data, which didn’t give them much of a chance. The methods commonly found in business forecasting software are designed so they can adapt to changes in trends or seasonal patterns if these occur, but they need plenty of data to do this well, if you limit the figures available to your statistical methods, you’re unlikely to be making judgmental changes from a reliable baseline.
When we analyzed the accuracy of the retailer’s adjustments against its statistical forecasts, they looked awful. The positive changes raised the average absolute error from 32 percent to 65 percent. Significantly, 83 percent of these adjustments were either too large or wrong-sided. Something odd was going on: Why would the forecasters of this major company work so hard to make mediocre statistical forecasts much worse?
Most people would probably consider a forecast to be an estimate of the most likely future demand. It turned out that the retailers’ forecasters were focusing on a different quantity. Often they were trying to determine a level of demand that had only a small chance of being exceeded — i.e., one that would limit stock-outs. Determining this would tell them how much inventory they needed to hold. For example, their system might forecast a demand of 500 units but they would adjust this to 550, reasoning that this level of inventory could cover anything but the most extreme level of demand. In effect, we’d measured the effectiveness of the forecasters’ adjustments unfairly, because they weren’t trying to predict actual demand.
But the retailer’s approach was still problematic. First, its forecasters had never clearly defined what they were forecasting. They called the adjusted figures “forecasts”, running the risk that other managers would wrongly interpret these as estimates of the most likely demand. Second, they had never determined what probability of a stock-out was appropriate in order to balance inventory-holding costs against the costs of disappointing customers. Nor had they done any analysis to see whether their adjustments were leading to over- or under-stocking.
Not So Positive
The adjustment of statistical forecasts is a key part of forecasting in most firms. It’s often impractical to use statistical methods to mode the effect of forthcoming events that you know are likely to have a big impact on demand. Human judgment has to bridge this gap and, if applied correctly, it can greatly improve accuracy. But our study has shown that the potential benefits are largely negated by excessive intervention and optimism. Indeed, had our non-retail forecasters been banned from making positive adjustments, their changes would have improved the average absolute error by over 20 percentage points.
Yet banning all positive adjustments is unrealistic for most firms. The answer is first to define carefully what you’re forecasting: Is it expected demand or is it the stock level required to cope with excessive demand? Then restrict yourself to key adjustments based on reliable market information and record the reasons for these changes. Lastly, statistical methods need plenty of data to prosper, so don’t starve them by restricting them to the past few months’ worth of data.
In the long run, better software, better market information and better-trained forecasters are needed. Companies are too often satisfied by software that fails to provide good baseline forecasts. CFOs need to ensure that their forecasts beat standard benchmark tests. If companies are to improve their forecasting, they need to take all these aspects seriously.
