Decision-making under uncertainty with Amazon Forecast

Next generation AI forecasting tools promise to help you see into the future, but there are some things you need to know to get the most from them.

Written by Daemonite Damien Jade Duff

Amazon Forecast is a fully managed cloud based machine learning solution for time-series forecasting. “Time-series forecasting” is when you get a computer (or a human) to look at historical data, such as in-store sales for items (and related data such as promotions), and predict future movements in that data. For example, here is a forecast of weekly sales in a real department store, generated by Amazon Forecast:

There are many applications for time-series forecasting, including demand prediction for stock, purchasing and workforce management, so it’s an important topic in business, and forecasting software is a necessary part of it.

But forecasting is basically guessing, and so at all times you’re dealing with a lot of uncertainty, and it’s important to consider how you would base your decisions on uncertain data. Modern forecasting tools such as Amazon Forecast have ways for dealing with this uncertainty built in. In this article we will give you an idea of what kind of results to expect from a forecasting engine, so that you can go on to think about how you can adapt them to your business.

Confidence intervals

Firstly, in addition to a forecast, forecasting tools like Amazon Forecast will give you a “confidence interval” as below:

If the the forecast is the forecasting engine’s attempt to get an estimate that will be close to the average future value over possible futures, this interval is made up of low and high forecasts, called the p10 and p90 forecasts, which are the engine’s attempt to get close to the 10th and 90th percentiles over possible future values. Just as you would want any actions you take in the future to maximise your profit according to your forecast, you would also want to ensure that they are compatible with this confidence interval.

Forecast accuracy

Obviously, even this confidence interval can be wrong, so your forecasting engine will provide you with an estimate of the average “error”, or wrongness, in your forecast. It does this by using “backtesting”. It forecasts for a period for which you actually have data and measures the difference between your forecast and the actual “ground truth” values, as below:

The average difference between your forecast and the ground truth values is your “error” measure — the measure of accuracy. In the above example, we calculate that the average percentage error in the forecast is 4.8%. We also calculate that the error in the low and high estimates in the interval (called p10 and p90 losses) are 1.7% and 3.0% respectively (note that these two errors are not really percentages but as they are analogous to the average percentage error in the forecast, it makes sense to present them as such). It is worthwhile conducting multiple backtests at multiple time horizons (Amazon Forecast is capable of this) and aggregating them — it is also worthwhile visualising all of the accuracy measures obtained, to get a feel for how the accuracy might fluctuate, and adjust our trust in the resulting forecasts accordingly.

Once you have a forecasting system set up and producing forecasts, you can also retrospectively check their performance using the same accuracy measures. We build this into all of our solutions because we know that it gives our clients dynamic insight into the performance of their forecasting systems over time.

Irreducible uncertainty

Despite these tools for managing uncertainty there is always the possibility that your forecasting system, whether human or AI, will just get it badly wrong. Almost every business was not able to predict the changes that came along with the COVID-19 pandemic and its vast ramifications — for example, in consumer behaviour and demand. The forecasting engines discussed above base their decisions on previous patterns and would not have been able to predict this event and, unless they had seen data from a previous pandemic or similar crisis, would not have been able to even hedge for the possibility.

This does not mean we must ignore all forecasts and fly utterly blind — rather, it means that, in addition to hedging according to confidence intervals and accuracy measurements, we must be prepared for the possibility that everything can change — as always, in business.

Find out more about ML & AI at Daemon
www.dae.mn/solutions/machine-learning/