Comment by justincormack
11 hours ago
For some kinds of forecasting it doesn't need to be exactly accurate - eg say you are forecasting traffic for how to scale a load balancer, it can be a bit off it will just cost you a little and you can adjust after - a lot of feedback systems dont need full accuracy. You wouldnt use this to forecast whether to buy shares. It basically replaces various traditional curve fitting tools. If you need a really good forecast yes you are going to build a domain model. At least this is my understanding...
> say you are forecasting traffic for how to scale a load balancer, it can be a bit off
Problem is you really don't even need ML for the curve fitting tools. Just ARIMA or Holt Winters is enough and likely faster. Heck, Kubernetes HPA doesn't even implement full PID, just P.
It seems like the main challenge is going to be finding domains where accuracy is important enough to spend inference token money on, but not so important that you'd hire a team to build a custom model. Or maybe that team borrows Timestral weights to fine tune to the domain, but they probably already have a different model customized to the problem?
Agree on the share-buying point. I keep a side project that tracks about twenty published bitcoin price models side by side and grades each one on its own forecast record. Two lessons carried over to any time-series benchmark for me. First, freeze every forecast as it stood the day it was made; refitting on later data flatters everything. Second, put the naive baseline (tomorrow looks like today) in the same table. The best-known model in my set, stock-to-flow, sits more than 100% off over the past year, and it looked brilliant in backtests. A 0.03 gain means little to me until I see it next to persistence.