Comment by bearjaws

Without seeing more this seems like it could be solved by not recomputing the entire history to add on data. Depends what kind of math you are doing however.

Some sort of check point system could likely save significant IO.

What am I missing that requires you to recompute all data every day?