Comment by mistercow

It's the hardest thing about building perf-related PoCs. Every time I've built a prototype to prove out an optimization, I've spent the entire duration of the project watching the benefit shrink, stressing that it would dwindle to nothing by the end. So far, I've been lucky and careful enough that I haven't convinced a team to expend massive resources on a benefit that turned out to be fictional, but I've had it happen enough times at the scale of a p-day or two that it always worries me.