Comment by t43562

I remember optimizing a huge build that took days. We optimised it to only hours on a cluster - it took years to get this right. We realised at the end of this that the packaging system (handled by another team) was taking about 2x longer than the build because there was one particularly huge zipfile that sat there compressing for ever and ever and ever while the rest of the cluster was idle.

As in this story we blew enormous effort optimizing something that was very difficult and because of our blinkers (and our management's blinkers) we missed that it was only able to achieve a limited result. We didn't look at the whole system. In the end it was never fixed because......siloed thinking...or something.