Comment by chaz

> Finally, we implemented another pipeline to load data into our key-value store for serving results. This mundane data copying ended up being one of the dominate items for the original development.

I'm always amazed by how difficult it is to simply get the data. Sounds so simple, but it's fraught with all kinds of issues in structure, timing, reliability, and scale -- and it's usually underestimated. Every BI project I've ever worked on was mostly spent simply getting the required data together into a single database. After that, it's a relative snap.

A bit off topic -- I'm guessing that healthcare.gov's biggest technical hurdles were similar. Simplistically put, it's a shopping comparison site and the UI/functionality is fairly trivial (which is why several HN comments suggested that a small team could built out the whole thing in days/weeks). But imagine if the data to support it was from 36 different legacy sources that were unreliable, poorly documented, and built/managed by completely different vendors. That's going to take up the majority of your time and frustration. Database-driven websites are easy if the data is already built for you.

Great article -- thanks for writing it.