Comment by bgentry
21 days ago
As somebody whose first day working at Heroku was the day this acquisition closed, I think it’s mostly a misconception to blame Salesforce for Heroku’s stagnation and eventual irrelevance. Salesforce gave Heroku a ton of funding to build out a vision that was way ahead of its time. Docker didn’t even come out until 2013, AWS didn’t even have multiple regions when it was built. They mostly served as an investor and left us alone to do our thing, or so it seemed those first couple years.
The launch of the multi language Cedar runtime in 2011 led to incredible growth and by 2012 we were drowning in tech debt and scaling challenges. Despite more than tripling our headcount in that first year (~20 to 74) we could not keep up.
Mid 2012 was especially bad as we were severely impacted by two us-east-1 outages just 2 weeks apart. To the extent it wasn’t already, reliability and paying down tech debt became the main focus and I think we went about 18 months between major user-facing platform launches (Europe region and eventually larger sized dynos being the biggest things we eventually shipped after that drought). The organization lost its ability to ship significant changes or maybe never really had that ability at scale.
That time coincided with the founders taking a step back, leaving a loss of leadership and vision that was filled by people more concerned with process than results. I left in 2014 and at that time it already seemed clear to me that the product was basically stalled.
I’m not sure how much of this could have been done better even in hindsight. In theory Salesforce could have taken a more hands on approach early on but I don’t think that could have ended better. They were so far from profitability in late 2010 that they could not stay independent without raising more funding. The venture market in ~2010 was much smaller than a few years later—tiny rounds and low valuations. Had the company spent its pre-acquisition engineering cycles building for scalability & reliability at the expense of product velocity they probably would have never gotten successful.
Even still, it was the most amazing professional experience of my career, full of brilliant and passionate people, and it’s sad to see it end this way.
It remains the greatest engineering team I've ever seen or had the pleasure to be a part of. I was only there from early 2011 to mid 2012 but what I took with me changed me as an engineer. The shear brilliance of the ideas and approaches...I was blessed to witness it. I don't think I can overstate it, though many will think this is all hyperbole. I didn't always agree with the decisions made and I was definitely there when the product stagnation started, but we worked hard to reduce tech debt, build better infrastructure, and improve... but man, the battles we fought. Many fond memories, including the single greatest engineering mistake I've ever seen made, one that I still think about until this day (but will never post in a public forum :)).
It was a pleasure working with you bgentry!
I'm just going to chime in here and say thank you, there still really isn't in my mind a comparable offering to heroku's git push and go straight to a reasonable production
I honestly find it a bit nuts, there's offerings that come close, but using them I still get the impression that they've just not put in the time really refining that user interface, so I just wanted to say thank you for the work you and the GP did, it was incredibly helpful and I'm happy to say helped me launch and test a few product offerings as well as some fun ideas
This!
It absolutely boggles my mind that nothing else exists to fill this spot. Fly and others offer varying degrees of easier-than-AWS hosting, but nobody offers true PaaS like Heroku, IMHO.
18 replies →
I’ve been using Render for close to 5 years, and it’s excellent. I can’t think of anything I use that it doesn’t do as well or better than Heroku did last I checked.
1 reply →
I agree with everything you said, and can only thank the founders for their tremendous insight and willingness to push the limits. The shear number of engineering practices we take for granted today because of something like Heroku boggles my mind.
I am forever grateful for the opportunity to work there and make it an effort to pass on what I learned to others.
1 reply →
> by 2012 we were drowning in tech debt and scaling challenges.
> the greatest engineering team I've ever seen
How do these two things reconcile in your opinion? In my view , doing something quickly is the easy part , good engineering is only needed exactly when you want things to be maintainable and scalable, so the assertions above don’t really make much sense to me.
It is hard to explain the impact of such massive growth over a 2-3 period. New features were coming online while old ones were being abused by overuse. For instance, we launched PostgreSQL in the cloud, something we take for granted today. Not only that, but we offered an insane feature set around "follow" and "forking" that made working with databases seem futuristic.
I remember when we launched that product we went to that year's PGCon and there were people in the crowd angry and dismissive that we would treat data that way. It was actually pretty confrontational. Products like that were being produced while we were also working on migrating away from the initial implementation of the "free tier" (internally called Shen). It took me and a few others months to replace it and ensure we didn't lose data while also making it maintainable. The resulting tool lovingly named "yobuko" ended up remaining for years after that (largely due to the stagnation and turn over).
Anyways, that was just a slice of it. Decisions made today are not always the decisions you wanted to be made tomorrow. Day0 is great, day100 comes with more knowledge and regret. :D
In general, my impression has been that you don't want to architect your solution at first for massive scaling, because:
* You probably aren't going to need it, so putting the effort into scaling means slowing down your delivery of the very features that would make customers want your solution.
* It typically slows down performance of individual features.
* It definitely significant increases the complexity of your solution (and probably the user-facing tooling as well).
* It is difficult to achieve until you have the live traffic to test your approach.
1 reply →
An organization without “tech debt” is not good at prioritizing.
Yea it seems like not much thought was put into scalability.
Tell us more about some of these ideas and approaches that changed you as an engineer! We'd love to hear!
Well many of them you may know in that they made their way into so many systems (though arguably without the refined UX of Heroku) but the two that come up the most and I am teaching others:
* The simpler the interface for the user, the more decisions you can make behind the scenes. A good example here is "git push heroku". Not only is that something every user (bot or human) can run, it is also easy to script, protect, and scale. It keeps the surface area small and the abstraction makes the most sense. The code that was behind that push lasted for quite some time, and it was effectively 1-2 Python classes as a service. But once we got the code into our systems, we could do anything with it... and we did. One of the things that blows my mind is that our "slug" (this is what we called the tarballs of code that we put on the runtimes) maker was itself a heroku app. It lived along side everyone else. We were able to reduce our platform down to a few simple pieces (what we called the kernel) and everything else was deployed on top. We benefited from the very things our customers were using.
* NOTE: This one is going to be hard to explain because it is so simple, but when you start thinking about system design in this way the possibilities start to open up right in front of you. The idea is that everything we do is effectively explained as "input -> filter -> output". Even down to the CPU. But especially when it comes to a platform. With this design mentality we had a logging pipeline that I am still jealous of. We had metrics flowing into dashboards that were everywhere and informed us of our work. We had things like "integration testing" that ran continuously against the platform, all from the users perspective, that allowed us to test features long before they reached the public. All of these things were "input" that we "filtered" in some way to produce "output". When you start using that "output" as "input" and chaining these things together you get to a place where you can design a "kernel" (effectively an API service and a runtime) and start interacting with it to produce a platform.
I remember when we were pairing down services to get to our "kernel" one of the Operations engineers developed our Chef so that an internal engineer needed maybe 5-7 lines of Ruby to deploy their app and get everything they needed. Simple input, that produced a reliable application setup, that could now get us into production faster.
Anyways, those are just a couple.
2 replies →
Absolutely agree and likewise buddy :)
Thanks for sharing your story. Those early days of using Heroku were really enjoyable for me. It felt so promising and simple. I remember explaining the concept to a lot of different people who didn't believe that the deployment model could be that simple and accessible until I showed them.
Then life went on, I bounced around in my career, and forgot about Heroku. Years later I actually suggested it for someone to use for a simple project once and I could practically feel the other developers in the room losing respect for me for even mentioning it. I hadn't looked at it for so long that I didn't realize it had fallen out of favor.
> That time coincided with the founders taking a step back, leaving a loss of leadership and vision that was filled by people more concerned with process than results
This feels all too familiar. All of my enjoyable startup experiences have ended this way: The fast growing, successful startup starts attracting people who look like they should be valuable assets to the company, but they're more interested in things like policies, processes, meetings, and making the status reports look nice than actually shipping and building.
The cancer of corporations: bureaucracy.
As corporations grow, they also increasingly need some level of process which can increasingly look a lot like bureaucracy.
Bureaucracy is a natural consequence of workforce growth. Respectfully, I think you are blaming the symptoms here, not the root cause.
Having been on a bigco team that underwent the same sort of headcount growth in a very short time I have to imagine that "more than tripling our headcount in that first year" was likely more a driver of the inability to keep up than a solution. That's not a knock on the talents of anyone hired; it's just exceedingly difficult to simultaneously grow a team that fast and maintain any kind of velocity regardless of the complexity of the problems you're trying to tackle. The culture and knowledge that enabled the previous team's velocity just gets completely diluted.
Thanks for a capsule tour through Heroku!
FWIW, the team that eventually created "Docker" was working at the same time on dotCloud as a direct Heroku competitor. I remember meeting them at a meet-up in the old Twitter building but couldn't tell you exactly which year that was. Maybe 2010 or 2011?
Yep, that team did great work. I remember having lunch at the Heroku office with the dotCloud team in 2011 or 2012 and also Solomon Hykes demoing Docker to us in our office’s basement before it launched. So much cool stuff was happening back then!
I remember Solomon's demoing at All Hands... it was such a crazy time for tech and innovation.
2 replies →
> people more concerned with process than results
Sounds like something Steve Jobs observed at apple https://youtu.be/l4dCJJFuMsE?si=QOCBUqcUPWu8AsAX
Link with spyware removed: https://youtu.be/l4dCJJFuMsE
I feel as if youtube itself might still contain some tracking.
People recommend invidious and piped but they are starting to have issues.
This video is archived on wayback machine (archive.org) and I feel as if that might be the best way to watch this video right now.
https://web.archive.org/web/20260207024055/https://www.youtu...
Edit: I have also uploaded it on preservetube when I got interested in what are some wayback machine alternatives (given that wayback machine suggests to not upload videos given storage constraints unless absolutely necessary)
So I found preservetube which is sort of intended for this and uploaded this video on it
https://preservetube.com/watch?v=l4dCJJFuMsE
Spyware?
3 replies →
I worked with some of the folks from there, and honestly you make it sounds like tech debt is inevitable consequence haunting projects from year one.
I disagree, I think the folks just did a sloppy job of "let's bungee strap it all together" for speed, instead of more serious planning and architecturing. They self-inflicted the tech debt on them and got drowned in the debt interest super fast.
There's someone out there who built the scalable version of Heroku at another garage startup. But we never heard of them because Heroku beat them to market.
Sure, but that bungee strapped slop got them pretty damn far
My point is that they would have gotten much farther. I'm not talking about architecture astronautics, just a little more thinking-ahead.
As far as the Salesforce acquisition goes, I'd be curious to see who made the decision to put Heroku into maintenance only mode.
I worked for a different part of Salesforce. I don't really feel like Salesforce did a ton of meddling in any of its bigger acquisitions other than maybe Tableau. I think the biggest missed opportunity was potentially creating a more unified experience between all of its subsidiaries. Though, it's hard to call that a failure since they're making tons of money.
It could be a case of post-founder leadership seeing that there's not a lot of room for growth and giving up. That happens a lot in the tech industry.