Comment by Karrot_Kream
15 hours ago
You're thinking about this at the wrong level. This org has made the decision to be AI-native. They aren't A/B testing being AI-native vs not being AI-native. That's the kind of thing "the market" does; if a company's markup is higher than another's it can attract more investment or its equity can trade for higher pricing on public markets.
I don't think it makes sense to A/B test "AI-native" vs other regimes of development unless you are a large, decentralized company like an F5 company or a Big Tech. How will a team of 25 (the size of the company if you read the thread) have enough output to even A/B test the difference?
It was the article that used metrics related to engineering velocity as evidence of success. It makes sense to ask how this is impacting their business metrics. The company is not in the business of shipping more code. I don’t see how asking how this ties back to core business results is thinking about it at the wrong level.
> It was the article that used metrics related to engineering velocity as evidence of success
Success for what? The article opened with: "Last week Will Larson (@Lethain ) asked how leaders are evaluating engineering managers for AI skills. My reply , a list of questions to think about, got a lot of bookmarks. More broadly, I've been thinking about how to build an AI-native engineering org for a while. Sharing some lessons from the last nine months of building at FurtherAI."
The opener is "how to build an AI-native engineering org" and the context is a company of roughly 22 engineers (from a reply.) Nothing here says "evidence of success"
> I don’t see how asking how this ties back to core business results is thinking about it at the wrong level.
In an company of 22 engineers, how would you make these measurements? Would you split your company into a velocity focused group and a quality focused group and make them compete? How? What would a customer motion look like? What would these groups share? This makes no sense. This is like asking a small company to A/B test teams that used a computer vs didn't use a computer.
A larger, more mature organization has the luxury of splitting business units into AI-native vs not-AI-native workflows and comparing COGS/business objectives against each other. I know of very large tech consulting services who did exactly this around the beginning of the cloud era to figure out which aspects of their business they wanted to place on clouds and which aspects they kept on-prem, and even then that was messy because part of the value of on-prem is the scale benefit you get by owning and operating your own DC. But no 22 person engineering team I saw A/B tested this out. Companies were founded either with a cloud-first philosophy or an on-prem philosophy.
If you can’t tell value at a 25 (or 10 or 5) person org, you will not be able to at a larger one.
If you weren’t making intentional decisions about cloud vs on prem… why? I worked with many small businesses that did.
This feels like a value-divorced outsourced engineering mindset. You have the power to understand if things are good ideas or working!
Idk, seems like it’s important to A/B test being “AI native” because it could go badly.
3 replies →