Comment by kubb
2 days ago
I’d wager 90% time spent at Google is fighting incidental organizational complexity, which is virtually unlimited.
2 days ago
I’d wager 90% time spent at Google is fighting incidental organizational complexity, which is virtually unlimited.
The phrase thrown around was “collaboration headwind”, the idea was if project success depends on 1 person with a 95% chance of success, project success also had a 95% chance. But if 10 people each need to succeed at a 95% chance, suddenly the project success likelihood becomes 60%…
In reality, lazy domain owners layered on processes, meetings, documents, and multiple approvals until it took 6 months to change the text on a button, ugh
Another side of this coin is that the expected payoff from a project depends on how many unrelated projects your organization is engaging in, which is deeply counterintuitive to most people.
Every project carries with it three possibilities: that of success, where the company makes money, that of failure, where the company does not, and that of a "critical failure", where the project goes so wrong that it results in a major lawsuit, regulatory fine or PR disaster that costs the company more than the project was ever expected to make.
If you're a startup, the worst that can happen to your company is the value going to 0. From an investor's perspective, there's not much of a difference between burning all the money ($10m) and not finding product-market-fit (normal failure), or your company getting sued for $3b and going bankrupt (critical failure). The result is the same, the investment is lost. For a large corporation, a $3b lawsuit is far more costly than sinking $10m into a failed project.
You can trade off these three possibilities against each other. Maybe forcing each release through an arduous checklist of legal review or internationalization and accessibility testing decreases success rates by 10%, but moves the "critical failure rate" from 1% to 0.5%. From a startup's perspective, this is a bad tradeoff, but if you're a barely-profitable R&D project at big co, the checklist is the right call to make.
This problem is independent from all the other causes to which bureaucracy is usually attributed, like the number of layers of management, internal culture, or "organizational scar tissue." Just from a legal and brand safety perspective, the bigger your org, the more bureaucracy makes sense, no matter how efficient you can get your org to be.
"Nothing breeds conservatism like having something to conserve" was a recurring line from a founder-type, complaining individuals had become less dramatically innovative as the "startup" succeeded and prospered. Though they had colleagues who thought the line misapplied. IIRC, fuzzily.
This is really insightful, I hadn’t considered this dynamic before.
I wonder if a related intensifier is that as a company grows larger it tends to follow the letter of the law rather than the spirit of the law, which results in less buffer space between the behavior and the law (and hence higher lawsuit risk).
I like the ideas here but I think the actual chance of getting sued for $3b is so small as to be negligible in the context of costs. It's also questionable how much the additional process/overhead moves the needle on that chance. Larger companies also have various "shields" against these sorts of lawsuits. E.g. they lobby politicians, they employ lawyers, they have legal and IP protection.
Just like anything else in life you want to look at the present value and then get insurance for huge risks.
That said agree that a startup can take more risk but I don't think that is the major factor explaining why larger companies tend to be process heavy and slower.
3 replies →
This is an excellent explanation of the culture of BigTech.
> lazy domain owners
Interesting. As a consultant for the most of the last 25 years, my experience is the domain owners are typically invested and have strong opinions on the things that impact their jobs.
Executive leadership, on the other hand, doesn't want to actually know the issues and eyes glaze over as they look at their watches because they have a tee time.
There's a culture of "I won't approve this unless it does something for me" at Google. So now changing the text on a button comes with 2 minor refactors, 10 obvious-but-ignored bugfixes, and 5 experiments that it is actually better.
While this sounds pretty frustrating, there is at least a small upside: at least you get to the obvious-but-ignored bugfixes.
Most smaller places don’t have the bandwidth and many larger ones don’t have the desire.
I’m not sure if that makes up for bugs potentially introduced in the refactors, though.
10 replies →
Coordination Headwind: https://komoroske.com/slime-mold/
Thanks, this may be the name they used it’s been a little while
The old "If you want to go fast, go alone. If you want to go far, go together."
Also why the optimal business strategy seems to be to go as far as you can alone and then bring on other people when you're running out of steam.
Well, good management/tech leadership is about making sure that the risks coming from individual failure points (10 people in your example) are recognized and mitigated, and that the individuals involved can flag risks and conflicts early enough so that the overall project success probability does not go down as you describe...
The assumptions in that math are wrong anyway. Once you depend on 10 people, the chance that they each achieve "95% successful execution" is 0.
This is only partially down to the impossibility of having every staff member on a project be A++ players.
There is coordination RISK not just coordination overhead. Think planning a 2 week trip with your spouse with multiple planes/trains/hotels, museum/exhibit ticket bookings, meal reservations, etc. Inevitably something gets misunderstood/miscommunicated between the two of you and therefore mis-implemented.
Now add more communication nodes to the graph and watch the error rate explode.
That's what the math is reflecting. Project succeeds if all of 10 people does their job well. Each person has a 95% chance of succeeding. 0.95^10 ~= 60%, and so the chance that all 10 people do their job successfully is ~60%.
Those jobs also include things like management and product design, and so the coordination risk is reflected in the 5% chance that the manager drops the ball on communication. (As a manager, I suspect that chance is significantly more than 5% and that's why overall success rates are even lower.)
2 replies →
And when you’re at a smaller company 90% of your time is fighting societal complexity, limit of which also approaches infinity, but at a steeper angle.
No greater Scott’s man can tell you that the reality is surprisingly complex, and sometimes you have resources to organize and fight them, and sometimes you use those resources wiser than the other group of people, and can share the lessons. Sometimes, you just have no idea if your lesson is even useful. Let’s judge the story on its merits and learn what we can from it.
Look, I've never had to design, build or maintain systems at the scale of a FAANG, but that doesn't mean I haven't been involved in pretty complicated systems (e.g., 5000 different pricing and subsidy rules for 5000 different corporate clients with individually negotiated hardware subsidies (changing all the time) and service plans, commission structure, plus logistics, which involves not only shipping but shipping to specific departments for configuration before the device goes to the employee, etc.
Arbitrarily, 95% of the time the issues were people problems, not technical ones.
You are right but it misses the flavor of the problem. I was a consultant in infosec to F500s for many years. Often solving a problem involves simply knowing the right person that has already thought about it or toiled on the problem or a similar one. But when there are 100,000 engineers it becomes an order of magnitude (or two!) more difficult and that puts forth unique challenges. You can still call them “people problems” and they often may be. However if you try to solve them the same way you might solve it at a smaller engineering org you will get and be nowhere and be poorer for the time you spent trying it. Ask me how I know lol. The technical problems are also like that. Almost everything has an analog or similar thing to what you are probably familiar with but it is scaled out, has a lot of unfamiliar edges and is often just different enough that you have to adjust your reasoning model. Things you can just do at even a typical f500 you can’t just do at big tech scale. Anyway, you are directionally correct and many of these wounds are self inflicted. But running a company like Google or Facebook is ridiculously hard and there are no easy answers, we just do our best.
1 reply →
I have a similar perspective. I think after a few years, it's the people things that have always been the hardest part of the job. That's probably why in the interviews, we always say things like: communication is key, culture fit, etc.
On the other hand, the good part of the job is solving complex technical problem with a team.
Equally important is the amount of time they save because of available abstractions to use like infra, tooling etc
I think I understand what you are saying, and I agree.
Google has all sorts of great infra projects that simplify complex domains. Those are solved problems now, so nobody notices it.
The existence of incidental complexity isn’t evidence that the counter factual is less complexity.