While it is still a stretch, the title should be "Google's promo committee is killing Kubernetes", since no other FAANG-equivalent is using or contributing to Kubernetes in a meaningful way.
The core point:
> It's too indirect, fixing a bug in kube-apiserver might retain a GCP customer or avoid a costly Apple services outage, but can you put a dollar value on that? How much is CI stability worth? Or community happiness?
While correct, the same is true for most projects at these companies. Very little work that a single engineer does can be assigned a provable revenue number. How does someone working on an internal build tool get promoted? Or a new training model? A large-scale refactor of a legacy codebase? (All of these are examples of very senior promos at my own FAANG company).
Kubernetes is no different. An engineer has to show how the work they did first and foremost aligns with their team's goals. If it doesn't, then well they shouldn't have been working on it in the first place. While working on open-source projects might be tolerated, even on company time, it makes sense that it won't put you on a path to promotion unless there is direct benefit to the company from it.
> While it is still a stretch, the title should be "Google's promo committee is killing Kubernetes", since no other FAANG-equivalent is using or contributing to Kubernetes in a meaningful way.
> Very little work that a single engineer does can be assigned a provable revenue number. How does someone working on an internal build tool get promoted?
While I generally agree, I've made changes in our internal build tools that I can easily put a number on, although it's not a revenue number. Make build process 10s faster, 1000 engineers run this 10x a day, you're saving more than an engineer day per day of time. That's a definite impact right there, assuming your management is on board with accepting it :)
Apple is #32 in contributor commits[1]. Between SAP and Glassdoor, with ~7000 commits. You need to start adding zeroes and then some to get to Google's number of commits.
We shouldn't just measure in commits. I'd be interested to hear the case made. Where has Apple lead in Kubernetes? What have been their major initiatives? That'd be interesting to hear.
> Make build process 10s faster, 1000 engineers run this 10x a day, you're saving more than an engineer day per day of time. That's a definite impact right there
Agreed, and you can measure and include similar statements regarding your Kubernetes contributions. However, if what you have to show for it is making the life of a hundred random startups easier, well why exactly should a Google promo committee care? Unless it is the specific goal of your business unit to make this impact, then the work is meaningful again.
Make build process 10s faster, 1000 engineers run this 10x a day
Depends where you are starting from. Taking the build from 100 to 90 seconds probably won't make a difference since people will be used to doing something else while waiting for the build to finish in the background. taking it from 10.1 to 0.1 seconds will make a huge difference since you can change your entire development process when builds are 'instant' and you can get constant feedback on any changes you make. Of course the second one is a hell of a lot harder.
> Make build process 10s faster, 1000 engineers run this 10x a day, you're saving more than an engineer day per day of time.
I upgraded my home internet to one 2x as fast as the old service, so that I would spend only half as much time online, but it didn't work. Speed up the build process and the build will just get more complicated and the test suite slower, until it once again slows to the point of being barely tolerable. ISTM that part of Go's early popularity was from its fast builds.
I think the tweet is a bit too much. I worked for GCP for several years before. Impact in promo is defined within the same group and aligned with the goals of the group. If improving the stability of k8s is among the goals, then no problem at all.
OSS projects are not built on top of anyone's mercy. For infra projects like k8s, companies just contribute what they need and Google is no exception.
> I worked for GCP for several years before. Impact in promo is defined within the same group
Did you personally go through promo from L6 to 7 or L7 to 8 since that's what the OP thread is about?(or have seen the process from sufficiently close).
I can see how your description of impact & promo would work when there's enough people inside the group to compare from (i.e other L4s, L5s, L6s), but when going L7 & L8 there might not be enough people in the group. Plus the folks at that level would be the one defining the goals, so it seems logical that they can't get promoted on that criteria.
Here it’s focused on k8s, but it’s not out of line if we generically apply it to AMZN. How many services look cool, have a great re:Invent kick-off, and then go KTLO a year later? I’m going to lay that at the feet of the AMZN promo process, which prioritizes creating something new over real and needed maintenance.
I’m going to further reduce this to lazy management. It’s easier to point to splashy announcements to show impact than it is to put in the work and help your direct reports quantity unglamorous maintenance work.
Not sure how long you've been at or spent at Amazon, but the promotion process doesn't prioritize building new things at all. It's true that people think complexity = build new things, but promotion criteria specifically encourage people to stick to simple things and build on top of what exists rather than invent something new for promotion's sake.
Even if you can prove a revenue number, it doesn't matter if they don't want to promote you. I've seen it explicitly stated at more than one company, that revenue impact is irrelevant and that you need to show a "large enough design and execution" to be promoted as a reason to just not give it out.
Not only that, but the "promo packet" and "promo committee" is fairly unique to Google. Other companies do things very differently, e.g. FB's twice-yearly (now yearly) manager-driven PSC+"calibration" cycle, which has fairly different incentives than Google-style committees of unknown engineering peers with arbitrary periodization.
Google's system is well known (at least, in my circle of colleagues!) to not promote healthy long-term maintainership, although it does incentivize point-in-time brilliance and solving deep complexity.
Google promos everywhere up to a certain level are in-org (IE not unknown engineering peers). They are done alongside rating calibrations.
Google promos everywhere up to an even higher level are basically in-PA.
For cloud, they are in-org to even a higher level. So the stuff talked about in this tweet would have been evaluated not just by cloud engineers, but almost certainly fairly local cloud engineers.
My experiencing having been on promo committees for google for 15 years is that for every case in my org i see of google incentivizing the wrong thing (or disincentivizing the right thing), i see probably 5-6 cases of it doing the right thing.
I see a lot more cases that are like:
1. Person builds new shiny thing for ... questionable reasons.
2. They make a mess of the world by not making it easy to migrate people to it and get rid of the old thing
3. Things are in a bad half-state, where both things have to be supported and the new thing does not provide obvious higher value.
4. Google doesn't promote them, they get pissed off about it and post on twitter about how Google didn't care about product excellence or something.
Than cases like:
1. Person toils away on the right thing forever
2. Person tries to get promoted for doing the right thing and having impact
3. Google turns them down
4. They get pissed off and post on twitter about how Google didn't care about about product excellence or something.
Externally, of course, people can't distinguish these two cases because they look the same (angry people on twitter).
I have been promoted from SWE III to Senior Director at Google, working only on open source until i was an L7.
> While working on open-source projects might be tolerated, even on company time, it makes sense that it won't put you on a path to promotion unless there is direct benefit to the company from it.
The thing is - this is all theater. It's mostly made-up storytelling. An individual engineer's self-review doesn't actually speak to their direct impact on the company's success.
It's fallacy to act like you can map commits to dollars. But there's a lot of money to be made by HNers for trying, so I'm sure they'll act like they can do the impossible.
Define "killing kubernetes"? It's still pretty successful and the adoption hasn't slowed in any way I can measure. I promise you that some site you used TODAY is running, at least part of it, on Kubernetes.
Google employees regularly get promoted, at all levels, based on their OSS work - Kubernetes and other projects. We have dozens of people who work on Kubernetes, in one area or another, at varying degrees of depth. Is that "killing" the project?
Of course it is never ENOUGH. I'd happily consume hundreds more people. :)
> While correct, the same is true for most projects at these companies. Very little work that a single engineer does can be assigned a provable revenue number. How does someone working on an internal build tool get promoted? Or a new training model? A large-scale refactor of a legacy codebase? (All of these are examples of very senior promos at my own FAANG company).
Which FAANG though? Google is usually singled out for this criticism more than the rest of FAANG combined. It’s possible that your experience at a non-Google FAANG might not be relevant for Google specifically.
(Note: I wouldn’t know first hand, I’ve never worked at a FAANG).
This thread is ridiculous. At FAANG, and everywhere really, promos are decided based on impact and influence. FAANG are just more likely to be working on open source projects than employees at other companies.
All impact is difficult to quantify, not just contributions to OSS. The only easily quantifiable achievements for an engineer are delivered projects that directly generate revenue. In the teams I’ve worked in, I’d say that covers perhaps a third of projects.
Let's pretend Linus Torvalds worked for Google, would he be able to get promoted for creating and maintaining Linux? As far as the bottom line goes it brings in zero dollars and costs the company his total compensation or more to maintain.
How would Linus quantify his revenue impact in his promotion packet and compete against another hypothetical engineer that improved ad targeting by 0.0X% and can directly measure an increase of revenue for the comapny in the millions of dollars?
I don't know why people here think you have to quantify your revenue impact in your promo packet. This is literally not a thing, and any number you put in there will be ignored. No one up the engineering chain cares about dollar figures when it comes to evaluating ICs.
As for the Linus question, plenty of engineers with similar profiles have worked for Google, and they don't do so as junior developers their entire career. Junio Hamano, who has been maintaining Git while at Google for the last 15 years, is the perfect example. Sundar Pichai started his career working on products that had zero or negligible revenue (Chrome, ChromeOS, Google Drive) and is now the CEO.
He wouldn't quantify his direct revenue impact, much as engineers who work on Borg and other infra don't. Or, he might point to a systems performance working group he helped chair and show that that working group improved performance of some syscall by n% (or added a new one that was n% faster) which when applied across all compute at Google has a wide revenue impact. (Or really he'd be at a leadership level where he'd need to show that the working group he lead did that a dozen times, but you get the idea).
When you have difficulty quantifying, you start qualifying, and that leads to some options for quantification as a second order effect.
"Lead the development and maintenance of custom OS underpinning our entire enterprise's technical stack. Worked with stakeholders across every domain within engineering to improve stability, performance, and feature set that enabled every part of the engineering org to better deliver" - "Examples of revenue visible effect are the debugging and patching of (performance improvement to the underlying OS) that reduced CPU utilization across the org, enabling a 3% reduction in fleet costs, allowing the company to save an estimated (estimate 3% reduction in fleet cost from before it was introduced)"
I can't speak to Google, but a good manager can work with their team to really highlight their achievements in a way the promo committee can understand. Admittedly, it still means learning and playing the system, but that's true regardless of the system you use.
Remember, too, that any impact on engineer time -also- has a direct provable reduction in cost. "Took 10 seconds off (process people wait on, build, test, whatever), saving an estimated (###) of engineering hours per year based on the number of teams using (part of process you affected)" should still be evaluated equivalent to revenue. If it isn't, you call it out, "leading to a reduction of engineering operating costs and saving the company roughly (estimate number of man hours * average cost of man hours)". So something that starts "saves effort" can be translated to "saves time" which can be translated to "saves money".
Also it's Google, so we should be looking to promote operating system authors who aren't white or male anyway.
Expecting hate for this, but I've been on the receiving end of company notifications (not at Google) where it was stated explicitly that the newly-opened distinguished engineer role was exclusively available to not-men.
(I'm generally-supportive of DEI efforts, but at sane companies those efforts wouldn't block qualified people from being promoted to their proper roles. Among the FAANGs, I have a lot more confidence in Facebook and Amazon's ability to operate sanely than I do Google. Prior evidence and all.)
Brendan Burns, the Kubernetes co-founder who currently works at Microsoft Azure as a Corporate Vice President disagrees the "killing Kubernetes" part.[1][2]
But yes, Big Tech's promo committee's short-sighted interests do NOT align well with multi-vendor sponsored open source projects' the long term prosperity. And it's no secrets that open source contributors are overworked. And it's not only affect K8s, but it affects across the board.[3]
> going L6->7 at Google is worth ~200k/year, 7->8 is ~400. Similar patterns at other places. People have kids, mortgages, student loans
Honestly this is a bit ridiculous. If making 500k as an L6 isn't good enough for you maybe you should be working on projects that provably bring your company revenue?
Exactly. If you are an L7 and making an average L7 salary, nobody should have to squint and tie themselves in knots to figure out the connection between your contribution and your employer’s revenue. You are a Ferrari that is purchased new each year.
There’s a place where you can work on things that don’t generate revenue but are morally/technically interesting: it’s called “academia”.
What is a professor's job other than revenue? Instead of selling a product, they have to beg for money instead. Alternatively, they sell their research. Publish or perish. Raise enough money for the school or you're out.
The Bay Area is unaffordable because most cities haven't allowed anywhere near enough housing to be built by land owners over the last five decades: https://techcrunch.com/2014/04/14/sf-housing/.
Return building rights to land owners and the problem will be solved. The insane thing is the way coalitions of existing owners conspire to reduce supply and thus increase the values of their assets, and the rest of us just go, "Oh, okay, that's cool, that you're conspiring to embeggar the rest of us."
Indeed. If lots of people with money want to live in a place and building more housing is illegal the price of existing housing will go up a lot. If you want the problem to go away you can either build more housing or make it so the people with money don't want to live there.
The problem is not the tech people, it is the boomers who lived here before us. They have manipulated the system, voted down every attempt to build more housing, and leveraged their good luck to engage in rent seeking and siphon off half of the "ridiculous tech salaries."
My old landlords had average non-tech jobs in the 80s, bought a house for under 100k. At some point they used it to buy a second house, and another, and another. Now they own 30+ houses and rent each for 4-7k/mo.
Old next door neighbor's house was owned by her dad, he was a public works employee and owns 6 houses in the bay area now.
Neighbors on the other side were an older couple with adult kids and multiple rental properties in Fremont. They bought a house for their adult son outright before we moved.
Market forces are moving in your favor. Most European companies have completely saturated their target markets for outsourcing (Barcelona, Estonia, Czech Republic) and Russia & Ukraine are suddenly unavailable options due to a war. Also the local markets have much more demand for engineers than are available to fill it.
There has been a large push in the last two years to recruit Americans and Canadians to fill their unmet needs and salaries have been climbing rapidly. Salaries for new job postings in Berlin are about where they were at in New York 3-4 years ago which is EXCELLENT for that market (seriously, two years ago salaries were maybe half of what they are now).
If you're in Europe and you haven't been interviewing you should look around and maybe even use a recruiter. My own recruiter has been feeding me opportunities across Europe with pretty much equal money to what I'm getting here in the States.
It's likely that if you haven't changed jobs in the last two years that you're leaving significant amounts of money on the table. That said, I know that chasing after money isn't fashionable or socially acceptable to most Europeans...but the new hires are going to get it -- why shouldn't you?
There's over 4 million (and counting) Ukrainian refugees in Europe right now, and quite a few of those people are software engineers. Then there's over 100k (and counting) Russians that rushed to leave the country before the new iron curtain goes down - and I wouldn't be surprised if most of those are software engineers.
Google created Kubernetes so potential competitors are dragged down with immense technical debt, of course they’re not going to award internal advancement of it.
Having participated in full on rearchitectures to Kubernetes several times at this point I can say that it wasn't justified a single time, even at the 10^4 microservice scale. Now having participated in several fairly-effortless rollouts of Nomad and arrived at a better place, it's funny to watch the rest of the industry cargo cult.
I actually don't believe that either of these solutions are going to be a long-term settled end state (though Nomad with Firecracker, or really any other Firecracker-centric solution that isn't k8s will have some legs) the industry falls on, but I agree wholeheartedly that Kubernetes is purely a distraction having seen the pain and the gnashing of teeth.
Every painless Kubernetes story that I've seen is at a scale where Kubernetes wasn't even necessary/justifiable and another solution would have been even simpler. But at least it's good for the resume.
And Helm charts are are akin K8s' own borgcfg. We're doomed to repeat our mistakes it seems.
To counter your argument. I did one full migration from AWS to GCP Kubernetes (GKE). The project was a huge success, simplifying our stack, deployments, logging, etc, etc.
We reduced our costs by 2/3rds, saving millions of years. Teams have been able to move onto feature work instead of maintaining custom deployment tooling. The Ops team 1/2 the size that it was before but is able to handle twice as many customers.
This is not really just the problem in Google. We might as well claim that promotion processes in big companies promote promotion-oriented work (yes, I intentionally use the word promotion this many times). The result is title inflation, lots of good employees leaving, and superfluous yet mediocre projects. I wouldn't say only unqualified people get promoted, though. Big companies have very different dynamics than small ones. Some people are just good at navigating big companies and are capable of aligning multiple organizations, for good or for bad. Big companies need such talent as well. As for individuals, the reward of getting promoted is not worth the effort, at least statistically. A much better alternative, is focusing on solving truly interesting and meaning problems in a blow-out company. The financial return and title bumps will follow naturally.
That's the most ridiculous bubble valley thing I read in a while, coming from a non-k8s using non-FAANG-employed European.
Which isn't meant dismissive of the problem at hand, I can totally see how it's happening and how it might be bad for the people involved in the project.
To phrase it differently - if the project is dying because it hinges on people working on a promotion that gives them a /bigger paycheck increase than my yearly income/ - well, good riddance. I'm staying on my side of the fence and continue to not buy a new Tesla every year.
> if the project is dying because it hinges on people working on a promotion that gives them a /bigger paycheck increase than my yearly income/ - well, good riddance. I'm staying on my side of the fence and continue to not buy a new Tesla every year.
An understandable position but this is simply how competitive environments work.
It’s a lot like the college application mania. There’s tons of gifted high school students who clearly can succeed without needing to volunteer in a foreign country every summer while playing oboe and competing in lacrosse at the state level.
But seats at Harvard are competitive, and those same talented and well-to-do high school seniors often only have college admissions as their ruler to measure their growth and self-worth. So they compete on every angle to try and get into Harvard rather than merely Dartmouth or heaven forbid BU.
It’s reasonable to think working adults should have measures of personal growth that don’t rely on climbing the corporate ladder, but conversely all those who *do* obsess with climbing the ladder will naturally be clustered around places of wealth/prestige, because that’s what they are optimizing for. Much like the Harvard application pool, you’ll find these types doing what it takes to stand out and land that 700k/yr L7 position at Google.
So I guess the answer is that the most stable and secure position for an open source project to be in, is one that does not rely on development being coupled to big tech. I imagine most open source folks could have told you that anyways :)
Kubernetes' biggest problem by far is its mess of accumulated technical debt. FAANG promo committees very likely undervalue FLOSS contributions, but this hits all of FLOSS pretty much equally; it hardly explains why k8s specifically would be in so much trouble.
I would go as far as to say it's not just limited to FLOSS contributors. In my experience at Google, not every service is profitable or brings in revenue, so it can be hard to justify "impact" on those teams. I certainly remember beating my head on how to spin my contributions as having "impact."
Kubernetes doesn't get patches from hyperscale companies because kubernetes is not really hyperscale software. If I recall correctly (probably don't), clusters can handle tens of thousands of container hosts. That's not manageable at large scale unless you're selling people individual kubernetes instances directly (which, to be fair, google and amazon are).
In my experience the thing that kills open source at big companies is the divergence of needs: The company needs massive scale software, but the community wants ease-of-use and features. Most open source models are a big drag on productivity (you can't just refactor all the callsites to do a migration!), and it's hard to change directions when the company's need does (since the other contributors will likely want the original software).
The tailscale people have been arguing this as well (with more data and better reasoning), but fundamentally you need different software when you're a scrappy startup or a medium size company or a giant behemoth.
What's preventing k8s from being useful in these larger scale deployments? GP was not very clear about that, they just said that it's not scaling enough.
Maybe I’m misunderstanding but why would you want to run one giant kubernetes cluster at hyper scale? I would think at hyper scale you would be running micro services with one kubernetes cluster per service per datacenter/region.
At hyper scale, efficiency matters a little. Small gains multiplied by big numbers become big gains. The overhead of running a cluster is not small, but more importantly - the overhead of managing nodes between services is real.
No, you would not want to run a cluster per service; Kubernetes is somewhat multitenant. You'd get better efficiency from running one cluster per datacenter, with all those teams in their own namespaces. Kubernetes can stack many containers per physical host, with widely varied workloads cooperatively sharing.
You wouldn't want to run one giant cluster, but at hyper scale you're talking about running thousands or tens of thousands of kubernetes clusters. That's the part that doesn't scale well, for a couple of reasons.
The biggest one is just mechanical: with that many clusters it will be hard to move capacity between clusters, and locality gets baked into everything you do (people do try to build around this, but it's awkward).
If each service or team runs their own kubernetes that's a lot of overhead: kubernetes will need something like 6-7 machines for the cluster (I don't have production experience with kube, spitballing here), so small teams or jobs will have terrible efficiency. Big teams will have to spend a lot of operational effort to manage their fleets.
It's worth noting that at hyperscale there will be individual jobs in a datacenter that are bigger than kubernetes handles comfortably. Handling this efficiently becomes very important, it's literally billions of dollars worth of hardware.
This is just the law of inertia at play. Big projects slow down. K8s isn’t just the K8s core, it’s a massive ecosystem, and when taken as a whole it’s unbelievably vast and growing at an amazing pace. As it does, the core of it moves slower, there are more stakeholders, more arguments. Small changes make for big problems and not everyone can be happy. Patches will keep coming, major changes will become eventually impossible. The death of Kubernetes makes for great clickbait, but most of us won’t see it in our lifetimes. Not when our banks run on 60 year old code. Kubernetes is too big to fail at this point.
As far as FAANG promo committees go, let them value what they value. Kubernetes is a direct revenue driver for many companies; it’s health is tied to billions of dollars in investments. Just because one cohort of contributors age out, cash out, or fall in love with someone doesn’t mean there is no one to take their place. The new people won’t do it the same way and that is okay, even great. I’m grateful for the vision and effort that have make K8s the platform it is, and if I can translate that into a contribution in the future I will.
Finally, I’ll say that people who really love working on the project may not get support to work on it full time, but then may find themselves able to retire sooner than many and have ample opportunity to contribute.
I basically can't stand the comment section of Kubernetes threads. Lots of big teams are getting big things done with k8s, and you would not know it from here.
Not sure why this says FAANG because it's talking about Google.
Calibration, promo committees, feedback and the like are intended to create equivalent expectations on impact across different orgs. The dirty little secret however is that at best it has limited success.
The best advice I can give anyone in such an organization is to be liked by your manager and their manager. If that's true, good things will tend to happen. If it's not, good things will be a lot less frequent.
Put another way: you can take the exact same set of objective facts and use them to say a person did a good job or a bad job. There's a popular meme about feedback at Google that goes something like:
> This project would've failed without this person. It failed anyway but it definitely would've failed without them.
The difference ultimately boils down to whether or not they like you.
Here are a few Google-specific tidbits worth knowing:
1. Ratings are fit to a curve across a sufficiently large pool, typically at the director level and usually over 100-150+ people. This means there will be a percentage range of people who get Meets All, Exceeds Expectations, Greatly Exceeds, etc. This is intended to stop ratings inflation;
2. A consequenc eof (1) is that ultimately you are competing with people in your org for those better ratings. This can create some perverse incentives and a toxic environment;
3. It is almost always better to let something blow up and come and fix it rather than preventing that from ever happening. The first will get you a lot of recognition. The latter will get you almost none;
4. Promos at Google are stack-ranked. Each committee gets 10-15 packets that will be for a particular level. The committee will rank those packets. After that the promotion target will come into play. This is set by management and was allegedly cut as a cost-saving measure when Ruth Porat came on board. If it's 20% then the top 20% from that ranking process be promoted.
You will find people who serve on those committees who say this isn't how it works and they'll argue they're evaluating if someone is operating at the next level or not. This is partially true. Thes packets will be divided between promote, don't promote and on-the-bubble. The on-the-bubble group will be sufficiently large to allow for the promotion target to be met;
5. For SWEs. L5 is the "terminal" level, meaning there is an expectation of growth to that level. L3->L4 and L4->L5 once went through promo committee but now don't. Management within orgs decide this. These too have target percentages and there have been cases where the promot rate has been too "high" and orgs have been told to cut back on promotions to meet the targets;
6. There is a massive backup at L5->L6. Because of the low target percentages the impact required keeps going up and really you need your management to really push for this to happen. There are limited slots so you may be waiting eyars and again this is why them liking you matters so much. Google is full of L6s who got promoted 5-10+ years ago that would never make the grade by today's standard. For really old cases you can find archives of why there were promoted and you'll find cases like "promoted unit testing".
I say all this because the author of this thread seems to fundamentally misunderstand how this process works.
The promo stack rank goes the other way. You decide on the promote vs no promote part first and then rank based on confidence. I’ve absolutely been in sessions where 100% of people were promoted. You don’t rank and then apply a bar based on a target promo rate.
There are certain classes of work that are completely ignored by most companies such as build, automation, developer tooling and infrastructure cost cutting. These are not flashy features that you can show off to customers but they can pay in dividends year over year.
(Personally, I have saved millions a year in infrastructure costs and my promotions/compensation were not close to what a sales person would get if they brought in millions of recurring revenue)
K8s is a great example of what murders open source: monolithic platforms that exist unto themselves, but have some seemingly benign model allowing arbitrary "plugins" to pretend it's not really a monolith in sheep's clothing. It looks like open source and has the right license, but it ends up just mirroring corporate interests rather than those of the Commons that the open source movement was created to support.
If you're going to have monolithic platforms anyway (and you will, because designing for modularity is hard, whereas hacking together some random monolithic code is easier) they should at least be open source rather than proprietary.
Actually I disagree. Any time there's an Open Source tool that does everything for free, it's extremely hard to justify paying for a proprietary product that does it much better, because companies are cheap. There's many open source projects like this that are just terrible but a company will always use them first because they're free.
The answer isn't modularity, it's composeability, which is a significant difference. A modular program requires "integration" (tight coupling of APIs/ABIs) whereas a composeable program has loose interfaces which require virtually no "integration".
Drone.io is one example; you can implement any "plugin" purely by creating a container with an initial CMD entry that reads environment variables, and that program can interface back with Drone a number of ways (STDIN/STDOUT, REST API, database). Another is any application that just reads in or spits out simple line-by-line instructions or a JSON blob. Unix pipe based programs are the penultimate example. The dumber the interface, the easier it is to compose.
This is the case with most major products from most major tech companies. There is a reason you don't (often) see steady incremental progress on products, but rather big flashy releases that never quite get their bugs cleaned up unless they are wildly* successful.
* Where wildly is defined as a success level which would leave any rational founder very wealthy, and very content.
Hopefully Kubernetes does die. Most companies write shit slow software and then think they're big smart boys because they have to buy $20,000/month of servers to make it not be unusable. In many shops it could and should be replaced by a handful number of efficient servers that do not suck shit.
Oh man, another rant that boils down to "big evil corporations are KILLING open source software by not supporting it in the specific ways and in the specific amounts that I want."
This seems like a strictly Google only problem. And by extension this is the problem with promotion committees that are removed from day to day of the candidate getting promoted. People are ridiculously bad at measuring growth based on some obscure company wide criteria - which is abstract at best and useless at worst at a company large enough as Google.
I'm at AWS and the promotion process here is largely contained within the candidate's org which is at best two levels higher but in the same business. So if your team's business relies on contributions to FOSS, its impact is measurable and leaders and the 'committee' can easily tell you if your contributions are at the next level.
Go figure, contributions to a multi-vendor consortium that just barely makes a marginal profit for an already marginal business unit (GCP) are not as highly valued as things that can be directly connected to revenues. This is not surprising in the least.
Perhaps obviously, Mr. Kantrowitz hasn't quit, or been poached by a competitor, so I would say Google's (cynical) comp and promo policy is working out just great for them. Whatever personal interest keeps Mr. Kantrowitz going is working out for them!
While it is still a stretch, the title should be "Google's promo committee is killing Kubernetes", since no other FAANG-equivalent is using or contributing to Kubernetes in a meaningful way.
The core point:
> It's too indirect, fixing a bug in kube-apiserver might retain a GCP customer or avoid a costly Apple services outage, but can you put a dollar value on that? How much is CI stability worth? Or community happiness?
While correct, the same is true for most projects at these companies. Very little work that a single engineer does can be assigned a provable revenue number. How does someone working on an internal build tool get promoted? Or a new training model? A large-scale refactor of a legacy codebase? (All of these are examples of very senior promos at my own FAANG company).
Kubernetes is no different. An engineer has to show how the work they did first and foremost aligns with their team's goals. If it doesn't, then well they shouldn't have been working on it in the first place. While working on open-source projects might be tolerated, even on company time, it makes sense that it won't put you on a path to promotion unless there is direct benefit to the company from it.
> While it is still a stretch, the title should be "Google's promo committee is killing Kubernetes", since no other FAANG-equivalent is using or contributing to Kubernetes in a meaningful way.
Apple puts Kubernetes on https://opensource.apple.com as a project they "lead and contribute to"…
> Very little work that a single engineer does can be assigned a provable revenue number. How does someone working on an internal build tool get promoted?
While I generally agree, I've made changes in our internal build tools that I can easily put a number on, although it's not a revenue number. Make build process 10s faster, 1000 engineers run this 10x a day, you're saving more than an engineer day per day of time. That's a definite impact right there, assuming your management is on board with accepting it :)
Apple is #32 in contributor commits[1]. Between SAP and Glassdoor, with ~7000 commits. You need to start adding zeroes and then some to get to Google's number of commits.
We shouldn't just measure in commits. I'd be interested to hear the case made. Where has Apple lead in Kubernetes? What have been their major initiatives? That'd be interesting to hear.
[1] https://k8s.devstats.cncf.io/d/9/companies-table?orgId=1
6 replies →
> Make build process 10s faster, 1000 engineers run this 10x a day, you're saving more than an engineer day per day of time. That's a definite impact right there
Agreed, and you can measure and include similar statements regarding your Kubernetes contributions. However, if what you have to show for it is making the life of a hundred random startups easier, well why exactly should a Google promo committee care? Unless it is the specific goal of your business unit to make this impact, then the work is meaningful again.
6 replies →
Make build process 10s faster, 1000 engineers run this 10x a day
Depends where you are starting from. Taking the build from 100 to 90 seconds probably won't make a difference since people will be used to doing something else while waiting for the build to finish in the background. taking it from 10.1 to 0.1 seconds will make a huge difference since you can change your entire development process when builds are 'instant' and you can get constant feedback on any changes you make. Of course the second one is a hell of a lot harder.
1 reply →
> Make build process 10s faster, 1000 engineers run this 10x a day, you're saving more than an engineer day per day of time.
I upgraded my home internet to one 2x as fast as the old service, so that I would spend only half as much time online, but it didn't work. Speed up the build process and the build will just get more complicated and the test suite slower, until it once again slows to the point of being barely tolerable. ISTM that part of Go's early popularity was from its fast builds.
I think the tweet is a bit too much. I worked for GCP for several years before. Impact in promo is defined within the same group and aligned with the goals of the group. If improving the stability of k8s is among the goals, then no problem at all.
OSS projects are not built on top of anyone's mercy. For infra projects like k8s, companies just contribute what they need and Google is no exception.
> I worked for GCP for several years before. Impact in promo is defined within the same group
Did you personally go through promo from L6 to 7 or L7 to 8 since that's what the OP thread is about?(or have seen the process from sufficiently close).
I can see how your description of impact & promo would work when there's enough people inside the group to compare from (i.e other L4s, L5s, L6s), but when going L7 & L8 there might not be enough people in the group. Plus the folks at that level would be the one defining the goals, so it seems logical that they can't get promoted on that criteria.
6 replies →
Here it’s focused on k8s, but it’s not out of line if we generically apply it to AMZN. How many services look cool, have a great re:Invent kick-off, and then go KTLO a year later? I’m going to lay that at the feet of the AMZN promo process, which prioritizes creating something new over real and needed maintenance.
I’m going to further reduce this to lazy management. It’s easier to point to splashy announcements to show impact than it is to put in the work and help your direct reports quantity unglamorous maintenance work.
Not sure how long you've been at or spent at Amazon, but the promotion process doesn't prioritize building new things at all. It's true that people think complexity = build new things, but promotion criteria specifically encourage people to stick to simple things and build on top of what exists rather than invent something new for promotion's sake.
6 replies →
Even if you can prove a revenue number, it doesn't matter if they don't want to promote you. I've seen it explicitly stated at more than one company, that revenue impact is irrelevant and that you need to show a "large enough design and execution" to be promoted as a reason to just not give it out.
Not only that, but the "promo packet" and "promo committee" is fairly unique to Google. Other companies do things very differently, e.g. FB's twice-yearly (now yearly) manager-driven PSC+"calibration" cycle, which has fairly different incentives than Google-style committees of unknown engineering peers with arbitrary periodization.
Google's system is well known (at least, in my circle of colleagues!) to not promote healthy long-term maintainership, although it does incentivize point-in-time brilliance and solving deep complexity.
The system you talk about is mostly dead.
Google promos everywhere up to a certain level are in-org (IE not unknown engineering peers). They are done alongside rating calibrations. Google promos everywhere up to an even higher level are basically in-PA.
For cloud, they are in-org to even a higher level. So the stuff talked about in this tweet would have been evaluated not just by cloud engineers, but almost certainly fairly local cloud engineers.
My experiencing having been on promo committees for google for 15 years is that for every case in my org i see of google incentivizing the wrong thing (or disincentivizing the right thing), i see probably 5-6 cases of it doing the right thing.
I see a lot more cases that are like:
1. Person builds new shiny thing for ... questionable reasons.
2. They make a mess of the world by not making it easy to migrate people to it and get rid of the old thing
3. Things are in a bad half-state, where both things have to be supported and the new thing does not provide obvious higher value.
4. Google doesn't promote them, they get pissed off about it and post on twitter about how Google didn't care about product excellence or something.
Than cases like:
1. Person toils away on the right thing forever
2. Person tries to get promoted for doing the right thing and having impact
3. Google turns them down
4. They get pissed off and post on twitter about how Google didn't care about about product excellence or something.
Externally, of course, people can't distinguish these two cases because they look the same (angry people on twitter).
I have been promoted from SWE III to Senior Director at Google, working only on open source until i was an L7.
2 replies →
> While working on open-source projects might be tolerated, even on company time, it makes sense that it won't put you on a path to promotion unless there is direct benefit to the company from it.
The thing is - this is all theater. It's mostly made-up storytelling. An individual engineer's self-review doesn't actually speak to their direct impact on the company's success.
It's fallacy to act like you can map commits to dollars. But there's a lot of money to be made by HNers for trying, so I'm sure they'll act like they can do the impossible.
Define "killing kubernetes"? It's still pretty successful and the adoption hasn't slowed in any way I can measure. I promise you that some site you used TODAY is running, at least part of it, on Kubernetes.
Google employees regularly get promoted, at all levels, based on their OSS work - Kubernetes and other projects. We have dozens of people who work on Kubernetes, in one area or another, at varying degrees of depth. Is that "killing" the project?
Of course it is never ENOUGH. I'd happily consume hundreds more people. :)
> (All of these are examples of very senior promos at my own FAANG company).
you have your own FAANG company? remember always that "your" company will drop you in a moment.
Azure and AWS k8s offerings aren't a meaningful usage? Asking earnestly.
> While correct, the same is true for most projects at these companies. Very little work that a single engineer does can be assigned a provable revenue number. How does someone working on an internal build tool get promoted? Or a new training model? A large-scale refactor of a legacy codebase? (All of these are examples of very senior promos at my own FAANG company).
Which FAANG though? Google is usually singled out for this criticism more than the rest of FAANG combined. It’s possible that your experience at a non-Google FAANG might not be relevant for Google specifically.
(Note: I wouldn’t know first hand, I’ve never worked at a FAANG).
This thread is ridiculous. At FAANG, and everywhere really, promos are decided based on impact and influence. FAANG are just more likely to be working on open source projects than employees at other companies.
All impact is difficult to quantify, not just contributions to OSS. The only easily quantifiable achievements for an engineer are delivered projects that directly generate revenue. In the teams I’ve worked in, I’d say that covers perhaps a third of projects.
Let's pretend Linus Torvalds worked for Google, would he be able to get promoted for creating and maintaining Linux? As far as the bottom line goes it brings in zero dollars and costs the company his total compensation or more to maintain.
How would Linus quantify his revenue impact in his promotion packet and compete against another hypothetical engineer that improved ad targeting by 0.0X% and can directly measure an increase of revenue for the comapny in the millions of dollars?
I don't know why people here think you have to quantify your revenue impact in your promo packet. This is literally not a thing, and any number you put in there will be ignored. No one up the engineering chain cares about dollar figures when it comes to evaluating ICs.
As for the Linus question, plenty of engineers with similar profiles have worked for Google, and they don't do so as junior developers their entire career. Junio Hamano, who has been maintaining Git while at Google for the last 15 years, is the perfect example. Sundar Pichai started his career working on products that had zero or negligible revenue (Chrome, ChromeOS, Google Drive) and is now the CEO.
I chaired ->6/7 promo committee for years.
Revenue would come up as a kind of impact probably less then 1/2 the time. Mostly from ppl in ads.
This whole thing is nonsense. The most Sr and influential ppl tended to come from web search and core infra, with no link to revenue in sight.
3 replies →
He wouldn't quantify his direct revenue impact, much as engineers who work on Borg and other infra don't. Or, he might point to a systems performance working group he helped chair and show that that working group improved performance of some syscall by n% (or added a new one that was n% faster) which when applied across all compute at Google has a wide revenue impact. (Or really he'd be at a leadership level where he'd need to show that the working group he lead did that a dozen times, but you get the idea).
When you have difficulty quantifying, you start qualifying, and that leads to some options for quantification as a second order effect.
"Lead the development and maintenance of custom OS underpinning our entire enterprise's technical stack. Worked with stakeholders across every domain within engineering to improve stability, performance, and feature set that enabled every part of the engineering org to better deliver" - "Examples of revenue visible effect are the debugging and patching of (performance improvement to the underlying OS) that reduced CPU utilization across the org, enabling a 3% reduction in fleet costs, allowing the company to save an estimated (estimate 3% reduction in fleet cost from before it was introduced)"
I can't speak to Google, but a good manager can work with their team to really highlight their achievements in a way the promo committee can understand. Admittedly, it still means learning and playing the system, but that's true regardless of the system you use.
Remember, too, that any impact on engineer time -also- has a direct provable reduction in cost. "Took 10 seconds off (process people wait on, build, test, whatever), saving an estimated (###) of engineering hours per year based on the number of teams using (part of process you affected)" should still be evaluated equivalent to revenue. If it isn't, you call it out, "leading to a reduction of engineering operating costs and saving the company roughly (estimate number of man hours * average cost of man hours)". So something that starts "saves effort" can be translated to "saves time" which can be translated to "saves money".
Junio has been promoted over the years at Google for building and maintaining Git.
I do not believe he has had any problem doing so. (nor is it a competitive process)
Revenue is not the only way to quantify impact.
Plenty of infrastructure teams promote plenty of people at Google and other companies despite not directly selling products.
Guido got paid to work on python
Yes, very obviously. There are L8s and L9s who have built their careers on just working with the Linux community, let alone running the whole thing.
Also it's Google, so we should be looking to promote operating system authors who aren't white or male anyway.
Expecting hate for this, but I've been on the receiving end of company notifications (not at Google) where it was stated explicitly that the newly-opened distinguished engineer role was exclusively available to not-men.
(I'm generally-supportive of DEI efforts, but at sane companies those efforts wouldn't block qualified people from being promoted to their proper roles. Among the FAANGs, I have a lot more confidence in Facebook and Amazon's ability to operate sanely than I do Google. Prior evidence and all.)
Brendan Burns, the Kubernetes co-founder who currently works at Microsoft Azure as a Corporate Vice President disagrees the "killing Kubernetes" part.[1][2]
But yes, Big Tech's promo committee's short-sighted interests do NOT align well with multi-vendor sponsored open source projects' the long term prosperity. And it's no secrets that open source contributors are overworked. And it's not only affect K8s, but it affects across the board.[3]
[1]: https://www.linkedin.com/in/brendan-burns-487aa590
[2]: https://twitter.com/brendandburns/status/1511841189686771716
[3]: https://twitter.com/kantrn/status/1511791447091003395
Well yes, what did you expect a corporate VP with skin in the game would say publicly on Twitter?
The OP then agrees that Microsoft stands out as supporting OSS more than competitors. https://twitter.com/kantrn/status/1511844921921077249
> going L6->7 at Google is worth ~200k/year, 7->8 is ~400. Similar patterns at other places. People have kids, mortgages, student loans
Honestly this is a bit ridiculous. If making 500k as an L6 isn't good enough for you maybe you should be working on projects that provably bring your company revenue?
Exactly. If you are an L7 and making an average L7 salary, nobody should have to squint and tie themselves in knots to figure out the connection between your contribution and your employer’s revenue. You are a Ferrari that is purchased new each year.
There’s a place where you can work on things that don’t generate revenue but are morally/technically interesting: it’s called “academia”.
What is a professor's job other than revenue? Instead of selling a product, they have to beg for money instead. Alternatively, they sell their research. Publish or perish. Raise enough money for the school or you're out.
You are a Ferrari that is purchased new each year.
That's a pretty cheap way of looking at yourself.
This is insane. I can't understand these numbers. No wonder the Bay Area has become completely unaffordable for long-term residents.
The Bay Area is unaffordable because most cities haven't allowed anywhere near enough housing to be built by land owners over the last five decades: https://techcrunch.com/2014/04/14/sf-housing/.
Return building rights to land owners and the problem will be solved. The insane thing is the way coalitions of existing owners conspire to reduce supply and thus increase the values of their assets, and the rest of us just go, "Oh, okay, that's cool, that you're conspiring to embeggar the rest of us."
Google makes $5.5 billion of profit per month. I think they can afford to pay some of their higher performers more.
Indeed. If lots of people with money want to live in a place and building more housing is illegal the price of existing housing will go up a lot. If you want the problem to go away you can either build more housing or make it so the people with money don't want to live there.
The problem is not the tech people, it is the boomers who lived here before us. They have manipulated the system, voted down every attempt to build more housing, and leveraged their good luck to engage in rent seeking and siphon off half of the "ridiculous tech salaries."
My old landlords had average non-tech jobs in the 80s, bought a house for under 100k. At some point they used it to buy a second house, and another, and another. Now they own 30+ houses and rent each for 4-7k/mo.
Old next door neighbor's house was owned by her dad, he was a public works employee and owns 6 houses in the bay area now.
Neighbors on the other side were an older couple with adult kids and multiple rental properties in Fremont. They bought a house for their adult son outright before we moved.
Wrong culprit. There’s no new developments. Need more homes built. Simple.
As someone in Europe, I am so sick of the implied salary pissing contest in every thread.
Market forces are moving in your favor. Most European companies have completely saturated their target markets for outsourcing (Barcelona, Estonia, Czech Republic) and Russia & Ukraine are suddenly unavailable options due to a war. Also the local markets have much more demand for engineers than are available to fill it.
There has been a large push in the last two years to recruit Americans and Canadians to fill their unmet needs and salaries have been climbing rapidly. Salaries for new job postings in Berlin are about where they were at in New York 3-4 years ago which is EXCELLENT for that market (seriously, two years ago salaries were maybe half of what they are now).
If you're in Europe and you haven't been interviewing you should look around and maybe even use a recruiter. My own recruiter has been feeding me opportunities across Europe with pretty much equal money to what I'm getting here in the States.
It's likely that if you haven't changed jobs in the last two years that you're leaving significant amounts of money on the table. That said, I know that chasing after money isn't fashionable or socially acceptable to most Europeans...but the new hires are going to get it -- why shouldn't you?
There's over 4 million (and counting) Ukrainian refugees in Europe right now, and quite a few of those people are software engineers. Then there's over 100k (and counting) Russians that rushed to leave the country before the new iron curtain goes down - and I wouldn't be surprised if most of those are software engineers.
6 replies →
How can we get access to your recruiter?
As someone not at FAANG, I'm right there with you.
Google created Kubernetes so potential competitors are dragged down with immense technical debt, of course they’re not going to award internal advancement of it.
Having participated in full on rearchitectures to Kubernetes several times at this point I can say that it wasn't justified a single time, even at the 10^4 microservice scale. Now having participated in several fairly-effortless rollouts of Nomad and arrived at a better place, it's funny to watch the rest of the industry cargo cult.
I actually don't believe that either of these solutions are going to be a long-term settled end state (though Nomad with Firecracker, or really any other Firecracker-centric solution that isn't k8s will have some legs) the industry falls on, but I agree wholeheartedly that Kubernetes is purely a distraction having seen the pain and the gnashing of teeth.
Every painless Kubernetes story that I've seen is at a scale where Kubernetes wasn't even necessary/justifiable and another solution would have been even simpler. But at least it's good for the resume.
And Helm charts are are akin K8s' own borgcfg. We're doomed to repeat our mistakes it seems.
To counter your argument. I did one full migration from AWS to GCP Kubernetes (GKE). The project was a huge success, simplifying our stack, deployments, logging, etc, etc.
We reduced our costs by 2/3rds, saving millions of years. Teams have been able to move onto feature work instead of maintaining custom deployment tooling. The Ops team 1/2 the size that it was before but is able to handle twice as many customers.
Can you elaborate more? Why it was not justified? The more detailed the better (If possible, concrete issues that you faced) ?
8 replies →
I am finally having to deal with Kubernetes, and the only thing positive I can say is that I miss WebSphere and WebLogic.
This is not really just the problem in Google. We might as well claim that promotion processes in big companies promote promotion-oriented work (yes, I intentionally use the word promotion this many times). The result is title inflation, lots of good employees leaving, and superfluous yet mediocre projects. I wouldn't say only unqualified people get promoted, though. Big companies have very different dynamics than small ones. Some people are just good at navigating big companies and are capable of aligning multiple organizations, for good or for bad. Big companies need such talent as well. As for individuals, the reward of getting promoted is not worth the effort, at least statistically. A much better alternative, is focusing on solving truly interesting and meaning problems in a blow-out company. The financial return and title bumps will follow naturally.
https://threadreaderapp.com/thread/1511791378497384454.html
That's the most ridiculous bubble valley thing I read in a while, coming from a non-k8s using non-FAANG-employed European.
Which isn't meant dismissive of the problem at hand, I can totally see how it's happening and how it might be bad for the people involved in the project.
To phrase it differently - if the project is dying because it hinges on people working on a promotion that gives them a /bigger paycheck increase than my yearly income/ - well, good riddance. I'm staying on my side of the fence and continue to not buy a new Tesla every year.
> if the project is dying because it hinges on people working on a promotion that gives them a /bigger paycheck increase than my yearly income/ - well, good riddance. I'm staying on my side of the fence and continue to not buy a new Tesla every year.
An understandable position but this is simply how competitive environments work.
It’s a lot like the college application mania. There’s tons of gifted high school students who clearly can succeed without needing to volunteer in a foreign country every summer while playing oboe and competing in lacrosse at the state level.
But seats at Harvard are competitive, and those same talented and well-to-do high school seniors often only have college admissions as their ruler to measure their growth and self-worth. So they compete on every angle to try and get into Harvard rather than merely Dartmouth or heaven forbid BU.
It’s reasonable to think working adults should have measures of personal growth that don’t rely on climbing the corporate ladder, but conversely all those who *do* obsess with climbing the ladder will naturally be clustered around places of wealth/prestige, because that’s what they are optimizing for. Much like the Harvard application pool, you’ll find these types doing what it takes to stand out and land that 700k/yr L7 position at Google.
So I guess the answer is that the most stable and secure position for an open source project to be in, is one that does not rely on development being coupled to big tech. I imagine most open source folks could have told you that anyways :)
Kubernetes' biggest problem by far is its mess of accumulated technical debt. FAANG promo committees very likely undervalue FLOSS contributions, but this hits all of FLOSS pretty much equally; it hardly explains why k8s specifically would be in so much trouble.
I would go as far as to say it's not just limited to FLOSS contributors. In my experience at Google, not every service is profitable or brings in revenue, so it can be hard to justify "impact" on those teams. I certainly remember beating my head on how to spin my contributions as having "impact."
Kubernetes doesn't get patches from hyperscale companies because kubernetes is not really hyperscale software. If I recall correctly (probably don't), clusters can handle tens of thousands of container hosts. That's not manageable at large scale unless you're selling people individual kubernetes instances directly (which, to be fair, google and amazon are).
In my experience the thing that kills open source at big companies is the divergence of needs: The company needs massive scale software, but the community wants ease-of-use and features. Most open source models are a big drag on productivity (you can't just refactor all the callsites to do a migration!), and it's hard to change directions when the company's need does (since the other contributors will likely want the original software).
The tailscale people have been arguing this as well (with more data and better reasoning), but fundamentally you need different software when you're a scrappy startup or a medium size company or a giant behemoth.
Having operated all the way up to "hyperscale", I agree with your point completely.
What's preventing k8s from being useful in these larger scale deployments? GP was not very clear about that, they just said that it's not scaling enough.
9 replies →
Maybe I’m misunderstanding but why would you want to run one giant kubernetes cluster at hyper scale? I would think at hyper scale you would be running micro services with one kubernetes cluster per service per datacenter/region.
At hyper scale, efficiency matters a little. Small gains multiplied by big numbers become big gains. The overhead of running a cluster is not small, but more importantly - the overhead of managing nodes between services is real.
No, you would not want to run a cluster per service; Kubernetes is somewhat multitenant. You'd get better efficiency from running one cluster per datacenter, with all those teams in their own namespaces. Kubernetes can stack many containers per physical host, with widely varied workloads cooperatively sharing.
7 replies →
You wouldn't want to run one giant cluster, but at hyper scale you're talking about running thousands or tens of thousands of kubernetes clusters. That's the part that doesn't scale well, for a couple of reasons.
The biggest one is just mechanical: with that many clusters it will be hard to move capacity between clusters, and locality gets baked into everything you do (people do try to build around this, but it's awkward).
If each service or team runs their own kubernetes that's a lot of overhead: kubernetes will need something like 6-7 machines for the cluster (I don't have production experience with kube, spitballing here), so small teams or jobs will have terrible efficiency. Big teams will have to spend a lot of operational effort to manage their fleets.
It's worth noting that at hyperscale there will be individual jobs in a datacenter that are bigger than kubernetes handles comfortably. Handling this efficiently becomes very important, it's literally billions of dollars worth of hardware.
> I would think at hyper scale you would be running micro services with one kubernetes cluster per service per datacenter/region.
While I loathe the term hyper scale... even a single AZ in a single region could be 50,000 machines. k8s is a dumpster fire at that size.
1 reply →
This is just the law of inertia at play. Big projects slow down. K8s isn’t just the K8s core, it’s a massive ecosystem, and when taken as a whole it’s unbelievably vast and growing at an amazing pace. As it does, the core of it moves slower, there are more stakeholders, more arguments. Small changes make for big problems and not everyone can be happy. Patches will keep coming, major changes will become eventually impossible. The death of Kubernetes makes for great clickbait, but most of us won’t see it in our lifetimes. Not when our banks run on 60 year old code. Kubernetes is too big to fail at this point.
As far as FAANG promo committees go, let them value what they value. Kubernetes is a direct revenue driver for many companies; it’s health is tied to billions of dollars in investments. Just because one cohort of contributors age out, cash out, or fall in love with someone doesn’t mean there is no one to take their place. The new people won’t do it the same way and that is okay, even great. I’m grateful for the vision and effort that have make K8s the platform it is, and if I can translate that into a contribution in the future I will.
Finally, I’ll say that people who really love working on the project may not get support to work on it full time, but then may find themselves able to retire sooner than many and have ample opportunity to contribute.
I basically can't stand the comment section of Kubernetes threads. Lots of big teams are getting big things done with k8s, and you would not know it from here.
Not sure why this says FAANG because it's talking about Google.
Calibration, promo committees, feedback and the like are intended to create equivalent expectations on impact across different orgs. The dirty little secret however is that at best it has limited success.
The best advice I can give anyone in such an organization is to be liked by your manager and their manager. If that's true, good things will tend to happen. If it's not, good things will be a lot less frequent.
Put another way: you can take the exact same set of objective facts and use them to say a person did a good job or a bad job. There's a popular meme about feedback at Google that goes something like:
> This project would've failed without this person. It failed anyway but it definitely would've failed without them.
The difference ultimately boils down to whether or not they like you.
Here are a few Google-specific tidbits worth knowing:
1. Ratings are fit to a curve across a sufficiently large pool, typically at the director level and usually over 100-150+ people. This means there will be a percentage range of people who get Meets All, Exceeds Expectations, Greatly Exceeds, etc. This is intended to stop ratings inflation;
2. A consequenc eof (1) is that ultimately you are competing with people in your org for those better ratings. This can create some perverse incentives and a toxic environment;
3. It is almost always better to let something blow up and come and fix it rather than preventing that from ever happening. The first will get you a lot of recognition. The latter will get you almost none;
4. Promos at Google are stack-ranked. Each committee gets 10-15 packets that will be for a particular level. The committee will rank those packets. After that the promotion target will come into play. This is set by management and was allegedly cut as a cost-saving measure when Ruth Porat came on board. If it's 20% then the top 20% from that ranking process be promoted.
You will find people who serve on those committees who say this isn't how it works and they'll argue they're evaluating if someone is operating at the next level or not. This is partially true. Thes packets will be divided between promote, don't promote and on-the-bubble. The on-the-bubble group will be sufficiently large to allow for the promotion target to be met;
5. For SWEs. L5 is the "terminal" level, meaning there is an expectation of growth to that level. L3->L4 and L4->L5 once went through promo committee but now don't. Management within orgs decide this. These too have target percentages and there have been cases where the promot rate has been too "high" and orgs have been told to cut back on promotions to meet the targets;
6. There is a massive backup at L5->L6. Because of the low target percentages the impact required keeps going up and really you need your management to really push for this to happen. There are limited slots so you may be waiting eyars and again this is why them liking you matters so much. Google is full of L6s who got promoted 5-10+ years ago that would never make the grade by today's standard. For really old cases you can find archives of why there were promoted and you'll find cases like "promoted unit testing".
I say all this because the author of this thread seems to fundamentally misunderstand how this process works.
The promo stack rank goes the other way. You decide on the promote vs no promote part first and then rank based on confidence. I’ve absolutely been in sessions where 100% of people were promoted. You don’t rank and then apply a bar based on a target promo rate.
There are certain classes of work that are completely ignored by most companies such as build, automation, developer tooling and infrastructure cost cutting. These are not flashy features that you can show off to customers but they can pay in dividends year over year.
(Personally, I have saved millions a year in infrastructure costs and my promotions/compensation were not close to what a sales person would get if they brought in millions of recurring revenue)
Threadrip to make 'this could have been a blog post' more readable:
https://the.rip/@kantrn-1511791378497384454-121542889
https://unrollthread.com/t/1511791378497384454/
There's a perception that this is true, and like all perceptions it is based in reality, but it is not the reality itself.
We have plenty of data that shows that people DO get promoted at all levels based on OSS work.
K8s is a great example of what murders open source: monolithic platforms that exist unto themselves, but have some seemingly benign model allowing arbitrary "plugins" to pretend it's not really a monolith in sheep's clothing. It looks like open source and has the right license, but it ends up just mirroring corporate interests rather than those of the Commons that the open source movement was created to support.
If you're going to have monolithic platforms anyway (and you will, because designing for modularity is hard, whereas hacking together some random monolithic code is easier) they should at least be open source rather than proprietary.
Actually I disagree. Any time there's an Open Source tool that does everything for free, it's extremely hard to justify paying for a proprietary product that does it much better, because companies are cheap. There's many open source projects like this that are just terrible but a company will always use them first because they're free.
The answer isn't modularity, it's composeability, which is a significant difference. A modular program requires "integration" (tight coupling of APIs/ABIs) whereas a composeable program has loose interfaces which require virtually no "integration".
Drone.io is one example; you can implement any "plugin" purely by creating a container with an initial CMD entry that reads environment variables, and that program can interface back with Drone a number of ways (STDIN/STDOUT, REST API, database). Another is any application that just reads in or spits out simple line-by-line instructions or a JSON blob. Unix pipe based programs are the penultimate example. The dumber the interface, the easier it is to compose.
This is the case with most major products from most major tech companies. There is a reason you don't (often) see steady incremental progress on products, but rather big flashy releases that never quite get their bugs cleaned up unless they are wildly* successful.
* Where wildly is defined as a success level which would leave any rational founder very wealthy, and very content.
Great, the sooner we get rid of that yaml spaghetti, and container everywhere, the better.
Hopefully Kubernetes does die. Most companies write shit slow software and then think they're big smart boys because they have to buy $20,000/month of servers to make it not be unusable. In many shops it could and should be replaced by a handful number of efficient servers that do not suck shit.
I love open source but I’m happy if we take the credit for killing kubernetes :)
> killing Kubernetes
Really? Finally some good news in these troubled times.
Oh man, another rant that boils down to "big evil corporations are KILLING open source software by not supporting it in the specific ways and in the specific amounts that I want."
It sounds like FAANG has reinvented tenure and promotion processes from academia…and the resulting inherent pressures.
A moment to reflect on Han’s critique of publish or perish academia?
What's the alternative to k8s then?
[dead]
As someone that's manually stitched together some of what Kubernetes provides (before it even existed), I very much appreciate that Kubernetes exists.
I mean you should really be demoted for continuing to push the k8s microservices crap.
This seems like a strictly Google only problem. And by extension this is the problem with promotion committees that are removed from day to day of the candidate getting promoted. People are ridiculously bad at measuring growth based on some obscure company wide criteria - which is abstract at best and useless at worst at a company large enough as Google.
I'm at AWS and the promotion process here is largely contained within the candidate's org which is at best two levels higher but in the same business. So if your team's business relies on contributions to FOSS, its impact is measurable and leaders and the 'committee' can easily tell you if your contributions are at the next level.
L7+ tech generally has in and out of org reviewing the promo. One for context, one for keeping the bar even.
Yes FB is entirely in org too. It's weird to go out of org.
Go figure, contributions to a multi-vendor consortium that just barely makes a marginal profit for an already marginal business unit (GCP) are not as highly valued as things that can be directly connected to revenues. This is not surprising in the least.
Perhaps obviously, Mr. Kantrowitz hasn't quit, or been poached by a competitor, so I would say Google's (cynical) comp and promo policy is working out just great for them. Whatever personal interest keeps Mr. Kantrowitz going is working out for them!
He does not appear to work at any of the FAANG companies: https://www.linkedin.com/in/noahkantrowitz/.