Comment by keyle
3 days ago
I've been in those companies where "struggling departments" ended up getting all the praises and raise in budgets the following quarter because of the heroic saves they did, and raising awareness on how important they are... For stuff they totally caused on themselves.
Meanwhile, my perfectly purring department was struggling to keep the lights on.
It's a serious problem in this industry due to the disconnect between non-technical management (who understands how to double click) and engineering (who holds the company standing).
<insert IBM story about IT department cost cuts>
I'm not sure how we solve this, other than having management come from engineering.
By building pain into the system. If your hands dealt with injury directly without sending pain signals up to your brain, you'd never change the behaviour that led to that harm or reconsider your priorities. Like it or not, sometimes the best thing for an organisation isn't to just fix every problem and prevent it from bubbling up; it needs to be treated like a learning opportunity for org leadership, which means sending the pain signals upward before just repairing it.
Building the right incentives around that can be tricky, those incentives need to ensure the highest levels of management aren't themselves disincentivising their directs & their departments from surfacing pain & problems - but it's also pretty common for people to mask those signals purely out of a well-intentioned desire to help. It's important to coach people on the idea that in large group sizes, it's more efficient to let certain kinds of problems play out and not be so reactive to them.
Too many companies ground their performance incentives & processes around oversimplified ideas that don't match the reality of human behaviour
+10,000%
Often, 'leaders' make mistakes and people below suffer the consequences. It is important to let these leaders deal with the pain caused by their decisions from their cluelessness about how things work.
The problem is it's systemic. Ultimately, pain needs to come from outside. As long as society rewards incompetence, we'll have incompetent organizations.
No it does not need to come from the outside. If you're an underfunded IT department and your network has an issue twice a week, you will get that funding. If you're heroically obscuring the fact that things are falling apart you won't. That means even if you could somehow, heroically fix it, it isn't perceived as such if nobody ever felt the problem and saw you fix it.
This is a pain signal. Some IT dude saying things are crap in every meeting is not.
6 replies →
Honestly? I’m just super direct with the exec team now after trying to do this dance. Obviously this is not allowed at every company, I’m just lucky to be at a place where the company culture allows for this.
I’ll ask for something preventative or that otherwise hardens our systems. They ask “is it a need?” and I’ll say something like “we can function without, but that means we have a 5-10% chance in the next 6mo of having a major failure and embarrassing ourselves in front of a live audience in the thousands as well as our client.” They then decide how much that risk is worth to them, and whatever they decide is kind of out of my hands at that point. If the thing I warned them of comes because they didn’t pay for it, I can point to the receipts (though I’ve never had to, we’re small enough people remember those conversations).
60% of the time they just get what I need maybe? But ultimately it’s about CYA. Tell them what’s up, tell them what the solution is, tell them what the consequences are if they don’t do the solution, and make them decide.
Again this obviously depends on company culture and structure, but I can’t imagine on the only person who can do this!
An example that not all companies are run by idiots. The job market is not a healthy market though, where its more important to know ppl then to be great at some skill. But if leadership sucks just leave if you can, that will fix the problem.
1 reply →
> By building pain into the system. If your hands dealt with injury directly without sending pain signals up to your brain, you'd never change the behaviour that led to that harm or reconsider your priorities.
I don't think that's it. Emergent problems require attention and action from leadership, who in turn can make the problem visible to higher ups. This creates signal, and positive feedback when the problem is fixed or mitigated.
If the problem doesn't exist to begin with, there is no signal. Managers don't get to show their fast-acting skills, and there are no heroics to speak of.
So ultimately poorly maintained and managed projects who deliver fixes for problems of their own doing create a perverse incentive, whereas no one is lauded or promoted for doing normal day-to-day things.
Well I think it is even more complex. If you're a plumber in a rotten system of pipes the whole company depends on, you can fix issues day in and day out, without speaking a word and they will notice everything is a bit unreliable and thus you do a bad job. You could do the exact same work, but make a big thing about every major fix, warn people a week ahead, give them the feeling the company depends on it and then do the exact same work and tell them how you fixed it. Suddenly you did a good job, despite you literally doing the exact same thing with your hands.
The difference is how it was communicated. Most non-Tech/non-infrastructure-people got no clue about these things. If they know you're battling the demons of plumbing on their behalf they will thank you, if you're the weird guy that has smeared dirt in the face and is seen once a week while the plumbing fails ever so often, guess what.
That means even if the problems and their fixes remain the same, the communication around them really matters. Tech people can be extremely bad with this. And if we're talking IT it is really the plumbing that holds the company together.
"You knew there was a problem, and didn't fix it? You're here because I don't know programming. Fired."
I think we've worked with very different kinds of people...
> If your hands dealt with injury directly without sending pain signals up to your brain, you'd never change the behaviour that led to that harm or reconsider your priorities.
At some point in one's early single-digit they learn that touching hot stuff hurts. They start to avoid stuff that they know is hot, but still come in contact with hot stuff accidentally. Later they learn techniques minimizing probability of touching hot stuff even by accident. By the time one reaches twenty or so, the only times a person burns themselves is really by being way too reckless.
> Like it or not, sometimes the best thing for an organization isn't to just fix every problem and prevent it from bubbling up; it needs to be treated like a learning opportunity for org leadership, which means sending the pain signals upward before just repairing it.
Should we accept that management as a whole is in general more clueless than your average teenager? The "learning opportunity" should, ideally, happen exactly once, realistically once in a very rare while.
> It's important to coach people on the idea that in large group sizes, it's more efficient to let certain kinds of problems play out and not be so reactive to them.
You are conflating two things here, I guess. Yes, some "problems" are not worth to be fixed proactively or at all, but that has very little to do with group sizes, it's a "simple" cost-benefit tradeoff. As groups grow the left hands tend to become increasingly unaware of what the right is doing and that is the primary reason why we have management class in the first place.
The problem OP raises is attention span of the metaphorical gold fish in the management layers. Even if a department does everything in their power to communicate impending problems, do risk weighed cost-benefit analyses, get proactive treatments pre-approved by higher management, the same higher management forgets the risks and costs savings once they have been mitigated, effectively incentivizing firefighting. Some teams gradually fall into eternal firefighting and burn out, others start manufacturing fires to get rewarded. The biggest problem is that it is nearly impossible to tell the two apart.
Unlike a teenage child, management has the unfortunate effect of being made up of people who can leave the company, forget past experience, etc. So you do kind of have to treat them like a child who needs continuous feedback and signals.
For a more broad example than IT cost center stuff, you can look at how some large companies go through cycles of arrogance with their customer bases, launch a product that fails, and then are humbled enough to try and pivot and earn good will back. Microsoft is always somewhere in this cycle for instance. The organization can never really learn this lesson permanently and will "regress" from time to time based on financial pressure or greed or some other impulse.
The Dunning Kruger effect suggests that the people who caused the pain are also those least likely to feel it.
That's why you change it to make the pain work. This does need CEO-level cooperation to implement, but i think it is possible.
1 reply →
Dunning Kruger's paper didn't even show the Dunning Kruger effect everyone loves.
3 replies →
In my 35+ years in IT, the "hero attitude" was the one in the top three I most hated traits in a person working with or for me. And talking about traits, I considered crucial to always have in my teams a "saboteur" engineer - the one who thoght, found, come up with all the way we could break a design, service, infra components, app, etc., when all the others were designing or operating for perfect or normal conditions.
Genuine heroism - a willingness to step up when needed - isn't a bad thing in itself. But /needing/ a hero just to function means the system is fundamentally broken. Maybe it's a bad process, maybe it's understaffing, maybe it's neglected maintenance, maybe it's a lack of contingency planning. But there's no reason everyone can't go home at 5 PM every night and still get things done.
Love the "saboteur" approach. I honestly want to be one my own career in IT, but as you have rightfully conveyed, "hero attitude" is what gets you visibility!
At a previous job the CEO/owner had the idea that you'd get some percentage of any cost saving your could find as a bonus. Something like 20% of the savings for the first year.
My colleague in the IT department had one idea, replace our commercial certificates with Let's Encrypt and drop the EV requirement. In total he'd stand to get a bonus of a little over €2000. He never got the money, because things like that was part of his job apparently.
Wow, that's pretty silly. 2000 Euros is almost nothing in the grand scheme of things, and it would have showed that the policy was sincere.
Even dumber, they've now got a disgruntled employee, and everyone around them knows they were cheated by the company.
If the policy is wrong and needs to be more specific, pay it out this time and change the policy. Don't just break your word.
The policy they think they've implemented is stupid. "Save money in someone else's department" is just going to create a ton of anger as people rush to step on each other's toes, and then those people have to constantly re-justify all the decisions they've made.
It's absolutely brain-dead.
2 replies →
I think a good place to start is tracking all the proactive things being done and reporting them. At least then maybe someone will see why it’s quiet, because you’ve anticipated the problems and stopped them before they start.
When things come up with other teams, you’ll have a catalog of tasks that were done to show why you didn’t have the same issue. The work was done, just at a better time to avoid downtime.
> start is tracking all the proactive things being done and reporting them
Speaking from experience, this does nothing. If you're at a company that is okay with average performers, then absolutely, 100%, fix all the bugs in advance, make the system rock solid and stable, prevent downtime, be a good engineer.
If on the other hand if you're at a company where 10% of people must get stack ranked and PIP, or at a company where "meets expectations" actually means you're going to get the stick, and you're supposed to be "redefining" expectations every year ... then yeah, don't do anything preventative. The optics are better when you take the 3am on-call and fix the issue (that you secretly knew in the first place would happen some time in the future in your coworker's code, and already knew how to fix -- but don't actually fix it until it surfaces). Be the savior that the VPs praise in the next meeting, that's your insurance against the PIP.
They set the rules of the game, you just play the game. The rules were their choice. They could have chosen different rules.
I'm sorry about your experience.
Personally, I only rehire people from projects that went smoothly, not ones where I had to make the urgent phone call.
Teams that "just work" are highly valued. They clear up my attention for other things.
5 replies →
I refuse to play those games. If they want to fire me for avoiding problems instead of sacrificing my sleep, fine. I’ll go stock shelves at Walmart.
If someone is constantly playing the hero, I see that as incompetence. If the boss can’t see that, they are also incompetent. I have no respect for “leaders” who don’t know how to get out of the firefight.
I’ve made some high profile appearances, working 18 hour days on 4 day long outages, from vendor issues I was no part in causing. I figure that gives me some good will on playing hero without willingly creating problems for myself. I’m too old to manufacture stress for the optics.
For what it’s worth, with the right boss, I have had proper reporting work. Everything ran smooth and work was relaxed. My boss would regularly tell me I should take 3 months off because we were so far ahead of everyone. He would occasionally get bored and lob a grenade into the works to cause some chaos, but since everything else was running so smooth we were able to sort them out and keep going. People who couldn’t explain what they were doing were always getting yelled at and assumed to be doing nothing.
2 replies →
> They set the rules of the game, you just play the game.
Obviously the only winning move here is not to play. Things like stack ranking are a perversion and no amount of compensation would be worth working for a company like that. If you choose to play, you're complicit in the moral abomination.
1 reply →
I run the media tech at an university solo. Needless to say I am underfunded. But more importantly, the infrastructure was underfunded too. I made it my first policy to also report near misses up the chain with their full implications, e.g. a list of events that we would not have been able to make.
E.g. that time a central media controls power supply broke down which would have made using one of the most prestigious rooms impossible. I fixed it myself by swapping in a spare power supply from a used unit, then went on to remind them twice a year that we are now living on borrowed time and I take no responsibilities if a fault I predict to happen and get no funds to fix will in fact happen. 4 years later I got the funds.
Having stuff costs money. Everybody wants to invest funds once, but nobody wants to keep paying for maintenance.
The tragedy is that “nothing broke” looks like “nothing was done” to people far enough away from the system.
Things keep breaking - "What are we paying you for?"
Nothing ever breaks. - "What are we paying you for?"
Management can choose their burden.
Isn’t this a universal problem though, not just software industry? Even at home, if one kid just does his thing quietly but another kid is difficult, what do we say? “John has his problems but he is trying, we gotta encourage him”. While his brother gets no praise or attention for just doing his thing quietly without fuss.
When things run smoothly, very few people notice. When things break, everyone notices
I thought that was where the phrasing:
"John! Why can't you be more like your brother?!"
comes from.
when you do things right, people won’t be sure you’ve done anything at all.
As a middle child this rings true for me.
Econometricians can solve it, bc we can create rigorous models that map causal inputs to output.
It’s extremely advanced technology, though, and most CEOs would rather rent seek / camp than give up some decision-making power (and very few are even aware it’s possible).
Do you have any good sources for this I'd be interested in learning more
I used AI to unpack it a bit here: https://statwonk.com/econometricians-can-build-decision-engi...
I'd generally point to econometrics and statistics applied to business. The key activity is causal inference and then the context determines the mix of econo vs. stats required to help the org make high-quality decisions to increase output or make it more lucrative or higher-quality.
Ah, yes - the person who comes in at 7AM and gets shit done by 11 is a slacker, and the one in the office just doing nothing after 6PM is the hard worker. Same thing.
You can't fix this. Out of sight, out of mind. It is hard-wired into us. It's all about the optics, and will always be.
> I'm not sure how we solve this, other than having management come from engineering.
I disagree with the implied idea here that "engineers are better managers". The solution is to have good management, not to assume that "engineers are better managers". I have seen good and bad managers, and in both groups there were engineers and non-engineers.
Engineers may not be better managers but it's not easy to really manage something you don't have any insight in.
> It's a serious problem in this industry
s/in this industry//
This thinking eventually results in The Scream Test. When the screams come as a system fails that is when they act on it.
Alas, for many parts of society there is a large amount of people that would rather be reactive than proactive. It means it is easier today but harder long term.
Managers will let you get away with anything if you time your reports correctly. They also don't want to sit in meetings where they are reminded of better outsourcing alternatives and they chose to dogfeed instead.
We've become too comfortable, since actual toil is no longer seen in the company: Manufacturing is overseas, customer support is overseas, logistics is an afterthought with established guarantees. Thus we want the mild weather and smooth meetings. If your engineering team is too smooth, maybe you should already branch out to help other related but "struggling" teams to get your hands dirty and noticed.
itemize the problems you are preventing
"Accounted for X situation" "Added gaurdrails to protect against Y"
When working as a business analyst i have to do this sort of thing all hte time or else id get no credit for half my work
> I've been in those companies where "struggling departments" ended up getting all the praises and raise in budgets the following quarter because of the heroic saves they did, and raising awareness on how important they are... For stuff they totally caused on themselves.
This is a very game-able system, and I'd wager a decent amount that any senior engineers on those teams know exactly what they are doing. In a lot of (broken, but aren't they all) management structures, it's better to be seen to swoop in with the save than to quietly fix it ahead of time.
And if your management is structuring rewards like this, it leads to your seniors anticipating a bunch of these failures, lining up 90% of the fix before hand, so that they can jump on the oncall escalation with a 100% "Hail Mary" of a fix...
> It's a serious problem in this industry
It’s not a problem in this industry, it’s a problem everywhere.
> I'm not sure how we solve this, other than having management come from engineering.
You mean the engineers who are causing the chaos you’re complaining about?
Engineers aren’t some magic group of people who know better than others - we’re just as fallible as other people.
Which they should. I've been lucky enough to work at places that had great non-technical managers that promoted based on great execution, as well as highly technical managers that also promoted based on great execution.
Now I'm at the other kind of place and it sucks. They'll fire the performative engineers though during layoff season. It's almost like they like playing politics until it really matters.
Same. I watched a manager fail upwards till he practically ran the entirety of engineering. All his projects failed, got expanded and restarted with more budget and more devs until he ran everything. Meanwhile the teams that actually wrote working services got their budgets frozen and lost headcount.
I guess the point of view is that if a department is well running, it means it is overressourced. So you reduce the ressources until it's breaking point, just enough for it to not fail. A jaded service manager told me it was part of its official training: if the clients was too satisfied that meant that human ressources were wasted on them, so he had to spin plates between clients. I guess it was economically optimal.
This is a short-term view approach and can really hurt a company on the long term.
It's also why US car companies are a wreck.
I feel like this is a cultural symptom and something many people are hoping to solve in healthcare. Basically we treat solving problems as amazing rather than preventing problems. You get rewarded if you treat a sickness instead of keeping a healthy person healthy.
This is the same thing. We need to reward things never going wrong as a society since this is pervasive.
> something many people are hoping to solve in healthcare
Respectfully, the solution is don't smoke, exercise, eat well, sleep, avoid stressors... These aren't easy problems but their solution isn't at the individual patient level and is a simple question of capital and political will.
The 'hope' envisages a product to temporize the solution while extracting large payments.
Nope, I believe you are wrong: a path where we, for example, forbid smoking because the statistics point at it correlating with many health problems, is a world where we use the same statistical tool to prescribe human behavior to the last detail. It is not just about smoking, alcohol, late night dancing, switching sex partners, fast driving on a track, paragliding, skydiving, climbing, car driving, bicycle driving, motor biking, even staying late for astronomical observations (sleep patterns?)... all carry insignificant risk when looked at statistically.
> ...avoid stressors...
Most stress is caused by a conflict between our expectations/motivations and the reality (everyone else's).
2 replies →
Since it's entirely possible (extremely likely?) the "problem" would never materialize, this is quite reasonable.
SpaceX almost has a full grip on the planetary consciousness extinction problem.
lol. I hate presentations. I like to run a tight ship. But that does not shine, so they made me do presentations every quarter. If you do some work, you must "take" credit. It is kinda a need when you manage people since you need to build their careers.
I finally moved on to be an IC. Same story, same pressure :) You need to present to directors not because they need to know, but because your managers have a quota of N presentations per quarter, and if you back out, someone else needs to step up.
Needless to say my productivity reduces by half and sometimes to almost zero during the week or fortnight of presentations every quarter.
You define "productivity" as coding.
The business defines it as "meetings, presentations, support, coding, whatever".
Your productivity remains at 100% when you are doing what they want.
I get that you thought you were hired as a coder, and thus measure your productivity by that. That's what I thought too. I ended up doing a lot of support (which is good, but that's another thread). Until I recalibrated my definition of productivity that frustrated me. When I realized that support was productivity I got much less frustrated.
When did I say I code?
I have been on the industry for 35 years. I have seen my share of technology evolutions and o have seen the work from a dozen different dimensions. If after all that time, I find the process painful, just trust me -- they can't change me, and I can't change them. You take the warts with the wins and move on. 2-3 bad weeks, 10 good weeks. Life moves on to next quarter. Complete CEO mindset :)
2 replies →
I feel that disconnect is everywhere, when the suits dont see anything and act on reports
>>I'm not sure how we solve this, other than having management come from engineering.
Given the whole point of management is to work to ensure their own survival and growth, it would in their interest to kill genuine competition when its coming up.
Who wants to raise their new competition and lose to them, no one!
Track leading indicators, pricing them if possible.
Car industry best practices
I believe it's a problem in most industries and even humanity in general. I don't believe it's a business problem at all.
Heroes are lauded even if they solve problems they themselves are the cause of - which is conveniently either forgotten or denied - or they are solving non-issues that are deemed important by the ignorami-class. Politics, for example, is utterly dominated by this dynamic.
It's the first instinct: let the expert run the show. However, one of the (many) ways to let a complex project fall apart completely is to hand over full control to engineers. I'm one myself, but I know what I'm good at and what not. Dunning-Kruger is often mentioned in these discussions, but don't forget it also applies to engineers that often lack any management or leadership experience of any appreciable kind. They vastly overestimate their ability to handle management and organization-wide issues and tend to not only miss the forest for the trees but actually miss the trees for the leaves.
"Unix: A History and a Memoir" by Brian Kernighan actually mentions how proper management was crucial to their success. It's a detail that's frequently conveniently forgotten by the engineers who think themselves better than the "suits". For the record, I don't claim engineers are the primary problem, but it's not just management's either. Quotes like "who holds the company standing" and "who understands how to double click" are enormous smells and IMO make quite clear what's happening here.
I don't have ready-made solutions unfortunately, but I do wish we would look further than "it's the suits". It's a systemic, human problem that I believe is a natural result of operating under informational constraints and, very human, cognitive biases on all sides.
Bell Labs is an outlier in basically every aspect. Mr Kernighan lists stability of the environment with regard to funding, structure, mission as well as technical competence of the management as main drivers of the culture. This is just not the reality in companies that look for financial results on a quarterly basis and where the executives are MBA types.
If one of the most successful engineering organizations in history attributes part of its success to capable management, that undermines simplistic narratives where management is inherently the problem and engineers would naturally thrive if left alone.
If anything, the Bell Labs example supports the idea that exceptional outcomes require both strong technical talent and strong management working together.
Not saying the "MBAs" are helping the situation, but the hero developers and their resume driven development practices aren't exactly angels either.
1 reply →
could an AI product solve this?