At one place, there was some important order processing taking place. As is fairly typical, couldn't rely on getting all the required info. Or critically, getting all the required info correctly. Some extra data, slightly missing pieces but enough to work, etc. etc. Some could be pretty gross. We built validation to massage some inputs, modify processing, what have you to address as many of those as we could think of. The team put in some metrics to identify which validations were "triggered" for each order. Neat. If we'd add more, we'd add a date to it.
It was great. We reported those stats so anyone could see anytime, but we'd also send out some comms about it every now and then. It also helped tremendously when a coworker or customer or whoever would say "Oh no! What happens if XYZ?" and we'd say no worries, we already addressed it, and we prevented #### orders from getting stuck for XYZ.
It showed that the team was thoughtful, that we were invested in making this work, that work needed to continue to keep things running smoothly, and we had data to back that up.
Really helped switch the conversation in the org because folks could see it. If someone pointed out that we hadn't thought of something, it honestly was more about what do we do vs. why didn't we think of this. (Yes, there is a comment here about poor org thinking or blame culture, but some of that exists everywhere.) More proactive. Recognition of preventive quality. People gave us accolades and it bubbled up to some good and real recognition at higher levels too.
I'm mangling the words here a bit but I hope you get the idea.
There is a simple but undeniable element of genius to the idea of, instrumenting the number of validations triggered. I would hazard a guess that 90+% of teams would focus on an "order success rate" type of metric and call it a day, instead of a "# of times bad data was handled" metric, which is exactly how you escape this trap of good work going unnoticed.
Thanks for sharing! Shamelessly stealing and applying this concept at work next week!
Yeah. Another way I would put it is to also look at inputs that strayed off the happy path. There may be something to learn from them as well, maybe something that adjusts the happy path.
That's exactly where we started. It worked or it didn't, right? The team was super curious to know, in fact, how many of the edge cases happened. That piece right there had some unexpected value as well. And it grew out from there.
Went through this exact scenario at my job recently. As the tech lead/architect of an org I reviewed a bunch of recently released projects and identified areas that needed critical improvements because they had major reliability/performance issues. One team had multiple releases at the top of the list, and their PM and eng manager (and really everyone else up the chain) simply ignored all concerns because they had to prioritize feature updates instead.
Fast forward a few months – I was on vacation and shit blew up. There was a sev 1 escalation, several customers were pissed, the CEO/CTO were involved. The team in question – the same one that wrote the shoddy code and ignored all warnings bells – worked round the clock and got the service back up. Now they are heroes for their effort and the same manager I mentioned has a great reputation at the company for being so active and communicative and showcasing his leadership during the outage.
Performing some heroics once in a while shows someone can be relied on. However, if heroics are the norm, it’s either bad work or bad management, and it should be looked at with some scrutiny.
I also find it more impressive when someone fixes someone else’s problem. I’m not one to shower praise on someone for fixing their own mistake, nor do I expect any praise if I fix my own mistake. I would be apologizing to everyone for screwing up in the first place.
I’ve started doing something like this more frequently, but it is in the interest of job security / cover-my-butt rather than pettiness.
I build something that depends on many systems that others have created and that uses data from probably about every dataset we have. What I build also has very high customer visibility. So whenever something breaks upstream, the first place customers notice the problem is in the system I maintain. Psychologically, people begin to develop a mental association around your system being a “problem” if it keeps getting brought up as the starting point of SEV discussions and customer tickets.
As a result, a lot of what I do now is defensive. I investigate and reverse engineer upstream codebases to identify likely failure modes, and I spend hours analyzing datasets of questionable origin for data quality issues and inconsistencies. I document and date stamp all of the problems that I find, file bug reports and assign them to the relevant teams, and write proposals describing potential solutions to what I see as large architectural design flaws that will come back to haunt us at some point.
All of this work is promptly ignored with “not enough bandwidth right now” or “not a priority compared to feature development”. Which is fine. I document all of those responses too.
Then eventually, something breaks in a big way that is again first noticed by customers within the system I am responsible for. In the past, I would immediately drop what I was doing and scramble into investigation mode for a few days to prove I wasn’t the root cause of the X hundred thousand dollar issue, er... I mean the blameless SEV review & postmortem... but more recently my preemption seems to be paying off, and lately I just reply to the panic with a bunch of links to old Slack threads (where we already discussed the issue), the documents and proposals I created (that no one read), or the bug reports I filed (that were never followed up on).
Perhaps it does come across as a bit petty, but I try to be as polite as I can, and from my perspective it’s an improvement over the previous situation. The only downside is that all of this preventative work takes away time from the primary work that I was hired to do.
This can be framed constructively, do an analysis of what was fixed during the outages from the originally identified and now lobby for funding to fix the rest.
I've been contemplating the issue in the headline, as pertains to my value. When I help somebody in 40 mins with something they've been stuck on for 3 months, my value is clear to everyone. When I work there the whole time, and nobody ever gets stuck for 3 months, my value is unclear. Don't know how to deal with this paradox.
The educational system i was put through was set up to teach that outcome scales more or less linearly with effort and time. The first lesson after graduation is that this is not so. Increased effort and time most likely primarily yields more effort and time being expected of you with lagging compensation.
Value and opportunity are chaotic processes in effort and time.
All we can do is to try to maintain the levels of workload such that we have clarity of mind to seize opportunities when they reveal themselves. Honest and balanced colleagues help there, but that is ultimately a missions for yourself only.
Helpful partners and friends who try to find opportunities most definitely can help (and in unbalanced relationships, can do it all), like the market research team at a startup.
Or worse: People get stuck quite frequently and ask you for help pretty quickly. You get everyone unstuck, but your own work falls behind and when your boss's boss asks for metrics on developers you have few points delivered and few LOC changed. Your boss tries to explain, but your head is the one that rolls next when layoffs happen.
As a senior engineer I've learned that is my job. I take on much less work than anyone else because I know I will be called on to solve those problems.
This is in fact known in management literature: assign your best people to the least important project. That way the second best can grow to become the best, while the best are always free to help out if your fourth most important project has issues - it isn't a big deal if your least important project doesn't get done on time and if they manage to finish it so much the better. (your best are also free should sales discover a short window where a quick feature can bring in a large sale - though this is obviously easy to abuse)
which is why one should not self-sacrifice. It garners no reward for one. Secondly, if the boss doesn't realize how much of an enabler you are, then it's time to start looking for a new job before the layoffs even starts as a thought in the boss' head.
I've done that as an hourly contractor, I fixed their 6 month problem on my first day, I hoped they would then hire me for more hours on other stuff but they were like nope, that was all we needed. They did tell some other companies that I was good at this kind of stuff but nothing came of it. My first and last day in contracting for small companies.
The trick is to do these sort of engagements as a fixed price contract. You could have probably done an assessment in a hour or two. Then you charge on what you estimate is the value to the company.
Really good bosses make up for this. They push teamwork and collaboration, but at the same time they know the details of what every individual is doing and how they each contribute to the whole, and they can ~accurately compensate/promote/terminate. This prevents demoralization of team members. Team members psychologically need to be recognized for their individual contribution.
These bosses tend to be competent ICs who became team leads, they are best positioned to judge the ICs they manage because they themselves are masters at the craft.
I think an underrated way to deal with this phenomenon is proper self-marketing. Talk endlessly what you have done to prevent catastrophe. Describe the avoided catastrophes vividly so that people get a clear picture.
Sometimes you can't quantify actual avoided things. For example you can't prove that a regulation stove actually prevented what could have been a kitchen fire.
Work for yourself. The fewer bugs you create, the faster you iterate. And the faster you iterate, the better your chances of finding product/market fit.
1.) Create a spreadsheet with all of the features of your group's/company's products(s) listed in rows.
2.) Create a column for every team member in the group and highlight the lead developers for each feature.
3.) Then ask each team member to add a checkmark in their column for every feature for which they would be willing to be on hook for 24x7 triage pager duty.
Over the long term - the most valuable contributor(s) on the team will be the one(s) with the most checkmarks next to the features they led development on (i.e. they write understandable code and document well) combined with the most checkmarked rows in their column (i.e. they proactively seek to understand other peoples codebases).
Kind of ignores the risk externalities have on projects no? If you tried your best to deliver something, but another teams priorities changed, or the business/market shifted, or the project got bogged down in planning/politics outside of that team members control, or if the team needs to integrate with a system (external or legacy) that’s out of their control and is notoriously flaky and torpedoes their velocity and morale.
Don't help anyone too much until a manager asks you to help. If someone is really pressing help, tell them that you are busy, but they can ask the manager to tell me to help. This way, you always get maximum credit. Every single manager that I have worked for claimed they were "different" and did not suffer from it. Usually, I give them many chances to see my value. After I am overlooked too many times, I revert to this selfish mode explained previously. In all such cases, my recognition from the manager improves. Sigh.
I don't know about that, you might want to receive some help yourself right? In some cases it might be better to work at a place that values cooperation more than competition. You sound like you would thrive better at such a work culture.
However, often leads just need something in order to know what you are doing, especially managers that don't really closely work together with their team. It can help to just mention it to your lead. Because it is easy to see what you done, but not how you have helped somebody. Just mention it in whatever recurrent meeting you have. And if helping out takes more time, I'd say it is only fair to have whoever is responsible for delivery involved in prioritization, because then it is at the detriment of whatever you are working on, which might be more important.
Often, it takes less than 15-30 minutes to help somebody get unstuck. I wouldn't enjoy working at a place where people would refuse to help me with something like that, or I would be so pressed to achieve things that I can't spare 30 minutes out of my day to help someone out.
Life and its rewards aren't perfect. Work with honest, intelligent people; genuinely do your best; your days will be much better and the odds will be with you.
On the same note, it's so discouraging when someone who is not/neither takes over, those people leave (one way or another), and you have to start the search again.
Or when someone is intelligent enough you don't figure out they're not honest until it's too late. <shiver>
Anyone who's been around the block a time or two will accumulate those experiences. I hope I've not been jaded by mine.
This is good advice that’s well intentioned, but (sorry), it can be interpreted as elitist, and in a way that’s detrimental to the reader.
I am no way suggesting that this is the intention or belief of the parent, but while I’ve got more miles on my odometer than I’d prefer, they’ve informed me that “reasonable” is better than “intelligent.”
My god how I’ve found that working with reasonable people is so much healthier, more productive and rewarding than working with the unreasonable* intelligent folks.
*I fully grant to my current and former colleagues, friends and associates that I have been irredeemably unreasonable any number of times. Consider this a small thanks :).
Great approach!
For the past 2 years I keep experimenting with different ways to track day to day productivity. It has helped me tremendously in assessing how I should feel at the end of the day about my work day and the amount of time I put towards work. (I tend to overwork myself significantly). It is of course a lot harder to apply this approach to individual productivity tracking across my teams. But I do now believe, based on self experiments and within smaller teams, that there is generally a lack of visibility of knowledge worker productivity. Especially within larger companies. WFH and hybrid makes this more crucial to have for both management and self assessments on the IC level.
This is a great answer. The only thing I would add is that you can take mental note when you accomplish something that anticipates and preempts future needs and challenges. It is ok to make brief mention of these things from time to time when appropriate. If nothing more it helps coworkers and bosses realize that you are putting in some thought and effort for the needs of others and to make things go more smoothly behind the scenes.
Doing this effectively requires tact. Try to mostly bring up your silent efforts casually, and be judicious about how often and in what situation you mention it. If it gets interpreted as an "I told you so," an excuse, self-importance, etc, then it will probably do more harm than good in terms of your standing with others.
Might be helpful to some degree. But won't save you if your organization's culture simply doesn't value your work or more generally doesn't care about proactive, methodical improvements that have no flash or immediate payoff.
I'm going to add: in a place where they have the bandwidth to be honest and intelligent. Sometimes good people are in bad situations. It's important to recognize this.
Another perspective: if you are not in a position to change the rules of the game, understand the rules of the game and play that game, not another one that you would like to play or that you think is more fair to play. Alternatively, if it is not the game you want to play, find a place where that game, or a similar one, is played.
Your managers need to understand the work that you're doing. When I review someone's code and I see them fixing something potentially catastrophic but that never happened, I definitely remember that and congratulate them. When I see a team member ask a question that makes me or the team rethink implementation strategy. That is very valuable. When a code reviewer finds a potentially critical bug in my code, or just a nuanced bug, I remember and appreciate. Because I fully understand that those are potential p0 or p1 issues.
And great engineers are consistently good at this type of passive, keeping the lights on work, but it does not reflect on their quantifiable work, so orgs do not include this in performance reviews. It is up to your manager to recognize this and advocate for you.
Even when they do recognize it it’s unlikely to be weighed the same as the engineer who heroically stays up all night fixing a critical issue.
Especially with many organizations focused on data or metrics for performance reviews or promotions being able to say you fixed an outage that was costing $X million per hour comes across much better than a vague counterfactual notion that your high code quality prevented Y such outages in the first place.
I went on holiday for 2 weeks. Came back to find the team had spent a week on a 5 minute task that had been clearly documented. Nothing had moved forward. Spoke to my manager that day and from that day forward it was clear what my value was. Eventually lead the team and helped the team develop processes and initiative so they could be productive in my absence.
Up until that point I knew my value but I don’t think anyone else quite got it.
If your solutions tend to be more stable from the get-go, it will get noticed at most places. There are always going to be that odd company where that's not true, but in my experience, simply doing higher quality work gets appreciated.
Well it's not 'clear' when every promotional process I've encountered doesn't reward such forward-thinking preventative maintenance compared to someone who postures themselves as a hero going around putting out manufactured fires.
Good and bad times will do their usual sinusoidal dance but your slope will be pointing up if you zoom out.
There will be unfair promotions outside of your team, reductions or freezes in yours (companies have tendency to throw people at problems = inflate dysfunctional parts; if you have well performing team - they won’t let you grow short term; just wait it out, they will eventually).
Me neither. And I've had a similar issue with the first case. You do your work and help someone finish theirs, people still think you only did you job (even if people do credit you, you still have normal impact).
The psychology of workplace is quite subtle and almost backward. There's a motto "squeaky wheel gets the grease" .. to the point I'm thinking of trying some games. Like designers who produce crappy options and one good to toy with higher ups thinking they decide. There are some ideas in that vein so you flip the relationship and benefit from it instead of bleeding.
Value is in the eye of the beholder is what I discover. Got to play the game.
If you are an IC, you play it up with the right people. Those right people fight for you instead of the other way around. When the “we need to cut the fat” talks come around, suddenly you are not on the chopping block. But that also assumes the people fighting for you are not on the chopping block.
Soft skills across the industry is highly under valued. Knowing when to pick a fight or hold your tongue is also important (ie, reading the room).
I hate it but you got to do what you got to do to get that cheddar
It's possible to phrase the value proposition to align with what you want to deliver.
You can phrase it as "I help teams deliver on time by making sure they don't get stuck." Or "I increase a team's velocity by preventing mistakes that stop work."
It's how you sell yourself and then, how you tell the story of what happened. Being able to tell a compelling story is important.
The issue is how do you provide the evidence for that. In other terms, how do you differentiate your effect from that of Homer Simpson's Bear Patrol and Lisa's tiger-repellant rock [0]?
My anecdata is to be vocal about your works, like having short knowledge sharing sessions to show it to your team/manager. Don’t be afraid of little things, your team would learn something from it.
Modern “agile” systems don’t work well with you. They want you to deliver X points of features in 2 weeks. Sounds like you’d be better off as a consultant? But then you need to do all that biz dev stuff! Or find a great CTO to work for (or be the CTO) who understands. You only need one job.
In the end people ended up simply inflating the value of stories.
In my previous team I had to argue with a person that 2 story points to add translation keys were ridiculous.
To which the EM argued that 2 was okay because he also needed to write few accompanying tests.
I have nightmares thinking about this stuff. It was literally quicker to add the translation keys and that pointless test than to even discuss and vote the story and debate the points..
Think of it this way: when a principal engineer is hired from outside the company, what is the forecasted value of that engineer? How was that determined, since they have yet to do anything? If Andrej Karpathy joined your AI startup, people won't say this person has unclear value. It's because of the impact they've had previously and what other people have to say about that person. Value is your reputation.
An adjacent problem is how to know that you've really prevented issues/others from getting stuck with whatever you were working on. No system is perfect and engineering is also an art of knowing when to skip which corners. If nobody sees value in your preventive work then maybe there isn't any?
I am dealing with similar. I can provide solutions 10x faster than anyone. Everyone looks up to me, and are dependent on me for solutions to everything.
However it is clear that noone is improving and that the process IS me.
What I've been trying to do is make sure that my scope and role is fully clarified, and any "extra" activity that I perform is documented and flagged. Anything that becomes a "common" activity implies a missing part of the process - be it a role that is missing, or a skillset that is lacking.
Before you start thinking that it is pretentious or self serving, it's perhaps the opposite - you owe it to the process and the team/business/organisation to make them see what you're putting in, else they fail to find the gap.
It's not paid off yet, but hopefully it will yield results within the next 6 months.
One piece of advice, I found myself this person once - it’s the road to burn out city to be the critical path answer to everyone on the team’s challenges. You sound like someone who is very generous with your time and support. In my experience once people have found a critical path there is no point at which they “stop.” This isn’t because they’re trying to hurt you, they’ve just found the answer so to them it doesn’t seem wrong.
It will take even more of your time, but you ought to consider practicing giving less answers and asking more questions of those who seek your help to try to help them unpack the issues themselves. It will feel more tiring at first, but you’ll gradually help the others learn and also create a small bit of friction that will encourage them to try their own solution or two before seeking you.
Management asks are separate/ they can actually reward you with compensation and promotions for this extra work. But the team asking for help won’t stop when you get more comp unless you start teaching that you’re not the answer.
If they don't pay you to fix those problems then don't fix those problems until they start paying you for it. You'll have to deal with the psychological torture of watching things go horribly wrong. So it's up to you how to deal with it.
Good leaders. Or, worse: Consultants. That is, it takes either an outsider or a self-critical leader to affect change? Those doing the planning are always optimistic [0] about their decisions, processes, evaluations, and progress. https://en.wikipedia.org/wiki/Planning_fallacy
Other problem on similar spectrum, when you have a good performance and your boss notice you, their expectations can increase and increase until you unable to deliver, then suddenly you're bad.
Agree with your point, solving problems gets you points, avoiding them doesn't. My cynical view of the headline is, that a lot of people do get credit for solving problems that never really existed, simply either fabricating (intentionally or not) easy to solve problems or vastly overblowing problems they just happen to have a "solution" for.
If ypu successfully fight a fire, you are a hero, if you prevent fires it is just normal and nothing special.
I'm currently in an unrelated field, but with a similar tension of unclear value while things work. The way to deal with this has been to keep a tally of checks on important systems, whether it's daily, weekly, monthly, with links to undesired outcomes should the process that has been checked fail. This list of you checking the processes is one way to "prove" your value, at least for ppl who wouldn't necessarily understand naturally.
Security/insurance paradox: when you do some jobs perfectly, it feels like you're doing nothing. Sadly, the only (ethical) way to counter such narrative is to spend a good amount of your time making reports showing how well everything is working and how it's not something you can just operate autonomously without disaster (to answer the inevitable "well the job is done, what are we paying you for?)
1. Donald Rumsfeld was right when he said “A’s hire A’s, B’s hire C’s”.
2. Environments where people can’t tell the difference between 1x and 1000x engineers lead to performative work and arsonist firefighting.
3. When folks in your organization can’t recognize (and reward) high impact, it’s probably time to get out, largely due to the first two points.
It's mostly a question of finding organizations that recognize and value that sort of contribution. It also helps if you track who you work with and what you help out on every day, so when review time comes along you can list not only your 'regular job' wins, but all the wins you helped teammates get.
Presumably that's not -all- you do? Or did you sit around for 3 months going quietly mad waiting?
Assuming you are doing other things make sure that is visible.
Also why did your colleague wait 3 months to ask you for help? A couple days would be OK, a month would be crazy. You should poll them and not wait for an interrupt.
Depending on your context, if you are at the top of the foodchain as described here, minus the recognition for it, one option is to simply add grist to the mill. Creative problems you can leave for the team to sweat and for you to come and slamdunk.
The problem may be in the relative value contribution. A senior eng that rights a project on 3 days which had been going on for 6 months is incredibly high value/he work.
A Sr eng who spends 6 months coaching a junior engineer has much lower value/hour.
I think there’s a “this is life” aspect and a “this organization is immature aspect”.
A mature organization respects the process that prevents getting stuck for 3 months. But… they may be more stable and less nimble. Boring orgs don’t like heroes.
Unfortunately one of the best ways is to toot your own horn. If you work with good folks, it will be appreciated for you to present a tricky problem you solved, or to present a design you put forth that would save everyone time.
The lean mantra "Make work visible" is not only good at an organization level but also as professional feedback level. Although if used openly for promotions it will cease to be a good metric.
Think that it is the management's fault if they misjudge you. They have the chance to judge you correctly, yet they misjudge you. Their fault entirely.
At least you're likely not a person who works at a freelance agency where you purposefully do poor work or break things, so then you can bill more time "fixing" the problem - until the client gets the wiser at least, or there's another more oblivious client you can grift much more easily who doesn't know better?
The above seems to be how the top billing agencies on Upwork function, with fewer staff you can grift more people because the hours being billed aren't honest, and so you can have more clients and reach the "top" earnings position faster and more easily; with the same practices existing at least on one of the two prior platforms before oDesk and Elance merged.
Easy - half a$$ your code and when it breaks - swoop in, "fix things" (actually do it right) and play the role of hero! (I've seen so-called "Rock Stars" at places I worked do this over and over)
In high school I actually had a friend teach me this in a humorous way. In football we did sprints (the kind where you physically run) at the end. He said -- half ass the sprints. Then on the last sprint, run as fast as you can, you'll have more energy left, and when they see you running fastest you'll get all the credit. He was doing it as a joke, but lo and behold he out runs everyone on the last sprint by a lot, and the coach specifically pulls him aside as an example of how everyone else should carry themselves.
He's one of my favorite pranksters. But needless to say, the football team was not very good.
There's a famous green text on 4chan, where a user tells he got a sysadmin job which was so boring he started crashing stuff left and right and blocking an entire office even for entire days.
By the end of the day he would plug back something and come out the "servers room" saying he fixed that and get everybody's praise. Even got him two raises in the span of 18 months.
That's how crazy it is.
I know a variation of this sort of story, where a good sysadmin/DevOps team was halved and then the problems started. The company didn't have those issues exactly because they had a good surplus of eyes to handle everything.
Another option is to work at a consultancy where you hop aboard a new customer project every 3-6 months. Approach every customer as an opportunity to do some resume-driven development and pick a bunch of untested new technologies to experiment with. Be sure to do at least a couple of presentations to tell everyone about the hottest new things you are doing to bring value to the customers. Leave the project once it slowly starts sinking and then just keep hopping from customer to customer. You will be far away once the sea water starts coming through the windows and the non-technical people directing these projects will never figure out what you did.
I have seen that this is one of the most efficient ways to advance your career especially in larger consultancy companies with hundreds or thousands of different customers.
Don't do it right, double down on the flaws. You had a 3k SLOC single function (all in main) C program to do something that could be expressed cleanly and clearly in 200 SLOC. Some specific sequence of inputs leads to an error. Instead of tidying it up, removing the repetition that led to the mistake, you copy/paste everything again and add another 100 cases to your various switch/case statements (actually you use if/else because switch/case might make things clearer). The specific problem is solved, but in a year another buggy code path will be discovered and you'll have another chance to play hero. In 5 years it'll be 50k SLOC of C all in main, that could have been under 1k SLOC (still all in main). No one else will be able to fix it but you!
It is more easy: do your job right, but do not comment, fix or somehow improve work of your colleagues. Let them fail, aknowledge failure and only then come with fix, get all praise.
Never point to the possible issues at the code reviews, retirement refinements etc.
I have seen so many people do this wherever I have worked. In fact there is another thing that they do i.e. when management pulls in the schedule they will meekly agree to it because they already know they are going to half-a$$ their code. Now they are heroes much before as they are "yes-men" and will be targeted for promotion. Swooping in to fix things just adds to their already loaded credentials.
The ones who are honest and actually disagree are banished and their lives are made difficult. They are called all sorts of names the most important being "not a team player".
This is exactly what happens at many software companies. People,sometimes under pressure to meet deadlines, apply totally crazy fixes to stop one problem and create new problems.
Hah, that was in my archetypal description of the fake 10xer, that somehow they are able to swoop in and fix their spaghetti code, come what may, while no one else can make sense of it:
Having people thank me for being 'on top of' a problem that I already know about because I'm the one who pushed the button that broke it... It's gross. It feels gross.
Much happier when I can fix something quickly because I have some tools or logic in the codebase already that lets me do something quickly by being covered with sanity checks so I can zip along without driving us off a cliff.
I was just considering it, but never had a chance to see it in real life.
I'm dumbfounded about all this, you spend years learning and reading books about the craft but the only topic which will matter is how you can con the game ?.. weird.
I don't do this for one reason - I cannot hold others to a high bar if most of my code is half-assed.
I usually write my best code to show others it's possible to write good code and ship on time, with exceptions of course (and I usually document in code why its half assed)
This is a slippery slope, IMHO. It can easily get you to be the next "tactical tornado" of your team (cf. John Ousterhout, A Philosophy of Software Design).
For i in employment ensure there is someone who checks in with me every month when they read this line of code, else, stop code from running so they come to me.
Play to your leadership. Exceptions apply, but female bosses tend to appreciate averted crises more than male ones.
If your boss is male, let shit fail. You'll get hero points for responding to incidents.
A female boss will quickly suspect incompetence if things keep breaking (i.e. see how far half-assed DIY home repairs get you with your wife before she loses patience with you). Your hero points will come from mitigation.
This paradox comes up a lot in security. IME in this particular field the gender stuff is less relevant since everyone is paranoid. But when we run stuff up to C-levels, it's only the female execs and lawyers that really stop to consider possible issues-- the men just dismiss everything until it happens.
Another variation on the problem is the over-allocation of resources to preventing problems which actually happened once, versus those that are more severe but haven't happened yet.
This is a management problem, because no-one wants to be accountable for a repeat incident even if it was rational to be working on something else more important.
> Another variation on the problem is the over-allocation of resources to preventing problems which actually happened once, versus those that are more severe but haven't happened yet.
I've heard this called "institutional scarring" in a blog post somewhere. The idea is a small wound can be replaced with tough inflexible tissue. The jist of the blog post was that just because something happened, doesn't mean you have to change things to ensure it never happens again because that can be an over-reaction that really burdens your future. Accept that loss and that it just might happen again, but that may be better than onerously preventing it with certainty.
e.g., You were tasked with working on a ball of mud and it was miserable, so the next system you get the chance to build has to be the most scalable, modular, and cutting-edge thing ever, just to be safe.
Same thing constantly happens in governments, too. "Oh no, something that's been done for 200 years now caused issues once! We have to restrict/regulate/bureaucratize/outlaw it immadiately!"
There's a ton of reactionary legislation on the books that exists pretty much entirely because politicians wanted to be seen doing something. It's mostly crap.
A lot of gun legislation is like this. Whatever your view of guns is, there is no rationale for a "safe hand gun roster" like California's where the Glock 17 Gen 3 is on it, but the Gen 5 -- which introduces a basic safety improvement for left-handed people -- is not.
This is probably a bit too cynical; it's not just "politicians who want to be seen doing something"; the public often wants something done as well. And "we didn't do anything after that incident from ten years ago and now it happened again" is not a good look. Complex stories about trade-offs and the cure being worse than the disease often don't "play well" in the media, especially not with the opposition demanding that "something could have been done!" And corporations tend to be pathologically risk-averse.
Politicians, the media, corporations, and the public all have a part in this.
This is basically how all bureaucracy comes about. In startups everything is so new that problems haven't had time to happen. In big tech, because they have a tremendous knowledge base of previous incidents and the resulting safeguards, every single step seems mired in bureaucracy.
The thing is: it's very easy to play up the likelihood and severity of problems that are entirely imaginary. This can be just a bad habit, but also a deliberate tactic. In either case, a lot of effort, time, money, etc. is wasted.
Waiting for something to actually happen before allocating resources to preventing it is (to some degree) a rational policy.
I have worked in finance tech all my career, so I'm not sure how other orgs work, but this is extremely pertinent. Large investment banks react exactly like this.
I spent a miserable year trying to convince people that they were over-reacting to an outage and there was a very simple solution to the exact problem that occurred. But when senior managers see their jobs at risk because of a repeat, they'll mandate that the entire department review their code for similar issues and remediate. They'll also listen to the loudest voices who somehow come up with massively over-engineered solutions.
Another example, we had a password expire which caused an outage on our trading stack. The amount of effort that went into stupidly convoluted hand-crafted solutions ensuring this "didn't happen again" was laughable. And in the end, after more than a year of work, the whole thing was abandoned in favour of a much simpler centralised solution that should have been done from the start.
Reminds me of a place I used to work. Every time they asked for feedback, my refrain was "nothing here gets prioritized without a PIR (post-incident response)". Towards the end of my time there, whenever a PIR-related ticket showed up, I would mark it as a duplicate of whatever actual ticket I had dying in the backlog that would have prevented it. It really hurt team morale to have no influence in preventing foreseeable issues in our domain area. Most of the rest of the team had even stopped suggesting improvements altogether because management refused to let us pull our own tickets in.
This is wonderful description of the kind of hellhole corporate Scrum is turning into.
Agile was literally about doing things quickly and quick cycles of capability improvement. But Scrum is a worse version of the planning processes it meant to replace!
If anything, the way scrum lays out the work into immediate problems exacerbates the cycle. In the long run it just turns into a ticketing system where fires get pushed up and technical debt gets pushed down.
It even spits out a super easy-to-track, meaningless set of efficiency numbers for consultants/executives to min-max!
I still can't believe a scrum master is an actual FT job - last company I worked for had one FT scrum master for each project - so they ran the meetings and as far as I can tell, did nothing else - what the heck do they pretend to do for the rest of the week?
I went to all the same meetings, and was still supposed to actually develop software in between them all.
Agile seems to have ended up as a way for PMs to report upward to senior managers who also need to report upward.
You can see why. These people have to decide what will be worked on out of all the potential things that can be worked on. Someone has to make that decision. Will this feature make us money? What about that bit of work that doesn't add a feature but reduces resource costs? What about tech debt that I'm told is building up and slowing the ability to deliver features?
I'm not a senior manager, but ultimately someone up the chain is responsible for the company surviving and making money and paying our salaries. They are just like you and I, trying to make decisions based on what little information that can glean. So part of that is "what will this cost and how much will it be worth" vs "what will that other thing cost and how much will it be worth".
To that end, they need some way to estimate this. They latched on to agile as it was being promoted by tech as a way to do this. Whose fault is that?
And so with that came all the frequent estimations, are we on track, rituals, etc. Some people don't believe this should naturally follow. I agree. But somehow all those rituals have become part of the cult.
We abondanded scrum. We abandoned refinements and estimation of stories and story points etc. Now we meet with a PM once a month formally and as a team perform a t-shirt size estimate on where we are. Otherwise we update her as frequently as she asks (which isn't often) or we want. This gives power to us but because of that we're conscientious and make sure to inform her timeously when things are looking sketchy or whatever. Yes, we still have to give "estimates", because ultimately, senior management want to make decisions, but it is otherwise quite lightweight.
I found the idea Scrum being efficient when done right was true at one company I worked at. Everyone was committed to the process, the Scrum team included debt as a priority for 20% of the effort and everyone had a fairly accurate velocity that we could bake in an additional 20% of your work as your own interests, so stakeholder priorities filled that remaining 60%. Then we would pivot in some Sprints if any epic/team goals required a push to get done or other emergencies/bugs that required a change in priority.
I don't think it can't be done right, but it's been cargo-culted pretty hard at Fortune 500 companies. But the best scrums I have ever done have just been literally post-it notes on a whiteboard - I think Jira ads a level of complexity that is unsustainable for most orgs.
I've found it works better when you "build up" to Scrum by incrementally adding processes when you identify issues and relaxing the process if things are working well.
Adding a bunch of processes because you want to add processes doesn't provide value.
This is why in many corporate cultures it pays not to proactively stop a problem that you know how to fix if the problem is not in your immediate problem area. Let it be noticed, let it become someone else's emergency, then fix it. Much better path to a reward that way. Of course, you should also be planning to leave said organization, in the long run it won't do well.
I wouldn't be so sure. As an employee and team member, you'll be better perceived if you do your share of performative firefighting every one in a while. I wish it wasn't true, but it usually is. (Though at remote teams, it's less and less true, with the remote teams I was on, nobody really cared).
"wow, last week we had such nasty bug, impossible to track down, also caused production reliability issues. Tom stayed up all night, and finally pushed the fix at 3 in the morning."
Now, if you then say that it wouldn't have happened if 1. Tom didn't overcomplicate the system, 2. Tom learned our tools properly, 3. Tom actually understood the requirements and at least tested his changes at least once manually, 4. Wrote good code so that it's easy to troubleshoot 5. Wouldn't have forced a rushed PR review together with the product owner.
Nobody ever gets credit for fixing problems that never happened (2001) [pdf]
I'm reminded of this every time I see some YouTuber or other social media click bait claiming that the Y2K bug was no big deal.
The reason it was no big deal is because thousands of graybeards, like myself, stayed up many long nights for months ahead of time making sure things would work.
I still remember the tension during the countdown to midnight UTC. Then the tension rising again during the countdown to Eastern time. Then once more at local time.
Only when it was 2000 in Pacific did we unclench our cheeks.
Depending on which Y2K bug you're referring to, this very much happened and is an easily believable scenario. I don't think this is a comparable situation.
That's not what I remember programs looking like. You would have counters that went to 99 and no more because the UI wouldn't accommodate another digit, and programmers didn't know what they were doing, and figured it would be clearer if the software reflected its UI and couldn't handle over 99 at all, invalid UI means unrepresentable state.
Lots of stuff didn't care about the day, too, like you'd get 0997 and that meant September 1st 1997 because the first (or last) of the month was inferred, and you'd get goofy logic around year++ where you add 100 years whenever you want to increment the month, and the whole thing is written as a modulo 1200 but whenever the date is about to be 0000 you look at what's stored in year instead and add one to it instead of 100, because that way you have less variables to allocate and every "little bit" counts.
Everyone knows now, but lots of people writing programs didn't have any prior art, they were just good at messing with computers and sort of fell into programming by accident.
It was also to do with how dates were stored in very old databases that were conceived in the 70s when storage was at a premium - a lot of them just used two digits for year.
In about that same timeframe, Y2K is also really good example. Nothing of note really happened. But if people had just ignored it a lot of things probably would have.
Not "probably". I was involved at ground level fixing code for BP in '98. Can say without a doubt that bad things would have happened. And not because my salary depended on it - there was no shortage of opportunities to work on other things, obviously. It was legitimately an issue that would have crippled the energy industry (major players, and all their many, many dependents). I infer from this experience that the impact would have been the same on many other industries (e.g. finance, resource development) directly and indirectly.
But it is a great example. I still run into people who recall Y2K as an example of much ado about nothing. No, no!! It wasn't an issue for you because many people worked hard to prevent the problems.
Those problems were not terribly complex just pervasive and critical, requiring a high LOE. It wasn't like a huge moon-landing engineering problem to be held up as some accomplishment of humanity, more akin fixing a lot of dumb Challenger o-ring problems before a blow-up.
This is a very important case. There was an enormous investment in fixing Y2K, starting long before the day in question (e.g. financial instruments with expiration dates after Y2K had to be fixed before they became due), so on the day there was only a smattering of minor residual bugs.
There were a few jokes about it in the newspapers but by and large the public just ignored it.
I work on climate and hope / hoped that the same would happen, but it's looking like it will soon get everybody's attention regardless.
Y2K would negatively impact companies' bottom lines if not dealt with, directly, in a short and known timeline, in predictable ways.
Climate change will impact companies in different ways at different times. Oil companies, for instance, are incentivized to ramp up production as much as possible while also diversifying.
The way to get all companies to take appropriate actions is to not allow them to externalize costs. That means a significant carbon tax.
I'm sure I'm not the only one that can't wait for the 2038 problem to get closer. bank accounts gonna be jumping like a 3-peat Micheal Jordan when those consultation fee checks start rolling in.. provided the Boston Dynamics dogs haven't taken over yet
Since you "work on climate" I am sure it got your attention that there is a massive effort happening all across the board. R&D has made tremendous advances, whole industries are investing to get away from fossil fuels, and governments have ongoing programs to accelerate and support lange hydro, solar, and wind projects.
Some countries have not heard the call yet it seems (ones that like their role as perpetual victims and blame everyone else but themselves), but even just five years down the road, the world will already look very different for all the effort that's being made.
In strategic sysadmin and cybersecurity this is staple thinking, and a
useful part of what I teach now.
Problem for vigilant thinkers is the rewards, as explained in TFA,
are perverse.
People are content with technology to be magical. They don't see the
millions of people quietly repairing it and planning to keep it all
going smoothly.
To "bosses" cybersecurity only bring them problems, and cost them
money apparently doing nothing. The same for any defence force, until
you need it.
Did some nice interviews for international sysadmin day last summer
[1].
I didn't worry. Even took a trip to a different country. Though I can totally understand why some people (know you're joking) might have played it more conservatively.
Yeah. I've heard folk who should know better (looking at you, John C. Dvorak) call Y2K a nothingburger and suggest the effort to prevent problems was a total boondoggle.
Disclaimer: I was part of that effort. The funny thing was that I was brought back to a previous client to fix something that my previous efforts had literally caused. Took me 20 minutes to fix it once I saw what the problem was. And then there was a "while you're here, can you take a look at this ..." That lasted about two years until they closed the department and moved it to NY, NY.
"we were all worried about the ozone and nothing happened, this is just a repeat and people will stop talking about it any time now"
and my response is "'Nothing' happened because the governments acted fast and turned it around so now it's mostly just boring stuff like catching some factoryy that decides to go cheap and use banned chemicals to save a buck"
I thought this was going to be about Y2K, since it was written just after.
I worked for years in the late 90's on Y2K projects, helping to stop the UK's critical infrastructure from just stopping at midnight. Wales would not have water or gas without our efforts, for example.
But I've heard people since say things like "why did we spend all that money on Y2K, when it clearly wasn't a problem since nothing happened?" and even "Y2K was a hoax invented by the IT industry".
We won. We successfully stopped the Y2K bug, and it was hard work, and it wasn't obvious that we had caught it all by midnight. Yet rather than celebrating, some people saw it as evidence that we had been ripping them off. People are weird.
I know of several other issues like that. I myself made sure the highscore in a video game I was working on would work :-)
The thing that annoys me is that this is best case scenario when it comes to climate change. If we do in fact manage to avoid the apocalypse, all the "climate deniers" are going to feel vindicated.
Suppose on Sept.10, 2001, you were an airline executive who successfully forced installation of jimmy-proof, secure steel doors between the cockpit and the cabin?
The only "credit" you'd get is "This guy cost the airlines god-know-how-much money, and for what? An imaginary threat!"
The book "The Black Swan" by Nassim Nicholas Taleb contains this very example. It is a great read on why the improbable events have the biggest impact.
Having spent several years as a manager, as a once-again IC I now include lack of problems with my features in production in my written self-assessment. I'll often refer back to major features I shipped ~2 quarters ago with a statement like, "Project Foo continued to function as expected, scaling as designed under load while incurring zero production incidents." I'll often include the words "engineering and operations excellence" and "commitment to product quality" when doing so.
- Problems that people don't realize are happening (where people cannot relate visible symptoms to their root causes). You only get credit for addressing visible symptoms. Though, disturbingly, you will get credit even if the fix is only temporary... This creates an incentive to only address symptoms since not solving the root problem allows you to get credit over and over again for fixing the (recurring) symptoms... In fact, people will praise you even more as a 'Relentless hero who dedicated their entire life to the problem' LOL.
Also, nobody gets punished for creating new problems if the visible symptoms cannot be traced back to the root cause. That's why I oppose complexity in all matters - It obscures root causes and sends everyone off on a wild goose chase addressing symptoms... Complexity turns everything into a frustrating game of whac-a-mole.
The person causing a problem can even get the credit for solving symptoms of that problem.
This us an incentives problem, plain and simple. Its extremely difficult to quantify things that don't happen, and because we've been building our economies and societies around a heavy focus on quantitative metrics we end up only incentivizing people to solve problems. Even create problems on purpose if you need to, you'll get credit for fixing them.
Human Resources departments often run into this exact scenario. Many of the tasks that fall into HR are hard or impossible to quantify, and if done well they're also solving problems before they manifest.
I wonder if this is why some managers seem to go out of their way to generate chaos. I have no patience for it, and judge them harshly for it. They don’t get any credit if my book for “solving” problems they pulled out of their back side or willfully ignored planning for.
I once worked with a test manager that was effectively judged by how many bugs were found and how many tests were written. He was actually a nice guy just trying to play the game given to him, but if you didn't know this you would think he was doing exactly as you described.
There will always be a few bad eggs in the mix, but I'd check the incentives forced on them before judging too harshly.
I work on a large distributed infrastructure. I always joked that my team's projects and people's careers are outage-driven: the only time we become important and people get opportunities for promotion is when where were big-enough outages that executives have to invest heavily on reliability or scalability. Other time, we are just minions who must listen to and serve feature or product teams. Nobody listen to us when we ask a product team to implement a reliability contract in their shining product.
Pre-outage improvements, reliability defense in depth, eliminated scalability bottlenecks before they are hit, are all ignored by leadership and the company: it is just human nature that even though they understand you have to prepare for possible issues, if it hasn't happen yet, you won't take it seriously. I've seen this in many internal performance reviews and promotion committees. People who haven't ever got bitten badly by an outage may call these premature optimizations.
My background is systems operation. The exception I could see to this is when there are rewards/incentives/etc for uptime or non-change-window uptime. Usually this would be coupled to meeting the goal of 4 9's or 5 9's of uptime, meeting SLA requirements, etc.
Granted, in my time working for a major 3 letter tech company on account with a major finance company my highest exposure to VP/C*O types was fixing the fallout from a whole chain of corporate culture related mistakes. If I hadn't already had a job lined up elsewhere I probably could have gotten a good bump out of it. I doubt that would have happened if I'd identified and fixed the problems. Part of that busted corporate culture. We didn't have metrics around uptime/SLAs there though which considering the clients business was pretty ridiculous.
I run an ad agency. I only touch ad accounts with pre-existing spend snd results for that reason. If I take the client from $200 CAC to $100 CAC I'm a hero. If I go from scratch to $100 they don't know my value and it's much more work.
It seems like many of these TQM discussion miss one key factor of TPS.
Respect. Engineers and management respect line workers. They have a word for the expert line workers, ("craftsmen"), Takumi. https://www.allaboutlean.com/toyotas-takumi/ and Takumi have a valued place in the company.
Without that basic level of respect... i dunno.. seems like it wont work
I always get cynical when I see awards for software releases. Someone always gets an award for staying up late to fix a last minute problem. Normally the person that should have written good code in the first place so they wouldn't have had to come in late. The award should go to the person (likely people) who never got called in at all in the first place because we didn't find a bug in their code.
I'm both a developer and a security consultant, both jobs deal with this problem. Being a security person, you go from being a no-one until something happens, then every eye is on you. And with the dev side, I work mostly on backend, so my work goes unnoticed always, the frontend dev gets all the credit, no matter how many thousands lines of code I commit, if something looks prettier or just works it's because of the frontend dev.
There's two choices, you either don't give a flying fuck and deal with it, or you go as loud as you can to make yourself visible beforehand and try to get credit for your work, your choice.
In our org, this was brought up. People who had problems, then fixed them, were the ones being seen and recognized by higher level management, and awarded for fixing the problems, when they, often, shouldn't have happened in the first place.
Now we have something akin to a "person that prevents problems" award. It's a nice gesture, but, even though I received one, I think it's mostly nonsense. You don't get the face time/experience presenting with the higher level managers. It's almost impossible to prove that you prevented catastrophes, by working extra hours, or doing the right thing. Nobody likes a "Hey look, I told you so!" sort of perspective, no matter how it's framed.
To me, this is the most gratification-limiting aspect of being a software engineer: time is required to test quality of foresight/design. Rewarding retroactively, for great past decisions, doesn't happen. There are systems I designed, when making a pittance for compensation, that's still being used a decade later, across multiple companies, almost entirely unmodified. That, now objectively, good work wasn't seen or recognized by anyone paying me at the time, and can't be seen by anyone now. I burned good time for a "good job me!" as a reward. As I get older, that mental masturbation seems less desirable, since it's usually at the expense of the surprisingly limited life credits that we spend on those tasks. But along with most other nerds, I can hardly stop myself because it's an innate obsession.
Oh, maybe adversarial salaries. You each, meticulously, go through the others code, trying to find as many bugs as possible, in your spare time. Each bug you find transfers a percentage of their salary into yours. End result: everyone's code is perfect, and everyone hates each other. :D
Reminds me that one time I found 17 databases in our org without backups configured. All I got was a thanks from some devops guy and no manager even heard of the problem...
My manager just told me I got a 3/5 on my eval yet again because in the past six months
- I did everything that was asked of me and
- I did a lot of things that weren't asked of me and
- I didn't catch anything on fire and
- I put out someone else's fire and
- I prevented several fires from ever being started in the first place,
but that's not good enough. The person who started the fire that I put out is getting promoted because they showed initiative in starting the fire. I didn't show enough initiative putting it out for them or preventing fires on multiple other projects. Apparently creating two of my own projects that never caught on fire also didn't demonstrate enough initiative. From now on I'm applying for one new job per day on company time.
Don't take it personally, managers at bigger companies have to evaluate their teams on a bell curve. The whole team can't score a 5 and someone has to be worse than the group.
That is called stack ranking and even though it is used in a lot of companies it is widely considered a bad idea. A lot of large companies don't actually use it.
Your manager probably just doesn’t like you, find a new one if you can, there’s no point swimming against the current - you’ll just get tired and get nowhere
And people who comment their code. Even if it is just a little. I swear, 90% of my time in coding is just fiddling around with functions to see what they do so that I can actually pull them together in the right way. If there was just a little documentation this would greatly reduce my time. It should also be in every team's and company's best interest because people hours are expensive. It should also be in the best interest of an open source project because more people will be able to build on your stuff. It should also be in your own interest because taking the 30 seconds to a minute to write those lines interrupts you and allows you to rethink and verify your method. So not doing it is only in the interest of moving fast and writing spaghetti code. But you're not actually moving fast. You move fast at that moment, but not in the race.
As a software engineer who pays more attention on security than average, I often feel my work didn't get recognized. For example, if I successfully prevented a supply chain attack, nobody would say thanks to me for a thing that didn't happen, even when they see a competitor product gets attacked. Similarly, most C/C++ programmers do not really care about integer overflow. But I know we are no longer in the world that computer viruses are everywhere(though ransomware are still common). The, who made it better? The people who are not satisfied with stuff that "just works".
This is kind of a problem with preventive ethics in general. It seems to me that given two bad choices, the best option is actually neither - that is, to prevent the choice from being necessary. To use the common example of the trolley problem, the best answer isn’t to choose one option or another, it’s to prevent the trolley from being about to run someone over in the first place.
Unfortunately the visibility of these non-events means that their heroes go unnoticed. Noticing them would require a fundamental reorganization of how we perceive time and the future.
I like to classify what to prioritise by how many standard deviations from the happy path a certain issue is.
For example, I’m working on a pay-as-you-go SaaS - so the probability of using the SaaS while your account balance is low is 1-SD away from the happy path.
Figuring out if the user has JavaScript enabled (SaaS is an SPA) would mean the user first signed up with JavaScript enabled, then disabled it. Id put it at 3-SD away from the happy path.
Keep doing this and work on the bugs nearest to the happy path.
Of course, the 1-SD, 3-SD estimates are just hunches.
This happens in organizations that don't measure and respect precursors or leading indicators of success/failure -- organizations that do not practice organizational observability, to coin a phrase. Which is to say, most of them, because this is not something one thinks of until one has suffered its absence. Your managers have to think like SREs, and be technically sophisticated enough to model and address this problem.
Does anyone know any great management books on this subject?
The High Velocity Edge had a lot about this subject.
The most memorable example was at a hospital, nurses would occasionally pick up the wrong vial. They always noticed before injecting, but there were lots of close calls that didn’t get tracked anywhere. Once they started tracking that as a leading indicator, it became pretty obvious that the vials needed a better design to prevent even the possibility of grabbing the wrong one. Also, if the hospital never realized this problem and someone got hurt, they would’ve ended up with a much worse solution like “double check really close”
This is a great insight, the idea that management has to look at leading indicators for the org itself. Makes me think that beyond line management, a manger’s job is to behave as an investor and steward of the org, instead of getting involved in the details.
Not a management book per se, but Working Backwards popularised an Amazon approach of thinking about the organisation in terms of mechanisms. You might find that useful.
I’ve always wanted to dig into Scarlet Ink, Dave Anderson’s blog, but never have, and I suspect it’ll be quite useful for you too.
> This happens in organizations that don't measure and respect precursors or leading indicators of success/failure
This happens in organizations that become over reliant on metrics and indicators of success/failure instead of recognizing that these are fuzzy concepts and indicators are just that... indications, not answers.
Take a leaf from the OH&S book, as OH&S is all about stopping things from happening. Many work places have a huge "Days Since Last Lost Time Accident" sign hanging out the front. It increments every day and the aim is to get it as high as possible.
The same metric could be applied to "Days Since Last Stuff Up That Cost Us". It would be easy thing to apply to an office door or cubicle wall.
Reminds me of the millennium problem. There was a post on the large forum asking if that issue was over-hyped because we didn't see the world crumble. The top response was that it was handled because we threw a lot of money at the problem. I tend to think that was right, but there's no way I know how to verify.
Just responding to the title, but I think this is why it's important to have leadership with real "IC" experience. If leadership has an intuition for what's hard and what the failure modes are, they will be able to correctly divvy out acknowledgement more often than a "pure manager" type.
The same thing occurs for many optimizations. Unless something is unbearably slow and you speed it up 10x or 100x; most people won't notice.
You might spend some time making an often called function return in 1 millisecond instead of 2 or 3 and some might notice that things 'seem' snappier; but no one knows why.
[Time Spent on Improvement] is shown having a negative impact on [Time Spent Working].
But [Time Spent Working] should also have a negative impact going back to [Time Spent on Improvement].
This simple mutually negative feedback loop, i.e. a competitive relationship, better explains the bistable situation where either [Time Spent Working] reinforces itself, while holding [Time Spent on Improvement] down, or vice versa.
Feedback loops reinforce themselves, but there is only so much time, so there will be a strong tendency for one feedback loops to dominate the time. (And since the Time Spent Working -> Reduce Performance Gap loop operates much faster, cheaper in the short run, and is much easier to reason about and manage, it usually wins.)
My first ever job was a placement during university -- like a year in industry.
It was a regular-joe IT support job at an eLearning company (if they still exist).
The FD walked past me one day, having just left what was clearly a deeply uncomfortable meeting.
He stopped in his tracks and asked, quite abruptly, why I didn't ever seem to look busy. I replied - with the arrogance of a cocky 20 year old - that it was because, right now, everything is working as it should. Nothing is broken and people are working and productive, which is because of the work of the IT department.
The company later become insolvent and was dissolved. So it turns out it might have been him who should have been busier, and he was just taking it out on me.
This is something that really annoyed me at a company I worked at for a while. And it was far worse - it wasn't a case of not being rewarded, people who planned properly and got the job done were punished for their efforts. If a project got behind they would take people from a project that was on track or better to put them on to it. I pleaded for a super green status for the best projects on the basis that the team would finish sooner and then the entire team could work on something else nnd get it done well too. They could take people from the next tier but that would be an incentive to do well. Not a hope.
The opposite. Department heads get fired for allowing blow-ups in their area, that should have been avoided. "Getting credit" means keeping the job managing the department via preventing problems.
I’ve seen it mean the department head is told to cut staff. If things run smoothly with 100 people, let’s see how runs with 90, then 80, and so on. The preventative measures start taking a backseat as the staffing issues are felt. The cuts keep coming until something falls over and the department head is pushed out after failing to maintain the service.
On their start date, I would to tell new grads assigned SRE / DevOps roles that their job was thankless and at best if everything worked fine no one would notice them. However, if anything went wrong, they'd not only be on the hook to fix but would do so under tremendous pressure. "Who wants this responsibility?" I'd ask.
Those who decided they were up for it usually did a great job, about half opted out at the end of the conversation and were put in roles more suited to their ability to handle stress.
I'm curious about views from the crowd here on how to apply the recommendations from this article in the context of a startup
I've worked in both heavy industry (mining /metals production) as well as my own startup for the last 7 years
I definitely empathize with the view that diligently investing in people, and taking time to build sustainable processes that focus on the mid to long term is the ideal way to go... But when your runway is weeks, not months, and it's ship or die, seems hard to justify such a Rosy ideal
One time I worked on a team where we got rewarded for being "bug heros", because we resolved a lot of bugs. It was kind of funny because we also generated the bugs.
SAP S4/HANA- the "new" ERP software is hardcoded to calculate depreciation until 2045. As programmers you probably know it could be untill year 9999 or much further.
But the consultants will not be out of work. Also a way to push customers to upgrade.
Artificial problem.
On a side note, I there is in my opinion a place for a new ERP disruptor. However making it is hard. Still even those small fish often manage to get money.
This is a failure of internal accounting. If maintenance is reduced, the value of capital equipment declines faster. Production reject rate increases. Accounting systems are not good at catching that.
Repetitive manufacturing operations which capture many metrics about what a steady production process is doing may capture such info. Non steady state systems have a real problem.
There is an interesting twist to the issue if you are doing development support, having a customer ticket queue. If you are diligently fixing errors before they ship you can profit from that. The queue shrinks, the stress levels sink, management is happy with you.
However, if the number of tickets gets too low over some period of time, well, management gets ideas that you have free capacity. ;)
Nobody is a strong word. There are many competent engineering teams that understand the value of strong foundations.
However the problem is quite real in large companies where promo driven development takes over and the incentives are shifted from building what is valuable to customers to building what is valuable to get a promo.
This is true in a sense but also it isn't. Our institutions spend a lot of resources on preventing things from happening, are successful, and the people who work in these institutions get a paycheque every month.
One could frame the entire US military budget as preventative spend.
Or because the prevention ends up killing hundreds of thousands of innocent civilians. Oh, and it turns out the reason you supposedly went to war in the first place is completely fake.
The corollary (if you’re cynical) is that you get Massive Credit for waiting to deploy a solution until the problem becomes a Major Crisis. That is, there’s a big incentive to to prepare the solution preemptively but not deploy.
A better way to think about it is that you can trust the code you test. Y2K was a problem because it affected _everything_ because anywhere a date was used had to be checked. For mortgages, yes, banks did fix that code earlier but they did it because someone noticed a problem in 1980 and fixed that specific problem, not because they transformed how they wrote code and checked all of the millions of lines of other code which nobody had reported problems with.
Roll ahead a couple decades, and now you have even more code with lots of interconnected assumptions and the guy who originally wrote it retired to the beach at Margaritaville with zero interest in going back to Cleveland to talk about date math. If you’re lucky, a pile of cash could change that. If you weren’t, they passed in a boating accident last month.
My first tech job was working for a COBOL vendor in the mid 90s and heard a lot from customers and other people in the industry. Much of it was cosmetic (rolling from 1999 to 1900 on a display, etc.) but a lot of it would have had real consequences: not accepting credit cards with dates in the new millennium, calculating interest incorrectly, sending people incorrect notices or failing to send correct ones, etc. I remember a someone at Mastercard saying they wouldn’t have been able to processed credit card transactions at all after midnight if they hadn’t done several years of careful work.
The two things which helped were that it got enough attention to get the business people to pay for remediation, and there were enough dates in the future that people got wake-up calls in different industries. Mortgages were the earliest but closer in you had things like credit card expiration dates which caused enough problems that people got serious about deploying updates.
These days, I’m wondering how 2038 will go. We have a LOT more devices floating around and there are still plenty of new devices shipping with 32-bit embedded Linux which may never get updated. Hopefully most of that cheap IoT stuff won’t last another decade but I’m kind of something like an automotive company getting publicly outed for skimping on technical debt management and some people getting locked out of their smart houses.
I call these problems ticking time bombs. This way business has a catchy name to use for the thing they dont understand, its also very clear this needs to be looked at.
Its tricky to get credit when your skills are x10 or higher compared to your team. 2 companies ago, a team of 40 engineers took 4 years and $29M to do a generation product update. The next company I worked for took $1M 5 engineers and 2 years (product from scratch). My current company I did the whole thing myself in 14 months (@ < $0.5M), basically on autopilot. Unfortunately, the stakeholders are unaware that they got enormous value by being lucky of hiring the right person for the project. And I still fight management/budget constraits and hot headedness. Almost enough to drive me to next employer.
Reminds of when you see people questioning the usefulness of getting vaccines for some diseases because they have almost entirely disappeared for the past 50 years (especially in 1st world countries for some), without realizing that this is precisely the reason why they don't reappear.
If everyone is vaccinated and the disease disappears (and kills few people) anti-vaxxers think: “why do we vaccinate everyone? The disease didn't kill anyone... and it disappeared."
In headache case, the link between illness and cure is instantaneous.
But when it comes to vaccination, due to the time needed for everything to take effect (many people need to be vaccinated, it takes time) anti-vaccines are unable to link the vaccine to a cure for the disease.
For humans, time changes everything, even the ability to make a connection between illness and cure.
FTFY: nobody ever gets credit if they don't publicize their successes. By default, fixing problems before they happen doesn't make headlines, so you have to find clever ways to earn publicity. One politician classic: let it become a "problem" enough to earn publicity, THEN fix it. Two examples: Y2K and Healthcare.gov
spot on.
As a general rule of thumb I keep a form of activity/issues log and send a monthly summart to the finance guys and head of dept along with "expected issues".
Letting things crash and burn is a thing I learned from one of my managers a while back. It's great for raises, bouses and rep. Takes a good deal of effort not to fix things I know will break though.
We live in a world where everyone depends on the output
Of everyone else - none of us could live without
farmers and hauliers, doctors and street cleaners, all of whom
Build on a foundation from the past
Assigning value, billionaires collecting rent, that’s the flawed model
It’s not capitalism that makes this world possible (the assumption of I Pencil).
It’s the sharing, the trading, the leaving money on the table because it’s too complex to work out how much Elon Musk’s 3rd Grade Math teacher should get for teaching him the basics of finance.
You don’t get credit - you get to live in the modern world
If there are those who do not get to share equally in that world - that’s a bug not a feature.
It even gets worse sometimes. You detect a problem that poses a risk to your organization, manage to convince management that it's dangerous, you get some time to fix it and you fix it. Then, because it's fixed, nothing will happen. And then management frowns upon it and wonders why all this effort was needed.
A big scale example of this was the Corona vaccination. After the large vaccination campaign in 2020, hospitalisation numbers stabilised to levels that were also seen sometimes in previous years during some flu epidemic.
Which led to criticism from some people, who said that this was evidence that the whole Corona epidemic was a hoax, organised by a conspiracy of medicin manufactories and policitians who wanted to scare people to gain more power ..
"Great wisdom is not obvious, great merit is not advertised. When trouble is solved before it forms, who calls that clever? When there is victory without battle, who talks of bravery?" - Sun Tzu
If only! Lots of things seem obvious, like common sense, but they apparently aren't because the behavior of many organizations is to reward the firefighters, to push "work harder" over "work smarter", and other behaviors that (in the long term) impede their ability to improve or expand their capabilities.
I liked the quote in the 2015 submission that twic posted [0], it relates back to the "seems obvious" sentiment. A lot of good solutions are simple (or composed of simple elements in a sane and reasonable fashion). But arriving at those solutions is decidedly not simple or obvious if we go by the Rube Goldberg machines that people end up contriving instead.
My memory is a bit hazy but it's a good story.
At one place, there was some important order processing taking place. As is fairly typical, couldn't rely on getting all the required info. Or critically, getting all the required info correctly. Some extra data, slightly missing pieces but enough to work, etc. etc. Some could be pretty gross. We built validation to massage some inputs, modify processing, what have you to address as many of those as we could think of. The team put in some metrics to identify which validations were "triggered" for each order. Neat. If we'd add more, we'd add a date to it.
It was great. We reported those stats so anyone could see anytime, but we'd also send out some comms about it every now and then. It also helped tremendously when a coworker or customer or whoever would say "Oh no! What happens if XYZ?" and we'd say no worries, we already addressed it, and we prevented #### orders from getting stuck for XYZ.
It showed that the team was thoughtful, that we were invested in making this work, that work needed to continue to keep things running smoothly, and we had data to back that up.
Really helped switch the conversation in the org because folks could see it. If someone pointed out that we hadn't thought of something, it honestly was more about what do we do vs. why didn't we think of this. (Yes, there is a comment here about poor org thinking or blame culture, but some of that exists everywhere.) More proactive. Recognition of preventive quality. People gave us accolades and it bubbled up to some good and real recognition at higher levels too.
I'm mangling the words here a bit but I hope you get the idea.
There is a simple but undeniable element of genius to the idea of, instrumenting the number of validations triggered. I would hazard a guess that 90+% of teams would focus on an "order success rate" type of metric and call it a day, instead of a "# of times bad data was handled" metric, which is exactly how you escape this trap of good work going unnoticed.
Thanks for sharing! Shamelessly stealing and applying this concept at work next week!
Yeah. Another way I would put it is to also look at inputs that strayed off the happy path. There may be something to learn from them as well, maybe something that adjusts the happy path.
That's exactly where we started. It worked or it didn't, right? The team was super curious to know, in fact, how many of the edge cases happened. That piece right there had some unexpected value as well. And it grew out from there.
I hope it helps!
Went through this exact scenario at my job recently. As the tech lead/architect of an org I reviewed a bunch of recently released projects and identified areas that needed critical improvements because they had major reliability/performance issues. One team had multiple releases at the top of the list, and their PM and eng manager (and really everyone else up the chain) simply ignored all concerns because they had to prioritize feature updates instead.
Fast forward a few months – I was on vacation and shit blew up. There was a sev 1 escalation, several customers were pissed, the CEO/CTO were involved. The team in question – the same one that wrote the shoddy code and ignored all warnings bells – worked round the clock and got the service back up. Now they are heroes for their effort and the same manager I mentioned has a great reputation at the company for being so active and communicative and showcasing his leadership during the outage.
Performing some heroics once in a while shows someone can be relied on. However, if heroics are the norm, it’s either bad work or bad management, and it should be looked at with some scrutiny.
I also find it more impressive when someone fixes someone else’s problem. I’m not one to shower praise on someone for fixing their own mistake, nor do I expect any praise if I fix my own mistake. I would be apologizing to everyone for screwing up in the first place.
Well, you could dust off the old email and 'mistakenly' send it again. Though it will seem petty, a small fraction might rethink the past few months.
I’ve started doing something like this more frequently, but it is in the interest of job security / cover-my-butt rather than pettiness.
I build something that depends on many systems that others have created and that uses data from probably about every dataset we have. What I build also has very high customer visibility. So whenever something breaks upstream, the first place customers notice the problem is in the system I maintain. Psychologically, people begin to develop a mental association around your system being a “problem” if it keeps getting brought up as the starting point of SEV discussions and customer tickets.
As a result, a lot of what I do now is defensive. I investigate and reverse engineer upstream codebases to identify likely failure modes, and I spend hours analyzing datasets of questionable origin for data quality issues and inconsistencies. I document and date stamp all of the problems that I find, file bug reports and assign them to the relevant teams, and write proposals describing potential solutions to what I see as large architectural design flaws that will come back to haunt us at some point.
All of this work is promptly ignored with “not enough bandwidth right now” or “not a priority compared to feature development”. Which is fine. I document all of those responses too.
Then eventually, something breaks in a big way that is again first noticed by customers within the system I am responsible for. In the past, I would immediately drop what I was doing and scramble into investigation mode for a few days to prove I wasn’t the root cause of the X hundred thousand dollar issue, er... I mean the blameless SEV review & postmortem... but more recently my preemption seems to be paying off, and lately I just reply to the panic with a bunch of links to old Slack threads (where we already discussed the issue), the documents and proposals I created (that no one read), or the bug reports I filed (that were never followed up on).
Perhaps it does come across as a bit petty, but I try to be as polite as I can, and from my perspective it’s an improvement over the previous situation. The only downside is that all of this preventative work takes away time from the primary work that I was hired to do.
2 replies →
This can be framed constructively, do an analysis of what was fixed during the outages from the originally identified and now lobby for funding to fix the rest.
It might be better to keep it low-profile. Go to a superior with the story of "told ya so".
3 replies →
I can this the "tragedy of software development": the pyromaniacs are the firefighters.
I've been contemplating the issue in the headline, as pertains to my value. When I help somebody in 40 mins with something they've been stuck on for 3 months, my value is clear to everyone. When I work there the whole time, and nobody ever gets stuck for 3 months, my value is unclear. Don't know how to deal with this paradox.
The educational system i was put through was set up to teach that outcome scales more or less linearly with effort and time. The first lesson after graduation is that this is not so. Increased effort and time most likely primarily yields more effort and time being expected of you with lagging compensation.
Value and opportunity are chaotic processes in effort and time.
All we can do is to try to maintain the levels of workload such that we have clarity of mind to seize opportunities when they reveal themselves. Honest and balanced colleagues help there, but that is ultimately a missions for yourself only.
Helpful partners and friends who try to find opportunities most definitely can help (and in unbalanced relationships, can do it all), like the market research team at a startup.
Or worse: People get stuck quite frequently and ask you for help pretty quickly. You get everyone unstuck, but your own work falls behind and when your boss's boss asks for metrics on developers you have few points delivered and few LOC changed. Your boss tries to explain, but your head is the one that rolls next when layoffs happen.
As a senior engineer I've learned that is my job. I take on much less work than anyone else because I know I will be called on to solve those problems.
This is in fact known in management literature: assign your best people to the least important project. That way the second best can grow to become the best, while the best are always free to help out if your fourth most important project has issues - it isn't a big deal if your least important project doesn't get done on time and if they manage to finish it so much the better. (your best are also free should sales discover a short window where a quick feature can bring in a large sale - though this is obviously easy to abuse)
> your own work falls behind
which is why one should not self-sacrifice. It garners no reward for one. Secondly, if the boss doesn't realize how much of an enabler you are, then it's time to start looking for a new job before the layoffs even starts as a thought in the boss' head.
Great workers know when they can help others without falling behind themselves. Or have good communication skills to explain what they were doing.
6 replies →
I've done that as an hourly contractor, I fixed their 6 month problem on my first day, I hoped they would then hire me for more hours on other stuff but they were like nope, that was all we needed. They did tell some other companies that I was good at this kind of stuff but nothing came of it. My first and last day in contracting for small companies.
The trick is to do these sort of engagements as a fixed price contract. You could have probably done an assessment in a hour or two. Then you charge on what you estimate is the value to the company.
9 replies →
Really good bosses make up for this. They push teamwork and collaboration, but at the same time they know the details of what every individual is doing and how they each contribute to the whole, and they can ~accurately compensate/promote/terminate. This prevents demoralization of team members. Team members psychologically need to be recognized for their individual contribution.
These bosses tend to be competent ICs who became team leads, they are best positioned to judge the ICs they manage because they themselves are masters at the craft.
I think an underrated way to deal with this phenomenon is proper self-marketing. Talk endlessly what you have done to prevent catastrophe. Describe the avoided catastrophes vividly so that people get a clear picture.
Sometimes you can't quantify actual avoided things. For example you can't prove that a regulation stove actually prevented what could have been a kitchen fire.
Self-promotion is incredibly difficult for some people.
4 replies →
Better yet, be in a work culture where people happily give credit to and praise others.
Work for yourself. The fewer bugs you create, the faster you iterate. And the faster you iterate, the better your chances of finding product/market fit.
One approach to showing value is as follows:
1.) Create a spreadsheet with all of the features of your group's/company's products(s) listed in rows.
2.) Create a column for every team member in the group and highlight the lead developers for each feature.
3.) Then ask each team member to add a checkmark in their column for every feature for which they would be willing to be on hook for 24x7 triage pager duty.
Over the long term - the most valuable contributor(s) on the team will be the one(s) with the most checkmarks next to the features they led development on (i.e. they write understandable code and document well) combined with the most checkmarked rows in their column (i.e. they proactively seek to understand other peoples codebases).
Couldn't there then be a risk that now some people want to avoid higher risk projects, and just build the simpler features, and get more checkmarks
What if the table in fact shows which people are best at dodging the hard work
Combined with showing who has the most friends in the office (giving checkmarks to features built by one's friends)
2 replies →
Kind of ignores the risk externalities have on projects no? If you tried your best to deliver something, but another teams priorities changed, or the business/market shifted, or the project got bogged down in planning/politics outside of that team members control, or if the team needs to integrate with a system (external or legacy) that’s out of their control and is notoriously flaky and torpedoes their velocity and morale.
Don't help anyone too much until a manager asks you to help. If someone is really pressing help, tell them that you are busy, but they can ask the manager to tell me to help. This way, you always get maximum credit. Every single manager that I have worked for claimed they were "different" and did not suffer from it. Usually, I give them many chances to see my value. After I am overlooked too many times, I revert to this selfish mode explained previously. In all such cases, my recognition from the manager improves. Sigh.
I don't know about that, you might want to receive some help yourself right? In some cases it might be better to work at a place that values cooperation more than competition. You sound like you would thrive better at such a work culture.
However, often leads just need something in order to know what you are doing, especially managers that don't really closely work together with their team. It can help to just mention it to your lead. Because it is easy to see what you done, but not how you have helped somebody. Just mention it in whatever recurrent meeting you have. And if helping out takes more time, I'd say it is only fair to have whoever is responsible for delivery involved in prioritization, because then it is at the detriment of whatever you are working on, which might be more important.
Often, it takes less than 15-30 minutes to help somebody get unstuck. I wouldn't enjoy working at a place where people would refuse to help me with something like that, or I would be so pressed to achieve things that I can't spare 30 minutes out of my day to help someone out.
Life and its rewards aren't perfect. Work with honest, intelligent people; genuinely do your best; your days will be much better and the odds will be with you.
> Work with honest, intelligent people
That's such good advice.
On the same note, it's so discouraging when someone who is not/neither takes over, those people leave (one way or another), and you have to start the search again.
Or when someone is intelligent enough you don't figure out they're not honest until it's too late. <shiver>
Anyone who's been around the block a time or two will accumulate those experiences. I hope I've not been jaded by mine.
> Work is with honest, intelligent people
Yes.
3 replies →
> Work with honest, intelligent people
This is good advice that’s well intentioned, but (sorry), it can be interpreted as elitist, and in a way that’s detrimental to the reader.
I am no way suggesting that this is the intention or belief of the parent, but while I’ve got more miles on my odometer than I’d prefer, they’ve informed me that “reasonable” is better than “intelligent.”
My god how I’ve found that working with reasonable people is so much healthier, more productive and rewarding than working with the unreasonable* intelligent folks.
*I fully grant to my current and former colleagues, friends and associates that I have been irredeemably unreasonable any number of times. Consider this a small thanks :).
19 replies →
Great approach! For the past 2 years I keep experimenting with different ways to track day to day productivity. It has helped me tremendously in assessing how I should feel at the end of the day about my work day and the amount of time I put towards work. (I tend to overwork myself significantly). It is of course a lot harder to apply this approach to individual productivity tracking across my teams. But I do now believe, based on self experiments and within smaller teams, that there is generally a lack of visibility of knowledge worker productivity. Especially within larger companies. WFH and hybrid makes this more crucial to have for both management and self assessments on the IC level.
1 reply →
That hasn't been my experience. I don't get to choose who I work with, my days are stressful and unfulfilling, and my career has stagnated.
3 replies →
This is a great answer. The only thing I would add is that you can take mental note when you accomplish something that anticipates and preempts future needs and challenges. It is ok to make brief mention of these things from time to time when appropriate. If nothing more it helps coworkers and bosses realize that you are putting in some thought and effort for the needs of others and to make things go more smoothly behind the scenes.
Doing this effectively requires tact. Try to mostly bring up your silent efforts casually, and be judicious about how often and in what situation you mention it. If it gets interpreted as an "I told you so," an excuse, self-importance, etc, then it will probably do more harm than good in terms of your standing with others.
Might be helpful to some degree. But won't save you if your organization's culture simply doesn't value your work or more generally doesn't care about proactive, methodical improvements that have no flash or immediate payoff.
reality feels like there is a mix of people with different motivations. I think the paradox exist because the odds are not clear
But most of the time you can not chose with whom to work
3 replies →
I'm going to add: in a place where they have the bandwidth to be honest and intelligent. Sometimes good people are in bad situations. It's important to recognize this.
Another perspective: if you are not in a position to change the rules of the game, understand the rules of the game and play that game, not another one that you would like to play or that you think is more fair to play. Alternatively, if it is not the game you want to play, find a place where that game, or a similar one, is played.
>Work with honest, intelligent people
Not too many of them around, are there? You can have either honesty or intelligence but not both.
However, those who choose to be dishonest are more possible to be your leaders.
> your days will be much better and the odds will be with you.
In the context of one's effort valuation it seem to be a very bad advice. Unfortunately.
9 replies →
> Work with honest, intelligent people
I'll let you know when I find this communist utopia.
1 reply →
Your managers need to understand the work that you're doing. When I review someone's code and I see them fixing something potentially catastrophic but that never happened, I definitely remember that and congratulate them. When I see a team member ask a question that makes me or the team rethink implementation strategy. That is very valuable. When a code reviewer finds a potentially critical bug in my code, or just a nuanced bug, I remember and appreciate. Because I fully understand that those are potential p0 or p1 issues.
And great engineers are consistently good at this type of passive, keeping the lights on work, but it does not reflect on their quantifiable work, so orgs do not include this in performance reviews. It is up to your manager to recognize this and advocate for you.
Even when they do recognize it it’s unlikely to be weighed the same as the engineer who heroically stays up all night fixing a critical issue.
Especially with many organizations focused on data or metrics for performance reviews or promotions being able to say you fixed an outage that was costing $X million per hour comes across much better than a vague counterfactual notion that your high code quality prevented Y such outages in the first place.
I went on holiday for 2 weeks. Came back to find the team had spent a week on a 5 minute task that had been clearly documented. Nothing had moved forward. Spoke to my manager that day and from that day forward it was clear what my value was. Eventually lead the team and helped the team develop processes and initiative so they could be productive in my absence.
Up until that point I knew my value but I don’t think anyone else quite got it.
That's not true, your value is clear just subtle.
If your solutions tend to be more stable from the get-go, it will get noticed at most places. There are always going to be that odd company where that's not true, but in my experience, simply doing higher quality work gets appreciated.
Well it's not 'clear' when every promotional process I've encountered doesn't reward such forward-thinking preventative maintenance compared to someone who postures themselves as a hero going around putting out manufactured fires.
1 reply →
Time.
Your value will emerge with time.
Just keep calm and carry on.
Good and bad times will do their usual sinusoidal dance but your slope will be pointing up if you zoom out.
There will be unfair promotions outside of your team, reductions or freezes in yours (companies have tendency to throw people at problems = inflate dysfunctional parts; if you have well performing team - they won’t let you grow short term; just wait it out, they will eventually).
Me neither. And I've had a similar issue with the first case. You do your work and help someone finish theirs, people still think you only did you job (even if people do credit you, you still have normal impact).
The psychology of workplace is quite subtle and almost backward. There's a motto "squeaky wheel gets the grease" .. to the point I'm thinking of trying some games. Like designers who produce crappy options and one good to toy with higher ups thinking they decide. There are some ideas in that vein so you flip the relationship and benefit from it instead of bleeding.
Value is in the eye of the beholder is what I discover. Got to play the game.
If you are an IC, you play it up with the right people. Those right people fight for you instead of the other way around. When the “we need to cut the fat” talks come around, suddenly you are not on the chopping block. But that also assumes the people fighting for you are not on the chopping block.
Soft skills across the industry is highly under valued. Knowing when to pick a fight or hold your tongue is also important (ie, reading the room).
I hate it but you got to do what you got to do to get that cheddar
It's possible to phrase the value proposition to align with what you want to deliver.
You can phrase it as "I help teams deliver on time by making sure they don't get stuck." Or "I increase a team's velocity by preventing mistakes that stop work."
It's how you sell yourself and then, how you tell the story of what happened. Being able to tell a compelling story is important.
The issue is how do you provide the evidence for that. In other terms, how do you differentiate your effect from that of Homer Simpson's Bear Patrol and Lisa's tiger-repellant rock [0]?
[0] https://www.youtube.com/watch?v=xSVqLHghLpw
1 reply →
My anecdata is to be vocal about your works, like having short knowledge sharing sessions to show it to your team/manager. Don’t be afraid of little things, your team would learn something from it.
Agreed but selling your work is an art. Sometimes folks try and sell it so often that it makes me wonder if they are really getting anything done.
Having a boss that loves to sell his team's work to his boss also works wonders.
Modern “agile” systems don’t work well with you. They want you to deliver X points of features in 2 weeks. Sounds like you’d be better off as a consultant? But then you need to do all that biz dev stuff! Or find a great CTO to work for (or be the CTO) who understands. You only need one job.
I always laughed about those X points targets.
In the end people ended up simply inflating the value of stories.
In my previous team I had to argue with a person that 2 story points to add translation keys were ridiculous.
To which the EM argued that 2 was okay because he also needed to write few accompanying tests.
I have nightmares thinking about this stuff. It was literally quicker to add the translation keys and that pointless test than to even discuss and vote the story and debate the points..
2 replies →
Think of it this way: when a principal engineer is hired from outside the company, what is the forecasted value of that engineer? How was that determined, since they have yet to do anything? If Andrej Karpathy joined your AI startup, people won't say this person has unclear value. It's because of the impact they've had previously and what other people have to say about that person. Value is your reputation.
It’s dealt with by knowledgeable managers that recognize and award your skill according.
If you don’t have such a manager, perhaps find them. Or become them.
An adjacent problem is how to know that you've really prevented issues/others from getting stuck with whatever you were working on. No system is perfect and engineering is also an art of knowing when to skip which corners. If nobody sees value in your preventive work then maybe there isn't any?
I am dealing with similar. I can provide solutions 10x faster than anyone. Everyone looks up to me, and are dependent on me for solutions to everything.
However it is clear that noone is improving and that the process IS me.
What I've been trying to do is make sure that my scope and role is fully clarified, and any "extra" activity that I perform is documented and flagged. Anything that becomes a "common" activity implies a missing part of the process - be it a role that is missing, or a skillset that is lacking.
Before you start thinking that it is pretentious or self serving, it's perhaps the opposite - you owe it to the process and the team/business/organisation to make them see what you're putting in, else they fail to find the gap.
It's not paid off yet, but hopefully it will yield results within the next 6 months.
I do hope you’re recognized.
One piece of advice, I found myself this person once - it’s the road to burn out city to be the critical path answer to everyone on the team’s challenges. You sound like someone who is very generous with your time and support. In my experience once people have found a critical path there is no point at which they “stop.” This isn’t because they’re trying to hurt you, they’ve just found the answer so to them it doesn’t seem wrong.
It will take even more of your time, but you ought to consider practicing giving less answers and asking more questions of those who seek your help to try to help them unpack the issues themselves. It will feel more tiring at first, but you’ll gradually help the others learn and also create a small bit of friction that will encourage them to try their own solution or two before seeking you.
Management asks are separate/ they can actually reward you with compensation and promotions for this extra work. But the team asking for help won’t stop when you get more comp unless you start teaching that you’re not the answer.
1 reply →
Quit your job and work for yourself.
If they don't pay you to fix those problems then don't fix those problems until they start paying you for it. You'll have to deal with the psychological torture of watching things go horribly wrong. So it's up to you how to deal with it.
> When I work there the whole time, and nobody ever gets stuck for 3 months, my value is unclear.
Well, a problem that hasn't happened yet is a risk, right? So you can apply risk assessment math to it:
Value = Estimated risk of the problem occurring * Cost if it happens
> Don't know how to deal
Good leaders. Or, worse: Consultants. That is, it takes either an outsider or a self-critical leader to affect change? Those doing the planning are always optimistic [0] about their decisions, processes, evaluations, and progress. https://en.wikipedia.org/wiki/Planning_fallacy
[0] not always bad: https://en.wikipedia.org/wiki/Hiding_hand_principle
Other problem on similar spectrum, when you have a good performance and your boss notice you, their expectations can increase and increase until you unable to deliver, then suddenly you're bad.
See also: The Locksmith's Paradox - https://medium.com/@pk.patrick.kelly/the-locksmith-paradox-6...
"When you do things right, people won't be sure you've done anything at all."
1 reply →
If the headline were actually true...
Agree with your point, solving problems gets you points, avoiding them doesn't. My cynical view of the headline is, that a lot of people do get credit for solving problems that never really existed, simply either fabricating (intentionally or not) easy to solve problems or vastly overblowing problems they just happen to have a "solution" for.
If ypu successfully fight a fire, you are a hero, if you prevent fires it is just normal and nothing special.
I'm currently in an unrelated field, but with a similar tension of unclear value while things work. The way to deal with this has been to keep a tally of checks on important systems, whether it's daily, weekly, monthly, with links to undesired outcomes should the process that has been checked fail. This list of you checking the processes is one way to "prove" your value, at least for ppl who wouldn't necessarily understand naturally.
No one knows your true value but yourself. Move accordingly.
Security/insurance paradox: when you do some jobs perfectly, it feels like you're doing nothing. Sadly, the only (ethical) way to counter such narrative is to spend a good amount of your time making reports showing how well everything is working and how it's not something you can just operate autonomously without disaster (to answer the inevitable "well the job is done, what are we paying you for?)
How do you get people to read such reports?
1. Donald Rumsfeld was right when he said “A’s hire A’s, B’s hire C’s”. 2. Environments where people can’t tell the difference between 1x and 1000x engineers lead to performative work and arsonist firefighting. 3. When folks in your organization can’t recognize (and reward) high impact, it’s probably time to get out, largely due to the first two points.
It's mostly a question of finding organizations that recognize and value that sort of contribution. It also helps if you track who you work with and what you help out on every day, so when review time comes along you can list not only your 'regular job' wins, but all the wins you helped teammates get.
Presumably that's not -all- you do? Or did you sit around for 3 months going quietly mad waiting?
Assuming you are doing other things make sure that is visible.
Also why did your colleague wait 3 months to ask you for help? A couple days would be OK, a month would be crazy. You should poll them and not wait for an interrupt.
It's always worth making some effort at promoting your value; no one else is likely to do it for you.
Depending on your context, if you are at the top of the foodchain as described here, minus the recognition for it, one option is to simply add grist to the mill. Creative problems you can leave for the team to sweat and for you to come and slamdunk.
The problem with doing something right the first time is nobody appreciates how hard it was.
The problem may be in the relative value contribution. A senior eng that rights a project on 3 days which had been going on for 6 months is incredibly high value/he work.
A Sr eng who spends 6 months coaching a junior engineer has much lower value/hour.
I think there’s a “this is life” aspect and a “this organization is immature aspect”.
A mature organization respects the process that prevents getting stuck for 3 months. But… they may be more stable and less nimble. Boring orgs don’t like heroes.
Unfortunately one of the best ways is to toot your own horn. If you work with good folks, it will be appreciated for you to present a tricky problem you solved, or to present a design you put forth that would save everyone time.
The lean mantra "Make work visible" is not only good at an organization level but also as professional feedback level. Although if used openly for promotions it will cease to be a good metric.
> Don't know how to deal with this paradox.
Think that it is the management's fault if they misjudge you. They have the chance to judge you correctly, yet they misjudge you. Their fault entirely.
Your value gets revealed in your absence. To leave and the company tanks- is a expression of the value shadow you throw.
Having a carefree life of no technical debt is not its own reward? Firefighting is stressful :)
The solution is to be a shameless self-marketer of all the work you do.
This does seem like the antithesis of useful, productive engineering work.
3 replies →
I hate to ask the obvious question, but maybe your job title is wrong?
At least you're likely not a person who works at a freelance agency where you purposefully do poor work or break things, so then you can bill more time "fixing" the problem - until the client gets the wiser at least, or there's another more oblivious client you can grift much more easily who doesn't know better?
The above seems to be how the top billing agencies on Upwork function, with fewer staff you can grift more people because the hours being billed aren't honest, and so you can have more clients and reach the "top" earnings position faster and more easily; with the same practices existing at least on one of the two prior platforms before oDesk and Elance merged.
Go into consulting.
Easy - half a$$ your code and when it breaks - swoop in, "fix things" (actually do it right) and play the role of hero! (I've seen so-called "Rock Stars" at places I worked do this over and over)
In high school I actually had a friend teach me this in a humorous way. In football we did sprints (the kind where you physically run) at the end. He said -- half ass the sprints. Then on the last sprint, run as fast as you can, you'll have more energy left, and when they see you running fastest you'll get all the credit. He was doing it as a joke, but lo and behold he out runs everyone on the last sprint by a lot, and the coach specifically pulls him aside as an example of how everyone else should carry themselves.
He's one of my favorite pranksters. But needless to say, the football team was not very good.
4 replies →
There's a famous green text on 4chan, where a user tells he got a sysadmin job which was so boring he started crashing stuff left and right and blocking an entire office even for entire days.
By the end of the day he would plug back something and come out the "servers room" saying he fixed that and get everybody's praise. Even got him two raises in the span of 18 months.
That's how crazy it is.
I know a variation of this sort of story, where a good sysadmin/DevOps team was halved and then the problems started. The company didn't have those issues exactly because they had a good surplus of eyes to handle everything.
They realized only later the mistake.
7 replies →
Another option is to work at a consultancy where you hop aboard a new customer project every 3-6 months. Approach every customer as an opportunity to do some resume-driven development and pick a bunch of untested new technologies to experiment with. Be sure to do at least a couple of presentations to tell everyone about the hottest new things you are doing to bring value to the customers. Leave the project once it slowly starts sinking and then just keep hopping from customer to customer. You will be far away once the sea water starts coming through the windows and the non-technical people directing these projects will never figure out what you did.
I have seen that this is one of the most efficient ways to advance your career especially in larger consultancy companies with hundreds or thousands of different customers.
3 replies →
Don't do it right, double down on the flaws. You had a 3k SLOC single function (all in main) C program to do something that could be expressed cleanly and clearly in 200 SLOC. Some specific sequence of inputs leads to an error. Instead of tidying it up, removing the repetition that led to the mistake, you copy/paste everything again and add another 100 cases to your various switch/case statements (actually you use if/else because switch/case might make things clearer). The specific problem is solved, but in a year another buggy code path will be discovered and you'll have another chance to play hero. In 5 years it'll be 50k SLOC of C all in main, that could have been under 1k SLOC (still all in main). No one else will be able to fix it but you!
9 replies →
It is more easy: do your job right, but do not comment, fix or somehow improve work of your colleagues. Let them fail, aknowledge failure and only then come with fix, get all praise.
Never point to the possible issues at the code reviews, retirement refinements etc.
5 replies →
I have seen so many people do this wherever I have worked. In fact there is another thing that they do i.e. when management pulls in the schedule they will meekly agree to it because they already know they are going to half-a$$ their code. Now they are heroes much before as they are "yes-men" and will be targeted for promotion. Swooping in to fix things just adds to their already loaded credentials.
The ones who are honest and actually disagree are banished and their lives are made difficult. They are called all sorts of names the most important being "not a team player".
This is exactly what happens at many software companies. People,sometimes under pressure to meet deadlines, apply totally crazy fixes to stop one problem and create new problems.
2 replies →
Hah, that was in my archetypal description of the fake 10xer, that somehow they are able to swoop in and fix their spaghetti code, come what may, while no one else can make sense of it:
https://news.ycombinator.com/item?id=18462325
Having people thank me for being 'on top of' a problem that I already know about because I'm the one who pushed the button that broke it... It's gross. It feels gross.
Much happier when I can fix something quickly because I have some tools or logic in the codebase already that lets me do something quickly by being covered with sanity checks so I can zip along without driving us off a cliff.
2 replies →
Maybe I'm the only one who notices people whose work "just works". Those people are solid gold.
2 replies →
I was just considering it, but never had a chance to see it in real life.
I'm dumbfounded about all this, you spend years learning and reading books about the craft but the only topic which will matter is how you can con the game ?.. weird.
That's literally how promo packs work in FAANG (minus the N) and other adjacent tech companies.
1 reply →
I don't think you're wrong, but god what a waste of time. Can't we just fix the actual problem?
2 replies →
I don't do this for one reason - I cannot hold others to a high bar if most of my code is half-assed.
I usually write my best code to show others it's possible to write good code and ship on time, with exceptions of course (and I usually document in code why its half assed)
Would you feel good about being that kind of engineer, if the external validation was great enough?
8 replies →
This is a slippery slope, IMHO. It can easily get you to be the next "tactical tornado" of your team (cf. John Ousterhout, A Philosophy of Software Design).
don't even need to half a$$ it, just use timebombs.
https://en.wikipedia.org/wiki/Time_bomb_(software)
Usually only works if you're the only dev, unless you get creative with counters like the original devs that made some nice cash fixing it all for y2k
(officially: don't do this)
2 replies →
Is it important that no one knows it was the Rock Star’s fault in the beginning?
For i in employment ensure there is someone who checks in with me every month when they read this line of code, else, stop code from running so they come to me.
[dead]
Play to your leadership. Exceptions apply, but female bosses tend to appreciate averted crises more than male ones.
If your boss is male, let shit fail. You'll get hero points for responding to incidents.
A female boss will quickly suspect incompetence if things keep breaking (i.e. see how far half-assed DIY home repairs get you with your wife before she loses patience with you). Your hero points will come from mitigation.
This paradox comes up a lot in security. IME in this particular field the gender stuff is less relevant since everyone is paranoid. But when we run stuff up to C-levels, it's only the female execs and lawyers that really stop to consider possible issues-- the men just dismiss everything until it happens.
Another variation on the problem is the over-allocation of resources to preventing problems which actually happened once, versus those that are more severe but haven't happened yet.
This is a management problem, because no-one wants to be accountable for a repeat incident even if it was rational to be working on something else more important.
> Another variation on the problem is the over-allocation of resources to preventing problems which actually happened once, versus those that are more severe but haven't happened yet.
I've heard this called "institutional scarring" in a blog post somewhere. The idea is a small wound can be replaced with tough inflexible tissue. The jist of the blog post was that just because something happened, doesn't mean you have to change things to ensure it never happens again because that can be an over-reaction that really burdens your future. Accept that loss and that it just might happen again, but that may be better than onerously preventing it with certainty.
I called the dev form of this “tech trauma.”
e.g., You were tasked with working on a ball of mud and it was miserable, so the next system you get the chance to build has to be the most scalable, modular, and cutting-edge thing ever, just to be safe.
I remember this blog post, it was also a chapter in their book Rework https://signalvnoise.com/archives2/dont_scar_on_the_first_cu...
1 reply →
Same thing constantly happens in governments, too. "Oh no, something that's been done for 200 years now caused issues once! We have to restrict/regulate/bureaucratize/outlaw it immadiately!"
There's a ton of reactionary legislation on the books that exists pretty much entirely because politicians wanted to be seen doing something. It's mostly crap.
A lot of gun legislation is like this. Whatever your view of guns is, there is no rationale for a "safe hand gun roster" like California's where the Glock 17 Gen 3 is on it, but the Gen 5 -- which introduces a basic safety improvement for left-handed people -- is not.
This is probably a bit too cynical; it's not just "politicians who want to be seen doing something"; the public often wants something done as well. And "we didn't do anything after that incident from ten years ago and now it happened again" is not a good look. Complex stories about trade-offs and the cure being worse than the disease often don't "play well" in the media, especially not with the opposition demanding that "something could have been done!" And corporations tend to be pathologically risk-averse.
Politicians, the media, corporations, and the public all have a part in this.
2 replies →
Yeah, the legendary politician's fallacy:
1) We must do something.
2) This is something.
3) Therefore, we must do this.
3 replies →
This is basically how all bureaucracy comes about. In startups everything is so new that problems haven't had time to happen. In big tech, because they have a tremendous knowledge base of previous incidents and the resulting safeguards, every single step seems mired in bureaucracy.
This is why death is a feature and not a bug.
The thing is: it's very easy to play up the likelihood and severity of problems that are entirely imaginary. This can be just a bad habit, but also a deliberate tactic. In either case, a lot of effort, time, money, etc. is wasted.
Waiting for something to actually happen before allocating resources to preventing it is (to some degree) a rational policy.
yeah that to the Ai doomers who are making six figures off of it
2 replies →
I have worked in finance tech all my career, so I'm not sure how other orgs work, but this is extremely pertinent. Large investment banks react exactly like this.
I spent a miserable year trying to convince people that they were over-reacting to an outage and there was a very simple solution to the exact problem that occurred. But when senior managers see their jobs at risk because of a repeat, they'll mandate that the entire department review their code for similar issues and remediate. They'll also listen to the loudest voices who somehow come up with massively over-engineered solutions.
Another example, we had a password expire which caused an outage on our trading stack. The amount of effort that went into stupidly convoluted hand-crafted solutions ensuring this "didn't happen again" was laughable. And in the end, after more than a year of work, the whole thing was abandoned in favour of a much simpler centralised solution that should have been done from the start.
Reminds me of a place I used to work. Every time they asked for feedback, my refrain was "nothing here gets prioritized without a PIR (post-incident response)". Towards the end of my time there, whenever a PIR-related ticket showed up, I would mark it as a duplicate of whatever actual ticket I had dying in the backlog that would have prevented it. It really hurt team morale to have no influence in preventing foreseeable issues in our domain area. Most of the rest of the team had even stopped suggesting improvements altogether because management refused to let us pull our own tickets in.
This is wonderful description of the kind of hellhole corporate Scrum is turning into.
Agile was literally about doing things quickly and quick cycles of capability improvement. But Scrum is a worse version of the planning processes it meant to replace!
If anything, the way scrum lays out the work into immediate problems exacerbates the cycle. In the long run it just turns into a ticketing system where fires get pushed up and technical debt gets pushed down.
It even spits out a super easy-to-track, meaningless set of efficiency numbers for consultants/executives to min-max!
> Scrum is a worse version of the planning processes it meant to replace!
You say this as if it was a coincidence.
I'm allowed to say such things. Some of my best friends are scrum masters.
I still can't believe a scrum master is an actual FT job - last company I worked for had one FT scrum master for each project - so they ran the meetings and as far as I can tell, did nothing else - what the heck do they pretend to do for the rest of the week?
I went to all the same meetings, and was still supposed to actually develop software in between them all.
Agile seems to have ended up as a way for PMs to report upward to senior managers who also need to report upward.
You can see why. These people have to decide what will be worked on out of all the potential things that can be worked on. Someone has to make that decision. Will this feature make us money? What about that bit of work that doesn't add a feature but reduces resource costs? What about tech debt that I'm told is building up and slowing the ability to deliver features?
I'm not a senior manager, but ultimately someone up the chain is responsible for the company surviving and making money and paying our salaries. They are just like you and I, trying to make decisions based on what little information that can glean. So part of that is "what will this cost and how much will it be worth" vs "what will that other thing cost and how much will it be worth".
To that end, they need some way to estimate this. They latched on to agile as it was being promoted by tech as a way to do this. Whose fault is that?
And so with that came all the frequent estimations, are we on track, rituals, etc. Some people don't believe this should naturally follow. I agree. But somehow all those rituals have become part of the cult.
We abondanded scrum. We abandoned refinements and estimation of stories and story points etc. Now we meet with a PM once a month formally and as a team perform a t-shirt size estimate on where we are. Otherwise we update her as frequently as she asks (which isn't often) or we want. This gives power to us but because of that we're conscientious and make sure to inform her timeously when things are looking sketchy or whatever. Yes, we still have to give "estimates", because ultimately, senior management want to make decisions, but it is otherwise quite lightweight.
It is so liberating.
I found the idea Scrum being efficient when done right was true at one company I worked at. Everyone was committed to the process, the Scrum team included debt as a priority for 20% of the effort and everyone had a fairly accurate velocity that we could bake in an additional 20% of your work as your own interests, so stakeholder priorities filled that remaining 60%. Then we would pivot in some Sprints if any epic/team goals required a push to get done or other emergencies/bugs that required a change in priority.
I don't think it can't be done right, but it's been cargo-culted pretty hard at Fortune 500 companies. But the best scrums I have ever done have just been literally post-it notes on a whiteboard - I think Jira ads a level of complexity that is unsustainable for most orgs.
2 replies →
I've found it works better when you "build up" to Scrum by incrementally adding processes when you identify issues and relaxing the process if things are working well.
Adding a bunch of processes because you want to add processes doesn't provide value.
> Scrum
I always thought this just meant to scrumble every week to get things done with a weekly standup?
I believe it's actually a rugby term, and the idea is to wrestle things across the line at a regular cadence as a team.
The unit of work is even called a sprint because the idea is specifically to commit to very intense units of work.
1 reply →
I would call it "the hellhole that corporate performance assessment is turning into". Scrum is just a tool
Reminds me of this cartoon (which I have posted on my office): https://naksecurity.medium.com/the-detriments-of-hero-cultur...
This is why in many corporate cultures it pays not to proactively stop a problem that you know how to fix if the problem is not in your immediate problem area. Let it be noticed, let it become someone else's emergency, then fix it. Much better path to a reward that way. Of course, you should also be planning to leave said organization, in the long run it won't do well.
I think that over time people realize who has a fire drill every month, and who quietly just gets things done.
I wouldn't be so sure. As an employee and team member, you'll be better perceived if you do your share of performative firefighting every one in a while. I wish it wasn't true, but it usually is. (Though at remote teams, it's less and less true, with the remote teams I was on, nobody really cared).
"wow, last week we had such nasty bug, impossible to track down, also caused production reliability issues. Tom stayed up all night, and finally pushed the fix at 3 in the morning."
Now, if you then say that it wouldn't have happened if 1. Tom didn't overcomplicate the system, 2. Tom learned our tools properly, 3. Tom actually understood the requirements and at least tested his changes at least once manually, 4. Wrote good code so that it's easy to troubleshoot 5. Wouldn't have forced a rushed PR review together with the product owner.
You'll just sound like a bitter, know it all.
Nobody ever gets credit for fixing problems that never happened (2001) [pdf]
I'm reminded of this every time I see some YouTuber or other social media click bait claiming that the Y2K bug was no big deal.
The reason it was no big deal is because thousands of graybeards, like myself, stayed up many long nights for months ahead of time making sure things would work.
I still remember the tension during the countdown to midnight UTC. Then the tension rising again during the countdown to Eastern time. Then once more at local time.
Only when it was 2000 in Pacific did we unclench our cheeks.
Depending on which Y2K bug you're referring to, this very much happened and is an easily believable scenario. I don't think this is a comparable situation.
I still don't quite buy that. Surely it's because computers mostly use epoch time for dates, not dd/mm/yy?
Guess we'll find out in 2038.
That's not what I remember programs looking like. You would have counters that went to 99 and no more because the UI wouldn't accommodate another digit, and programmers didn't know what they were doing, and figured it would be clearer if the software reflected its UI and couldn't handle over 99 at all, invalid UI means unrepresentable state.
Lots of stuff didn't care about the day, too, like you'd get 0997 and that meant September 1st 1997 because the first (or last) of the month was inferred, and you'd get goofy logic around year++ where you add 100 years whenever you want to increment the month, and the whole thing is written as a modulo 1200 but whenever the date is about to be 0000 you look at what's stored in year instead and add one to it instead of 100, because that way you have less variables to allocate and every "little bit" counts.
Everyone knows now, but lots of people writing programs didn't have any prior art, they were just good at messing with computers and sort of fell into programming by accident.
Many old programs written last century did not use epoch time.
https://en.wikipedia.org/wiki/Year_2000_problem#On_1_January...
1 reply →
Interesting that everyone is responding to the first half of your comment. I think 2038 is going to be more interesting than Y2K...
https://en.wikipedia.org/wiki/Year_2038_problem
It was also to do with how dates were stored in very old databases that were conceived in the 70s when storage was at a premium - a lot of them just used two digits for year.
Now they do at least
2 replies →
In about that same timeframe, Y2K is also really good example. Nothing of note really happened. But if people had just ignored it a lot of things probably would have.
Not "probably". I was involved at ground level fixing code for BP in '98. Can say without a doubt that bad things would have happened. And not because my salary depended on it - there was no shortage of opportunities to work on other things, obviously. It was legitimately an issue that would have crippled the energy industry (major players, and all their many, many dependents). I infer from this experience that the impact would have been the same on many other industries (e.g. finance, resource development) directly and indirectly.
But it is a great example. I still run into people who recall Y2K as an example of much ado about nothing. No, no!! It wasn't an issue for you because many people worked hard to prevent the problems.
Those problems were not terribly complex just pervasive and critical, requiring a high LOE. It wasn't like a huge moon-landing engineering problem to be held up as some accomplishment of humanity, more akin fixing a lot of dumb Challenger o-ring problems before a blow-up.
This is a very important case. There was an enormous investment in fixing Y2K, starting long before the day in question (e.g. financial instruments with expiration dates after Y2K had to be fixed before they became due), so on the day there was only a smattering of minor residual bugs.
There were a few jokes about it in the newspapers but by and large the public just ignored it.
I work on climate and hope / hoped that the same would happen, but it's looking like it will soon get everybody's attention regardless.
Y2K would negatively impact companies' bottom lines if not dealt with, directly, in a short and known timeline, in predictable ways.
Climate change will impact companies in different ways at different times. Oil companies, for instance, are incentivized to ramp up production as much as possible while also diversifying.
The way to get all companies to take appropriate actions is to not allow them to externalize costs. That means a significant carbon tax.
I'm sure I'm not the only one that can't wait for the 2038 problem to get closer. bank accounts gonna be jumping like a 3-peat Micheal Jordan when those consultation fee checks start rolling in.. provided the Boston Dynamics dogs haven't taken over yet
2 replies →
On climate change, it already did work once with the hole in the ozone layer.
People ask why they don't hear about it anymore -- oh, it's because we listened and banned CFCs and fixed the hole.
The mitigations were done so well today the public remembers Y2K as a “scam”.
1 reply →
> I work on climate and hope / hoped that the same would happen, but it's looking like it will soon get everybody's attention regardless.
Unfortunately those fixes are slated for Humanity 2.0, Homo extinctus.
Since you "work on climate" I am sure it got your attention that there is a massive effort happening all across the board. R&D has made tremendous advances, whole industries are investing to get away from fossil fuels, and governments have ongoing programs to accelerate and support lange hydro, solar, and wind projects.
Some countries have not heard the call yet it seems (ones that like their role as perpetual victims and blame everyone else but themselves), but even just five years down the road, the world will already look very different for all the effort that's being made.
the readiness paradox:
If you are prepared, nothing interesting happens, life moves on, people opened a notebook and pressed a few buttons.
If you are not prepared, the texas grid freezes, people die, people lose their savings, and "nobody could have imagined it would be this bad".
Recommend "Left of Bang"
In strategic sysadmin and cybersecurity this is staple thinking, and a useful part of what I teach now.
Problem for vigilant thinkers is the rewards, as explained in TFA, are perverse.
People are content with technology to be magical. They don't see the millions of people quietly repairing it and planning to keep it all going smoothly.
To "bosses" cybersecurity only bring them problems, and cost them money apparently doing nothing. The same for any defence force, until you need it.
Did some nice interviews for international sysadmin day last summer [1].
[0] https://www.goodreads.com/book/show/22663095-left-of-bang
[1] https://cybershow.uk/episodes.php?id=15
What, what do you mean nothing happened? I'm in my underground bunker, do you think it would be safe to come out and look around?
You guys are all in your Y2K underground bunkers still, right?
My advice looking back over the past 25 years . . . stay in the bunker.
Yeah, it's the year 24, don't worry about it. A human is probably still able to type this.
I didn't worry. Even took a trip to a different country. Though I can totally understand why some people (know you're joking) might have played it more conservatively.
Yeah. I've heard folk who should know better (looking at you, John C. Dvorak) call Y2K a nothingburger and suggest the effort to prevent problems was a total boondoggle.
Disclaimer: I was part of that effort. The funny thing was that I was brought back to a previous client to fix something that my previous efforts had literally caused. Took me 20 minutes to fix it once I saw what the problem was. And then there was a "while you're here, can you take a look at this ..." That lasted about two years until they closed the department and moved it to NY, NY.
At least I got credit in terms of billable hours.
While I often enjoyed reading Dvorak columns in PCMag, he was always more someone who liked to provoke people than offer thoughtful commentary.
Another example is ozone depletion.
yeah, I've had that discussion
"we were all worried about the ozone and nothing happened, this is just a repeat and people will stop talking about it any time now"
and my response is "'Nothing' happened because the governments acted fast and turned it around so now it's mostly just boring stuff like catching some factoryy that decides to go cheap and use banned chemicals to save a buck"
I thought this was going to be about Y2K, since it was written just after.
I worked for years in the late 90's on Y2K projects, helping to stop the UK's critical infrastructure from just stopping at midnight. Wales would not have water or gas without our efforts, for example.
But I've heard people since say things like "why did we spend all that money on Y2K, when it clearly wasn't a problem since nothing happened?" and even "Y2K was a hoax invented by the IT industry".
We won. We successfully stopped the Y2K bug, and it was hard work, and it wasn't obvious that we had caught it all by midnight. Yet rather than celebrating, some people saw it as evidence that we had been ripping them off. People are weird.
I know of several other issues like that. I myself made sure the highscore in a video game I was working on would work :-)
The thing that annoys me is that this is best case scenario when it comes to climate change. If we do in fact manage to avoid the apocalypse, all the "climate deniers" are going to feel vindicated.
Suppose on Sept.10, 2001, you were an airline executive who successfully forced installation of jimmy-proof, secure steel doors between the cockpit and the cabin?
The only "credit" you'd get is "This guy cost the airlines god-know-how-much money, and for what? An imaginary threat!"
The book "The Black Swan" by Nassim Nicholas Taleb contains this very example. It is a great read on why the improbable events have the biggest impact.
which is where I got it (forget the attribution, though, but the name "Taleb" seems to trigger some people).
Wow, really puts things into perspective.
Having spent several years as a manager, as a once-again IC I now include lack of problems with my features in production in my written self-assessment. I'll often refer back to major features I shipped ~2 quarters ago with a statement like, "Project Foo continued to function as expected, scaling as designed under load while incurring zero production incidents." I'll often include the words "engineering and operations excellence" and "commitment to product quality" when doing so.
Nobody gets credit for fixing:
- Problems that never happened.
- Problems that people don't realize are happening (where people cannot relate visible symptoms to their root causes). You only get credit for addressing visible symptoms. Though, disturbingly, you will get credit even if the fix is only temporary... This creates an incentive to only address symptoms since not solving the root problem allows you to get credit over and over again for fixing the (recurring) symptoms... In fact, people will praise you even more as a 'Relentless hero who dedicated their entire life to the problem' LOL.
Also, nobody gets punished for creating new problems if the visible symptoms cannot be traced back to the root cause. That's why I oppose complexity in all matters - It obscures root causes and sends everyone off on a wild goose chase addressing symptoms... Complexity turns everything into a frustrating game of whac-a-mole.
The person causing a problem can even get the credit for solving symptoms of that problem.
Or: The Preparedness paradox [1] / "There is no glory in prevention"
[1] https://en.wikipedia.org/wiki/Preparedness_paradox
and the related discussion: https://news.ycombinator.com/item?id=32466346
[dead]
This us an incentives problem, plain and simple. Its extremely difficult to quantify things that don't happen, and because we've been building our economies and societies around a heavy focus on quantitative metrics we end up only incentivizing people to solve problems. Even create problems on purpose if you need to, you'll get credit for fixing them.
Human Resources departments often run into this exact scenario. Many of the tasks that fall into HR are hard or impossible to quantify, and if done well they're also solving problems before they manifest.
I wonder if this is why some managers seem to go out of their way to generate chaos. I have no patience for it, and judge them harshly for it. They don’t get any credit if my book for “solving” problems they pulled out of their back side or willfully ignored planning for.
I'm sure there are some that do this.
I once worked with a test manager that was effectively judged by how many bugs were found and how many tests were written. He was actually a nice guy just trying to play the game given to him, but if you didn't know this you would think he was doing exactly as you described.
There will always be a few bad eggs in the mix, but I'd check the incentives forced on them before judging too harshly.
1 reply →
I work on a large distributed infrastructure. I always joked that my team's projects and people's careers are outage-driven: the only time we become important and people get opportunities for promotion is when where were big-enough outages that executives have to invest heavily on reliability or scalability. Other time, we are just minions who must listen to and serve feature or product teams. Nobody listen to us when we ask a product team to implement a reliability contract in their shining product.
Pre-outage improvements, reliability defense in depth, eliminated scalability bottlenecks before they are hit, are all ignored by leadership and the company: it is just human nature that even though they understand you have to prepare for possible issues, if it hasn't happen yet, you won't take it seriously. I've seen this in many internal performance reviews and promotion committees. People who haven't ever got bitten badly by an outage may call these premature optimizations.
My background is systems operation. The exception I could see to this is when there are rewards/incentives/etc for uptime or non-change-window uptime. Usually this would be coupled to meeting the goal of 4 9's or 5 9's of uptime, meeting SLA requirements, etc.
Granted, in my time working for a major 3 letter tech company on account with a major finance company my highest exposure to VP/C*O types was fixing the fallout from a whole chain of corporate culture related mistakes. If I hadn't already had a job lined up elsewhere I probably could have gotten a good bump out of it. I doubt that would have happened if I'd identified and fixed the problems. Part of that busted corporate culture. We didn't have metrics around uptime/SLAs there though which considering the clients business was pretty ridiculous.
Ah the old ops justification problem
Everything is working => "What are we even paying you for?"
Everything breaks => "What are we even paying you for?"
Only take. No give. - Dog
I run an ad agency. I only touch ad accounts with pre-existing spend snd results for that reason. If I take the client from $200 CAC to $100 CAC I'm a hero. If I go from scratch to $100 they don't know my value and it's much more work.
It seems like many of these TQM discussion miss one key factor of TPS.
Respect. Engineers and management respect line workers. They have a word for the expert line workers, ("craftsmen"), Takumi. https://www.allaboutlean.com/toyotas-takumi/ and Takumi have a valued place in the company.
Without that basic level of respect... i dunno.. seems like it wont work
I always get cynical when I see awards for software releases. Someone always gets an award for staying up late to fix a last minute problem. Normally the person that should have written good code in the first place so they wouldn't have had to come in late. The award should go to the person (likely people) who never got called in at all in the first place because we didn't find a bug in their code.
You guys get awards?
I'm both a developer and a security consultant, both jobs deal with this problem. Being a security person, you go from being a no-one until something happens, then every eye is on you. And with the dev side, I work mostly on backend, so my work goes unnoticed always, the frontend dev gets all the credit, no matter how many thousands lines of code I commit, if something looks prettier or just works it's because of the frontend dev.
There's two choices, you either don't give a flying fuck and deal with it, or you go as loud as you can to make yourself visible beforehand and try to get credit for your work, your choice.
Only previous submission that had a discussion (and it's worth reading through):
https://news.ycombinator.com/item?id=8940820 - Jan 24, 2015 (50 comments)
In our org, this was brought up. People who had problems, then fixed them, were the ones being seen and recognized by higher level management, and awarded for fixing the problems, when they, often, shouldn't have happened in the first place.
Now we have something akin to a "person that prevents problems" award. It's a nice gesture, but, even though I received one, I think it's mostly nonsense. You don't get the face time/experience presenting with the higher level managers. It's almost impossible to prove that you prevented catastrophes, by working extra hours, or doing the right thing. Nobody likes a "Hey look, I told you so!" sort of perspective, no matter how it's framed.
To me, this is the most gratification-limiting aspect of being a software engineer: time is required to test quality of foresight/design. Rewarding retroactively, for great past decisions, doesn't happen. There are systems I designed, when making a pittance for compensation, that's still being used a decade later, across multiple companies, almost entirely unmodified. That, now objectively, good work wasn't seen or recognized by anyone paying me at the time, and can't be seen by anyone now. I burned good time for a "good job me!" as a reward. As I get older, that mental masturbation seems less desirable, since it's usually at the expense of the surprisingly limited life credits that we spend on those tasks. But along with most other nerds, I can hardly stop myself because it's an innate obsession.
Award everybody with a bonus that can be deducted with every bug report
Oh, maybe adversarial salaries. You each, meticulously, go through the others code, trying to find as many bugs as possible, in your spare time. Each bug you find transfers a percentage of their salary into yours. End result: everyone's code is perfect, and everyone hates each other. :D
1 reply →
Reminds me that one time I found 17 databases in our org without backups configured. All I got was a thanks from some devops guy and no manager even heard of the problem...
Yup
My manager just told me I got a 3/5 on my eval yet again because in the past six months
- I did everything that was asked of me and
- I did a lot of things that weren't asked of me and
- I didn't catch anything on fire and
- I put out someone else's fire and
- I prevented several fires from ever being started in the first place,
but that's not good enough. The person who started the fire that I put out is getting promoted because they showed initiative in starting the fire. I didn't show enough initiative putting it out for them or preventing fires on multiple other projects. Apparently creating two of my own projects that never caught on fire also didn't demonstrate enough initiative. From now on I'm applying for one new job per day on company time.
Don't take it personally, managers at bigger companies have to evaluate their teams on a bell curve. The whole team can't score a 5 and someone has to be worse than the group.
It sounds like the fire starter should get the worse score. Of course, we’re only hearing one side of the story.
That is called stack ranking and even though it is used in a lot of companies it is widely considered a bad idea. A lot of large companies don't actually use it.
Your manager probably just doesn’t like you, find a new one if you can, there’s no point swimming against the current - you’ll just get tired and get nowhere
I don’t get credit for my tiger-repelling rock even though we haven’t seen any tigers.
Are these magic rocks for sale?
I tend to prefer working with people whose stuff "just works". I take that back. I strongly prefer working with them.
And people who comment their code. Even if it is just a little. I swear, 90% of my time in coding is just fiddling around with functions to see what they do so that I can actually pull them together in the right way. If there was just a little documentation this would greatly reduce my time. It should also be in every team's and company's best interest because people hours are expensive. It should also be in the best interest of an open source project because more people will be able to build on your stuff. It should also be in your own interest because taking the 30 seconds to a minute to write those lines interrupts you and allows you to rethink and verify your method. So not doing it is only in the interest of moving fast and writing spaghetti code. But you're not actually moving fast. You move fast at that moment, but not in the race.
Documentation exists in one of these states:
1. incomplete
2. wrong
3. missing
My goal is to write code that is so clear it doesn't need documentation.
17 replies →
As a software engineer who pays more attention on security than average, I often feel my work didn't get recognized. For example, if I successfully prevented a supply chain attack, nobody would say thanks to me for a thing that didn't happen, even when they see a competitor product gets attacked. Similarly, most C/C++ programmers do not really care about integer overflow. But I know we are no longer in the world that computer viruses are everywhere(though ransomware are still common). The, who made it better? The people who are not satisfied with stuff that "just works".
I think you misunderstand what I mean by "just works". It means it doesn't have bugs in it. That includes security holes.
This is kind of a problem with preventive ethics in general. It seems to me that given two bad choices, the best option is actually neither - that is, to prevent the choice from being necessary. To use the common example of the trolley problem, the best answer isn’t to choose one option or another, it’s to prevent the trolley from being about to run someone over in the first place.
Unfortunately the visibility of these non-events means that their heroes go unnoticed. Noticing them would require a fundamental reorganization of how we perceive time and the future.
I like to classify what to prioritise by how many standard deviations from the happy path a certain issue is.
For example, I’m working on a pay-as-you-go SaaS - so the probability of using the SaaS while your account balance is low is 1-SD away from the happy path.
Figuring out if the user has JavaScript enabled (SaaS is an SPA) would mean the user first signed up with JavaScript enabled, then disabled it. Id put it at 3-SD away from the happy path.
Keep doing this and work on the bugs nearest to the happy path.
Of course, the 1-SD, 3-SD estimates are just hunches.
This happens in organizations that don't measure and respect precursors or leading indicators of success/failure -- organizations that do not practice organizational observability, to coin a phrase. Which is to say, most of them, because this is not something one thinks of until one has suffered its absence. Your managers have to think like SREs, and be technically sophisticated enough to model and address this problem.
Does anyone know any great management books on this subject?
The High Velocity Edge had a lot about this subject.
The most memorable example was at a hospital, nurses would occasionally pick up the wrong vial. They always noticed before injecting, but there were lots of close calls that didn’t get tracked anywhere. Once they started tracking that as a leading indicator, it became pretty obvious that the vials needed a better design to prevent even the possibility of grabbing the wrong one. Also, if the hospital never realized this problem and someone got hurt, they would’ve ended up with a much worse solution like “double check really close”
This is a great insight, the idea that management has to look at leading indicators for the org itself. Makes me think that beyond line management, a manger’s job is to behave as an investor and steward of the org, instead of getting involved in the details.
Not a management book per se, but Working Backwards popularised an Amazon approach of thinking about the organisation in terms of mechanisms. You might find that useful.
I’ve always wanted to dig into Scarlet Ink, Dave Anderson’s blog, but never have, and I suspect it’ll be quite useful for you too.
> This happens in organizations that don't measure and respect precursors or leading indicators of success/failure
This happens in organizations that become over reliant on metrics and indicators of success/failure instead of recognizing that these are fuzzy concepts and indicators are just that... indications, not answers.
Take a leaf from the OH&S book, as OH&S is all about stopping things from happening. Many work places have a huge "Days Since Last Lost Time Accident" sign hanging out the front. It increments every day and the aim is to get it as high as possible.
The same metric could be applied to "Days Since Last Stuff Up That Cost Us". It would be easy thing to apply to an office door or cubicle wall.
I beg to differ. If you have a metric on "time since last outage", people will be incited to hide problems.
There is extensive writing on this subject http://rachelbythebay.com/w/2021/06/01/count/
Reminds me of the millennium problem. There was a post on the large forum asking if that issue was over-hyped because we didn't see the world crumble. The top response was that it was handled because we threw a lot of money at the problem. I tend to think that was right, but there's no way I know how to verify.
Just responding to the title, but I think this is why it's important to have leadership with real "IC" experience. If leadership has an intuition for what's hard and what the failure modes are, they will be able to correctly divvy out acknowledgement more often than a "pure manager" type.
The same thing occurs for many optimizations. Unless something is unbearably slow and you speed it up 10x or 100x; most people won't notice.
You might spend some time making an often called function return in 1 millisecond instead of 2 or 3 and some might notice that things 'seem' snappier; but no one knows why.
Diagram 5 can be improved.
[Time Spent on Improvement] is shown having a negative impact on [Time Spent Working].
But [Time Spent Working] should also have a negative impact going back to [Time Spent on Improvement].
This simple mutually negative feedback loop, i.e. a competitive relationship, better explains the bistable situation where either [Time Spent Working] reinforces itself, while holding [Time Spent on Improvement] down, or vice versa.
Feedback loops reinforce themselves, but there is only so much time, so there will be a strong tendency for one feedback loops to dominate the time. (And since the Time Spent Working -> Reduce Performance Gap loop operates much faster, cheaper in the short run, and is much easier to reason about and manage, it usually wins.)
My first ever job was a placement during university -- like a year in industry.
It was a regular-joe IT support job at an eLearning company (if they still exist).
The FD walked past me one day, having just left what was clearly a deeply uncomfortable meeting.
He stopped in his tracks and asked, quite abruptly, why I didn't ever seem to look busy. I replied - with the arrogance of a cocky 20 year old - that it was because, right now, everything is working as it should. Nothing is broken and people are working and productive, which is because of the work of the IT department.
The company later become insolvent and was dissolved. So it turns out it might have been him who should have been busier, and he was just taking it out on me.
This is something that really annoyed me at a company I worked at for a while. And it was far worse - it wasn't a case of not being rewarded, people who planned properly and got the job done were punished for their efforts. If a project got behind they would take people from a project that was on track or better to put them on to it. I pleaded for a super green status for the best projects on the basis that the team would finish sooner and then the entire team could work on something else nnd get it done well too. They could take people from the next tier but that would be an incentive to do well. Not a hope.
The opposite. Department heads get fired for allowing blow-ups in their area, that should have been avoided. "Getting credit" means keeping the job managing the department via preventing problems.
I’ve seen it mean the department head is told to cut staff. If things run smoothly with 100 people, let’s see how runs with 90, then 80, and so on. The preventative measures start taking a backseat as the staffing issues are felt. The cuts keep coming until something falls over and the department head is pushed out after failing to maintain the service.
But then you need upper management that appreciates the "risk potential" of every area, and what is required to keep that risk at bay.
Except TSA security theater. They see their budgets grow and seen as essential, for preventing problems that never happened.
But in this case, if those problems were to happen, they wouldn't be stopped by their measures.
On their start date, I would to tell new grads assigned SRE / DevOps roles that their job was thankless and at best if everything worked fine no one would notice them. However, if anything went wrong, they'd not only be on the hook to fix but would do so under tremendous pressure. "Who wants this responsibility?" I'd ask.
Those who decided they were up for it usually did a great job, about half opted out at the end of the conversation and were put in roles more suited to their ability to handle stress.
I'm curious about views from the crowd here on how to apply the recommendations from this article in the context of a startup
I've worked in both heavy industry (mining /metals production) as well as my own startup for the last 7 years
I definitely empathize with the view that diligently investing in people, and taking time to build sustainable processes that focus on the mid to long term is the ideal way to go... But when your runway is weeks, not months, and it's ship or die, seems hard to justify such a Rosy ideal
Let the debate begin!
One time I worked on a team where we got rewarded for being "bug heros", because we resolved a lot of bugs. It was kind of funny because we also generated the bugs.
SAP S4/HANA- the "new" ERP software is hardcoded to calculate depreciation until 2045. As programmers you probably know it could be untill year 9999 or much further.
But the consultants will not be out of work. Also a way to push customers to upgrade.
Artificial problem.
On a side note, I there is in my opinion a place for a new ERP disruptor. However making it is hard. Still even those small fish often manage to get money.
This is a failure of internal accounting. If maintenance is reduced, the value of capital equipment declines faster. Production reject rate increases. Accounting systems are not good at catching that.
Repetitive manufacturing operations which capture many metrics about what a steady production process is doing may capture such info. Non steady state systems have a real problem.
There is an interesting twist to the issue if you are doing development support, having a customer ticket queue. If you are diligently fixing errors before they ship you can profit from that. The queue shrinks, the stress levels sink, management is happy with you.
However, if the number of tickets gets too low over some period of time, well, management gets ideas that you have free capacity. ;)
Nobody is a strong word. There are many competent engineering teams that understand the value of strong foundations.
However the problem is quite real in large companies where promo driven development takes over and the incentives are shifted from building what is valuable to customers to building what is valuable to get a promo.
This is true in a sense but also it isn't. Our institutions spend a lot of resources on preventing things from happening, are successful, and the people who work in these institutions get a paycheque every month.
One could frame the entire US military budget as preventative spend.
>One could frame the entire US military budget as preventative spend.
A decent percentage of the population thinks the country should reduce military spending, because they are undervaluing the preventative spend.
Or because the prevention ends up killing hundreds of thousands of innocent civilians. Oh, and it turns out the reason you supposedly went to war in the first place is completely fake.
For the auditory learners, here is a summarized audiobook of this PDF :
https://player.oration.app/c3feabfb-d437-4545-ae17-235d50d61...
The corollary (if you’re cynical) is that you get Massive Credit for waiting to deploy a solution until the problem becomes a Major Crisis. That is, there’s a big incentive to to prepare the solution preemptively but not deploy.
There's one thing I don't get about Y2K:
It was supposed to be banks with lots of COBOL code being impacted, right?
Banks issue 30-year mortgages.
So, why weren't banks being impacted in the early 1970s with mortgages due to be retired in the early 1900s?
A better way to think about it is that you can trust the code you test. Y2K was a problem because it affected _everything_ because anywhere a date was used had to be checked. For mortgages, yes, banks did fix that code earlier but they did it because someone noticed a problem in 1980 and fixed that specific problem, not because they transformed how they wrote code and checked all of the millions of lines of other code which nobody had reported problems with.
Roll ahead a couple decades, and now you have even more code with lots of interconnected assumptions and the guy who originally wrote it retired to the beach at Margaritaville with zero interest in going back to Cleveland to talk about date math. If you’re lucky, a pile of cash could change that. If you weren’t, they passed in a boating accident last month.
My first tech job was working for a COBOL vendor in the mid 90s and heard a lot from customers and other people in the industry. Much of it was cosmetic (rolling from 1999 to 1900 on a display, etc.) but a lot of it would have had real consequences: not accepting credit cards with dates in the new millennium, calculating interest incorrectly, sending people incorrect notices or failing to send correct ones, etc. I remember a someone at Mastercard saying they wouldn’t have been able to processed credit card transactions at all after midnight if they hadn’t done several years of careful work.
The two things which helped were that it got enough attention to get the business people to pay for remediation, and there were enough dates in the future that people got wake-up calls in different industries. Mortgages were the earliest but closer in you had things like credit card expiration dates which caused enough problems that people got serious about deploying updates.
These days, I’m wondering how 2038 will go. We have a LOT more devices floating around and there are still plenty of new devices shipping with 32-bit embedded Linux which may never get updated. Hopefully most of that cheap IoT stuff won’t last another decade but I’m kind of something like an automotive company getting publicly outed for skimping on technical debt management and some people getting locked out of their smart houses.
I call these problems ticking time bombs. This way business has a catchy name to use for the thing they dont understand, its also very clear this needs to be looked at.
Its tricky to get credit when your skills are x10 or higher compared to your team. 2 companies ago, a team of 40 engineers took 4 years and $29M to do a generation product update. The next company I worked for took $1M 5 engineers and 2 years (product from scratch). My current company I did the whole thing myself in 14 months (@ < $0.5M), basically on autopilot. Unfortunately, the stakeholders are unaware that they got enormous value by being lucky of hiring the right person for the project. And I still fight management/budget constraits and hot headedness. Almost enough to drive me to next employer.
Corollary - create fake problems then solve them.
So become a management consultant? ; )
Or mid mgmt at tech co
Volvo probably is an exception in the car world.
Great article, and much of this resonates - but what template has been used here? It looks like it was created using LATEX.
In all seriousness, do companies which don't suffer from this dysfunction exist? If so, are they hiring?
This may be one of the biggest challenges facing human civilization. Climate Change, Financial Bubbles, etc.
To get credit for fixing a problem that never happened, usually you need to make things a lot worse.
Companies say they care about security, but don’t pay as much
Reminds of when you see people questioning the usefulness of getting vaccines for some diseases because they have almost entirely disappeared for the past 50 years (especially in 1st world countries for some), without realizing that this is precisely the reason why they don't reappear.
Like the button logic that works but never had problems?
Theoretically this is the military's entire job.
This explains anti-vaxxers beautifully.
If everyone is vaccinated and the disease disappears (and kills few people) anti-vaxxers think: “why do we vaccinate everyone? The disease didn't kill anyone... and it disappeared."
So if you have a headache and you take an aspirin and the headache disappears you're going to conclude it's from the aspirin, right?
In headache case, the link between illness and cure is instantaneous.
But when it comes to vaccination, due to the time needed for everything to take effect (many people need to be vaccinated, it takes time) anti-vaccines are unable to link the vaccine to a cure for the disease.
For humans, time changes everything, even the ability to make a connection between illness and cure.
That's why I deploy buggy products
Y’all don’t celebrate Petrov Day?
Life is not fair move on
It’s the Jonah paradox.
FTFY: nobody ever gets credit if they don't publicize their successes. By default, fixing problems before they happen doesn't make headlines, so you have to find clever ways to earn publicity. One politician classic: let it become a "problem" enough to earn publicity, THEN fix it. Two examples: Y2K and Healthcare.gov
spot on. As a general rule of thumb I keep a form of activity/issues log and send a monthly summart to the finance guys and head of dept along with "expected issues". Letting things crash and burn is a thing I learned from one of my managers a while back. It's great for raises, bouses and rep. Takes a good deal of effort not to fix things I know will break though.
I call this “I, Pencil Socialism”.
We live in a world where everyone depends on the output Of everyone else - none of us could live without farmers and hauliers, doctors and street cleaners, all of whom Build on a foundation from the past
Assigning value, billionaires collecting rent, that’s the flawed model
It’s not capitalism that makes this world possible (the assumption of I Pencil).
It’s the sharing, the trading, the leaving money on the table because it’s too complex to work out how much Elon Musk’s 3rd Grade Math teacher should get for teaching him the basics of finance.
You don’t get credit - you get to live in the modern world
If there are those who do not get to share equally in that world - that’s a bug not a feature.
We need to fix the bug.
Nice article!
This is why you don't travel back in time and kill Hitler
It even gets worse sometimes. You detect a problem that poses a risk to your organization, manage to convince management that it's dangerous, you get some time to fix it and you fix it. Then, because it's fixed, nothing will happen. And then management frowns upon it and wonders why all this effort was needed.
A big scale example of this was the Corona vaccination. After the large vaccination campaign in 2020, hospitalisation numbers stabilised to levels that were also seen sometimes in previous years during some flu epidemic. Which led to criticism from some people, who said that this was evidence that the whole Corona epidemic was a hoax, organised by a conspiracy of medicin manufactories and policitians who wanted to scare people to gain more power ..
Welcome to Infosec.
[dead]
[dead]
[dead]
[dead]
[dead]
"Great wisdom is not obvious, great merit is not advertised. When trouble is solved before it forms, who calls that clever? When there is victory without battle, who talks of bravery?" - Sun Tzu
[dead]
[dead]
[flagged]
If only! Lots of things seem obvious, like common sense, but they apparently aren't because the behavior of many organizations is to reward the firefighters, to push "work harder" over "work smarter", and other behaviors that (in the long term) impede their ability to improve or expand their capabilities.
I liked the quote in the 2015 submission that twic posted [0], it relates back to the "seems obvious" sentiment. A lot of good solutions are simple (or composed of simple elements in a sane and reasonable fashion). But arriving at those solutions is decidedly not simple or obvious if we go by the Rube Goldberg machines that people end up contriving instead.
[0] https://news.ycombinator.com/item?id=8940820#8941621
just aint true if you file a bug report