Comment by kendallpark
7 years ago
> They’re more likely to fight for what they believe in, challenge the status quo, and push the organization to make painful but necessary changes, he says.
A long time ago, I had a new employer that wanted me to hotfix non-critical code changes to a live, popular website via FTP with no version control, no nothing. Like a single typo could crash a high traffic site. I told them this was extremely dangerous and irresponsible, and refused to do any changes until we got the system under version control with a reasonable way to deploy changes. This caused a huge standoff and I basically put my foot down and told them that I wouldn't be deploying any changes until I got the client's website under version control--they could dock my pay for the extra time it took, I didn't care. Anyhow, I set up version control and a deployment strategy that we could rollback (this only took a day or so). Then I made the changes and educated the team on how to safely push changes to the site.
A month or so later they decided to not renew my contract, but by then the feeling was quite mutual.
I told this story while interviewing at what would be my next employer and it worked as points in my favor.
EDIT: FWIW I consider myself a generally agreeable giver until a certain threshold is crossed. One such threshold is stuff that moves into what I consider to be ethically irresponsible behavior. Perhaps you would call this a "selectively agreeable/disagreeable giver." There are "disagreeable givers" that pick their battles wisely in terms of when to apply friction and when to go along with the team. On the flip side there are those that will apply friction to every little thing and might relish in complaining and "I told you so"s and cowboying up their own solutions for every small disagreement.
EDIT2: Clarifying that the "hotfixes" were non-critical, normal code changes.
I am impressed you stood your ground on a technical quality issue. One thing though - if you find yourself in this kind of situation in the future, try to inform your employer of the technical risks (data corruption, data loss, lack of accountability when multiple people edit files), and get them to sign off in writing that they agree to the risks. That way, they cannot blame you if the system ever becomes screwed up.
I'm I the only idiot who would back up the files and make the change, and thinks that the hero contractor's wrong here? Version control is nice, but it's not a prerequisite to shipping, and not using one isn't "extremely dangerous and irresponsible" unless you're working on control systems at a nuclear power plant.
Maybe you shouldn't make the change until you have a Selenium suite, automated rollback, blue/green deployments, a bulletproof status page, and an on-call rotation with a regularly tested runbook. Maybe we should just sit on our hands and make internal improvements until things are perfect.
Reductio ad absurdum. I shoved a code base into version control and set up a super simple but safe deployment strategy. No fancy anything. It put off non critical development for a day and made development faster because I could make sure my code worked locally and not just on the live site.
Some companies value hyper-compliance, some companies value employees that speak up in cases of negligence. You have the freedom to choose your own preferences.
Furthermore, it IS possible to respectfully communicate your concerns and position without being a heroic jackass. My manager on that project and I were on good terms when I left and he served as a recommendation for my next job. The situation wasn't nearly as exaggerated as you're making it.
Version control is a must have for any project involving more than a few files or a few developers. It's simply not acceptable to work without version control in this decade.
Yes, the approach was, in practice, wrong. Make the backup, get the patch out, then put in the deployment system on the side.
Yes this is the best approach IMO. CYA (cover your ass) by explaining the risks, take backups (file & db) and push the hotfix. Sometimes you have to work with what you got while moving toward an ideal.
You then sell them the version control benefits and get them to sign off on that scope instead of being an obstructionist.
Yes, this is good advice. I was a doe-eyed fresh dev back then and in an advantageous position in the sense that nothing was riding on this job. Getting fired wouldn't have been the worst thing in the world. That said, had I gone ahead and made a mistake doing cowboy development on a live site... it would've fallen in my head I would've lost the contract anyway.
As I mentioned in another comment, I work in health now. You really can't turn a blind eye to these things in my current field. You might say I found a field more calibrated to my speed-quality balance than the old job.
1 reply →
I think there's a trade off there. There's something to be said for "getting shit done" in the short term, voraciously arguing why it was bad afterwards.
Weirdly enough, "getting shit done fast" were the exact words used in the meeting where it was explained why my contract wouldn't be renewed.
Like I said, the feeling was mutual. Every company and person can decide how deeply they're willing to cut corners. My speed-quality equilibrium and this company's speed-quality equilibrium did not match.
(But seriously can I get a show of hands on HN as to how many devs would be willing to accept FTPing non-versioned code changes to a live, high-traffic website as a standard of practice? I don't think my speed-quality equilibrium is extreme by any means.)
It's not a good practice, but I've done it. The thing is, most live high traffic websites are doing things that are absolutely pointless and the world would not be greatly harmed if they went down. It might even be improved!
It's not my job to prevent some assholes from taking whatever risks they want with their piece of shit non-critical website, once they're in full possession of the facts.
Actual critical systems where lives are on the line are another story completely.
You have to decide for yourself how much you can tolerate and still respect yourself. I'd have made the same choice as OP, and because I know that about myself, I keep fuck-off money in my bank account in case I ever need to take a stand that gets me fired. Self-respect is more important to me than any one job.
Yeah, I'm with you there. Like at the end of the day, I don't want to work for a company that treats a client's site with such gross negligence. The client was trusting us to manage the site responsibly and we would have been majorly violating that trust with those actions. To me, it felt like an ethical issue, I wasn't even thinking of legal ramifications (which I imagine there are some). I was very young then.
Nowadays I'm in the medical field. If you let standards slip to "get shit done fast," you can kill someone.
I don’t yet have that luxury, but this is one of my big overriding goals - FI by early 40s.
I’ve been in one or two situations where I would have walked if not for my mortgage and being sole provider with a young family.
Absolutely hated not having that choice, remembering how disempowering that was keeps me on track to work towards that freedom.
(I am not in SV, and good options at my level in my location take a few months to find)
1 reply →
There's also a thing called negligence for which you can quite rightly be dismissed and sued for.
Then all you are left with is shit.
It’s hardly “extremely dangerous.” Hotfixes like that are very common on for technically unsophisticated web sites. If a typo did crash it, what’s the big deal? Just fix it and move on.
How critical was this website? How much would people be put out if it was down for a few minutes?
With all due respect, I don't want to go into all the details of a previous employer and their relationship with that client so that people on the internet can retrospectively tell me whether I made the right call or not.
Suffice to say, as I've leveled up as a developer over the years, there are many decisions I've later realized were naive or misinformed.
With all the knowledge and experience I have now: I have zero regrets about the above standoff.
FWIW if someone told a dev to FTP-cowboy develop at that company now, they'd be the one getting fired. When I was there, the company was in the midst of transitioning into more serious web dev practices.