Comment by ice109
8 years ago
Lots of people in the thread are commenting how surprised they are that a junior dev has access to production db. Both jobs I've had since graduating gave me more or less complete access to production systems from day one. I think in startup land - where devops takes a back seat to product - it's probably very common.
I work for a large bank as OPS engineer. The idea that I could even read a production database without password approval from someone else is too crazy to consider. Updating or deleting takes a small committee and a sizeable "paper trail" to approve.
Sometimes when I read stories like these, I think it's no wonder a company like WhatsApp can have a billion customers with less than 100 employees. And then I make some backups to get that cozy safe feeling again.
Probably because your industry is regulated.
Which doesn't abdicate responsibility from the CTO of the company to have practices in place that could have prevented this. While I'm going to hold my breath on being threatened with legal claims, to be fired for something that any person in the building could have done doesn't sound like a conducive environment to work in.
absolve not abdicate. you abdicate a throne, you are absolved of responsibility
You also abdicate responsibilities; ie you leave them, as a king leaves a throne. You can abdicate a philosophical position, just means you reject it.
Eg https://en.m.wiktionary.org/wiki/abdicate
2 replies →
I didn't look at comments until today, I love that the primary thread I spawned was the semantic differences between absolve and abdicate. I appreciate the criticism, I learned quite a bit about the differences :)
I think people are trying to say abrogate. I've seen this usage a few times lately.
2 replies →
Correct, regarding his usage as a synonym for excuse, forgive or mitigate.
However, you can abdigate a responsibility. One can say for example that the CTO abdigated his responsibility to ensure the production database was protected and backed up.
Abdicate is correct here. It also means "to cast off", although that meaning is seldom used anywhere other than the phrase "abdicating responsibility".
Edit: not sure why I'm being down voted. I am correct: https://www.merriam-webster.com/dictionary/abdicate
1 reply →
> I think in startup land - where devops takes a back seat to product - it's probably very common.
It is, but it need not be. It's pretty easy to set up at least some backup system in such a way that whoever can wipe the production systems can't also wipe the back-up.
In this particular case, the OP indicated his team size was 40 and the whole company was about 100. I'd argue that's beyond "everyone has prod access" size.
https://www.reddit.com/r/cscareerquestions/comments/6ez8ag/a...
Its a pretty gross error to me to have direct db access. Obviously in any stack you could push code that affects the db catastrophically anyway, but in dev mode you should never connect directly to production database, not only for this error but for general data integrity.
CTO needs to put a theatre to not get fired, because he is ultimately responsible.
As a junior noc analyst at a Fortune 500 company, I had root access to almost the entire corporate infrastructure from day one. Databases, front ends, provisioning tools, everything.
It's not about having access to the production database. It's about having an example script that can do catastrophic things and having the production username and password in the example docs.
I also had production access to everything from day one. The first thing I did was setup the hosts files in the various dev servers - including my own computer so I couldn't access the databases from them.
I have to remote into another computer to access them.
In that case don't you think they should informed him that this is a production environment so don't fuck it up? Giving your junior dev a front seat on your production without proper communication is a disaster waiting to happen.
I think in startup land - where devops takes a back seat to product - it's probably very common
Perhaps with hipster databases like MongoDB that are insecure out of the box, but most grown-up, sensible DBs have the concept of read-only users, and also it is trivial to set up such that you can e.g. DELETE data in a table without being able to DROP that table.
I'll wager any startup that does like you say has devs that do SELECT * FROM TABLE; on a 20-col, million+ row table only actually wanting one value from one row, but they filter in their own code... Yes I have seen this more times than I care to count.
I agree.
But not in the commands to run in a local-dev-setup-guide that purges the db it points to.
If anyone should be fired for that, it's the CTO. He must suck at his job at the very least, and junior dev should get an apology.
> I think in startup land - where devops takes a back seat to product - it's probably very common.
Not focusing on devops and putting your prod db credentials in plain sights are VERY different things. It's really really easy to do, especially if you are using Heroku or something like that. Same thing goes with database backups. I worked at multiple startups (YC and non) and they all had the basics nailed down, even when they were just 3-5 employees
The problem isn't so much that a junior had access to the production Db. The problem is that the junior's dev setup had access to the production Db and could nuke the whole thing with a few misplaced keystrokes. I'm working on a product currently where I am the only dev. I have a pretty large production Db. I also have a smaller clone of that same Db on my local machine for development purposes. I can only access the production Db by directly shelling into the machine it's running on or performing management commands on one of the production worker machines(which I also need to shell into). This was not very difficult to set up and ensures that my development environment cannot in any way affect the production environment.
Also, why even distribute the production credentials at all? Only the most senior DBAs or devs should have access to production credentials.
I've done about ~40 or so technology due diligence projects for investors of tech companies. You'd be amazed how many security flaws there are out there. One of the most simple ones - storing production credentials in the git repo.
Sure, there exist reasons to do that. It's still a bad idea, but, ok.
But there is no reason to write the production DB credentials in a document, especially as an "example". That is monumentally dumb. It amounts to asking for this to happen.
We give everyone access to production systems, but even if someone deleted everything from production, we can restore everything in ~20 minutes (this has happened), and if that process fails, we have backups on s3 that can be restored in a couple of hours (and this is tested regularly, but thankfully hasn't happened yet), and even if that fails...
There's a reason why it's called disaster recovery and prevention.
Why try to justify stupid behavior and absent security controls with the idea that your downtime is "only" 20 minutes? How silly.
I've only had one job after college, and I am still there almost a year after being hired. The first few months I only had access to my own local copy of the production DB. Though there's a reason or two why I wouldn't be outright trusted one of them stemming from me being a junior.
Wow, really?!? I've never granted access to prod databases on day one, junior or not.
I thought that was just SOP.
Can you send me your password again, I forgot it. Also, please reply to those emails from emily - or just delete them... they are cluttering up your inbox, and I am tired of having to sort through your guys' personal crap.
Thanks
Same. I had instructions from a competent developer however. I would still blame whoever allowed production access as part of application setup, as well as the fact there isn't a process to back up this production data.
It's not that this shouldn't happen, but that it does happen and has to be dealt with as the potential impact scales up. Having production creds on day 1 isn't the same as day 500.
Perhaps, but in 2017 it is gross fiduciary malpractice not to have backup systems in place for production data and code. It would be grounds for a shareholder suit against the principals.
Common does not mean it's correct or has to be or whatever. Trump thinks climate change is hoax, does not mean that it's convincing...