Comment by megous
3 years ago
At $29 * 7 * 12 per year it became way cheaper for us to just piece together functionality we needed from nginx+cgit+homegrown database to store users/repos/acl/push info and a few git hooks written in a few lines of PHP. The cost is now independent of the number of developers using the system.
So far it did cost ~1 month of paying for github in dev time and I can't imagine it costing much more when we'll want to add some automation on top of the list of accepted pushes/ref updates, for which we did not have a need for so far.
Certainly beats installing 1GiB debian package of selfhosted gitlab and having to figure out why some stupid ruby service is eating increasing amounts of hundreds of MiB of RAM on an empty gitlab instance while doing nothing at all.
That's $2.5k/yr that can be put into something else.
When developers do operations... I guess. :D
I mean I dislike the gitlab pricing increases, and we are kind of regretting sticking with them (especially since everyone on the Gitlab instance has to pay for a seat, even when not using any of the premium features) considering how Github is cheaper and has a higher release velocity at this point... But to be very fair, we never saw what you describe about memory leaks or slow ruby services. Migrations are also usually silent and done in the background. I guess the software itself is pretty heavy, but predictably so (RAM usage rarely spikes, for example).
We have a 400+ seats instance, with a decade worth of code and it just works 99.99% of the time, and upgrades are generally painless. Though our instance is pretty vanilla, and internal to our corporation. The runners are more finnicky and buggier, especially since we have tons of different build targets but that's besides the point if we are just talking about using it as a dumb git repo. And if you only need it for that purpose why even pay instead of using the community edition?
(I'm in the AI team but I sometimes help the devops team that takes care of the instance to customize our AI pipelines, so I'm familiar with gitlab and with my colleagues experience with it)
The company I'm currently at uses a self hosted instance, which due to our "security"* constraints we're grossly behind in updates. Not to say that's Gitlabs fault at all, but the amount of issues we have near daily is hilarious (if it didn't simultanously crush productivity).
That said, I recall issues with the Gitlab Enterprise Support / whoever we contact about our license etc, mostly to do with slow or poor communication. Requests would sit for a month before we got a reply, often dismissive or generally unhelpful, though we _were_ on an outdated version so I don't blame them.
I vaguely recall an email about support ending on self hosted instances? I can't recall the details, but I know it triggered an internal investigation into moving away from Gitlab. EDIT: Pretty sure I'm remembering self hosted Jira, a quick Google search shows Jira EOL but no Gitlab.
All of that said, I largely blame my company for the failures here. I wouldn't expect any company to support self hosted, outdated versions. The support issues were annoying, but I'm also not sure how much I can blame Gitlab for that. I'm also struggling to remember the details, so take this as one mans hazy annecdote.
* film industry, security by obscurity in the worst ways, leads to incredibly outdated and neglected tech (we only _just_ transitioned to Python3, and that's only the core services)
At least parts of what you say ring true in my experience following GitLab releases more closely (e.g. 1-2 months behind). I would highly recommend not self-hosting if you're going to go with GitLab. Performance issues will appear and disappear between updates, and sometimes on a whim and you're never quite sure what you did or didn't do.
I did find the support staff to be fairly responsive, but most of that time felt like me collecting diagnostic information with little actionable material, and sometimes I would have to explain the same thing multiple times in the same support ticket because it switched hands.
If you do still opt to self-host, dig into their documentation: there are little nuggets and hacks they use internally that you'll want to use to get the right performance out of it.
3 replies →
My bad experience was mostly just with installing gitlab on a 1GiB RAM VM, to see how it will fare and how easy it is to manage. I expect it to work for people who don't mind throwing a 16GiB+ RAM machine at it. But our dataset is currently like ~200 MiB and simple relatively dumb git hosting just works much better in our case.
GitLab should run fine on a machine with 4GB memory - this is the smallest recommended memory allocation, spec'd for up to 500 users. 2GB tends to work okay for testing but 1GB is indeed probably too small for all of the services to start. Postgres actually tends to be the long pole in the tent on small systems.
1 reply →
The problem with a homegrown system is that every developer working within it will see it as a barrier rather than a cost saving. Whenever they can't do something that's trivial on Github/Gitlab it'll be a reminder that the company would rather save $xxx/year than make their job easier.
Ultimately, having a simple, well-integrated, industry standard stack that includes the same tooling every other company uses is a perk of working at a well-funded or profitable company. People leave for less.
> it'll be a reminder that the company would rather save $xxx/year than make their job easier.
My company has already made this crystal clear by switching to MS Teams.
Every morning we begin stand-up with a ritual, our whole team simultaneously muttering something arcane whilst manipulating the inner workings of Teams, hoping to appease the MS gods.
Shortly after the ritual is complete you hear a cacophony of "fucking teams wouldn't find my headset/camera", and the day begins.
2 replies →
A developer costs about 250K a year all up. That doesn’t include opportunity cost.
49, 40 hour weeks a year = $127.55 an hour.
29* 7 * 12 / (127.55) = 19.09 hours
If it takes you more than 2.3 dev days (100% productive) you’re negative ROI doing it yourself.
This math doesn’t even factor in the opportunity cost of doing this.
>> A developer costs about 250K a year all up.
Gotta love HN math. Calculations to two decimal places, starting with a number plucked from the high end of a distribution with standard deviation of at least 100K.
> the high end of a distribution
What a dev costs is not the same as what an advertised salary for a dev is. This is closer to the middle of the distribution for dev costs in the US.
Ok, so at 80K a year, that’s a 3x multiplier.
So annually you have to spend < 3*19 = 57 hours on your custom built source control and CI to come out on top.
Can’t be done outside of “we hired the OG devs of Gitlab/hub/etc”
1 reply →
> A developer costs about 250K a year all up.
I currently cost more like 80K in euro's if I reason from my employer's side. So tell me, how are you getting to the 250K exactly?
Europe exists as well and even in the US there are enough companies that don't pay FAANG salaries.
Even if you throw out SF salaries as a wild outlier, this isn't actually that ridiculous. An average quality mid-career dev (5-10 years exp.) in a second tier market like Chicago/Denver/Austin/Boston can pretty easily make 170-200 in cash. "[A]ll up" is the key here tho; there's a big non-salary component in the US that doesn't exist in Europe. Tack on health insurance and the total cost to the company will easily blow past 250 right there. Plus, you're probably giving them some equity and a yearly bonus.
I'll be honest, it does sometimes blow my mind to see how low salaries are over there when I look at job postings and Who's Hiring and the like. I'm jealous of a lot of things Europeans take for granted, but it's wild to me to see senior positions in major European capitals paying the amount that I made two years out of a bootcamp.
1 reply →
US salaries don’t include employer payroll taxes, unemployment insurance, health insurance, retirement matching, etc. Total cost can easily be 2x advertised salary.
Don't tell them it becomes 40k after taxes or people here won't believe you.
6 replies →
A developer:
in the US
in specific rich cities in California, New York, Texas, Washington, and Colorado
at a handful of tech companies which are currently bedevilled by lay-offs
averaged across all roles
GitLab doesn't have regional pricing.
That assume that gitlab is friction free and takes zero time to use?
Maybe something like gitea+teamcity solution is better and cheaper and has less opportunity cost?
I like gitlab, but it feels they have been walking in the wrong direction for many many years and the consequences are starting to pile up.
I think the point is too many times developers are a penny wise and a pound foolish when it comes to build vs buy. Because they can build, and it is often fun if you haven't done it before.
Though I often don't take my own advice, I try to ask is what I'm about to build core to our ROI/product, is there an existing solution that gets us 80% there, and yes, what is the cost.
If it is not core to the business, won't drive revenue, and cost are not outrageous, which is a company by company truth, then I much prefer to spend X time building new product than recreating an existing product/tool.
One final point - don't forget to consider the maintenance costs, which in the long term often greater then the initial investment. If your CI goes down, you just blew away and savings you were planning on.
There are also maintenance costs with the off the shelf solutions. E.g. Gitlab self-hosted runners don’t handle very large artifacts well and frequently caused CI to timeout. We had to roll our own artifact management system.
In general I agree with you off-the-shelf is usually better but sometimes a custom tailored solution rather than a generic system can be better.
Their solution is not a monthly recurring event.
Their solution will require a developer to ssh into VMs to update, debug, upscale etc.
1 reply →
Yeah, now its 24/7 ops headache.
1 reply →
Hehe, even at US prices we'd break even after half a year.
Average salary where I live is $21k/yr with average devs taking maybe $25-40k/yr. Top jobs being ~$65k
The classic HN Dropbox comment, never gets old.
… sometimes, I do think it's merited. We switched to Github Actions for our CI and it's been … mixed. It was a hell of a lot of work to switch (GHA is significantly different than what we were moving from and … buggy) and even then, some of the pricing on GHA high. 24¢/GiB/mo for storage[1] which is like 8×? 10×? the sort of cloud baseline, 50¢/GiB data transfer [2], and the runners are similarly costly, IMO. I'm not sure if GHA wins out, in part because we're not done switching.
You can do self-hosted runners aaaand… welcome to infra/ops?
A little NIH syndrome would do the industry good. We need more stuff invented. The current stuff is mediocre.
[1]: https://docs.github.com/en/billing/managing-billing-for-gith...
[2]: I think it's only egress, and only from packages though?
Yes, but all we used gitlab for was git repo hosting with push access control for about 40 repositories to allow sharing code via push/pull and not much else (almost no issues/pull requests). It's ridiculous to pay $2500/yr so that 7 people can do push/pull over slow growing set of ~200MiB of data when it can be handled in much more performant way by a 36$/yr (+ a small one time setup cost) VM we already had for running other things.
Why were you paying for it then?
GitLab has a CE version that’s free for commercial use. It sounds like you just want a pretty interface for git repos with some merge flow.
1 reply →
Why don’t you just run gitlab community edition or gitea
Gitlab is massive piece of software, and has inordinate HW requirements for the job we'd be using it for. Also it would not run on the VM we already have set up for other services, backed up, etc.
Gitea is fun and all, but it's not in debian repo, so would require manual updates. Annoying.
2.5k doesn't pay 1 day of a developer in a top-of-market startup.
I need to up my rate :D