Comment by plqbfbv
21 hours ago
To all of those proposing self-hosted GitLab: we did it for 6+ years in my company, and it's not always a smooth sailing. We had our own runners and we made it auto-upgrade across docker images daily before business start. It mostly worked really well, except those few times were a Docker upgrade had to be rolled back, or that one time the bundled pg_shared_buffers was set at 1MB by default, making schema upgrades impossible for bigger instances, or a version major would break pipeline expectations forcing to upgrade 200+ repos at a time (we pinned to major afterwards). Lately I was also receiving an almost weekly "critical patch" newsletter due to critical/high vulnerabilities, which I can only imagine are due to LLM running over the code and identifying bugs.
That said, I wish we hadn't migrated to GH, our self-hosted instance had WAY less downtime despite being perhaps a bit slower (mgmt saving money) and required a bit more toil: GH is nowhere near Enterprise-ready and it feels a downgrade across the board. GL has better access granularity, better docs, better integrations, and you can clearly see the UI received a lot of attention (although it does take 10m with a new account to pin the proper items in the maze of sub-menus that is the sidebar). You can also look at the code and help out if needed, and/or simply provide a patched version to your image via a docker mount.
If you're really looking at self-hosting GitLab for a smallish team (up to 50-100 ppl), prepare at the very least a 16GB machine (best 32GB) with 4 cores and a decent SSD, and at least a small team (1-3 people) that can maintain it properly or jump at it at any moment. For runners, a small k3s cluster is ideal to make use of all the resources you can throw at it without worrying about managing the runner state/configuration.
> That said, I wish we hadn't migrated to GH, our self-hosted instance had WAY less downtime despite being perhaps a bit slower (mgmt saving money) and required a bit more toil
The key thing that drove our decision to self host gitlab was that WE were the ones in control of when things happened. We didn't do upgrades during times where we really needed our git infra up, and we were in control of our response time to incidents. We were not at the mercy of some other company's upgrade schedule and incident resolution process.
Of course, it comes with added workload and responsibility, but as the above comment says, it was not that onerous. While we paid for Gitlab Enterprise for a long time, we eventually switched to the free tier as that company was winding down (for reasons unrelated to our choice of git hosting), and the free tier actually has pretty much everything we needed.
Never forget the sudden GitLab price-hike and flavor feature changes just a few years ago. I was literally about to go from free to paid, when the pricing doubled nearly overnight. I left for Gitea.
https://news.ycombinator.com/item?id=35144974
Have you looked at migrating to Forgejo? It’s a fork of Gitea though I think it has diverged significantly since the fork.
I migrated my Gitea instance to Forgejo, it was pretty painless
2 replies →
Setting up custom GitLab runners can be cumbersome in the beginning! We faced that multiple times with various customer projects for which we were using GitLab.com managed repos. That's why my colleague and I recently built a "GitLab Runners as a Service" [0] -- simply use your self-managed GitLab or GitLab.com account to login, adding pipeline runners is a 1-click operation! Under the hood we provision a Hetzner machine for you runners and automatically connect your GitLab group or project to it. Happy to hear your feedback!
[0] https://rocketrunner.io/
This is pretty cool and one of the few things I couldn't find a viable alternative to in-house hosting so far. The only thing your "GitLab Runners as a Service" is missing is ISO 27001 (and optionally SOC 2) certification. That makes it again easier for your customer to get / maintain their ISO 27001.
(So far I used this cloud.init script to spin up and upgrade GitLab-Runner instants: https://gitlab.com/21analytics/gitlab-runner-cloud-init/-/bl...)
Ha, let us look into ISO 27001 certification and see what it takes to obtain it (and SOC 2). Thank you for this valuable pointer! If you could just sign in (accounts are free and you can even try us out for the first 48h and not be charged), then we could message you to your email address once we have an update here.
2 replies →
I find the gitlab runner registration process a lot simpler than the github process, and a lot more flexible when you are self hosting the gitlab instance. We used kubernetes runners for both, and gitlab was clearly better.
The worst part about the github runner setup is that there is no built in support for using your own cache if you are using github.com and want to run your own runners. In order to use your own cache store for your runner jobs, you have to patch the runner image because the cache location is hard coded. Gitlab lets you choose your cache location as a standard feature.
Right! That's why running your owners runners is so valuable as you've got full control over the cache location which clearly makes subsequent pipeline runs super fast.
Looks pretty cool!
P.S.: With AdBlockers enabled I cannot navigate on your site at all on Safari mobile. Seems like your navigation uses to much JS to provide a link.
Thank you for building this, looks like it could be really helpful. We've been running our self-hosted GitLab with almost 99.99% uptime and a few runners but the configuration and update process has been a bit cumbersome. Will take a look at rocket runner.
99.99% uptime is impressive! So far our runners are likewise very reliable! It's just a Hetzner machine with our stack on top after all. Thanks for trying us out and please drop us an in-app message for any questions or comments!
ex-GitLabber here, and yes this is a pretty common scenario. GitLab as a product is pretty good, however GitLab Inc, is not great at supporting the broader community especially the SMB-Mid tier self-hosted users. (If you're big enterprise you'll get white glove support)
It's a bummer because they had so much potential to dethrone GitHub a few years ago but really missed the opportunity to position themselves as a viable alternative. That said I still use gitlab.com for my personal projects over anything owned by Microsoft.
As someone who used to submit issues to Gitlab just to see them stagnate for 6-8 years and then ceremonially get closed after being stuck in management/release bump hell (along with many 'me toos' from both Gitlab premium support staff / 3rd party contributors) I agree with your statement 100%.
I also self-hosted GitLab for some years, eventually ended up moving over to Gitea + WoodpeckerCI + Nexus (with Nexus for a while). Well there was DroneCI in the middle for a bit there as well, but Woodpecker is perfectly okay and more free in regards to what you can do. Only thing is that not that much other software out there supports Gitea, for example Kepler only knows about GitHub, GitLab, Jira, Trello and Linear.
Not bad resource wise at all for something serving 50 to 100 people. Don't forget a second machine for HA or at minimum DR !
Yeah, it's mind blowing to me that GH still doesn't have a branch viewer in each repo.
[dead]
I'd also say that gitlab runners are the biggest failure point in gitlab and take some work to micro-manage. If I was setting up from scratch I would consider gitlab for source control, but some hosted service for CI
Really? I've been very happy with how easy GitLab Runners are to manage. We were running 20,000 jobs a week on 6 bare metal boxes and we had tons of spare capacity. Simple Ansible script to deploy the runner and register it. Tags for t-shirt sizes to set the amount of CPU/memory allocated to each job container.
Maybe running on Kubernetes or some autoscaling solution is more painful.
When it works, it's great. When it doesn't, good luck figuring out why.
When you fork a project, it's really not clear in the UI why it's stalling (because it uses your configured runner, i.e. none, not the project's).
When a runner fails, it's not really clear why. gitlab-runner logs are mostly useless in terms of connectivity to the gitlab instance, displaying what projects/repos it's effectively associated with. And i don't think it even has a simple test command to test the docker setup, you just have to trigger jobs until it fails in the middle of hundreds of lines of logs.
Maybe give https://rocketrunner.io a try? We made super easy and fully managed GitLab CI/CD runners for GitLab.com or self-hosted GitLab servers :-)
(Sorry for posting this twice, but we'd really love to get more user feedback)
We did tun our own installation at work, supporting ~120 developers.
A single vm, 8 cpu cores and 64gb memory, using ssd disks on a local vmware cluster.
It worked okay but:
1. Was only exposed to the corporate network and some vpn connections
2. We did the upgrades late at night once every like six months, taking a snapshot of the whole thing from vmware side before touching anything
It worked pretty well and was relatively low maintenance. Frankly a very pleasant experience.
I’m using github and bitbucket at work nowadays and frankly i miss that experience, both as an user and as an administrator.
Edit: btw i left that job and that gitlab installation in 2022, no idea how things changed in the meantime.
> To all of those proposing self-hosted GitLab: ... it's not always a smooth sailing.
> (we pinned to major afterwards)
I mean OK. But also, if I want a reliable service in my business, I wouldn't typically auto-upgrade docker images nightly to "*".
> I mean OK. But also, if I want a reliable service in my business, I wouldn't typically auto-upgrade docker images nightly to "*".
Well fair, but we had been using it for 4+y at the time before running into that, and the rails-based migration scripts basically never failed once despite being migrated across 6 majors and countless minors/patches. Being a small company, from that PoV it gave us very little downtime for essentially zero security work to maintain it. Of course we had daily (and tested) backups too, just in case, but we never had to use them.
Overall, the downtime of GL, even with auto-upgrading and the issues above, was perhaps 1.5 work-days across 6+ years. GH exceeded that budget in the first 2 months after migrating this year...
4 years of running production GitLab on nightly is very impressive! I would never have imagined you could last that long. Clearly the quality of GitLab releases is much higher than I imagined.
gitlab.com and selfhosted gitlab in various companies in the last few years
Ok, with DSCI as monolith application where git and CI are the same server, 32GB RAM VM is more then enough , no need to host runners, no need in k8s cluster, no need in dedicated maintain team, so no extra costs on devops tasks ... Also with general programming languages for CI pipeline you are in full control and simplicity ...
This thread is about full-featured forges like github/gitlab.
That DSCI is not a forge - it's a task runner attached to gitweb frontend. There are _none_ of the forge-like features like user management, pull requests, etc... Even in CI area, most of the features are missing: from the quick perusal of the doc, even something as basic as "have runner on remote machine" or "run two jobs at once" is not implemented.
- you don’t need runners on remote machines unless you you have hundreds of jobs and which never happens for small or medium size dev teams - job matrix / parallel jobs are barely needed in commercial development , only in open source , but still not in every project - pull requests and user management are coming soon
Whoever downvoted this ^^ my comment I need your reasoning … I don’t know why people hate simplicity ) ( or love complexity )
it is a simpler set of services at the expense of a more complex user experience
1 reply →