← Back to context

Comment by znpy

6 years ago

I know about that, and we used to use that but had to move away from it because it created a lot of scalability problems (mind you, most of them due to disk usage).

With harbor we can save docker images layers to switft, the openstack flavor of object storage (S3). That solves a lot of scalability problems.

AFAIK gitlab ships the docker Registry underneath so the problems stay, mostly. I think that harbor does the same. I skimmed the harbor source and it seems that it forwards http requests to the docker registry if you hit the registry API endpoints.

Haven't looked at Quay but as far as I know wherever there's the docker registry you'll have garbage collection problems.

One note on the side: I think that quay missed their chance to become the goto docker registry. Red Hat basically open sourced it after harbor had been incubated in the cncf (unsurprisingly, harbor development has skyrocketed after that event).

4 comments

znpy

Reply
jschorr  6 years ago

(Co-founder of Quay here)

Scalability and garbage collection are actually two of the main areas of focus Quay has had since its inception. As you mentioned, most modern Docker registries such as Quay and Harbor will automatically redirect to blob storage for downloading of layers to help with scale; Quay actually goes one step further and (for blobs recently pulled) will skip the database entirely if the information has been cached. Further, being itself a horizontally scalable containerized app, Quay can easily be scaled out to handle thousands of requests per second (which is a very rough estimate of the scale of Quay.io)

On the garbage collection side, Quay has had fully asynchronous background collection of unreferenced image data since one of its early versions. That, plus the ability to label tags with future expiration, means you can (reasonably) control the growth problem around images. Going forward, there are plans to add additional capabilities around image retention to help mitigate further.

In reference to your note: We are always looking for contributors to Project Quay, and we are starting a bug bash with t-shirts as prizes for those who contribute! [1]

[1] https://github.com/quay/quay#quay-bug-bash

Edit: I saw the edit below and realized the bug bash is listed as ending tomorrow; we're extending it another month as we speak!

  • znpy  6 years ago

    Hey, thank you so much for your reply.

    My message wasn't meant to dismiss quay, I hope it didn't come across like that.

    I'll be giving a look at what's repo in during my vacation... I haven't loaded that many images on my private harbor registry, it might still make sense to switch :)

    Edit: I just realized that the bug bash ends tomorrow... :/

    • jschorr  6 years ago

      Not at all!

      I thought I'd just give some interesting background info and not so subtly ask people to contribute to the project :D