Comment by Giefo6ah
8 days ago
Ceph solves the distributed consistent block storage problem very well. But I hardly ever need that problem solved, it's way more often that I need a distributed highly available blob storage, and Ceph makes the wrong tradeoffs for this task.
Ceph is fundamentally an object storage system. RBD (block devices) are built on top of that layer.
Yeah. 100% this. Remember, Ceph's storage nodes are called OSD, as in Object Storage Daemon.
The biggest reasons to not use Ceph are:
- You plan on using <=30 disks on <=3 computers. Some of the petabyte scale stuff is just in the way at that scale. Even more so if you're looking at a single computer.
- You don't have at least fractional ops staff that will look at a dashboard regularly. In this world, you're better off using a cloud service.
- You're 100% satisfied with the S3 interface and will never want anything else than its write-once objects. Ceph's writable object support won't gain you anything in that world, and genuinely makes the distributed systems problems involved much harder. Ceph was architected as a distributed filesystem and excels as a networked block store for virtual machines. Ceph can do S3-compatibility, but another implementation can cut off a big chunk of the functionality and provide just S3-compat, and simpler can be better.
I doubt many software projects mentioned in this conversation have gone through the extensive stress testing with glitching nodes while trying to maintain performance that Ceph has; simple systems can be too simple. Ceph is quite well battle-hardened by now. Ceph's dedicated QA hardware pool is likely bigger than many competing projects have tried as a cluster size!
Disclaimer: ex-Ceph-developer.