Comment by jabart
2 years ago
Clickhouse open source and cloud user. My understanding is that the cloud version uses S3. Which would mean that they have very specific tenant pattern and code to run that version. This may be why lightweight work a specific way in that environment, or they need a way to test it at scale that would be hard through a feature flag in the open source product. Lightweight deletes were released to both, and previous roadmaps listed updates as upcoming.
You should be using ReplacingMergeTree if you are doing updates at the current moment.
> You should be using ReplacingMergeTree if you are doing updates at the current moment.
Indeed. Altinity and other community users like ContentSquare made numerous contributions to make it more usable. It's a promising approach to updates at scale and has improved markedly over the last few months.
That said you can't currently use RMT very efficiently in S3 because of overall limitations in MergeTree S3 table storage. We need to think about whether the improvements we're proposing will also enhance RMT. Thanks for bringing that up.
Yeah, its the don't treat S3 like a disk issue. I'm looking at S3 only for cold storage but need AWS VPC Gateway Endpoint support for S3 access since we are on-premise.
*Yes you can use a vpc gateway but need public IPs to waste to setup the BGP/IP routes.
What's the "AWS VPC Gateway Endpoint" support you refer to? Do you mean an S3 proxy? If so one of our PRs that just merged may be helpful. [0]
[0] https://github.com/ClickHouse/ClickHouse/pull/51749
2 replies →
> Lightweight deletes were released to both
Because they were contributed by community member, not ClickHouse Inc core team.
https://github.com/ClickHouse/ClickHouse/pull/37893
Commits show both, yes a community member kicked things over with the initial PR and seems like a team effort to get that feature launched.