Comment by zX41ZdbW
5 days ago
ClickHouse provides idempotent atomic inserts, so the insertion of the same batch can be safely repeated. The client can provide its own idempotency token or rely on the block hash.
5 days ago
ClickHouse provides idempotent atomic inserts, so the insertion of the same batch can be safely repeated. The client can provide its own idempotency token or rely on the block hash.
Afaik this is always best effort, e.g.: https://clickhouse.com/docs/operations/settings/settings#ins...
"For the replicated tables by default the only 100 of the most recent blocks for each partition are deduplicated"
This doesn't work under failure conditions either (again afaik), e.g. if the clickhouse server fails.
100 is the default, and can be changed at runtime.
The deduplication works regardless of server restarts, and it does not matter when a request goes to another replica, as it is implemented with a distributed consensus (RAFT) via clickhouse-keeper.
ah. interesting. So some hash of the batch is recorded in the distributed log after the batch has been written? to disk? Isn't there still a race there?
At least intuitively this seems very hard to guarantee something more than "at least once" but I might be missing something.
1 reply →