Comment by Intralexical

`zpaq add archive.zpaq new.mbox -fragment 0 -method 3` is great for this. It splits the input into fragments averaging 1024 bytes in size [0], which catches up to ~90% of redundancy. The remaining ~10% is packed and compressed into 64MB (max) blocks that are added to the .zpaq.

The resulting artifact is a single .zpaq file on disk. This file is only ever appended to, never overwritten, so it plays nice with Restic's own chunked deduplication. Plus it won't flood the filesystem with inodes and it suffers less small files overhead than TFA's solution.

Granted I suspect TFA splitting on the e-mail headers may be chunking more efficiently. Though, unless I skimmed the linked GitHub too fast, it looks like TFA's solution also doesn't use any solid compression to exploit redundancy across chunks. And I trust zpaq as a general purpose tool more than a one-off just for a single use case. The code does look clean, though, nice work.

[0] Average fragment size is 1024*2^N. If the most of the data is attachments that don't change, you can probably use a higher `-fragment N` to have less overhead keeping track of hashes. `-method 3` is a good middle ground for backups. `-m5` gets crazy high compression ratios, but also crazy slow speed. Old versions of ingested files are shadowed by default; use `-all` when you want to list/extract them.