Comment by singularity2001
1 day ago
switching to copy on write worktrees is the biggest game changer it should become common knowledge pretty soon and it's already available on most operating systems
1 day ago
switching to copy on write worktrees is the biggest game changer it should become common knowledge pretty soon and it's already available on most operating systems
You also need a deduplication, because with simple copy on write you will have a copy for writing the same - change a byte in a file, get a copy, then change it back, get a duplicate.
Deduplication in ZFS needs 5G of RAM for 1TB of logical data - if you expect your agents to produce 2TB of data, you should provide 10G of RAM for deduplication.
Why are your agents expected to produce 2TB of data? Not criticizing, I'm curious. Outside of classifiers I don't have any of my agents produce data, they produce scripts to produce data so it can be deterministic
change a byte ... change it back, get a duplicate.
What are realistic scenarios in which you change a byte back?
He's saying the build artifacts themselves are consuming the space
thanks to the parent you're replying to, I discovered there's numerous projects that try to retrofit CoW into worktrees + also fill in the gaps for mostly-shared data like build artifacts! I've decided to take lane (https://github.com/lukeed/lane) for a spin as I type to see how it goes.