Comment by news_hacker

1 year ago

I've been using bkt (https://github.com/dimo414/bkt) for subprocess caching. It has some nice features, like providing a ttl for cache expiration. In-pipeline memoization looks nice, I'm not sure it supports that

1 comment

news_hacker

Reply
aktau  1 year ago

I was not aware of bkt. Thanks for the link. It seems very similar to memo, and has more features:

  - Explicit TTL
  - Ability to include working directory et al. as context for the cache key.

There do appear to be downsides (from my PoV) as well:

  - It's a rust program, so it needs to be compiled (memo is a bash/zsh script and runs as-is).
  - There's no mention of transparent compression, either in the README or through simple source code search. I did find https://github.com/dimo414/bkt/issues/62 which mentions swappable backends. The fact that it uses some type of database instead of just the filesystem is not a positive for me, I prefer the state to be easy to introspect with common tools. I will often memo commands that output gigabytes of data, which is usually highly compressible. Transparent compression fixes that up. One could argue this could be avoided with a filesystem-level feature, like ZFS transparent compression. But I don't know how to detect that in a cross-FS fashion.

I opened https://github.com/dimo414/bkt/discussions/63 so the author of bkt can perhaps also participate.