Comment by jedberg
1 year ago
If you'll allow me to shill my company for a second, I'd be interested if you could take a look at DBOS[0] and tell me if it might simplify your problem. It's designed for making microservices a lot easier to reason about, and might help make your system a lot simpler.
That seems quite clever but for this specific use case it likely won’t solve the difficult bits. Namely I am using threads and subprocesses mixed with asyncio and PyAV to transcode real time video. Does DBOS allow you to store the internal state of Python modules written in C? How does it deal with recreating subprocesses and threads?
It saves the inputs and outputs of the function, not the internal state, so most of what you asked isn't directly relevant. We do have async support coming soon though.
That being said, real time video processing is probably not a use case that we shine on. We'd be really good for managing everything around that, but not the actual video.
Yeah and that of course is the core thing here. Everything else is both quick and safe to retry/idempotent but when you interrupt a demuxer or an encoder, etc. you can’t restart it without feeding the container headers at least to a new instance.