Comment by Uptrenda

I'm building an open source NAT traversal and networking framework called P2PD. Built from the ground up to allow things like multi-network interface applications, improved network programming in Python, and if people want it: an easy way to bypass NATs. The thing is: it depends on public servers for some of this which tends to change a lot, causing errors when they're all down.

What I'm building at the moment is a server monitoring solution for STUN, TURN, MQTT, and NTP servers. I wanted to allow the software for this to be portable. So I wrote a simple work queue myself. Python doesn't have linked-lists which is the data structure I'm using for the queues. They allow for O(1) deletes which you can't really get on many Python data structures. Important for work items when you're moving work between queues.

For the actual workers I keep things very simple. I make like 100 independent Python processes each with an event loop. This uses up a crap load of memory but the advantage is that you can parallel execution without any complexity. It would be extremely complex trying to do that with code alone and asyncio's event loop doesn't play well with parallelism. So you really only want one per process.

Result: simple, portable Python code that can easily manage monitoring hundreds of servers (sorry didnt mean for that to sound like chatgpt, lmao, incidental.) The DB for this is memory-based to avoid locking issues. I did use sqlite at first but even with optimizations there were locking issues. Now, I only use sqlite for import / export (checksums.)

Not anything special by HN standards but work is here: https://github.com/robertsdotpm/p2pd_server_monitor

I'm at the stage now where I'm adding all the servers to monitor to it. So fun times.