Comment by openasocket
4 days ago
Cool! Looks like is used dynamic loading. Which is certainly workable. One downside is that while dynamic loading is well-supported, dynamic unloading is generally not. Meaning if you are using dynamic loading and re-loading for hot updates the server process will start to accumulate memory. Not a huge issue of the server process is periodically restarted, but can potentially cause weird bugs.
You might be able to make something more robust by forking and loading the modules in a separate process. Then do something fancy with shared memory between the main process and the module processes. But I haven’t looked into this much
Once you start forking wouldn't it make more sense to just exec? Woops, CGI.
Doesn't dlclose unload things...?
> If the reference count drops to zero and no other loaded libraries use symbols in it, then the dynamic library is unloaded.
It's implementation defined. The dlclose may be a noop and that's fine as far as POSIX is concerned.
So generally you have no guarantee if your libc actually unmaps the shared object, due to various reasons. There are ways to get it to unload, but that entails digging around platform specific dlopen flags and ensuring symbols in your shared object doesn't use certain load types (unique/nodelete). Thread local storage/destructors also further complicate things.
Some libcs like musl dlclose don't do anything for example and just leave things to be unloaded on program exit.
Look at musl, dlclose marks a library as invalid, so when reallocations happened next the pointers get reused. It's garbage collected, but things get unloaded.
Though, you do seem to be correct that POSIX allows things to remain in the address space:
> The use of dlclose() reflects a statement of intent on the part of the process, but does not create any requirement upon the implementation, such as removal of the code or symbols referenced by handle. Once an object has been closed using dlclose() an application should assume that its symbols are no longer available to dlsym(). All objects loaded automatically as a result of invoking dlopen() on the referenced object shall also be closed if this is the last reference to it.
> Although a dlclose() operation is not required to remove structures from an address space, neither is an implementation prohibited from doing so. The only restriction on such a removal is that no object shall be removed to which references have been relocated, until or unless all such references are removed. For instance, an object that had been loaded with a dlopen() operation specifying the RTLD_GLOBAL flag might provide a target for dynamic relocations performed in the processing of other objects-in such environments, an application may assume that no relocation, once made, shall be undone or remade unless the object requiring the relocation has itself been removed. [0]
[0] https://pubs.opengroup.org/onlinepubs/009696799/functions/dl...
1 reply →
Oh really interesting, will look into it! Was afraid it would leak memory.
You might not necessarily need even shared memory - it's possible to pass a file descriptor over a socket on all modern unices (and albeit with a completely different API also on Win32) so your control process could do an accept(), maybe read the headers in there, then talk to module processes to determine what the desired approach is, and then hand over the http socket so the module process can do whatever it needs to with it.
When I imagine how I'd use this in my head the imagined design rapidly gets much more complicated than what you're currently doing, and I'm not at all arguing that that complexity is necessarily worth it ... but there's also all sorts of cool+weird things you could implement that way that would be exceedingly tricky otherwise, so I figured I'd point it out anyway :D
(the example of code that does the relevant magic to get fds across a socket that immediately springs to mind is https://fastapi.metacpan.org/source/MLEHMANN/IO-FDPass-1.3/F... - yes, warning, it's inside a perl extension, but I see no reason that would impede you borrowing the C parts if it was useful ;)
Initially I did use fork a lot to allow each handler in its own “process” mainly because I wanted to isolate it as well in a container fashion. However the overhead became to much at the start when I just wanted it to work. Will be looking into it again!
1 reply →