Comment by PxldLtd

This is a really good point. Competitors in this space have a lot of resources so there's a tightrope to walk if you go the OSS core route. Any of these competitors could leverage your core and provide many more features than you could reasonably implement.

BSL/BUSL seems like a good fit for licensing here. It's technically source available instead of open source but just adds the layer that a competitor can't be built using your core. Otherwise the core is free to modify and fork. AGPL might be an option but I fear it would scare off a lot of companies in the space who have policies against AGPL licensed code but you'd get to keep advertising as OSS.