Comment by baby
2 years ago
I really wonder why it’s so expensive to run. I always hear things about scaling but I used to run a top 500 alexia website and it was just a php app running on a mutualized offer for $5/month. Lots of manual caching though but still.
My wild guess is that either the stack is not really optimal (last I heard it was java) or they do other costly things at scale (sgx?)
I guess, then the question is how real time was the website. Was it as real time as supporting, instant messaging, voice/video calls etc
Oh I forgot that signal is not just about forwarding messages. I’m wondering how much the VOIP costs.
FTA: "Signal spends around $2.8 million dollars per year on bandwidth to support sending messages and files (such as photos, videos, voice notes, documents, etc.) and to enable voice and video calls."
Don't forget media!
> the stack is not really optimal (last I heard it was java)
how's java relevant here?
Java in theory and in synthetic benchmarks: damn near as lean and mean as C.
Every actual Java project: “oh, did you want that memory and those cycles for something else? Yeah, sorry, I need them all. Why no, I’m not actually doing anything right now, why do you ask?”
In this case we don’t need to speculate at all. Signal is open source. Back when I was at Twilio we even did some at-scale experiments with running Signal. The intensive parts have absolutely nothing to do with Java because the server logic is relatively simple. The hard parts of Signal are the database storage/retrieval and the encryption.
100% true in my experience. Literally anything else is far better when it comes to bloat, including C#, RoR etc.
Increasing the Java heap size just makes it so that when garbage collection eventually hits, it causes an even more massive slowdown across the entire application.
You can't send an sms yourself like you can an email. Instead of setting up a server, you have to work with a telco provider (an aggregator specifically). Any SMS service eventually hands off to one of these. Many SaaS SMS providers are just frontends for legacy telco services. They charge insane fees because they can, that is all there is to it.
Sending mass email is still difficult. Its probably easier to pay a provider than set up and establish reputation for yourself. But they don't charge near the rates. Last time I compared rates it was something like 10x-100x to send an sms compared to an email, but it has been a while.
> Many SaaS SMS providers are just frontends for legacy telco services.
I worked on an automated SMS marketing system back in the day so I have seen this in action, at scale. This would be stuff like "text LAKERS to 12345 for Lakers updates"- we didn't handle the Lakers but we did handle many sports teams. Though I wasn't privvy to the financial side, I got the sense that the per-text cost ended up being manageable at scale, but this is because we were one organization who would apply the rules onto our own customers, and if we failed to do so properly we risked losing the interconnects to the various carriers. We typically used a single contracted "aggregator" service which provided a unified API for the carriers. When I left, we were using OpenMarket.
When you have a self-service SaaS offering such as Twilio, the per-text costs are going to go up because the barriers for sending unwanted texts (or fail to follow the rest of the rules mandated by the TCPA) is so much lower, and Twilio has to address that organizationally which adds cost.
Additionally, Twilio does not purchase short codes (ie 12345) which means its harder for the carriers to track bad behavior across their network. There is an initial cost (fairly high) to acquiring a short code, though you can also share short codes across customers in some cases. Acquiring a single short code and sending all messages from that short code would likely reduce costs.
I would love to see more detail from Signal about what sort of SMS interconnection they are using, because directly connecting with an aggregator instead of a SaaS offering (if they haven't already) could save a lot of money, and they are definitely at the scale that would allow for it. And given that they only use it for account verification and are a non-profit, it seems likely they could get a good deal since the risk of TCPA violations is effectively zero.
Yeah, aggregator is a very industry specific term, so I just merged into teclo provider. But yeah, all the issues with short codes, national laws, and reputation, makes it very complex. I worked at a company like Twillio that had contracts with different aggregators across the world, and sold a platform to manage SMS interactions. They added a layer to make ensure customers respected opt-out keywords, or opt-in for specific countries, so it would help manage TCPA (and other) violations. I imagine this helped keep costs down. We would definitely fire customers for trying to get around the safeguards.
I was on the support side, so I just saw when it went wrong, which was a lot.
> Additionally, Twilio does not purchase short codes (ie 12345) which means its harder for the carriers to track bad behavior across their network. There is an initial cost (fairly high) to acquiring a short code, though you can also share short codes across customers in some cases. Acquiring a single short code and sending all messages from that short code would likely reduce costs.
Twilio offers short codes, but short codes are country specific, and the costs for sending to the US are low anyway < ~ $0.01/message for most services, lower with volume; IIRC, short code messaging costs were half, but then you've got some overseas destinations where it's $0.10/message and that's real money.
Maybe they should flip it on its head - get a thousand? Ten thousand? numbers that can accept SMS and tell people to "text 473843 to this number" to verify.
It's usually even more expensive to support receiving messages than sending them, beyond keywords like Unsubscribe. If you want any sort of threading its going to be extra. Also its extra for dedicated shortcodes. When you get an SMS from a random shortcode, there might be multiple companies using that code, but they mix the pools enough that its unlikely you will receive two messages from two companies from the same code. Also shortcodes are usually country/region locked. So if you want to international support, you need to buy shortcodes in multiple regions, and different regions have different telco laws. On top of that, provisioning is very manual compared to the modern cloud.
I supported a marketing platform for a while, and it was so much easier to send an email than an sms.
SMS sender isn't generally something you can trust. If you get the SMS directly from the carrier that's responsible for the number, and you have reason to trust their SMS sending to verify the sender, then yes. But in countries with number portability, you still need to pay to lookup the carrier responsible for a number.
And you'll need to maintain ingress numbers in all the countries you support, and maybe numbers per carrier, depending, and you'll need to tell the user the right number to text to ... it's a lot, and it might not work well or might not save much money.
That's in fact how iMessage does phone number verification. It works really poorly internationally.
how is that in any way comparable? it's not about java vs php
Java is likely the most optimized part of the stack.
Many startups move up to the jam when there is little else that has optimized performance and efficiency like the jvm for 20-30 years.
Of courses this is a moot conversation if you’ve never used Java at scale. Apple and others are Java houses.
Java is entirely performant if you treat it right, and many of the problems with GC in J8 are fixed in later versions.
You can push Java very far.
Of course you can also write horribly ugly code in it.
You can write horribly ugly code in most languages.
But the secret of JVM existing as an option is eventually learned by most who scale.