Comment by Dunedan

2 years ago

> Storage: $1.3 million dollars per year.

> Servers: $2.9 million dollars per year.

> Registration Fees: $6 million dollars per year.

> Total Bandwidth: $2.8 million dollars per year.

> Additional Services: $700,000 dollars per year.

Signal pays more for delivering verification SMS during sign-up, than for all other infrastructure (except traffic) combined. Wow, that sounds excessive.

Twitter said that's why they got rid of the SMS 2FA. They said it was costing millions to have that enabled for them.

https://www.cnn.com/2023/02/18/business/twitter-blue-two-fac...

  • > Twitter said that's why they got rid of the SMS 2FA. They said it was costing millions to have that enabled for them.

    Previous Twitter employees have said that this is incorrect. Because Twitter began as an SMS-only (and then SMS-first) application (remember 40404?), they very early on established direct-connection infrastructure for sending SMS, meaning that they have a marginal cost of literally $0.00/message in most markets. Twitter still has to maintain that infrastructure, because they didn't get rid of SMS 2FA - they just restricted it to Twitter Blue users, so the overhead is still the same.

    Almost nobody else who delivers SMS today has that infrastructure, because it doesn't make sense for most services to build.

    The only place where Twitter was paying significant amounts for SMS was due to SMS pump schemes, which is a consequence of Twitter gutting its anti-spam detection, resulting in them paying for SMS pumping which was previously blocked.

    • > they very early on established direct-connection infrastructure for sending SMS, meaning that they have a marginal cost of literally $0.00/message in most markets.

      I am very, very interested to understand how that works, because without more detail or sources I'm calling bullshit. I definitely understand how Twitter could have greatly reduced their per-message fee with telecom providers, but at the end of the day Twitter is not a telecom and is still at the mercy of whoever is that "last mile" for actually delivering the SMS to your phone, so I don't understand how they have no marginal cost here. Happy to be proven wrong.

      15 replies →

Signal agrees: (from the article:)

... legacy telecom operators have realized that SMS messages are now used primarily for app registration and two-factor authentication in many places, as people switch to calling and texting services that rely on network data. In response to increased verification traffic from apps like Signal, and decreased SMS revenue from their own customers, these service providers have significantly raised their SMS rates in many locations, assuming (correctly) that tech companies will have to pay anyway.

...

These costs vary dramatically from month to month, and the rates that we pay are sometimes inflated due to “toll fraud”—a practice where some network operators split revenue with fraudulent actors to drive increased volumes of SMS and calling traffic on their network. The telephony providers that apps like Signal rely on to send verification codes during the registration process still charge their own customers for this make-believe traffic, which can increase registration costs in ways that are often unpredictable.

  • SMS has become a kind of real-world PoW (proof of work) mechanism. A phone number typically has a recurring fee to keep it working. So a live number indicates that someone is spending money (a proxy for effort) to maintain it.*

    It still seems like a lot of money to spend on simple, old technology, but from the PoW perspective, making it cheaper would defeat its purpose.

    *Which is why many sites reject Google Voice numbers, for example, for SMS verification.

  • > In response to increased verification traffic from apps like Signal, and decreased SMS revenue from their own customers, these service providers have significantly raised their SMS rates in many locations, assuming (correctly) that tech companies will have to pay anyway.

    There's nothing that requires tech companies to use SMS for registration or for 2FA. The normal way to do it is by email, which continues to be free. For Signal, there is no need to do 2FA registration at all.

    Signal is ideologically committed to publicizing your phone number, and apparently they'd rather pay $6 million to hold to their commitment than just... not do that.

SMS rates are absolutely bonkers considering the technical way they're transmitted. The US is an outlier in SMS rates actually being reasonable (usually unlimited or close to) for consumers - but for the rest of the world the insane mark up on that communication method has mostly obsoleted it...

That'd be all well and good... the technology would die naturally, but all my American relatives continue to stubbornly use iMessage.

  • > for the rest of the world the insane mark up on that communication method has mostly obsoleted it...

    For P2P communication. SMS is alive and well for B2C messaging, most importantly for 2FA OTP delivery, but also as a first line of defense against spam/bot account creation.

    It's not a good solution to either problem, but it's slightly better than nothing (which apparently makes it good enough for many), so I suspect we're stuck with it for now.

    > That'd be all well and good... the technology would die naturally, but all my American relatives continue to stubbornly use iMessage.

    iMessage is not SMS, though. It just uses phone numbers as identifiers, but so do many other popular over-the-top messengers, including the most popular one globally.

    • To clarify - iMessage does not use SMS if you're going from Apple to Apple device and both devices have data/wifi available. iMessage refuses to support messaging to Android clients and defaults to SMS for these messages.

      I've got an Android phone so all iMessage transmissions come across as SMS (or MMS).

      26 replies →

    • > For P2P communication. SMS is alive and well for B2C messaging, most importantly for 2FA OTP delivery, but also as a first line of defense against spam/bot account creation.

      In Brazil, businesses use Whatsapp to communicate with consumers. You order pizza and book doctor appointments over whatsapp

  • > stubbornly use iMessange.

    Personally, I prefer it over downloading yet another client, dealing with additional credentials, wondering about who can access my messages, and so on and so forth…

    And all that just to message the handful of people that I know who use <popular in other country third party app>.

    • If only someone would release a universal protocol that the app's native messaging apps could utilize to eliminate the need for these 3rd party messaging apps. Oh, right, it's called RCS and Apple refuses to support it.

      16 replies →

  • I think I understand your comment, since iMessage isn't SMS, but defaults to SMS for those not using it.

    There are opensource self hosted solutions like BlueBubble that allow reasonably secure communication through iMessage to the other chat platforms on desktop/Android etc. I have zero affiliation, but I know others who happily use it. There are also less secure and paid solutions I can't speak to.

    https://bluebubbles.app/faq/

  • For the purpose of 2FA and account registration let’s view it as a tax for fraud prevention, where the real value in SMS is in verifying someone’s identity rather than transmitting messages

    • If SMS actually worked for this purpose, it would be acceptable. However, SMS provides no guarantees about: 1) If it actually gets delivered 2) If it is delivered to the intended recipient 3) 1 and 2 without anyone reading or tampering the message while in transit

      Now, even if stars align, your SMS ends up on a route where nobody is mitm-ing or hijacking it, the telco systems work and it gets delivered, it is STILL not a guarantee of identity. It simply verifies that you have somehow got access to a particular phone number.

  • Just because consumers get unlimited SMS doesn’t mean businesses get that. The telcos are ruthless about extracting their pound of flesh at business rates.

Phone numbers have become the de facto version of "Internet stamps" for identity verification.

They are near-ubiquitous on a per-user level, but hard to accumulate without significant cost. (Unlike email addresses.)

But the down side is that phone verification tends to be on a per-service level. So, for instance, Signal incurs these costs when they verify their users, and every other service incurs these same costs when they verify _their_ users.

There are a number of businesses out there that are trying to act as clearinghouses, where they verify the users once, then allow the users' verified profiles to be confirmed by multiple services.

I wonder if any of those could be used to reduce these "registration" costs.

  • > but hard to accumulate without significant cost

    Varies heavily by region. The shop opposite my house has ~50 SIM cards on the shelf, for £0.99/ea.

  • Phone number verification is used to verify the user's registration intent, so not really.

    • "Sign in with $Clearinghouse" could bring you to a page that prompts whether you want to share a user ID or the phone number, as required, with that service.

      The clearing house verifies you only once, or once a year, instead of every time. If the clearing house were to be a nonprofit, perhaps even set up by Signal themselves to spread costs with similar services, that has to be cheaper.

      It also gives users confidence that only a randomized user ID was shared, so it won't be used for cross-service correlation and tracking, if the service didn't actually need your phone number but only some identifier.

    • A Flow:

      > Service A => User: Please Enter Your Phone Number and Email

      > Service A => Clearinghouse: Please verify phone number XXX wants to sign up for an account with us

      > Clearinghouse => User (SMS): Please respond with the Email you used at signup to confirm you want an account with Service A

      Later...

      > Service B => User: Please Enter Your phone number and Email

      > Service B => Clearinghouse: Please verify phone number XXX wants to sign up for an account with us

      > Clearinghouse => User (Email): Please verify you want an account with Service B

      Not saying it's great (providing email twice is annoying), but it's something.

      5 replies →

  • A service that requires a telephone number simply shouldn't be called an Internet service. It can't be used purely over the Internet.

    Telephone numbers are fundamentally incompatible with privacy. Signal's leadership knows this, but they don't appear to care.

I really wonder why it’s so expensive to run. I always hear things about scaling but I used to run a top 500 alexia website and it was just a php app running on a mutualized offer for $5/month. Lots of manual caching though but still.

My wild guess is that either the stack is not really optimal (last I heard it was java) or they do other costly things at scale (sgx?)

  • I guess, then the question is how real time was the website. Was it as real time as supporting, instant messaging, voice/video calls etc

  • > the stack is not really optimal (last I heard it was java)

    how's java relevant here?

    • Java in theory and in synthetic benchmarks: damn near as lean and mean as C.

      Every actual Java project: “oh, did you want that memory and those cycles for something else? Yeah, sorry, I need them all. Why no, I’m not actually doing anything right now, why do you ask?”

      2 replies →

  • You can't send an sms yourself like you can an email. Instead of setting up a server, you have to work with a telco provider (an aggregator specifically). Any SMS service eventually hands off to one of these. Many SaaS SMS providers are just frontends for legacy telco services. They charge insane fees because they can, that is all there is to it.

    Sending mass email is still difficult. Its probably easier to pay a provider than set up and establish reputation for yourself. But they don't charge near the rates. Last time I compared rates it was something like 10x-100x to send an sms compared to an email, but it has been a while.

    • > Many SaaS SMS providers are just frontends for legacy telco services.

      I worked on an automated SMS marketing system back in the day so I have seen this in action, at scale. This would be stuff like "text LAKERS to 12345 for Lakers updates"- we didn't handle the Lakers but we did handle many sports teams. Though I wasn't privvy to the financial side, I got the sense that the per-text cost ended up being manageable at scale, but this is because we were one organization who would apply the rules onto our own customers, and if we failed to do so properly we risked losing the interconnects to the various carriers. We typically used a single contracted "aggregator" service which provided a unified API for the carriers. When I left, we were using OpenMarket.

      When you have a self-service SaaS offering such as Twilio, the per-text costs are going to go up because the barriers for sending unwanted texts (or fail to follow the rest of the rules mandated by the TCPA) is so much lower, and Twilio has to address that organizationally which adds cost.

      Additionally, Twilio does not purchase short codes (ie 12345) which means its harder for the carriers to track bad behavior across their network. There is an initial cost (fairly high) to acquiring a short code, though you can also share short codes across customers in some cases. Acquiring a single short code and sending all messages from that short code would likely reduce costs.

      I would love to see more detail from Signal about what sort of SMS interconnection they are using, because directly connecting with an aggregator instead of a SaaS offering (if they haven't already) could save a lot of money, and they are definitely at the scale that would allow for it. And given that they only use it for account verification and are a non-profit, it seems likely they could get a good deal since the risk of TCPA violations is effectively zero.

      2 replies →

    • Maybe they should flip it on its head - get a thousand? Ten thousand? numbers that can accept SMS and tell people to "text 473843 to this number" to verify.

      3 replies →

  • Java is likely the most optimized part of the stack.

    Many startups move up to the jam when there is little else that has optimized performance and efficiency like the jvm for 20-30 years.

    Of courses this is a moot conversation if you’ve never used Java at scale. Apple and others are Java houses.

    • Java is entirely performant if you treat it right, and many of the problems with GC in J8 are fixed in later versions.

      You can push Java very far.

      Of course you can also write horribly ugly code in it.

      1 reply →

I did my part to help reduce costs by switching to the decentralized alternative, Session.[0]

Bonus: Session does not demand users' phone number. Also no bundled cryptocurrency.[1]

[0] https://getsession.org/

[1] https://www.stephendiehl.com/blog/signal.html

> we can rent server infrastructure from a variety of providers like Amazon AWS, Google Compute Engine, Microsoft Azure

Moving off cloud services to lower-cost provider like Hetzner, Vultr and DigitalOcean might provide a lot of cost savings.

I also imagine they're using managed SMS services from one of these clouds, and moving off them to a combination of local SMS gateways in each country can also further reduce costs (and in one case I've personally observed, by upto two orders of magnitude). This obviously pushes a lot of complexity on Signal's side, but is usually worth it.

  • Any idea what prevents Signal from using cheaper alternatives?

    Edit: I meant moving off cloud to Hetzner, Vultr, DigitalOcean.

    • In business, you get what you pay for. Cheaper hosting might raise more issues that need to by handled by your employees, who also are expensive, and also the organization's focus gets disrupted. The hosting company / cloud vendor has an enormous economic advantage, with access to the entire hardware and software stack, the engineers who built it, people whose full-time job is operating it. Often it's cheaper to pay more for better.

      As I have to explain about open source, 'Free is only free if your time is worth nothing.' (And I use a lot of FOSS, it just not always the solution.)

      1 reply →

    • As I understand it, you have to often use multiple gateways based on which one is cheaper and can deliver your message to the recipient, and also take care of handling retries in case one gateway fails. This is not something you typically want to handle if you're not aware of it, and the process of having to talk to each vendor and figure out their limitations is tedious.

    • There's a lower bound on what these services can charge in the form of interconenction fees charged by the mobile service providers delivering the messages.

      In the US, that's effectively zero due to the US phone infrastructure largely using a shared-cost model, but in most other countries which use "sender pays", these fees can be significant.

    • DO, at least, has bad peering agreements that will cause you noticeable, unfixable (if you stay on DO…) persistent problems at large enough scale.

    • I use Hetzner, but they have a bad rep for killing services that attract too much attention, e.g. DMCA requests

  • So ... hire staff to manage that complexity?

    • Might not be cheaper at scale and truly globally.

      The loaded costs should have the numbers run.

      It would be a fascination under the covers look with signal.

Just wondering, are they relying on these big name cloud providers (AWS/Azure/GCP), known for predative traffic and storage pricing? Have they considered cheaper providers such as Backblaze B2 for storage and Hetzner/OVH for servers? The fees for storage, server and bandwidth could be cut by 80% if they did that.

Sounds like a great case to get the fuck away from SMS and phone numbers.

But hey, they still want your whole address book, and announce you're on signal to everyone else on signal.

The whole "secure" thing is a joke. Its all linked to your identity via your phone#.

  • Signal actually jumps through quite a few hoops in order to let you and your contacts are on Signal without Signal actually having access to a copy of your whole address book. It's even mentioned in TFA.

    I do agree about being linked to your phone number - doing it that way means not considering a lot of people's valid threat models. They are working on moving to usernames, though. It's in beta now.

    • > Signal actually jumps through quite a few hoops in order to let you and your contacts are on Signal without Signal actually having access to a copy of your whole address book. It's even mentioned in TFA.

      It doesn't say how it works. If Alice's phone can tell whether her contact Bob uses Signal without Alice and Bob doing any sort of a priori cryptographic exchange, why couldn't Signal itself do whatever Alice's phone is doing?

  • They want the address book because if you don't have engagement promotion features like that, there is no way to ever become remotely popular in the chat app space.

    Why is the security a joke? The data is e2e encrypted, and isn't related to a phone number in any way after registration. Do you know of a better way of combining privacy and anti-abuse measures? If you don't offload identity checks to telecom providers during registration some bad actor will immediately create a million accounts and send millions of spam messages and destroy the slim chance of this type of app to exist for free.

    • > They want the address book because if you don't have engagement promotion features like that, there is no way to ever become remotely popular in the chat app space.

      Intentionally ignoring the fact that Signal splatters your phone number to everyone else is a humongous problem. And you can even put your phone number block in your address book, and it'll tell you everyone who has Signal. This happens all the time, with Signal servers leaking all of this metadata.

      And doing "engagement promotion" is what companies do to sell more shit. So, exactly what are they "selling"?

      >Why is the security a joke?

      Metadata, pertaining to communication patters and to whom matters just as much as what's being said.

      And that metadata, like "your phone number" and "contact's phone number", and "when data is being sent to/from" is that metadata.

      > The data is e2e encrypted,

      > and isn't related to a phone number in any way after registration.

      Bullshit. I see new people hopping on signal fairly regularly. If that was true, it'd be a simple verify-once-and-delete. It aint.

      > Do you know of a better way of combining privacy and anti-abuse measures?

      I reject your claim of "privacy", with regards to metadata.

      Secondly, Tox has an alternate way to handle this, by allowing any number of accounts not tied to anything. Sure, it's a SHA256 id, but who cares. There, its secure AND anonymous.

      Basically, I look at Signal as "better than SMS, but not much". It's basically a way to keep the phone company from scanning messages.

I wish their justification for dropping SMS capability from their Android app to move away from phone numbers was a little more transparent about the obvious cost aspect rather than solely sticking to the patronizing "we're saving insecure messaging users from themselves" messaging they had. I found it pretty obnoxious. I think people generally get "valuable nonprofit + huge expense = not-sustainable = bad."

  • > their justification for dropping SMS capability from their Android app ... was a little more transparent about the obvious cost aspect

    I'm not following. Signal gets stung for the registration SMS costs because they send the SMS to the user. They don't pay when one user sends an SMS to another user. If you send an SMS, you're the one who pays.

    (I didn't realise they were moving away from phone numbers. Don't they they stay mandatory when PNP comes along?)

Why is it that SMS is so damn expensive? (or more specifically, what is it about Twilio et al's businesses that makes them cost so much?)

  • When you control access to the customer you can charge people a lot. Just like Apple can take 30% primarily because they’re the gatekeeper to iPhone users, telecoms are gatekeepers to their users so they can charge you a lot to text them. You don’t really have a choice. L

  • In the US, shafting customers as hard and fast as you can is the current business model. What are they going to do? Move to 1 or 2 remaining competitors with the exact same business model?

  • Nothing just profit and existing system access costs set by the incumbents.

> Signal pays more for delivering verification SMS during sign-up, than for all other infrastructure (except traffic) combined. Wow, that sounds excessive.

Particularly when the phone requirement is the biggest weakness in Signal.

Getting rid of it will make it substantially cheaper to operate and much more private. Win-win.

I wonder if you could do something clever such that you can have people volunteer their SIM for sending 2FA?

Funny, because that's the reason I can't use Signal - I don't have a phone number.

  • In case one isn't aware, you can get a $1/month throwaway phone number from Twilio for that purpose.

    • That's a neat workaround for the people that can figure that out, but doesn't change the underlying problem for the majority of users at all.

      2 replies →

What's it cost to be an SS7 peer for a year? Could they spin up their own "phone company" for the purpose of delivering SMS verification and nothing else, cheaper than they're paying someone else's markup?

  • What's expensive isn't (just) the technical infrastructure, it's termination/interconnection fees charged by the destination mobile networks.

    • Huh, I knew those existed for voice calls, didn't realize they applied to SMS too. Makes sense, though.

is there any way they can reduce that cost?

  • Yeah, decouple Signal user identity from the phone number.

    • This will probably never happen. One of the reasons WhatsApp blew up is because using a phone number as your source of identification means there's much less friction in the signup flow. No username/password to create and your social graph is already there in your contact list.

      My mom was able to get our entire extended family on Signal without my involvement, which is a testament to how easy that is.

      9 replies →

    • Phone numbers are the easiest login for people, especially in a world where not everyone has an email address.

      I know this will invite comments about usernames. I would like usernames a lot too.

      2 replies →

    • Which might be said to increase privacy. I suppose there's something to the point about combating spam. But surely there are other ways to do this, right?

      1 reply →

  • Send them via whatsapp. A lot of online services give an option to send OTP via whatsapp along with SMS/Email.

    • As far as I understand, this is even more expensive than SMS in many cases due to WhatsApp's B2C messaging fee structure.

      It's also not a great idea to make sign-ups for an instant messaging service contingent on having an account with another, competing service.