I've been thinking on and off about this problem space for about a decade now - having worked on google wave, ShareJS and ShareDB. The architecture I want is something like this:
- My data is stored at a well known URL on a machine that I own. If people don't want to host their own stuff, they can rent another person's computer.
- We need a standard protocol for "data that changes over time". This is a really obvious point once you start thinking about it - REST doesn't support realtime edits, and websockets / zmq / etc are all too low level. We need a standard way to express semantic changes (eg CRDT edits) and do catchup, that can work across multiple devices / applications / underlying protocols. I've been working on this as part of statecraft - https://github.com/josephg/statecraft but its still hidden behind all the documentation I haven't written yet.
- Then we need application-specific schemas to be published. Eg, there should be a standard calendar schema with events / reminders / whatever. Any calendar vendor could provide this. Then calendar apps could request on login from the user where the calendar data actually lives. Those apps could be web / mobile / desktop / whatever, because remember - we have a standard way to interoperate with data like this.
- Ideally the data would also be stored encrypted at rest. The server shouldn't need to be able to read any of the user's data.
You could build a peer to peer system whereby my desktop apps and phone share data with one another. But ideally, data should be accessible from any device at any time without worrying about whether your laptop is on or off. For that we need servers. You could make a single persistent server be a peer in a CRDT-based cluster of devices. That might be better - but its harder to implement and might run into issues with bandwidth and size (I don't want my phone to sync my whole photo library, etc). There are some generally unsolved problems here, but I don't think they're beyond us.
If you're working on this problem and want to chat, throw me an email - I'm me@josephg.com.
I feel the dat project (https://datproject.org/) ticks some boxes you want as a base protocol. Dat itself is an easy-to-use tool for syncing large files between machines, but its core (http://awesome.datproject.org/hypercore) gives you everything you need to build on top of.
With dat you have:
- urls to individual files, with the understanding that they will change over time
- built-in encryption and non-repudiability of every change
- storage wherever you want, no peer is more important than any other
Shared object updates should be deliverable by any protocol that works for a specific application, whether client/server, peer-to-peer, or store-and-forward.
At least the calendar aspect of this has existed for over a decade. Similarly there are standardized protocols for handing sever-based contact lists and email and chats. Sure most of those protocols are crappy but you’ll just use a library anyway so that’s not an excuse. The problem is not on the technical side. Basically all the big providers have decided a walled garden is the best long term strategy and have been dropping support for any standards-compliant data sharing.
14 years ago I was syncing my contact lists from my personal server to a flip phone over vCard format (I think? Don’t remember exactly, but the contacts equivalent of CalDAV). Nowadays you either need to oAuth into Google or whatever other centralized provider a user might have, with whichever proprietary API they may or may not give you access to.
As with so many other things, a shiny new technical solution won’t change the reality that big companies are not incentivized to support interoperability.
Apple support for CalDAV and CardDAV is in a good shape. The sync mechanism isn’t as good as the native integration with iCloud, but I see it’s related to the protocol.
I really get the motivation. And id pay both convenience and money. But whats the business model, or why would the existing user data silo megacorps ever want to interoperate? Or is it just a new old third way of self hosting for the motivated minority?
DVCS focus exclusively on files rather than JSON (or whatever). They’re diff-based rather than operation based. They aren’t designed to work in a real-time manner. They require humans to fix merge conflicts. (Even many trivial ones). They store every change for all time (and require that you do so by design). And commits are a heavyweight operation. If you made a git commit for every keystroke, git would struggle. Also the network protocols aren’t published, and usually aren’t designed to work over websockets / etc for web apps.
We need something like a dvcs, but not exactly that.
I use Syncthing¹ for years now, mainly to sync the notes I write on my phone to my laptop and vice versa, but also as a way to sync my photos to my PC. Or as a way to sync my keepass password safe to other locations.
It only really works when two of the sharing machines are online at the same time. I work around this by having a rasperry pi running 24/7 which does it's thing.
I always tried to avoid cloud based services, because I don't want to keep thinking about whether I can trust cloud providers and I kinda like the idea of a LAN beeing used for LAN things – you know.. local stuff.
Does Syncthing work reliably for you? I've been unable to make it work.
I've set it up to backup folders from my phone to a LAN storage drive (running it as a service on Windows, with the storage connected as a network drive), but it seems to "forget" the connection between the server and the phone. It says the folders are up to date, and won't pull the most recent files and photos from the phone.
I had problems on android too. if I recall right, it had to do with power saving/background connections. Somwhere in the android app settings you can allow syncthing to run in background or so, then it works
This has been a dream of mine for so long, it is great to see excitement and serious thought for it.
One hack to give this a try would be to play around with integrating APIs for cloud storage to go through a local cache system first, then do async synchronisation on demand. (Collaboration is not really enabled here but the rest of the principles in https://www.inkandswitch.com/local-first.html#practitioners become trivial). Essentially firebase but with developer facing API bridges, for dropbox, drive, Amazon, ftp, whatever. The twist: you have more control of the data and it doesn't necessarily go up to Firebase backend, the devices could even do the backend computations async. Then you can build on top of any cloud storage platform, even decentralize things. It gets even more exciting to think of decentralization.
I would like to additionally emphasize on the case for data being on the cloud where it is not necessary to be there. Location history comes to mind. I want it to be local-first, local-only.
I was hoping to read more about how merging of conflicts are done. The article tells us, that "users have an intuitive sense of human collaboration and avoid creating conflicts with their collaborators".
Regarding non-clear merging, it states that it's still an open research question of how to let the application or the user with a view of the change history merge the conflicts.
How to merge conflicts is probably the most important part in a non-trivial app.
Does anyone know of examples or research that has been done in this direction?
I thought a lot about this type of application when Dapps (decentralized applications) became a thing with ethereum. I tried to build one that is really decentralized and also works on multiple platforms.
Opposite to the analysis of this article, I have chosen a webapp that runs in the browser.
Users can send it around as they want and do not have to install anything. Also the app makes a call to a server which makes recommendations on updates.
For the data-storage, I had some trouble because there was no database out there which supported replication, json-import/export and encryption. That was the reason I created one. https://github.com/pubkey/rxdb
I've been working on software that meets at least most of the objectives this outlines, and for the same reasons.
The concept was blown off by my "Group Mentor" in Startup School last October, which was a bit disappointing, but it's good to see it being discussed here and maybe it will get some legs as time goes on.
I'll be releasing a simple app soon that, hopefully, demonstrates the advantages in a way that's easy to digest. At this point I'm not expecting much positive feedback though. It doesn't use any of the current trendy tech and there's nothing truly new or whizbanging about it. It is, however, fast, solid, easy to develop and modify, and runs on any server without having to install anything other than a web server.
This is brave to post here. The web is based on taking control away, and many businesses are based on that. Here we are at a place of investors in businesses.
In my spare time I've been working on something I currently call Distos (Distributed Operating System).
My goal is to create a sort of log database merged with an app platform which maintains an encrypted and authenticated log structure that powers functions in developer code that update/manipulate local stateful resources. The apps on a user's device get access to resources managed by the platform like KV stores or Sqlite databases. The App uses these log messages, which are filtered and provided by the platform, to update these stores locally and creates new logs on the user facing clients in order to make things happen both locally and remotely once the logs are synced.
I am convinced that "logs" are the future for personal data.
Take a look at Scuttlebutt [0] which works on very similar principles of authenticated log replication.
Also note that Lotus Notes [1] has been doing all this replicated encrypted app platform stuff since the early 90s.
I am working on a log-centric bitemporal database at JUXT which I think intersects with this problem space as well, see my recent tweet relating to the article [2].
Yes Scuttlebutt is a big inspiration among others. I believe the primary change/improvement I'm making is that it runs on the device and is also focused on the user's data instead running on a server or doing distributed social networking.
Reading [1] makes it seem to me like I might just be reinventing Lotus Notes.
If I had to bet on personal computers' and other devices' future, I would say that some years down the line--not many--only a barebones system will run on them so they can connect to an OS delivered through a cloud service. The turning point will be Google Stadia's success.
As much as I would prefer local-first as well as offline-first approaches, I reckon that the future will only have a place on the fringes for them.
The problem is to achieve acceptable quality of service for an online cloud OS service with all the responsive UIs and expected reliability, everything has to be done locally-first either way, using approaches like CRDTs. And even more so for Google, because their level of service quality delivered over public internet is nowhere near acceptable for an OS and will never be. But, of course, they can still offer locally-first OS as a service, not giving users any control.
I’ve worked in the industry of SaaS vs. Enterprise long enough to see Google Cloud (with all its compute power) get turned down because it requires the corporation to share its data.
In my experience, most customers end up trusting at least one cloud vendor. If not GCP (which does not have regions in France for example), they will trust Azure or AWS (who do). If they are a big e-commerce company, they will hate AWS but go to Azure, etc.
The reason is mostly the human cost of maintaining infrastructure, and the global lack of good people have the knowledge to do so.
Edit: Trusting hundreds of SaaS vendors vs. one major cloud platform is another debate though.
For latency it depends, having a fiber connection to a data center in a nearby city gives 0-2ms latency. Using WiFi adds about 50-100ms. Keyboard-to-computer 10-20ms, computer-to-screen 10-20 ms, software rendering 0-2 ms. So if you have a good Internet connection (and not using wifi) you wouldn't really notice if the "app" was running on your computer or on a server. Try for example ssh -X and start a GUI app/game on a server.
The history tend to repeat itself, maybe in a few years we will mostly be using realy lightweight mobile devices connected to a powerful server in some noisy data-center.
I do have a hate and love relationship with hardware though, running your own computer is one order of magnitude cheaper then running one in the cloud. So server hosting prices need to go down.
Interesting, but I really would love to see honesty and the drawbacks of the approach listed.
Saas software is not only an economic incentive, but also a UX win!
The user no longer need to update his software, you can deploy breaking changes (or security fixs) in seconds (as opposed to weeks, waiting that all users download the new release).
Security: the user no longer need to download random software from the internet, and a random malware on his computer will have hard time to access his online data.
I'm sure there is a lot more of advantages of the centralized model, but I feel it's unfortunate that they are never put under the light.
> The user no longer need to update his software, you can deploy breaking changes (or security fixs) in seconds (as opposed to weeks, waiting that all users download the new release).
The user can also no longer choose when to update the software, and you can deploy breaking changes in seconds. Meanwhile, on my computer, I can choose which software to update and when I want to do it, and I'll do so as to not impact my work flow when I don't have the time to adapt to the latest and greatest.
> Security: the user no longer need to download random software from the internet, and a random malware on his computer will have hard time to access his online data.
On the other hand, malicious parties interested in many users' data now have less work ahead of them. Some shitty engineering at LinkedIn and suddenly millions of users have their data leaked.
It is true that centralizing the data means that a team of experts can manage the security as opposed to an amateur like me, but it's been proven again and again that it's unreasonable to expect data that you share with a centralized third party to be secure and private.
On the flipside, if a new update breaks something, the user can't hold of upgrading their software. You could come to work one day and learn that the obscure sorting option you use doesn't work today.
Obviously a new patch won't be far away, but there is a certain stability to controlling the upgrading process yourself.
Wonderful article, RSS needs to make a comeback. Especially among friends and family who love to make large posts about important topics. I try to tell them to build a blog and then just link to articles that they write. Your article and others inspired me to finally just put together a system to make it easier to start blogging. I just mimic a social media platform, but since everything is committed to a repository using the JAMstack it could easily be converted to a full website. Any feedback would be wonderful. https://your-media.netlify.com/post/make-your-own-media/
Everything is owned by the end user. This is only providing a recipe for people to use.
I will also mention that https://www.stackbit.com/ is doing basically the same thing but more from a “Make life easier for Website designers” perspective.
My gripe really isn't with SaaS. My gripe is less about "app" software. My gripe is with printers, scanners, smart phones, IoT devices, TVs, Cricuts, which only work by sending all your data into the cloud...
If you can send my data into the cloud you should also give me the ability to easily mock your cloud API so I can also send the data somewhere else...
I found it rather confusing that "local-first" is defined here roughly as "real-time collaboration software that doesn't rely on a central server". But with this definition it's close to saying "CRDTs can be useful for their purpose".
The examples (MS Office, Trello, Dropbox, etc) also seemed strange to me: I'd think that neither an average MS Office user would care about privacy, data ownership, etc, nor an average nerdy user who cares about those would want to use something like MS Office or Trello. Then there's plenty of easier to solve and related issues that aren't yet solved (e.g., plain offline usage of some software, more widespread asynchronous collaboration), and the article talking about privacy and data ownership ends with "We welcome your thoughts, questions, or critique: @inkandswitch or [email protected]". Looks like a nice summary, but maybe a bit otherworldly.
This is already a thing and there are lots of developers putting effort into the `dat://` and `ipfs://` formats and protocols. Persistent naming in content-addressable networks that can be trusted is presently being tackled... and how to structure apps on these protocols...
I'd be down for working on projects in this space. I'm presently contributing some work into the Lean theorem prover where I'm hoping, with a bit of elbow grease, it will be fairly low-cost and attractive to build out more p2p protocols and libraries that meet our privacy and security demands.
I think this may be a great idea for indie software developers, like Timing or Standard Notes, in order to expand a particular offering. SaaS will probably rule for a lot of enterprise software, but there are always niches to be filled and some of them don’t make sense pricing wise to do a subscription. I like this!
This is excellent and gets me excited - many of those "ideal" principles were important to me when I designed and wrote my own iOS app Mindscope, basically a Workflowy-meets-Scapple app for visualizing your thoughts hierarchically. https://itunes.apple.com/us/app/mindscope-thought-organizer/...
It's an app I wrote primarily for myself, but it's been great to hear from lots of people who really "got" the vision themselves and use it a lot.
I simply love apps & sites that make immediacy and the feeling of "control" core values of the UI. Wish I had more time to give Mindscope more development love than I've been able to lately...working on that.
I have to say, there are some brilliant ideas presented here. Obviously I don’t know much about the details of CRDTs but going to dig deeper into them.
This is a lot of good work and thinking put in to a technical solution to something that’s not a technical problem.
The reason that software is online is a business one, not a technical one.
Software as a Service is impossible to pirate and generates continuous income rather than a single upfront fee. That’s all you really need to know to understand why there is less and less desktop software coming to the market these days.
So yeah, sure, if you were to build a piece of desktop software from a clean sheet of paper today, this is a really good guide on how to do that. But nobody is going to. Because it makes no business sense to do so.
Most people won't make use of this kind of research. However, CRDTs aren't just another way to architect the same kind of software. They are an inversion of the tropes and techniques we've zealously stuck to over the last decade, and they grant us brand new technical capabilities that no SAAS player will ever be able to offer:
• Offline-first support with real-time collaboration
• Real-time collaboration with local devices over Bluetooth/ad-hoc Wi-Fi
• End-to-end encrypted real-time collaboration without the server having access to any of your content
• Transport-agnostic sync: use Dropbox, iCloud, and Bluetooth all at the same time with no consistency issues
• The ability to switch to a different cloud provider with zero friction, and to grab your (mergeable, collaborative, and versioned) documents from your current cloud provider without conversion to any intermediary format
• Anxiety-free sync: the user can be 100% confident that their changes will never fail to merge, even if they spent a month in the bush editing their documents
These are off the top of my head, but there are many, many others. And they are features. If enough people build software using these tools, people will get used to them and start seeing the big players as annoying and clunky. ("Why can't I make changes to my spreadsheet when I go through a tunnel? Why did I lose all the changes I've been working on over the last hour? What do you mean this .doc file is just a link to a webpage?")
Is there Big Money in it? I don't know (or care), but I'm going to try hard to make sure that any software I write on the side follows these principles, and I hope others start to do the same.
You could have easily said that "the reason that software is online is a business one" about time sharing versus personal computing, and yet here we are. Focus on the user instead of your bottom line and you will (eventually) win.
I'd say it's even simpler: Software as a Service is a lot easier to develop than software that a third party has to install: Integration of build pipelines that deploy the software immediately are a blessing for bugfixing and you can build an insane backend out of third party dependencies noone but you has to ever install and connect to each other.
I don't like the privacy problems this generates, but in a business environment I want my stuff to work for my customers, and that means I do the hosting.
There is also the huge advantage of not having to install anything. It's already hard enough to convince my friends to switch to any open source videoconferencing in-browser solution, I would never go through having them install a whole application.
"No install" is a major selling point, unfortunately.
Keep in mind that it's possible to develop Software as a Service by selling software that the user has to install - we have done it with our accounting software Solar Accounts. This approach is probably more painful to develop than the traditional web app, but it does give our users features such as (a) end-to-end security and (b) storing data locally
- which are selling points for some users.
SAAS and the goals outlined in this article can be perfectly complementary if you let client-side JS do most of the work. Just because an app runs in the browser doesn’t mean that you also have to share all your data with the developer, or lose your ownership of it.
> The reason that software is online is a business one, not a technical one.
That's only partly true. Software gets more and more developed for the big corps in mind as the sole users. So web browsers are developed to make Google and FB happy. Kubernetes is developed to only work in AWS and Google Cloud really well. And so on.
All that hollows out the foundation that gives software usability to all of us. Eg. if big linux corps only focus on Kubernetes, Kubernetes only works in AWS and GC, then nobody puts effort into Linux anymore. It might even happen that linux gets unusable if you don't run it in such kind of environment. In the end it all depends how the majority of resources get spent.
So we come from today's tech level X everywhere to tech level X+Y in the cloud and tech level X-Z on local clusters. Then where will most of the data be? Where will the new hip stuff be? How can you develop new stuff with an incomplete local infrastructure?
What you lose is not the desktop software but the desktop. And you will lose it in a technical sense. You won't be able to rebuild it technically alone.
> a technical solution to something that’s not a technical problem
Why does a problem have to be 'technical' to have a technical solution? Skype, Wikipedia, Amazon, eBay...pretty much every major technical product I can think of solves non-technical problems, almost by definition.
> But nobody is going to. Because it makes no business sense to do so.
You are talking about a general trend as if it's ubiquitous. Yes there are strong business reasons why most software vendors have shifted to SaaS, but that doesn't obviate the fact that some users have problems for which local software is a better fit. The local software market may be smaller, but it exists, and it's incorrect to say it makes "no business sense" for anyone to operate in it.
New technology can be a prerequisite to solve certain non-technical problems, but it’s not the technology itself that solves the problem. Wikipedia is a good example. It probably wouldn’t work without the technology, but just building a wiki platform is not enough to get people to volunteer to collaborate on writing high quality encyclopedic articles. Arguably, the vast majority of the work that has gone into creating Wikipedia has not been technical.
Couldn't the same argument be made about why it made no sense to create the Linux operating system, since there was no business case to do so?
This may be a scenario where it's possible to create a communal good (for personal data ownership and control) which doesn't require businesses to lead the way.
In addition, it is feasible to distribute some-or-all of an application's software via the cloud while still keeping application data locally; we can be more nuanced here and keep a lot of the business benefits while caring more for users.
> we can be more nuanced here and keep a lot of the business benefits while caring more for users.
If you are talking about "business", then we should not talk about "users" but "customers". And it so happens that most "customers" (so far, at least) consider it more valuable if the business also is responsible in keeping (and securing) the data. For these customers, putting the responsibility of controlling the data on them is a burden, not a privilege.
Actually it's not true that local-first software cannot be businesses (or non-profit orgs but still revenue-generating), it's just that the money to be made in local-first is an order of magnitude (at least) smaller than with cloud software. On one hand, the money to be made is much smaller, but on the other hand, local-first is where the next big disruptions can happen, while still having a non-zero revenue. Look at open source or open data projects for some examples on this: Wikipedia, VideoLAN (VLC), OpenStreetMaps, etc.
> But nobody is going to. Because it makes no business sense to do so.
While your reasons make sense, they make sense from the supplier side. However, the ultimate decider of what makes "business sense" is not the supplier, it's the customer.
You can have all the supply-side advantages in the world, if your customers decide they want that other gal's product, you're toast.
It may not even make sense from a supplier side (sometimes). Instead of just writing the SW, the supplier now has to manage servers, deal with angry customers when downtime happens, backup, protect the servers from attackers, deal with scaling, deal with customers who don't like change, etc., etc.
With local SW, you can just write it and mostly forget about it, as long as OS stays compatible with your program.
But the customer is easily distracted. That's why we have things like Facebook where customers will happily give up their privacy in return for a little functionality.
There are SaaS business models with local/native desktop software, e.g. Adobe CC.
iOS AppStore-monetized software can run locally and be licensed as a subscription. Apple encourages local computation and continues to add both hardware and software for on-device processing.
x86 clients are adding SGX (hardware secure enclaves) which provide stronger mechanisms for endpoint license enforcement. This can support multiple business models, including subscriptions.
Local computation makes sense especially from a user experience perspective. However, another challenge arises from talent availability and willingness to work with this sort of deployment model. More complex deployments generally lead to the need of hiring really top notch engineers, which as we know is quite scarce. The next best thing might be infrastructure that makes the deployment easier but no one's really interested in investing in infrastructure tech without some tangible reward.
I once subscribed to a farming management program, developed by a french company that was launching to the cloud market.
Apparently they had bought several other smaller companies, integrated their software into their own program when useful, built huge servers for their cloud service that they also rent.
I knew the program was good - I was familiar with previous versions - so it was kind of a no brainer. The problem was the new cloud thing.
I am in favor of economic rational practices in agriculture (in addition to the normal functioning of markets), so that crops are chosen wisely between farmers and prices do not plunge because of all-or-nothing decisions, and sharing information between farmers is essential.
But data harvesting (no pun intended) of farming practices to the purpose of informing large companies and futures markets brokers of their possibilities can work against the interests of farmers and consumers. The way large corporations treat small farmers is ruthless and they negotiate no terms.
So if my and other farmers' data was being uploaded to the cloud without possibility of local storage, who had access to it? Every crop, production, pesticide, invoice, price was going there!
They wouldn't sell it, they told me. It is stored with e2ee, they assured me. If I ever forgot the password, data would be irrecoverable, they said.
But there was also the problem of internet connection issues, because farms are remote. Would it ever work?
The first thing I noticed after installing the program was that there no important message was shown that informed the user of what would happen if the password was lost. I typed in some new data, saved and exited. Then I started the program and reset the password with ease. There's no possibility of 3FA, either.
So I told them I could not use the program because of my poor internet connection, which was true. They installed a local version that was temporarily available.
But they took so long doing this that I developed my own modular managing system, based in - yes, that's right - spreadsheets. It fits my needs, it is completely portable and it works.
The formulas are getting cryptic, and there are still some things I have to do manually (like sorting lists or generating pdfs), but the files access each other and it works. I have no time to code anything and I don't think I have sufficient knowledge of database software (I am trying to find time to learn MySQL).
Maybe some day I'll make some GUI that handles all of this, in such a way that if everything else fails, I can still access and edit files manually.
As a thought, since you're just getting into database stuff as time permits, it's probably better to start out with something like SQLite and/or DB Browser for SQLite, if you prefer the GUI way of doing things.
On the flip side customers lose control over their environment.
If you work in regulated environments this can be s big deal and this validation process becomes captured (forced) by the saas provider.
If you ran your own you validate and keep that unless new requirements. No need to revalidate every time a new release is out and pay for that re-validation.
> Software as a Service is impossible to pirate and generates continuous income rather than a single upfront fee. That’s all you really need to know to understand why there is less and less desktop software coming to the market these days.
Sounds very cynical and short sighted to me.
Pros to customer: Nothing to install. Can be used on any operating system with a web browser. Updates and security patches are performed automatically. Can share documents and collaborate more easily.
Pros to developer: Easier to support multiple platforms. Easier to upgrade users and migrate data.
> So yeah, sure, if you were to build a piece of desktop software from a clean sheet of paper today, this is a really good guide on how to do that. But nobody is going to. Because it makes no business sense to do so.
Imagine having to install Trello, Google Docs, Slack etc. manually everywhere you wanted to use it, deal with updates yourself and ask people you wanted to collaborate with to do the same. That makes no sense in terms of ease of use.
>Imagine having to install Trello, Google Docs, Slack etc. manually everywhere you wanted to use it, deal with updates yourself and ask people you wanted to collaborate with to do the same. That makes no sense.
That's the way things worked for years, and the way most software still works (except through app stores) and it wasn't a madhouse of anarchy and suffering, it was perfectly fine.
Installing software "manually everywhere you want to use it" implies installing it once in 99% of cases, maybe twice in most of the rest, where "install" is a trivial process. Businesses will require that everyone use the same set of installed software, so that's not even an issue to begin with. It's not unreasonable to expect people to have a PDF reader, or Adobe software if you're working in graphics, for instance, or for Windows users to be able to read Windows documents.
Plus... you do know native software can handle automatic updates and security patches as well, right? And if you don't think people can share documents or collaborate easily with native software, I mean, that's been a thing since Napster and IRC. FTP and USENET. SAAS didn't solve any of those problems, because they were never problems.
`sudo apt get install trello gdocs slack` wouldn't be exactly taxing (I realise it d doesn't work that way, but it could). You can just have a web/net link with apt-url too.
I imagine it's similarly easy with chef/puppet, etc..
One of my problems on Ubu is that it's so easy to install stuff that I can acquire a lot of unused cruft; mind you it's mostly a mental problem now with disk sizes as they are.
I agree that SaaS is more convenient in the short term on both sides, but people have been using desktop-based software for a long time so I don't think it's accurate to say it makes no sense. Like anything else, there are tradeoffs.
Desktop software works "forever" (as long as the underlying OS and drivers don't break support for it, at least) and the longer you use it, the more value you get for your money. SaaS works until the company that maintains it stops supporting it, and the value scaling is basically the reverse of desktop software.
Interest? Very few people care to build medical billing software or crms for free in their spare time. So you get products that aren't as good. Even with blender, there's far better paid tools because people need money to live, so they put their efforts into things that make money.
The Software as a Service model is 100% compatible with what they have outlined, and in fact has been working well with many traditional desktop applications (Microsoft Office, Adobe Suite). Providing a good end user experience while taking care of caching, cross-device & offline syncing, conflict resolution etc. are still mostly unsolved problems, and any amount of progress in those areas is good.
The other reason is how it looks to business customers.
Buying a piece of software (one-off) almost always requires budgeting and capital-expenditure approval. Add in the mess of depreciation calculations and maintenance fees, and it's just a mess. Can be weeks of work just to buy £200 worth of software.
A subscription service usually gets budgeted and booked as recurring expenditure, and looks better on the balance sheet.
>Software as a Service is impossible to pirate and generates continuous income rather than a single upfront fee. That’s all you really need to know to understand why there is less and less desktop software coming to the market these days.
I'd argue that the increasing complexity of the stack involved in any given product has also brought us to a place where the kinds of thorough QA we had 20 years ago isn't really viable anymore, at least for most smaller orgs.
The ease with which you can fix a deployed product in a SaaS situation is a huge boon.
You can, but it's still easier to pirate said software.
And in Oracle's case, you're neglecting to mention their ability to bring lawyers to bear on any infringers, something not a lot of software companies may have the capital to do, compared to a SaaS model.
> But nobody is going to. Because it makes no business sense to do so.
If your only objective is making money, that's correct. If your aim is to achieve something more, you may still want to take this risk in spite of knowing you'll earn less.
"Software as a Service is impossible to pirate and generates continuous income rather than a single upfront fee. That’s all you really need to know to understand why there is less and less desktop software coming to the market these days."
The other reason is platform control: there's continuous tension between the desire of each OS vendor to differentiate their platform and make applications part of the desktop experience, and the desire of application vendors to be able to deliver to any customer using the minimum number of platforms.
SaaS vendors will build mobile apps (1-2 platforms), and desktop applications in Electron (kind-of 1 platform) as service clients, but I agree that they are not incentivized to build local-first applications.
Actual developers rarely see ongoing revenue from their past works. It's the business who sees the ongoing revenue, because odds are high that any individual developers have moved on with their career, and have been replaced by someone new to kick in 2-3 years of labour.
> Software as a Service is impossible to pirate and generates continuous income
Service as a Software Substitute (SaaSS)[1] is also inherently spyware, an increasingly popular feature as surveillance capitalism[2] infects our economy.
It is not inherently spyware, because fully homomorphic encryption exists. This in theory allows cloud computing providers to operate with no knowledge of their customer's data or computations. It's in general too computationally expensive to be practical, but that's not the same as being impossible.
>Service as a Software Substitute (SaaSS)[1] is also inherently spyware, an increasingly popular feature as surveillance capitalism[2] infects our economy.
Sure, but it's how business in the real world works as well. You can't enter a shop without being observed or even recorded, either by cameras or the shop personal / owner in smaller ones. Your behavior during a conversation is analyzed in realtime and reacted to, that's the basics of selling. There is no invisible commerce in the real world.
That's a fundamental reality that you can't just omit, doing so would be irrational treating the web as something purely abstract, which it isn't. It's an abstraction / extension of the real world, because it's part of it.
What the web needs is obvious boundaries and private spaces that are basically extensions of your own home and those of your friends, make it clear that you can't do x or y (going in public) without being observed and analyzed. Right now the only real indicator is the tracker count of adblock plugins.
Surveillance capitalism? Capitalism as an economic system does not surveil, governments and corporations do. If you do not like a service, you can opt out by not using it, unlike the government. If you have an issue with something used by millions of people, then you should consider the possibility that it actually may be the people who keep the service alive.
I try to do my part. I do not buy access to games on Steam, movies and TV series on Netflix, books on Amazon, I do not use Google, I do not use Facebook nor am I in contact with anyone who does, and so on.
I am open to discussion. Why is it specifically capitalism's fault? What other economic systems would be favored, and why?
There are local first E2E encrypted apps that do charge monthly fees. 1password is one example, bitwarden is another and mylio is a third. 1password syncs with 3 services, mylio syncs with your local NAS, other computers, amazon, etc.
They are definitely in the privacy space more than others, but it shows it's possible.
I thought about ideas like the post here myself, although I admit I haven't thought about it in the detail they have there and I am look forward to reading that fully over the weekend.
Anyway I don't think piracy is the problem per-se, but the question is: what is the commodity?
Because take Netlify for instance - picking on them, but you will see this sort of pattern everywhere:
1. Generous free tier to get you hooked.
2. $29 / month for something that costs $0.01 in terms of cloud compute.
3. $100 / month for something that costs $0.10 in terms of cloud compute.
Etc.
So really they are creating a fake scarcity and mark up stuff ridiculously.
But with local-first as I see it, you will get stuff like "filling in forms" or whatever software that has made companies into multi-million companies for free. And you will deploy that on cloud(s) of your choosing with no effort, and pay their pretty cheap per-transaction costs. Stuff that is trivial will rightfully become very cheap and commodity like.
This is probably a good thing for consumers, because for cloud companies and new startups to compete they'll need to offer truly innovative products. Do stuff that was really not possible before. Quantum computing for example.
That generous free tier probably costs more to run than all the paying customers combined, depending on how generous we're talking and the proportion of free to paying. There was likely also no revenue for a significant time, but still salaries to pay, and other costs to be borne.
If you really think the profit margin is 99.96% for that tier two customer, it should be pretty easy to create a competing business to provide the trivial service and charge a lot less. Like $0.02 and have a 50% profit margin :-) But I think you're underestimating how much it costs to run a business, even if compute costs are cheap.
Double stream of income in validated environments where the provider gets to validate on your behalf on their schedule. It’s nog like you needed that last useless feature, but hey, since it’s new we gotta validate it for you, pay me.
The reason that software is online is a business one, not a technical one.
True, but I host my data in the cloud because I don't want to pay the energy bill of the equipment at home. Despite not having concrete facts I believe it is more environmental friendly if we started hosting servers centralised in facilities instead of everyone doing at in their homes.
> I host my data in the cloud because I don't want to pay the energy bill of the equipment at home.
Then you pay someone else's energy bill.
> Despite not having concrete facts I believe it is more environmental friendly if we started hosting servers centralised in facilities instead of everyone doing at in their homes.
It's true that running 1000 instances of a software on the same big computer consumes less than running them on 1000 different computers, because they won't use all its resources at the same time. There's also less overhead if you can share their computations.
However, hosting everything at the same place has some environmental issues too. A datacenter needs cooling, and not all homes do. If there's a lot of data traffic between you and the software, that traffic needs to go through the internet instead of being local.
I'm not saying one way is better than the other; it probably depends on the kind of software we're talking about.
OT also works fine for this sort of stuff. OT algorithms are easier to implement ([1] for an implementation I wrote of OT over arbitrary JSON structures). OT just requires a central source of truth / a central authority. For local first stuff depending on how you design it you can have one of those - in the form of your server.
I've been thinking on and off about this problem space for about a decade now - having worked on google wave, ShareJS and ShareDB. The architecture I want is something like this:
- My data is stored at a well known URL on a machine that I own. If people don't want to host their own stuff, they can rent another person's computer.
- We need a standard protocol for "data that changes over time". This is a really obvious point once you start thinking about it - REST doesn't support realtime edits, and websockets / zmq / etc are all too low level. We need a standard way to express semantic changes (eg CRDT edits) and do catchup, that can work across multiple devices / applications / underlying protocols. I've been working on this as part of statecraft - https://github.com/josephg/statecraft but its still hidden behind all the documentation I haven't written yet.
- Then we need application-specific schemas to be published. Eg, there should be a standard calendar schema with events / reminders / whatever. Any calendar vendor could provide this. Then calendar apps could request on login from the user where the calendar data actually lives. Those apps could be web / mobile / desktop / whatever, because remember - we have a standard way to interoperate with data like this.
- Ideally the data would also be stored encrypted at rest. The server shouldn't need to be able to read any of the user's data.
You could build a peer to peer system whereby my desktop apps and phone share data with one another. But ideally, data should be accessible from any device at any time without worrying about whether your laptop is on or off. For that we need servers. You could make a single persistent server be a peer in a CRDT-based cluster of devices. That might be better - but its harder to implement and might run into issues with bandwidth and size (I don't want my phone to sync my whole photo library, etc). There are some generally unsolved problems here, but I don't think they're beyond us.
If you're working on this problem and want to chat, throw me an email - I'm me@josephg.com.
I feel the dat project (https://datproject.org/) ticks some boxes you want as a base protocol. Dat itself is an easy-to-use tool for syncing large files between machines, but its core (http://awesome.datproject.org/hypercore) gives you everything you need to build on top of.
With dat you have: - urls to individual files, with the understanding that they will change over time - built-in encryption and non-repudiability of every change - storage wherever you want, no peer is more important than any other
The Dat protocol is remarkably complex. [1]
Shared object updates should be deliverable by any protocol that works for a specific application, whether client/server, peer-to-peer, or store-and-forward.
[1] https://datprotocol.github.io/how-dat-works/
2 replies →
Dat is mentioned in the OP article[0]: the authors have built on top of hypercore[1] (part of the Dat project) to make Hypermerge[2]
[0] https://www.inkandswitch.com/local-first.html#crdts [1] https://github.com/mafintosh/hypercore [2] https://github.com/automerge/hypermerge
At least the calendar aspect of this has existed for over a decade. Similarly there are standardized protocols for handing sever-based contact lists and email and chats. Sure most of those protocols are crappy but you’ll just use a library anyway so that’s not an excuse. The problem is not on the technical side. Basically all the big providers have decided a walled garden is the best long term strategy and have been dropping support for any standards-compliant data sharing.
14 years ago I was syncing my contact lists from my personal server to a flip phone over vCard format (I think? Don’t remember exactly, but the contacts equivalent of CalDAV). Nowadays you either need to oAuth into Google or whatever other centralized provider a user might have, with whichever proprietary API they may or may not give you access to.
As with so many other things, a shiny new technical solution won’t change the reality that big companies are not incentivized to support interoperability.
Apple support for CalDAV and CardDAV is in a good shape. The sync mechanism isn’t as good as the native integration with iCloud, but I see it’s related to the protocol.
I'm not sure about the CDRT part, but otherwise I think Tim Berners-Lee's new SOLID project ticks those boxes? https://solid.inrupt.com/
(Except the encryption, I think.)
I really get the motivation. And id pay both convenience and money. But whats the business model, or why would the existing user data silo megacorps ever want to interoperate? Or is it just a new old third way of self hosting for the motivated minority?
If it's easy enough and faced with the ever-growing concern with surveillance and dark patterns, it might not only be the minority that is motivated.
> We need a standard protocol for "data that changes over time".
This already exists: it's called a DVCS.
DVCS focus exclusively on files rather than JSON (or whatever). They’re diff-based rather than operation based. They aren’t designed to work in a real-time manner. They require humans to fix merge conflicts. (Even many trivial ones). They store every change for all time (and require that you do so by design). And commits are a heavyweight operation. If you made a git commit for every keystroke, git would struggle. Also the network protocols aren’t published, and usually aren’t designed to work over websockets / etc for web apps.
We need something like a dvcs, but not exactly that.
I use Syncthing¹ for years now, mainly to sync the notes I write on my phone to my laptop and vice versa, but also as a way to sync my photos to my PC. Or as a way to sync my keepass password safe to other locations.
It only really works when two of the sharing machines are online at the same time. I work around this by having a rasperry pi running 24/7 which does it's thing.
I always tried to avoid cloud based services, because I don't want to keep thinking about whether I can trust cloud providers and I kinda like the idea of a LAN beeing used for LAN things – you know.. local stuff.
¹: https://syncthing.net/
Does Syncthing work reliably for you? I've been unable to make it work.
I've set it up to backup folders from my phone to a LAN storage drive (running it as a service on Windows, with the storage connected as a network drive), but it seems to "forget" the connection between the server and the phone. It says the folders are up to date, and won't pull the most recent files and photos from the phone.
I had problems on android too. if I recall right, it had to do with power saving/background connections. Somwhere in the android app settings you can allow syncthing to run in background or so, then it works
2 replies →
I’d love to try that, but unfortunately iOS hates open source software and it seems nobody has made a working app for iPhone yet...
Lack of SyncThing is my greatest pain point after having switched from Android to iOS.
This has been a dream of mine for so long, it is great to see excitement and serious thought for it.
One hack to give this a try would be to play around with integrating APIs for cloud storage to go through a local cache system first, then do async synchronisation on demand. (Collaboration is not really enabled here but the rest of the principles in https://www.inkandswitch.com/local-first.html#practitioners become trivial). Essentially firebase but with developer facing API bridges, for dropbox, drive, Amazon, ftp, whatever. The twist: you have more control of the data and it doesn't necessarily go up to Firebase backend, the devices could even do the backend computations async. Then you can build on top of any cloud storage platform, even decentralize things. It gets even more exciting to think of decentralization.
I would like to additionally emphasize on the case for data being on the cloud where it is not necessary to be there. Location history comes to mind. I want it to be local-first, local-only.
I keep an eye on this list for new additions: https://github.com/Kickball/awesome-selfhosted
Have found a couple really useful tools. My favorite so far is wallabag.
Thanks for sharing!
I was hoping to read more about how merging of conflicts are done. The article tells us, that "users have an intuitive sense of human collaboration and avoid creating conflicts with their collaborators". Regarding non-clear merging, it states that it's still an open research question of how to let the application or the user with a view of the change history merge the conflicts.
How to merge conflicts is probably the most important part in a non-trivial app. Does anyone know of examples or research that has been done in this direction?
I think the simplest strategy is to create "conflict documents" containing all conflicted data and present to the user for manual resolution.
This seems similar to what Evernote does
Paper from the same (co-)author (Martin Kleppmann) is at: https://arxiv.org/pdf/1608.03960.pdf
I thought a lot about this type of application when Dapps (decentralized applications) became a thing with ethereum. I tried to build one that is really decentralized and also works on multiple platforms.
Opposite to the analysis of this article, I have chosen a webapp that runs in the browser.
The App itself is just a mhtml-file that includes javascript, css and images. See https://en.wikipedia.org/wiki/MHTML
Users can send it around as they want and do not have to install anything. Also the app makes a call to a server which makes recommendations on updates.
For the data-storage, I had some trouble because there was no database out there which supported replication, json-import/export and encryption. That was the reason I created one. https://github.com/pubkey/rxdb
Did you look at PouchDB for data storage? It's designed for replication, has built-in JSON import and export, and their are encryption plugins.
RxDB is based on pouchdb
1 reply →
I've been working on software that meets at least most of the objectives this outlines, and for the same reasons.
The concept was blown off by my "Group Mentor" in Startup School last October, which was a bit disappointing, but it's good to see it being discussed here and maybe it will get some legs as time goes on.
I'll be releasing a simple app soon that, hopefully, demonstrates the advantages in a way that's easy to digest. At this point I'm not expecting much positive feedback though. It doesn't use any of the current trendy tech and there's nothing truly new or whizbanging about it. It is, however, fast, solid, easy to develop and modify, and runs on any server without having to install anything other than a web server.
This is brave to post here. The web is based on taking control away, and many businesses are based on that. Here we are at a place of investors in businesses.
In my spare time I've been working on something I currently call Distos (Distributed Operating System).
My goal is to create a sort of log database merged with an app platform which maintains an encrypted and authenticated log structure that powers functions in developer code that update/manipulate local stateful resources. The apps on a user's device get access to resources managed by the platform like KV stores or Sqlite databases. The App uses these log messages, which are filtered and provided by the platform, to update these stores locally and creates new logs on the user facing clients in order to make things happen both locally and remotely once the logs are synced.
I am convinced that "logs" are the future for personal data.
Take a look at Scuttlebutt [0] which works on very similar principles of authenticated log replication.
Also note that Lotus Notes [1] has been doing all this replicated encrypted app platform stuff since the early 90s.
I am working on a log-centric bitemporal database at JUXT which I think intersects with this problem space as well, see my recent tweet relating to the article [2].
[0] https://www.scuttlebutt.nz/applications
[1] https://web.archive.org/web/20170228160130/www.kapor.com/blo...
[2] https://twitter.com/refset/status/1124311089943019521
Yes Scuttlebutt is a big inspiration among others. I believe the primary change/improvement I'm making is that it runs on the device and is also focused on the user's data instead running on a server or doing distributed social networking.
Reading [1] makes it seem to me like I might just be reinventing Lotus Notes.
1 reply →
If I had to bet on personal computers' and other devices' future, I would say that some years down the line--not many--only a barebones system will run on them so they can connect to an OS delivered through a cloud service. The turning point will be Google Stadia's success.
As much as I would prefer local-first as well as offline-first approaches, I reckon that the future will only have a place on the fringes for them.
The problem is to achieve acceptable quality of service for an online cloud OS service with all the responsive UIs and expected reliability, everything has to be done locally-first either way, using approaches like CRDTs. And even more so for Google, because their level of service quality delivered over public internet is nowhere near acceptable for an OS and will never be. But, of course, they can still offer locally-first OS as a service, not giving users any control.
I’ve worked in the industry of SaaS vs. Enterprise long enough to see Google Cloud (with all its compute power) get turned down because it requires the corporation to share its data.
In my experience, most customers end up trusting at least one cloud vendor. If not GCP (which does not have regions in France for example), they will trust Azure or AWS (who do). If they are a big e-commerce company, they will hate AWS but go to Azure, etc.
The reason is mostly the human cost of maintaining infrastructure, and the global lack of good people have the knowledge to do so.
Edit: Trusting hundreds of SaaS vendors vs. one major cloud platform is another debate though.
Not in the financial industry AFAIK.
Saving costs is a major drive, I agree...
For latency it depends, having a fiber connection to a data center in a nearby city gives 0-2ms latency. Using WiFi adds about 50-100ms. Keyboard-to-computer 10-20ms, computer-to-screen 10-20 ms, software rendering 0-2 ms. So if you have a good Internet connection (and not using wifi) you wouldn't really notice if the "app" was running on your computer or on a server. Try for example ssh -X and start a GUI app/game on a server. The history tend to repeat itself, maybe in a few years we will mostly be using realy lightweight mobile devices connected to a powerful server in some noisy data-center. I do have a hate and love relationship with hardware though, running your own computer is one order of magnitude cheaper then running one in the cloud. So server hosting prices need to go down.
50ms for WiFi is some pretty horrible WiFi.
Interesting, but I really would love to see honesty and the drawbacks of the approach listed.
Saas software is not only an economic incentive, but also a UX win!
The user no longer need to update his software, you can deploy breaking changes (or security fixs) in seconds (as opposed to weeks, waiting that all users download the new release).
Security: the user no longer need to download random software from the internet, and a random malware on his computer will have hard time to access his online data.
I'm sure there is a lot more of advantages of the centralized model, but I feel it's unfortunate that they are never put under the light.
Some flip sides:
> The user no longer need to update his software, you can deploy breaking changes (or security fixs) in seconds (as opposed to weeks, waiting that all users download the new release).
The user can also no longer choose when to update the software, and you can deploy breaking changes in seconds. Meanwhile, on my computer, I can choose which software to update and when I want to do it, and I'll do so as to not impact my work flow when I don't have the time to adapt to the latest and greatest.
> Security: the user no longer need to download random software from the internet, and a random malware on his computer will have hard time to access his online data.
On the other hand, malicious parties interested in many users' data now have less work ahead of them. Some shitty engineering at LinkedIn and suddenly millions of users have their data leaked.
It is true that centralizing the data means that a team of experts can manage the security as opposed to an amateur like me, but it's been proven again and again that it's unreasonable to expect data that you share with a centralized third party to be secure and private.
On the flipside, if a new update breaks something, the user can't hold of upgrading their software. You could come to work one day and learn that the obscure sorting option you use doesn't work today.
Obviously a new patch won't be far away, but there is a certain stability to controlling the upgrading process yourself.
Wonderful article, RSS needs to make a comeback. Especially among friends and family who love to make large posts about important topics. I try to tell them to build a blog and then just link to articles that they write. Your article and others inspired me to finally just put together a system to make it easier to start blogging. I just mimic a social media platform, but since everything is committed to a repository using the JAMstack it could easily be converted to a full website. Any feedback would be wonderful. https://your-media.netlify.com/post/make-your-own-media/
Everything is owned by the end user. This is only providing a recipe for people to use.
I will also mention that https://www.stackbit.com/ is doing basically the same thing but more from a “Make life easier for Website designers” perspective.
My gripe really isn't with SaaS. My gripe is less about "app" software. My gripe is with printers, scanners, smart phones, IoT devices, TVs, Cricuts, which only work by sending all your data into the cloud...
If you can send my data into the cloud you should also give me the ability to easily mock your cloud API so I can also send the data somewhere else...
I found it rather confusing that "local-first" is defined here roughly as "real-time collaboration software that doesn't rely on a central server". But with this definition it's close to saying "CRDTs can be useful for their purpose".
The examples (MS Office, Trello, Dropbox, etc) also seemed strange to me: I'd think that neither an average MS Office user would care about privacy, data ownership, etc, nor an average nerdy user who cares about those would want to use something like MS Office or Trello. Then there's plenty of easier to solve and related issues that aren't yet solved (e.g., plain offline usage of some software, more widespread asynchronous collaboration), and the article talking about privacy and data ownership ends with "We welcome your thoughts, questions, or critique: @inkandswitch or [email protected]". Looks like a nice summary, but maybe a bit otherworldly.
The average MS Office user works for a company that most definitely cares about all those things.
This is already a thing and there are lots of developers putting effort into the `dat://` and `ipfs://` formats and protocols. Persistent naming in content-addressable networks that can be trusted is presently being tackled... and how to structure apps on these protocols...
I'd be down for working on projects in this space. I'm presently contributing some work into the Lean theorem prover where I'm hoping, with a bit of elbow grease, it will be fairly low-cost and attractive to build out more p2p protocols and libraries that meet our privacy and security demands.
I think this may be a great idea for indie software developers, like Timing or Standard Notes, in order to expand a particular offering. SaaS will probably rule for a lot of enterprise software, but there are always niches to be filled and some of them don’t make sense pricing wise to do a subscription. I like this!
The Holo / Holochain project is building this, and is planning to deploy this year, at scale.
This is excellent and gets me excited - many of those "ideal" principles were important to me when I designed and wrote my own iOS app Mindscope, basically a Workflowy-meets-Scapple app for visualizing your thoughts hierarchically. https://itunes.apple.com/us/app/mindscope-thought-organizer/...
It's an app I wrote primarily for myself, but it's been great to hear from lots of people who really "got" the vision themselves and use it a lot.
I simply love apps & sites that make immediacy and the feeling of "control" core values of the UI. Wish I had more time to give Mindscope more development love than I've been able to lately...working on that.
I have to say, there are some brilliant ideas presented here. Obviously I don’t know much about the details of CRDTs but going to dig deeper into them.
I actually like what manager.io does
This is a lot of good work and thinking put in to a technical solution to something that’s not a technical problem.
The reason that software is online is a business one, not a technical one.
Software as a Service is impossible to pirate and generates continuous income rather than a single upfront fee. That’s all you really need to know to understand why there is less and less desktop software coming to the market these days.
So yeah, sure, if you were to build a piece of desktop software from a clean sheet of paper today, this is a really good guide on how to do that. But nobody is going to. Because it makes no business sense to do so.
Most people won't make use of this kind of research. However, CRDTs aren't just another way to architect the same kind of software. They are an inversion of the tropes and techniques we've zealously stuck to over the last decade, and they grant us brand new technical capabilities that no SAAS player will ever be able to offer:
• Offline-first support with real-time collaboration
• Real-time collaboration with local devices over Bluetooth/ad-hoc Wi-Fi
• End-to-end encrypted real-time collaboration without the server having access to any of your content
• Transport-agnostic sync: use Dropbox, iCloud, and Bluetooth all at the same time with no consistency issues
• The ability to switch to a different cloud provider with zero friction, and to grab your (mergeable, collaborative, and versioned) documents from your current cloud provider without conversion to any intermediary format
• Anxiety-free sync: the user can be 100% confident that their changes will never fail to merge, even if they spent a month in the bush editing their documents
These are off the top of my head, but there are many, many others. And they are features. If enough people build software using these tools, people will get used to them and start seeing the big players as annoying and clunky. ("Why can't I make changes to my spreadsheet when I go through a tunnel? Why did I lose all the changes I've been working on over the last hour? What do you mean this .doc file is just a link to a webpage?")
Is there Big Money in it? I don't know (or care), but I'm going to try hard to make sure that any software I write on the side follows these principles, and I hope others start to do the same.
You could have easily said that "the reason that software is online is a business one" about time sharing versus personal computing, and yet here we are. Focus on the user instead of your bottom line and you will (eventually) win.
Can't say it better.
I welcome your thoughts on swarmdb http://github.com/gritzko/ron-cxx
That is syncable RocksDB with CRDTs inside. Pre-alpha.
Some other interesting projects: the DAT Project (https://www.datprotocol.com/, https://datprotocol.github.io/how-dat-works/) as well as Briar (https://briarproject.org/). I'm very interested in this space for all of the reasons you outline.
I'd say it's even simpler: Software as a Service is a lot easier to develop than software that a third party has to install: Integration of build pipelines that deploy the software immediately are a blessing for bugfixing and you can build an insane backend out of third party dependencies noone but you has to ever install and connect to each other.
I don't like the privacy problems this generates, but in a business environment I want my stuff to work for my customers, and that means I do the hosting.
There is also the huge advantage of not having to install anything. It's already hard enough to convince my friends to switch to any open source videoconferencing in-browser solution, I would never go through having them install a whole application.
"No install" is a major selling point, unfortunately.
5 replies →
Keep in mind that it's possible to develop Software as a Service by selling software that the user has to install - we have done it with our accounting software Solar Accounts. This approach is probably more painful to develop than the traditional web app, but it does give our users features such as (a) end-to-end security and (b) storing data locally - which are selling points for some users.
SAAS and the goals outlined in this article can be perfectly complementary if you let client-side JS do most of the work. Just because an app runs in the browser doesn’t mean that you also have to share all your data with the developer, or lose your ownership of it.
> The reason that software is online is a business one, not a technical one.
That's only partly true. Software gets more and more developed for the big corps in mind as the sole users. So web browsers are developed to make Google and FB happy. Kubernetes is developed to only work in AWS and Google Cloud really well. And so on.
All that hollows out the foundation that gives software usability to all of us. Eg. if big linux corps only focus on Kubernetes, Kubernetes only works in AWS and GC, then nobody puts effort into Linux anymore. It might even happen that linux gets unusable if you don't run it in such kind of environment. In the end it all depends how the majority of resources get spent.
So we come from today's tech level X everywhere to tech level X+Y in the cloud and tech level X-Z on local clusters. Then where will most of the data be? Where will the new hip stuff be? How can you develop new stuff with an incomplete local infrastructure?
What you lose is not the desktop software but the desktop. And you will lose it in a technical sense. You won't be able to rebuild it technically alone.
> a technical solution to something that’s not a technical problem
Why does a problem have to be 'technical' to have a technical solution? Skype, Wikipedia, Amazon, eBay...pretty much every major technical product I can think of solves non-technical problems, almost by definition.
> But nobody is going to. Because it makes no business sense to do so.
You are talking about a general trend as if it's ubiquitous. Yes there are strong business reasons why most software vendors have shifted to SaaS, but that doesn't obviate the fact that some users have problems for which local software is a better fit. The local software market may be smaller, but it exists, and it's incorrect to say it makes "no business sense" for anyone to operate in it.
New technology can be a prerequisite to solve certain non-technical problems, but it’s not the technology itself that solves the problem. Wikipedia is a good example. It probably wouldn’t work without the technology, but just building a wiki platform is not enough to get people to volunteer to collaborate on writing high quality encyclopedic articles. Arguably, the vast majority of the work that has gone into creating Wikipedia has not been technical.
Couldn't the same argument be made about why it made no sense to create the Linux operating system, since there was no business case to do so?
This may be a scenario where it's possible to create a communal good (for personal data ownership and control) which doesn't require businesses to lead the way.
In addition, it is feasible to distribute some-or-all of an application's software via the cloud while still keeping application data locally; we can be more nuanced here and keep a lot of the business benefits while caring more for users.
> Couldn't the same argument be made about why it made no sense to create the Linux operating system, since there was no business case to do so?
But there is a case to do so: https://www.joelonsoftware.com/2002/06/12/strategy-letter-v/
> we can be more nuanced here and keep a lot of the business benefits while caring more for users.
If you are talking about "business", then we should not talk about "users" but "customers". And it so happens that most "customers" (so far, at least) consider it more valuable if the business also is responsible in keeping (and securing) the data. For these customers, putting the responsibility of controlling the data on them is a burden, not a privilege.
1 reply →
> makes no business sense to do so
Actually it's not true that local-first software cannot be businesses (or non-profit orgs but still revenue-generating), it's just that the money to be made in local-first is an order of magnitude (at least) smaller than with cloud software. On one hand, the money to be made is much smaller, but on the other hand, local-first is where the next big disruptions can happen, while still having a non-zero revenue. Look at open source or open data projects for some examples on this: Wikipedia, VideoLAN (VLC), OpenStreetMaps, etc.
> But nobody is going to. Because it makes no business sense to do so.
While your reasons make sense, they make sense from the supplier side. However, the ultimate decider of what makes "business sense" is not the supplier, it's the customer.
You can have all the supply-side advantages in the world, if your customers decide they want that other gal's product, you're toast.
It may not even make sense from a supplier side (sometimes). Instead of just writing the SW, the supplier now has to manage servers, deal with angry customers when downtime happens, backup, protect the servers from attackers, deal with scaling, deal with customers who don't like change, etc., etc.
With local SW, you can just write it and mostly forget about it, as long as OS stays compatible with your program.
But the customer is easily distracted. That's why we have things like Facebook where customers will happily give up their privacy in return for a little functionality.
Please sir, can I have some more?
As long as the internet remains free and not completely controlled by a few big players.
> Software as a Service is impossible to pirate
There are SaaS business models with local/native desktop software, e.g. Adobe CC.
iOS AppStore-monetized software can run locally and be licensed as a subscription. Apple encourages local computation and continues to add both hardware and software for on-device processing.
x86 clients are adding SGX (hardware secure enclaves) which provide stronger mechanisms for endpoint license enforcement. This can support multiple business models, including subscriptions.
Local computation makes sense especially from a user experience perspective. However, another challenge arises from talent availability and willingness to work with this sort of deployment model. More complex deployments generally lead to the need of hiring really top notch engineers, which as we know is quite scarce. The next best thing might be infrastructure that makes the deployment easier but no one's really interested in investing in infrastructure tech without some tangible reward.
2 replies →
Adobe CC is desktop software with a monthly licensing fee.
1 reply →
That’s not saas
I once subscribed to a farming management program, developed by a french company that was launching to the cloud market.
Apparently they had bought several other smaller companies, integrated their software into their own program when useful, built huge servers for their cloud service that they also rent.
I knew the program was good - I was familiar with previous versions - so it was kind of a no brainer. The problem was the new cloud thing.
I am in favor of economic rational practices in agriculture (in addition to the normal functioning of markets), so that crops are chosen wisely between farmers and prices do not plunge because of all-or-nothing decisions, and sharing information between farmers is essential.
But data harvesting (no pun intended) of farming practices to the purpose of informing large companies and futures markets brokers of their possibilities can work against the interests of farmers and consumers. The way large corporations treat small farmers is ruthless and they negotiate no terms.
So if my and other farmers' data was being uploaded to the cloud without possibility of local storage, who had access to it? Every crop, production, pesticide, invoice, price was going there!
They wouldn't sell it, they told me. It is stored with e2ee, they assured me. If I ever forgot the password, data would be irrecoverable, they said.
But there was also the problem of internet connection issues, because farms are remote. Would it ever work?
The first thing I noticed after installing the program was that there no important message was shown that informed the user of what would happen if the password was lost. I typed in some new data, saved and exited. Then I started the program and reset the password with ease. There's no possibility of 3FA, either.
So I told them I could not use the program because of my poor internet connection, which was true. They installed a local version that was temporarily available.
But they took so long doing this that I developed my own modular managing system, based in - yes, that's right - spreadsheets. It fits my needs, it is completely portable and it works.
The formulas are getting cryptic, and there are still some things I have to do manually (like sorting lists or generating pdfs), but the files access each other and it works. I have no time to code anything and I don't think I have sufficient knowledge of database software (I am trying to find time to learn MySQL).
Maybe some day I'll make some GUI that handles all of this, in such a way that if everything else fails, I can still access and edit files manually.
As a thought, since you're just getting into database stuff as time permits, it's probably better to start out with something like SQLite and/or DB Browser for SQLite, if you prefer the GUI way of doing things.
https://sqlitebrowser.org
https://a-gentle-introduction-to-sql.readthedocs.io/en/lates...
SQLite is a bunch easier to get up and running than MySQL, and the data is all in one file you can copy around. So, easy to backup. :D
If you do eventually hit it's limits, it's big brother is PostgreSQL, which is a really powerful database system. :)
5 replies →
Thank you. Thank you for being self sufficient. Thank you for learning what you need to help yourself.
If you ever need a helping hand. Maybe with visualisation or analysis stuff shoot me a note.
1 reply →
This is applicable the more you try to optimise the business for profit, and, ironically, not "service".
It dehumanizes the customer.
With SaaS the customer is always running the latest version of the software.
You only need to support it, no need to worry about clients/customers still using a 6 year old version because they refuse to upgrade for some reason.
This also has downsides: for example, the user is confronted with an ever-changing UI, and other surprises.
1 reply →
On the flip side customers lose control over their environment.
If you work in regulated environments this can be s big deal and this validation process becomes captured (forced) by the saas provider.
If you ran your own you validate and keep that unless new requirements. No need to revalidate every time a new release is out and pay for that re-validation.
This is not always true.
My last company had at least one client lagging behind by about 100 releases.
I think it was browser compatibility and their internal enterprise firewall.
> Software as a Service is impossible to pirate and generates continuous income rather than a single upfront fee. That’s all you really need to know to understand why there is less and less desktop software coming to the market these days.
Sounds very cynical and short sighted to me.
Pros to customer: Nothing to install. Can be used on any operating system with a web browser. Updates and security patches are performed automatically. Can share documents and collaborate more easily.
Pros to developer: Easier to support multiple platforms. Easier to upgrade users and migrate data.
> So yeah, sure, if you were to build a piece of desktop software from a clean sheet of paper today, this is a really good guide on how to do that. But nobody is going to. Because it makes no business sense to do so.
Imagine having to install Trello, Google Docs, Slack etc. manually everywhere you wanted to use it, deal with updates yourself and ask people you wanted to collaborate with to do the same. That makes no sense in terms of ease of use.
>Imagine having to install Trello, Google Docs, Slack etc. manually everywhere you wanted to use it, deal with updates yourself and ask people you wanted to collaborate with to do the same. That makes no sense.
That's the way things worked for years, and the way most software still works (except through app stores) and it wasn't a madhouse of anarchy and suffering, it was perfectly fine.
Installing software "manually everywhere you want to use it" implies installing it once in 99% of cases, maybe twice in most of the rest, where "install" is a trivial process. Businesses will require that everyone use the same set of installed software, so that's not even an issue to begin with. It's not unreasonable to expect people to have a PDF reader, or Adobe software if you're working in graphics, for instance, or for Windows users to be able to read Windows documents.
Plus... you do know native software can handle automatic updates and security patches as well, right? And if you don't think people can share documents or collaborate easily with native software, I mean, that's been a thing since Napster and IRC. FTP and USENET. SAAS didn't solve any of those problems, because they were never problems.
6 replies →
`sudo apt get install trello gdocs slack` wouldn't be exactly taxing (I realise it d doesn't work that way, but it could). You can just have a web/net link with apt-url too.
I imagine it's similarly easy with chef/puppet, etc..
One of my problems on Ubu is that it's so easy to install stuff that I can acquire a lot of unused cruft; mind you it's mostly a mental problem now with disk sizes as they are.
I agree that SaaS is more convenient in the short term on both sides, but people have been using desktop-based software for a long time so I don't think it's accurate to say it makes no sense. Like anything else, there are tradeoffs.
Desktop software works "forever" (as long as the underlying OS and drivers don't break support for it, at least) and the longer you use it, the more value you get for your money. SaaS works until the company that maintains it stops supporting it, and the value scaling is basically the reverse of desktop software.
>But nobody is going to. Because it makes no business sense to do so.
I've got Blender open on my desktop. Could you explain the business sense behind Blender exisiting?
Interest? Very few people care to build medical billing software or crms for free in their spare time. So you get products that aren't as good. Even with blender, there's far better paid tools because people need money to live, so they put their efforts into things that make money.
9 replies →
Companies/Content producers that do not want to depend on a single vendor, or want to avoid the lock-in, and went on to support an open alternative?
Makes perfect business sense.
12 replies →
Donations and free time.
The Software as a Service model is 100% compatible with what they have outlined, and in fact has been working well with many traditional desktop applications (Microsoft Office, Adobe Suite). Providing a good end user experience while taking care of caching, cross-device & offline syncing, conflict resolution etc. are still mostly unsolved problems, and any amount of progress in those areas is good.
The other reason is how it looks to business customers.
Buying a piece of software (one-off) almost always requires budgeting and capital-expenditure approval. Add in the mess of depreciation calculations and maintenance fees, and it's just a mess. Can be weeks of work just to buy £200 worth of software.
A subscription service usually gets budgeted and booked as recurring expenditure, and looks better on the balance sheet.
In some environments (esp. the public sector) recurring (AKA "operational") expenditures are a lot harder to justify than capital expenditures.
An OpEx increase requires an increase in budget paid for by taxes, while CapEx can be covered by a one-time bond.
This leads to paying vendors millions for a software+support contract rather than hiring a few engineers.
Taxes might be different too. our municipality taxes my wife's biz as a percent of biz capital (US)
>Software as a Service is impossible to pirate and generates continuous income rather than a single upfront fee. That’s all you really need to know to understand why there is less and less desktop software coming to the market these days.
I'd argue that the increasing complexity of the stack involved in any given product has also brought us to a place where the kinds of thorough QA we had 20 years ago isn't really viable anymore, at least for most smaller orgs.
The ease with which you can fix a deployed product in a SaaS situation is a huge boon.
You can charge subscriptions for installed software as well. Oracle is an example, as is every other on premise Enterprise software
You can, but it's still easier to pirate said software.
And in Oracle's case, you're neglecting to mention their ability to bring lawyers to bear on any infringers, something not a lot of software companies may have the capital to do, compared to a SaaS model.
> But nobody is going to. Because it makes no business sense to do so.
If your only objective is making money, that's correct. If your aim is to achieve something more, you may still want to take this risk in spite of knowing you'll earn less.
"Software as a Service is impossible to pirate and generates continuous income rather than a single upfront fee. That’s all you really need to know to understand why there is less and less desktop software coming to the market these days."
The other reason is platform control: there's continuous tension between the desire of each OS vendor to differentiate their platform and make applications part of the desktop experience, and the desire of application vendors to be able to deliver to any customer using the minimum number of platforms.
SaaS vendors will build mobile apps (1-2 platforms), and desktop applications in Electron (kind-of 1 platform) as service clients, but I agree that they are not incentivized to build local-first applications.
> Because it makes no business sense to do so.
Apparently there are needs, that's why this guide even exists. Maybe not many but there are people willing to pay for better privacy/data protection.
But you are right, this won't generates continuous income, which is bummer for developers.
s/developers/software selling businesses/
Actual developers rarely see ongoing revenue from their past works. It's the business who sees the ongoing revenue, because odds are high that any individual developers have moved on with their career, and have been replaced by someone new to kick in 2-3 years of labour.
> Software as a Service is impossible to pirate and generates continuous income
Service as a Software Substitute (SaaSS)[1] is also inherently spyware, an increasingly popular feature as surveillance capitalism[2] infects our economy.
[1] https://www.gnu.org/philosophy/who-does-that-server-really-s...
[2] http://nymag.com/intelligencer/2019/02/shoshana-zuboff-q-and...
It is not inherently spyware, because fully homomorphic encryption exists. This in theory allows cloud computing providers to operate with no knowledge of their customer's data or computations. It's in general too computationally expensive to be practical, but that's not the same as being impossible.
https://en.wikipedia.org/wiki/Homomorphic_encryption
12 replies →
>Service as a Software Substitute (SaaSS)[1] is also inherently spyware, an increasingly popular feature as surveillance capitalism[2] infects our economy.
Sure, but it's how business in the real world works as well. You can't enter a shop without being observed or even recorded, either by cameras or the shop personal / owner in smaller ones. Your behavior during a conversation is analyzed in realtime and reacted to, that's the basics of selling. There is no invisible commerce in the real world.
That's a fundamental reality that you can't just omit, doing so would be irrational treating the web as something purely abstract, which it isn't. It's an abstraction / extension of the real world, because it's part of it.
What the web needs is obvious boundaries and private spaces that are basically extensions of your own home and those of your friends, make it clear that you can't do x or y (going in public) without being observed and analyzed. Right now the only real indicator is the tracker count of adblock plugins.
2 replies →
Surveillance capitalism? Capitalism as an economic system does not surveil, governments and corporations do. If you do not like a service, you can opt out by not using it, unlike the government. If you have an issue with something used by millions of people, then you should consider the possibility that it actually may be the people who keep the service alive.
I try to do my part. I do not buy access to games on Steam, movies and TV series on Netflix, books on Amazon, I do not use Google, I do not use Facebook nor am I in contact with anyone who does, and so on.
I am open to discussion. Why is it specifically capitalism's fault? What other economic systems would be favored, and why?
3 replies →
There are local first E2E encrypted apps that do charge monthly fees. 1password is one example, bitwarden is another and mylio is a third. 1password syncs with 3 services, mylio syncs with your local NAS, other computers, amazon, etc.
They are definitely in the privacy space more than others, but it shows it's possible.
I thought about ideas like the post here myself, although I admit I haven't thought about it in the detail they have there and I am look forward to reading that fully over the weekend.
Anyway I don't think piracy is the problem per-se, but the question is: what is the commodity?
Because take Netlify for instance - picking on them, but you will see this sort of pattern everywhere:
1. Generous free tier to get you hooked. 2. $29 / month for something that costs $0.01 in terms of cloud compute. 3. $100 / month for something that costs $0.10 in terms of cloud compute.
Etc.
So really they are creating a fake scarcity and mark up stuff ridiculously.
But with local-first as I see it, you will get stuff like "filling in forms" or whatever software that has made companies into multi-million companies for free. And you will deploy that on cloud(s) of your choosing with no effort, and pay their pretty cheap per-transaction costs. Stuff that is trivial will rightfully become very cheap and commodity like.
This is probably a good thing for consumers, because for cloud companies and new startups to compete they'll need to offer truly innovative products. Do stuff that was really not possible before. Quantum computing for example.
That generous free tier probably costs more to run than all the paying customers combined, depending on how generous we're talking and the proportion of free to paying. There was likely also no revenue for a significant time, but still salaries to pay, and other costs to be borne.
If you really think the profit margin is 99.96% for that tier two customer, it should be pretty easy to create a competing business to provide the trivial service and charge a lot less. Like $0.02 and have a 50% profit margin :-) But I think you're underestimating how much it costs to run a business, even if compute costs are cheap.
1 reply →
2. $29 / month for something that costs $0.01 in terms of cloud compute.
The major cost for paid tiers is not infrastructure, but user support.
1 reply →
> But nobody is going to. Because it makes no business sense to do so
An app that comes to mind is Omnifocus (iOS/macOS): https://www.omnigroup.com/omnifocus
Double stream of income in validated environments where the provider gets to validate on your behalf on their schedule. It’s nog like you needed that last useless feature, but hey, since it’s new we gotta validate it for you, pay me.
The reason that software is online is a business one, not a technical one.
True, but I host my data in the cloud because I don't want to pay the energy bill of the equipment at home. Despite not having concrete facts I believe it is more environmental friendly if we started hosting servers centralised in facilities instead of everyone doing at in their homes.
> I host my data in the cloud because I don't want to pay the energy bill of the equipment at home.
Then you pay someone else's energy bill.
> Despite not having concrete facts I believe it is more environmental friendly if we started hosting servers centralised in facilities instead of everyone doing at in their homes.
It's true that running 1000 instances of a software on the same big computer consumes less than running them on 1000 different computers, because they won't use all its resources at the same time. There's also less overhead if you can share their computations.
However, hosting everything at the same place has some environmental issues too. A datacenter needs cooling, and not all homes do. If there's a lot of data traffic between you and the software, that traffic needs to go through the internet instead of being local.
I'm not saying one way is better than the other; it probably depends on the kind of software we're talking about.
TLDR:
Local-first software is powered by CRDTs.
If you want to learn more about CRDTs, check out:
- https://github.com/automerge/automerge (author's project, legit)
- https://www.youtube.com/watch?v=yCcWpzY8dIA (deep technical talk)
- https://gun.eco/distributed/matters.html (my Cartoon Explainer)
OT also works fine for this sort of stuff. OT algorithms are easier to implement ([1] for an implementation I wrote of OT over arbitrary JSON structures). OT just requires a central source of truth / a central authority. For local first stuff depending on how you design it you can have one of those - in the form of your server.
[1] https://github.com/josephg/json1
Your work originally inspired me to get into this stuff. :)
<3 thanks so much for your amazing contributions to OSS.
OT are inferior to CRDTs in every single way. In 2019 people shouldn't even be looking at OT.
6 replies →
A cute dog is dancing in video Just watch this video and try to control your laugh http://bit.ly/2ZK1XVs
See how a Turkey saved the life of his friends Even those birds also have love in their hearts http://bit.ly/2WhAiZS
Sometimes our feet smell are so bad even we feel ashamed due to our feet smell Now you can get rid from this just follow this simple method https://zoomtips.blogspot.com/2019/04/Smelly-feet.html
Guy made world record He drive car on two wheels See his video how he is driving http://bit.ly/2ZOYaWX
See the friendship between cat and squirrel That's so cute see how much they are taking care of each others http://bit.ly/2VdHs4E
Very nice initiative ! it would help users a lot in saving their valuable data