Comment by lbutler
5 years ago
I have an application that uses sqlite as their file format but they've cleared the header so you can't open it directly.
I'd love to be able to make secondary applications like you've described but being enterprise software they don't want to make it too easy.
They obviously want to keep people locked in with their $40k per seat application!
I guess the first step is figuring out the page size and other bits the other meta data you set in the header [1].
I know I just have to sit down and understand the format better and I will eventually figure it out...
That's mean, but you'll just have to live with that. If someone takes active steps towards preventing people from making secondary apps/utils, then they're (a) jerks, and (b) allowed to do that.
Might be worth actually contacting them to ask why, if you can make the case that secondary applications will increase the value of their app, not decrease it.
I'm surprised that nobody's pointed out that there are actually valid reasons other than greed to obscure your file format. It's an implementation detail, not a contract. If customers begin relying on the implementation details, you end up with angry customers when you change the implementation details. A SQLite db without the header is basically a statement saying, "we are using the obvious file format here for our convenience, not for general purpose access. Screw around in here at your own risk."
If you modified their app's internal state db and screwed it up because they have designed their software with certain assumptions that aren't clear from just reading their db schema, that would be a nightmare for them to support. The easiest thing for them to do is just to try to discourage tampering with their internal state.
This is especially true if there's a chance that a market for secondary apps/utils will spring up. If that's to happen and be viable, they absolutely would want to put thought into what their supported interfaces are for those apps/utils, otherwise they will end up painted into a corner and unable to change their architecture without destroying a marketplace.
I really don't get this line of reasoning. If I do something with a product that is (maybe even explicitly) unsupported by the manufacturer I don't have a reasonable leg to stand on. We've recently had this with a customer where they used the concrete calls of our app to automate some of their stuff (we had had no public API for that action at that point because we hadn't had the need so far and so did no other customer or to be precise: no customer was prepared to shell out the cash for us to develop it) and after we changed something they suddenly weren't able to script the creation of customer entries in their installation. We had told them at least two years ago that the way they interacted with our system was but supported (we only noticed them automating stuff back then because it was throwing exceptions in our backend) and while we were nice enough to fix it this time because the fix was trivial we recommended that they should switch to our supported API. Two weeks ago theyir stuff broke again and we told them to use the API or fuck off.
1 reply →
I've never understood the "We can ship this but it's not contract" mentality. To my mind if you ship something it's contract. I think developers keeping that in mind would make the world full of more resilient code. You're making a contract w/ your future self, if nothing else.
To the point re: modifying internal state and screwing-up the application - If you're writing anything out to persistent storage you should assume that it's untrusted data when you read it back in. If for no other reason than physics itself is a malicious (or, at best, ambivalent) actor.
2 replies →
+1. I recently made a variation of this argument on my FOSS app. [1] If it were commercial software with support I'd feel even more strongly.
Philosophically, people should be able to do what they want on their machines. But expecting support (eg figuring out how their third-party software has corrupted my database) is another matter, so I can see why people would install a speed-bump or two...
[1] https://github.com/scottlamb/moonfire-nvr/issues/44#issuecom...
>That's mean, but you'll just have to live with that
I mean I know we're all on board with the idea of intellectual property actually being a thing now, but surely there are limits? I've seen people take the hard-line stance that if something is your property you should be able to dictate exactly under what situation it can be used, but there have to be limits to IP holders rights on some level, and I feel like reverse engineering a file format is a pretty reasonable place to draw that line.
> we're all on board with the idea of intellectual property actually being a thing now
We most certainly are not. I personally believe that intellectual property as a whole doesn't make sense in the 21st century and should be abolished.
> there have to be limits to IP holders rights on some level
There are. The laws generally recognize fair use and reverse engineering for interoperability.
> I feel like reverse engineering a file format is a pretty reasonable place to draw that line
Absolutely. Unfortunately, in the US it seems corporations can force people to give up their rights by making them agree to it. Therefore, "you must not reverse engineer our software" is a standard clause in every contract and it's not negotiable.
12 replies →
I didn't take "you'll just have to live with that" as a claim that the user couldn't reverse engineer it. It simply means that they've made it harder for the user to do so, and so the user has to live with it being harder.
1 reply →
> I know we're all on board with the idea of intellectual property actually being a thing now
I don't know that that's true at all! I'd say the pendulum is swinging in the opposite direction.
>we're all on board with the idea of intellectual property actually being a thing now
No. Intellectual property is not genuine property. It is a state granted monopoly and is antithetical to free market principles.
3 replies →
They aren't saying you have to live with the legal restriction, but rather the technical restriction of having to reverse engineer it without the headers
I think the limit is when they are using their IP to prevent you from owning your own data.
Even when a person is on board with intellectual property rights, there is still a distinction between the vendor created program and the user created data. Encoding the data in an undocumented or obfuscated file format may not exert legal rights over that data, but it effectively does so.
I can only see three reasons why someone would support intellectual property rights with respect to file formats: they believe the manipulation of data done by software implies a transfer of ownership of the data, at least in its modified form; they are making a cynical grab for control over the data; or they are incredibly naive.
(There are border cases, such as novel compression schemes, where how the data is stored is the product. That does not really matter when someone is using a file format as a simple container for data. If a file format is truly a border case, there should also be ample forewarning to the end user.)
I can understand why they do it, it's enterprise software so the more open you make it, then the less licences they can sell.
They do have a module you can purchase to run API calls and access their files/software but as you probably guessed that's another $40k license!
Most of my apps I build use this API, but for me to provide to other companies they need them to also buy the API extension.
I'd love to cut out the middle man and I'll do it eventually when I reverse engineer the header!
Although the circumventing of security measures would be illegal in many countries. (Like removing the copy protection of a game)
I don't get this mentality. You haven't created this system. You have no idea what it takes or else you would have created it yourself. You don't like the price, but you want it anyway. How is this, I mean come on, different from stealing? You even say it: cutting out the middle man. You want to remove their profits.
At $40K/seat, that must be some special software. Simple economics invites competition. Depending on the software, it may not take too many seats to incent the funding of an alternative.
Maybe this is the kind of software that requires huge development costs. But maybe it would be worth 20 seats' worth of customers joining forces to fund a team of 5 people to build you a competing app tailored to your specific needs/wants and completely under your control.
Granted, that could bump your costs from $800K/year to $1.6M+/year. But only short-term. Once your software is production-ready, you drop the costs of your current software. So think of it more like going from $8M/10 years to $6-10M/10 years but having complete control to add the features you want. And perhaps having the opportunity to recoup $millions/year by licensing to others. Or, open source it and give others the same kind of control while benefiting from the features they add. Spread your development costs across more seats to further lower your $/seat.
Or, look at the 100 employees your vendor currently has and lose heart, then hope somebody with deep pockets funds a competitor.
This particular software is for water utilities to model and simulate their water and waste water networks.
It's mostly used so utilites can forecast growth in their areas for the next 25+ years and see the impact on their networks and feed into their capital work projects.
A decently sized utility may spend up to $200M/yr on capital works so $40k isn't even a line item!
There is completion in the market but consultants are forced to use what their clients pick and most utilites aren't that price sensitive.
There are also open source alternatives by the EPA[1][2], and most commercial operators are just wrappers around this public domain software.
I'm trying to create FOSS to help view and run these models.
[1] https://en.m.wikipedia.org/wiki/EPANET
[2] https://en.m.wikipedia.org/wiki/Storm_Water_Management_Model
Thanks for sharing! Your FOSS projects [1] look awesome, wish you the best in building them out. I’m fascinated by examples of niche but critical software applications. It reminds me of Patrick MacKenzie’s thesis that there are more B2B software opportunities out there than you’d expect. [2]
[1]: https://github.com/modelcreate/
[2]: https://twitter.com/patio11/status/1213188681843892224
1 reply →
It sounds like they are using just a basic Linear Regression to forecast growth.
And to the uninitiated, this sounds like a very fancy word, that makes the software seem smart.
But really, it’s just drawing a straight line. And as you add more counts to your x axis, the Linear Regression “forecasts” what the value is on the y axis. This is the magical number, given by the computer, and is used to determine future load or capacity needs.
shrug I used to own a company (since sold) that sold software for $50k/seat. Most people who bought it also bought the automatic failover option, which was another $50k/seat.
The movie business doesn't even blink at that sort of cost if it there's even a small chance to prevent having to set up the remote shot again. The logistics, time, hiring, transport, accommodation, equipment, wages, etc. etc. etc. all make $50k a drop in the ocean.
We spent 2 years writing the software, developing the add-on hardware that helped, and touting it around various Post-production houses. It was used on Star Wars I, The Matrix, etc. Post houses started to take it on board as well. Then we were bought, and the product discontinued. C'est la vie.
I’ve been thinking that the next realm for movie studios is to use the random AI face generated people, that look real, but don’t exist.
These faces can be planted on other fake AI bodies, that move like real people.
Then, what you have is a background full of fake AI people. They look like real people. They move like real people. No more need to hire extras.
You can just have your primary actors act in a green screen. And virtually change the world all around them.
The first company that can commercialize this, is going to make a ton of money. And might even be able to gain first-mover advantage, as they lock in all the studios.
I might be wrong but I thought Dropbox used to do that as well. All the sync state is stored in a sqlite file with the header changed or removed.