Nutanix Objects violates MinIO’s open source license

4 years ago (blog.min.io)

MinIO is a fantastic tech and they seemed to be really patient to resolve this issue (waiting 3 years before doing actions). I'll continue to use them and recommend them everywhere I work. They really deserve respect... And to be paid for their hard work.

  • I'd like to hear Nutanix side of this story before sliding with one party. Awaiting for blog post from them :P

    Until then here is my spicy story: - In 2019: Minio Sales contacted Nutanix (like this user mentioned https://news.ycombinator.com/item?id=32152645) hoping for a nice big cheque.

    - 2019-2021: Nutanix cites Apache-2 license and refused to pay.

    - 2021: Minio changed its license to AGPL (probably few others like Nutanix)

    - 2021: Nutanix knows this and refuses to use AGPL version with their product.

    - 2022: Discussion went on for another year and nothing came out from Nutanix.

    - Now: Minio decided to publicly shame the company.

    • I worked on Objects at Nutanix for the last ~12 months. Nutanix had originally used the API server in MinIO to translate between the S3 REST API and internal RPCs. MinIO's claim on this blog post that "Nutanix Objects is built around MinIO object storage" is a gross exaggeration.

      By the time I joined in June 2021, MinIO was deprecated and we were using an in-house S3 REST API server. I am skeptical that any of the APGL code was distributed because we just weren't using it around the time that MinIO changed from Apache to AGPL.

    • Do you work for Nutanix? Why the brand new account for this? And then referencing another throwaway user that says they work there

    • MinIO is leveraging their switch to the AGPL license as a vehicle to extract immense seven figure plus licensing agreements from "big" companies who relied on non-AGPL versions of the code in the past.

      I don't know if that makes MinIO bad, better or good but it's all about money.

Around 2019, a lot of kubernetes distributions started popping up. They often bundle various open source solutions into one platform/PaaS and sell it to the end users. I wonder,

- What are the consequences for these companies?

- Do they share revenue with the open source projects?

- Can they simply distribute these services without any consequences?

- If not, When and How does a small open source project org enforce track and their license?

  • The consequences is that these companies get very rich and they eventually take-over the open-source project.

    See Redis for example, two Israeli dudes took the open-source Redis, made tons of money.

    Everyone is happy: the two founders became rich, the VCs became rich.

    What about the authors and contributors of Redis ? Well thank you for the gift. As a present you can have the privilege to work for us to keep maintaining your bugs. Don't complain too much.

    Then you can rewrite the history to make it sound like you created Redis and it's a win, while it's actually just a very smart dude in Italy who wrote most of the software using his own sweat and support from his employer (Pivotal).

    • > What about the authors and contributors of Redis ? Well thank you for the gift.

      He was eventually hired by Redis-the-company, allowed them to use the trademark (originally they were Redis Labs which was a compromise with him), went to their conferences, trained their Redis developers (who contributed to Redis-the-open-source), etc.. I assume he was happy with the deal as he spoke positively about them and chose to spend a lot of time with them, and eventually retired after I presume getting a nice amount of money from the decade-long adventure.

      1 reply →

    • These days liberal OSS licenses are really just free labor for this kind of thing. If you use a very liberal OSS license just make sure you are 100% OK with your work being appropriated this way, including having your name stripped from it and some hustler taking credit.

      In the long term I think this kind of behavior is going to kill open source for things beyond libraries and building blocks.

      Everything open on the Internet is destroyed by exploitation of one form or another: appropriation, spam, scams, etc. I've become fond of saying "the Internet is a dark forest."

    • If he didn’t want someone commercializing his software, he should used a different license. His own employer is is a commercial wrapper on an open source Project.

  • Note that projects in the cloud native space are mostly Apache-licensed. (For example, the CNCF only approves other licenses on an exception basis I believe.) In that case, so long as attributions/trademarks are honored (which oddly seems to not have been the case here), projects licensed in that way can be freely used with Kubernetes without other restrictions.

  • Companies are not doing anything wrong when they are profiting off a permissive license. Its only a problem if they do something which is not in line with the license.

    If people want to make money off open source, then dual licensing is a good way.

    Permissively licensing code and then when companies use it as they wish (due to the permissive license), complaining about it does not look good.

Kudos for MinIO team for spending THREE YEARS trying to resolve this. Shame that they had to resort to a public naming and shaming but sometimes corporate entities are tone deaf.

Now this is exposed the next question is if Nutanix Objects is just a MinIO wrapper then what value are they even proving here?

Open and shut case. Disappointing that the Nutanix engineers care so little about their peers.

  • If you think this is bad, you should see how their sales and marketing departments behave.

  • Please don't blame engineers on every single issue. The engineer may not even know there's an issue here. They may be assured by their boss or legal department that they are in the clear. They may not even think about such mundane things like licensing and stuff, that's what they have higher ups for.

    If someone is to blame, then it's the company leadership and legal department. As much as we want to make us engineers more important than we are, we are not decision makers. Blame should be put where it belongs.

    • > They may not even think about such mundane things like licensing and stuff

      Imagine a medical doctor or civil engineer claiming that knowing the laws of their professions is "mudane". That's why no one takes programers seriously.

      > we are not decision makers.

      You totally can decide to not work on stuff you are not comfortable with. It's not like there's a shortage of software engineering jobs.

      5 replies →

    • Software engineers are in a position of great privilege: if we can’t hold ourselves to account, what are we doing? Almost any software engineer put in a difficult position can get up and walk into another job — “it’s not my decision” is not an acceptable excuse for (almost any) software engineer.

      Blame lies with those who are complicit by choice, just as much as those who are directing the behaviour.

    • > They may not even think about such mundane things like licensing and stuff, that's what they have higher ups for.

      Oh, come on. Engineers these days are not stupid. While I agree that their boss could plainly lie to them that he bought a commercial license, it was more like, "What will we use for the underlying storage?" "Maybe MiniIO, they're S3-compatible and efficient." "Fine. Can we use their code, though?" "Sure, it's open source, and we are a *aaS business, so no problem." I saw this kind of thinking before.

    • "I was just following orders" is not considered a legitimate excuse. The engineers have agency and should be considered a "reasonable person".

      1 reply →

    • The engineers might understand the issue better than legal. Most engineers I've meet understand the spirit and intentions of the open source license far better than the legal teams, who are more interested in whether or not you could be successfully sued.

      One of the issues with open source software, from a branding perspective, is that you can technically be in the clear, but violate the social contract that implicitly exist in the community. Many companies fail to factor that part in when running licenses through legal.

    • The engineer who includes the binary is responsible for understanding the ramifications

    • You are being downvoted but I actually think there are some fair points that you are making.

      We use a lot of FOSS in our company. We pay licenses and contribute very little (our job isn't to improve gitlab or docker, we are shipping a software product on top of that), but I wouldn't know where exactly we are in the legal-illegal spectrum to save my life.

      I consider myself an employee, not an entrepreneur. If I was an entrepreneur, I sure would happily seek legal advice on what exactly is fair use of open source. But really, I wouldn't know who to trust on the free advice market to figure out what I'm allowed and not allowed to do when starting up. I have absolutely 0 interest in legal stuff and it's mostly scary and confusing to me (and that's probably why I don't do any entrepreneurship, not even a side hustle in consulting), and I wish I and other salary men would be given a break about what the company is doing.

      Nutanix shouldn't do what they are doing, but I don't think engineers should be to blame. At the end of the day, if an employee would have to go through everything that the company might not do perfectly right before deciding on a job, we would work nowhere. I wouldn't work for Oracle, but where to draw the line exactly ?

      5 replies →

They use a load of other FOSS software under the hood too, not least of which libvirt/KVM.

I wonder how many other licences they're violating this way.

  • libvirt uses LGPL and the KVM/linux kernel uses GPL. Both are fine to keep to yourself if you run it on your own machine and only expose it over the network.

    MinIO uses AGPL which explicitly includes network usage so Nutanix is forced to provide all patches and associated code.

    • Recent versions of MinIO use AGPL. Much of what they talk about here are issues with Apache licensed code. (The switch happened in April 2021).

      https://github.com/minio/minio/commits/master/LICENSE

      This really seems like Nutanix just didn’t include the MinIO NOTICES file in their OSS disclosures for some reason. Something so minor should have been an easy oversight to fix. Without actually testing out Nutanix, it’s hard to know if they are actually violating this part of the Apache license. MinIO isn’t included in their “open source packages we use” webpage, but that’s not where the NOTICES message would need to be included. Either way, it’s odd that things escalated like this.

      The newer AGPL versions of MinIO would offer its own licensing challenge for Nutanix (which is part of the reason for the switch to AGPL). But that’s not even what MinIO is focusing on in their post. MinIO also don’t show the version of their software that they claim Nutanix is using. And it’s very possible that Nutanix froze the minio version in April 2021 (quite likely the case).

Nutanix Objects does not use minio in the core data path. The presence of a binary in a kubernetes pod doesn't necessarily mean that the binary is being used or the fact that nutanix objects is nothing but a wrapper over minio. Earlier implementations did use minio purely as a S3 protocol adapter, i.e a protocol translator from S3 API to Nutanix internal storage protocol. This was something that was publicly acknowledged : https://blocksandfiles.com/2019/11/07/nutanix-objects-storag...

However, in later releases they seemed to have replaced the minio based protocol adapter to something that they developed in-house in C++ and have no longer using minio in their protocol stack.

  • I recently left Nutanix, but I worked on Objects for the past 12 months. The MinIO path was deprecated by the time that I joined. I don't have enough information to confidently side with either Nutanix or MinIO, but I'll clarify and confirm a few things:

    - As you said, MinIO was used to translate S3 REST API requests to internal RPCs.

    - MinIO was replaced with an in-house S3 API server.

    - I distinctly remember seeing a patch 6-12 months ago where MinIO was removed from the build.

  • ROFL if you see block and files as the official disclosure fron Nutanix is a great testament of how that company is run :) Try getting their OSD file and see if MinIO is listed :)

Will revoking their license stop Nutanix from using MinIO or will they have to go to court to get them to stop? I don't see any mention of a lawsuit in the post.

  • My intuition is that they're escalating progressively. Threats and lawsuits, as a general rule, make it more difficult to reach an amicable resolution. I'm inclined to interpret MinIO's response as a mature and prudent one.

  • First, revoke the license. That means they are no longer permitted to redistribute the code.

    If they then continue to redistribute it, they are committing a copyright violation. That’s when there is cause for a lawsuit.

    • >First, revoke the license. That means they are no longer permitted to redistribute the code.

      I wonder how it works. What is the act of revoking an open source license exactly? I assume they simply sent a letter and wrote a blog post? Pretty sure in my country it would have no legal force. Is it different in the US?

      1 reply →

    • As soon as they are in violation of the license, they no longer have permission to redistribute the code, because the license is the only thing that allows that and it only allows that if they are in compliance with the license. So there is cause for a lawsuit immediately, not after any other action.

      1 reply →

You'd think Nutanix would have the brains to change the names of the deployed binaries, which brings up an interesting question. How do you detect license violation if the violator has replaced the brand name across the codebase?

  • Alternate interpretation: Nutanix is fully compliant with the terms of Apache 2 and refused to be extorted into paying MinIO money.

    The press release is high on FUD (can’t revoke an irrevocable license, no evidence presented they have deployed the AGPLv3 version) and low on details why it took them three years to issue a press release when an injunction would have been granted pretty quick if Nutanix were truly in violation of the Apache license.

    I don’t claim to know the details but I do know a little bit the rights under Apache2 and (unless my understanding is incorrect) MinIO’s claims are baffling.

  • Patterns of strings, function names, other symbols and the entire call graph usually show up in the compiled binary, unless they apply some sort of obfuscator to the process.

  • I'd imagine without further obfuscation a visual diff would be quite telling in that situation

Any recommendations on MinIO forks or open-source alternatives with more welcoming licenses?

They changed their license from Apache recently. https://en.wikipedia.org/wiki/MinIO#Re-licensing

  • How is the AGPL not welcoming?[1]

    [1] https://drewdevault.com/2020/07/27/Anti-AGPL-propaganda.html

    • well the problem is that some companies who use the AGPL do actually write some fud about it. i.e. if I use min.io agpl and the user never interacts with it, just my program, but I never even modified min.io and I just use the binary interface, I would never be subjected to the release my program's code, I would just need to give a link to the min.io source code that I used.

      however every company thinks in its own way:

      https://itextpdf.com/how-buy/agpl-license

      > You may not deploy it on a network without disclosing the full source code of your own applications under the AGPL license. You must distribute all source code, including your own product and web-based applications.

      and they also kinda fork the license with this text:

      > When using iText 7 Community under AGPL, you must prominently mention iText and include the iText copyright and AGPL license in output file metadata, and also retain the producer line in every PDF that is created or manipulated using iText.

      I mean this makes it really hard to trust the license at all, what would happen if I build a sever that can modify/create/convert pdfs and release the source code and than I have another program that calls this server internally with a http client, is that still some kind of linking or is it more like mongodb?

      I would happily build something with agpl when I could use itext/ghostscript and build something like minio which could than be used behind the scenes, but if that is not possible or if it is a grey zone than I'm not sure if agpl is a cool license at all. every company who uses the agpl writes something different about it, thats why an "official" clarification would be really really cool.

    • It's only not welcoming to those who wish to close the source.

      It's the paradox of tolerance: you must not tolerate those who find others intolerable.

      4 replies →

  • There are quite a few Storage projects out there with a S3-compatible API storage, that can be used in place of Minio such as Ceph, Openstack Swift, SeaweedFS, etc..

    Of course, they all differ in subtle ways, so you have to try for yourself.

Can you revoke an Apache-2.0 copyright license? The terms say irrevocable, though it stipulates respecting the terms and conditions:

"2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form."

  • Apache also doesn’t require you to share source code. Just because a company publishes their code under an OSS license doesn’t automatically make them the good guys and unless there’s some critical context it seems like pure FUD.

    Afaik Apache only requires you to maintain the copyright when distributing in source form (ie you don’t need to mention the license in binary form) but I’m not a lawyer and maybe misread. The license is certainly irrevocable and patent indemnifying provided you don’t violate it.

    You can’t both try to engender good will by releasing your code as OSS and then simultaneously going after someone who would seem to be complying with the terms with FUD. To see the FUD most clearly:

    > and we believe they may also be in violation of the GNU AGPL v3 versions of MinIO

    If that were the case you’d actually be in a court of law enforcing the license rather than trying to sway any kind of public opinion.

    This almost certainly stems from their switch to AGPLv3 to ensure that cloud providers can’t use it as part of their own offering. That’s fair but also provides context on motivation.

    • > The license is certainly irrevocable and patent indemnifying provided you don’t violate it.

      That's essentially what I find contentious, is whether "subject to the terms and conditions of this license" it is irrevocable or it is irrevocable irrespective of whether the terms of the license are being violated. With its phrasing I assumed it's the latter.

      1 reply →

our organization was using Minio as an external S3 replacement and had contacted their sales once. When a decision was made to not go for the paid plan, we were legally threatened saying that we cannot even make remote calls to a AGPL software.

  • That's kind of the point of the AGPL

    • For a system that is a providing an object storage service how else are you going to use it? We had started using it when it was Apache2 and then we got stuck. Might as well just make it into a paid product/service and not play the open source card and earn creds from community.

      1 reply →

I wonder why these folks keep doing this. Do they believe nobody will find out? Or that even if they find out, they won't have to pay much so they factor it in? It's really hard to imagine for me.

Strange. You spent THREE YEARS in discussion with Nutanix and still say

> we believe they may also be in violation of the GNU AGPL v3 versions of MinIO.

`may also be`? you are not even 100% sure whether they are using your AGPL v3 version? I have no clue, what the heck you were discussing for 3 years.

Moving from Apache-2.0 to AGPLv3 is a clear trap for those who use Minio as part of their commercial offering. With AGPLV3 one need to "disclose your source code" where as its not required with Apache-2.

If you started your "Open source" project with AGPL it is a different thing but to start the project under Apache-2 and few years later introducing AGPLv3 is kind of lame and unethical, IMHO. https://github.com/minio/minio/discussions/12156

I'm shocked anyone would buy something from Nutanix.

Sometimes it's pretty cut and dry, just people using open source without attribution and hoping that nobody will find out. But why?

MinIO is licensed under AGPL (the current versions, at least): https://github.com/minio/minio/blob/master/LICENSE

It effectively mandates that the modified version needs to be made available: https://en.wikipedia.org/wiki/GNU_Affero_General_Public_Lice...

  The GNU Affero General Public License is a modified version of the ordinary GNU GPL version 3. It has one added requirement: if you run a modified program on a server and let other users communicate with it there, your server must also allow them to download the source code corresponding to the modified version running there.

So the logical first question is: why pick software that is using AGPL? Did the engineers/managers just not care? Did they miss it? I know for a fact that there are many out there who couldn't care less about licenses and compliance. Maybe companies haven't been strong armed into caring about licensing as much as they have been in regards to GDPR, for example?

Secondly, why should the modified version remain a "secret"? Would competition suddenly spring up? Or maybe the project contains tight coupling to the rest of the platform, which could be considered a security risk?

Why isn't open sourcing a modified version something that would take a few hours anyways, since then none of this would be an issue?

(disclaimer: I discuss SSPL below because I find it interesting; apologies for the tangent)

Honestly, the state of software licensing sometimes puzzles me. For example, MongoDB switched over to SSPL altogether: https://www.mongodb.com/community/licensing

  If you make the functionality of the Program or a modified version available to third parties as a service, you must make the Service Source Code available via network download to everyone at no charge, under the terms of this License. ...

Seems like that applies to even patches: https://github.com/mongodb/mongo

  MongoDB is free and the source is available. Versions released prior to October 16, 2018 are published under the AGPL. All versions released after October 16, 2018, including patch fixes for prior versions, are published under the Server Side Public License (SSPL) v1. See individual files for details.

DigitalOcean, for example, proudly advertises managed MongoDB as a service: https://www.digitalocean.com/products/managed-databases-mong...

And yet, to the best of my understanding, the entirety of the DigitalOcean platform isn't open source (even though many projects are): https://github.com/orgs/digitalocean/repositories

Or even anything that might have something to do with MongoDB in particular: https://github.com/orgs/digitalocean/repositories?q=mongo&ty...

It just feels like one of those "rules for thee, not for me" situations, since it wouldn't be feasible for small companies to compete with them. Edit: someone mentioned them probably running the enterprise version which is probably the explanation for this!

That said, the thought experiment of building a company (including all systems) as 100% open source is really interesting, whether such a thing would be feasible if people stopped caring about "guarding" their IP and whatnot.

  • > DigitalOcean, for example, proudly advertises managed MongoDB as a service: https://www.digitalocean.com/products/managed-databases-mong...

    > And yet, to the best of my understanding, the entirety of the DigitalOcean platform isn't open source (even though many projects are): https://github.com/orgs/digitalocean/repositories

    I don't think it's required to open source everything, only the bits that provide the MongoDB service. I don't know if they've done that.

    Also, the SSPL seems to be a little controversial [1] as it appears to want to relicence all software it's running near under itself.

    [1] https://en.wikipedia.org/wiki/Server_Side_Public_License

    • > I don't think it's required to open source everything, only the bits that provide the MongoDB service. I don't know if they've done that.

      Yes, that's my exact point - these things are sometimes full of finer points. I don't doubt that DigitalOcean have talked with MongoDB and have probably figured out some sort of a deal, or another way to offer it as a service (someone mentioned them using the enterprise version, where the terms are probably different).

      Though offering MongoDB as a service for a small no-name company all of the sudden seems impossible, unless they actually want to open soruce lots of their own code.

      > Also, the SSPL seems to be a little controversial [1] as it appears to want to relicence all software it's running near under itself.

      Of course, there was backlash to it even existing, much like larger companies didn't really like AGPL being a thing either.

      Then again, I guess one could argue that MongoDB definitely can create such a license, as a reaction against cloud platforms utilizing their solution: https://www.mongodb.com/blog/post/mongodb-now-released-under...

        This should be a time of incredible opportunity for open source. The revenue generated by a service can be a great source of funding for open source projects, far greater than what has historically been available. The reality, however, is that once an open source project becomes interesting, it is too easy for large cloud vendors to capture most of the value while contributing little or nothing back to the community. As a result, smaller companies are understandably unwilling to wager their existence against the strategic interests of the large cloud vendors, and most new software is being written as closed source.
      

      I don't really have a horse in that race, though in theory such a license would be good for the open source community, whilst its effects on larger cloud vendors are also pretty much clear. Of course, there is a lot of controversy around it and it's not considered "open source" at all by many.

      3 replies →

  • No idea about the relationship between MongoDB and DigitalOcean, but note that you've linked to the "community" edition; their site also shows an "enterprise" edition, which is more likely for a large entity like DigitalOcean to be using.

    • That's a great point, thanks for bringing it up! I've updated my original post with this detail as a possible explanation, since it's the one that makes the most sense.

      I guess that MinIO or any other company could also do something similar, have dual licenses, where interested parties can pay for commercial usage and whatnot.

  • nutanix uses the apache licensed version probably, which changed in 2021. so it's even more buzzling since they would've just needed a NOTICES file.