Comment by adrian_b
3 hours ago
I do not believe that your problems are typical for FreeBSD servers.
I have been using FreeBSD servers for around 30 years. Most of them had Intel NICs and I have used at least 5 or 6 different kinds of Supermicro motherboards, both with Xeon and with Epyc.
Most servers have worked 24/7, without being rebooted for years and without having any minute of downtime except when I did some hardware upgrade or kernel upgrade.
I do not doubt that you had the problems described, but there must be some very unusual circumstances that have caused this. I would not be surprised if there was some problem with the version of Supermicro BIOS of your motherboards, and not with FreeBSD, because I have seen many bugs in Supermicro firmware. Or perhaps you had some buggy version of Intel NICs.
There is one advantage of Linux over FreeBSD, which is not widely known. Linux has a huge database of known bugs in various peripheral devices, including Ethernet NICs, and when one of those is recognized it applies workarounds for the bugs.
Like any other operating system, FreeBSD also implements workarounds for peripheral interfaces with known bugs, but because it has a much smaller user base also its database of bugs includes much fewer bugs, typically only those that had been reported by FreeBSD users. Because of this, I have seen cases when some hardware devices did not work well in FreeBSD, while they worked well in Linux, and the reason was always because Linux knew that they must not be used in the standard way, but it applied the corresponding workaround for their bugs.
Only Windows is shielded from the problems caused by bugs, because the hardware vendors write themselves the Windows device drivers and include in them any required workarounds for their bugs.
This looks to be a theme in FreeBSD, specifically tweaking mbuf numbers because of exhaustion [1], [2], [3], [4] (and not to mention any posted on the freebsd bug tracker). The fact that you have options to tweak mbufs and limits in the first place just doesn't seem like a well thought out design -- in Linux you never have to specify skbuff (Linux's mbuf equivalent) numbers for any reason. I don't even know if my issue was hardware specific or just a FreeBSD-ism to do with mbufs, but it is a class of issue that just doesn't exist in Linux.
[1] https://serverfault.com/questions/335461/pfsense-mbuf-full-w...
[2] https://unix.stackexchange.com/questions/394876/how-to-fix-m...
[3] https://redmine.pfsense.org/issues/5553
[4] https://docs.netgate.com/pfsense/en/latest/hardware/tune.htm...
> There is one advantage of Linux over FreeBSD, which is not widely known. Linux has a huge database of known bugs in various peripheral devices, including Ethernet NICs, and when one of those is recognized it applies workarounds for the bugs.
Almost universal hardware support and workarounds for quirks is precisely the reason why everyone uses Linux even if they might want to use FreeBSD or something else. In other words, this advantage is known and is the reason why Linux is dominant in the server space today.
It is widely known that Linux has "better" hardware support for many peripheral devices, but few know what "better" means, i.e. that a lot of devices that you can buy have various bugs and in most cases their vendors do not document the bugs in any way, because they provide Windows drivers that contain workarounds for the bugs, and they do not care about other operating systems.
The users of other operating systems must discover the bugs and how to handle them by reverse engineering, and here Linux has the advantage of a much greater user base than any alternative, so in most cases the bugs will be identified by some Linux user, and then either the user or the maintainers of the corresponding Linux subsystem will implement a workaround for the bug.