Comment by Filligree
5 hours ago
How many PCIe lanes are you allocating?
The card is obviously 16-lane, but it also has two ports; 40Gb total. In a server that’s fine, but if you want 10G in a desktop you’ll have a problem.
I’m probably not telling you anything new. NICs using newer PCI generations are rare as hen’s teeth. It should be possible to do this with four lanes, but isn’t…
Unless you find a 25G dual-port card, in which case the single lane my secondary slots hand out does at least suffice for 10G one way.
PCIe is also a full duplex connection so 2x10G is still just 20G instead of 40G. For PCIe 2.0 an x8 connection should get you full bandwidth on both ports simultaneously while x4 will fall just short for simultaneously usage (but still higher than 1 port). Unless you're really hankering for that full 20G, in which case a 25G NIC is definitely the better pick, that means you can just slot it in an x4 slot off the chipset on even a standard desktop PC.
Funnily enough, if you want a dirt cheap PCIe 3.0 based card the MCX353A-QCBT and MCX354A-QCBT give 1/2 ports of 40G QSFP+. They support QSFP+ to SFP+ adapters, so you can plug a 10G SFP+ into the QSFP+ port, but they don't support 4x10G breakout unfortunately. I ended up using the 2 port variant in both of my NASes - one port is 40G between the 2 for dirt cheap fast backups and the other is adapted to 10G to connect to the rest of the home network.
I've started buying Intel E810s for most purposes, even for 10G links. (SFP28 ports are generally backward-compatible with SFP+ DACs and transceivers.) The ones you can get on eBay for cheap typically run Dell firmware but it's serviceable. An E810-XXVDA2 is Gen4 x8; as long as the host slot can physically accept the card connector you only need Gen4 x1 electrical for a single 10G link or Gen4 x2 for dual 10G or single 25G.
I'm only planning on using one of the SFP+ ports on each of the cards, the dual port cards were just more common and cheaper on eBay.
The specs say they require PCIe v2.1 x8 lane.
My Proxmox server is quite old and has a Gigabyte GA-X79-UP4 mobo and has loads of spare PCI slots. One slot is taken up by a generic graphics card as the Mobo has no on-board graphics. (I think I went for this mobo because of the number of SATA ports, but it was over 10 years ago so not entirely sure.)
My general Linux server is newer and has an ASUS Prime H610M-A D4 mobo. Only two PCI slots (not used at the moment) and so the Intel X540-DA2 will use up the PCIe 4.0 x16 slot leaving just a PCIe 3.0 x1 slot. But that's fine as this machine is just a CPU (i7-13700), 64GB RAM and a 2TB NVMe. Sticking a good graphics card in it for GPU related fun had been on my list for years but I never got around to it, now the prices are just insane so I'll ignore that for now or something second hand falls into my lap.