Comment by MrDarcy

2 days ago

I've moved on from Ubiquiti access points as well. Their U6 simply does not handle VLANs properly, they never acknowledged the issue let alone fixed it. See https://community.ui.com/questions/U6-IW-how-to-trunk-all-5-...

Their security issues in the past. Their failure to make the EdgeRouter handle DHCP and DNS properly. Etc...

I've since moved to cheap switches that support all port vlan trunks and LACP bonding, then just plug Proxmox into them and run OpenWRT in a VM for routing all the vlans. The Proxmox+OpenWRT combo even supports hot-plug virtual interfaces as more VLANs are lit up, they just pop up nicely in the web UI.

For the APs, TP-Link is less expensive and better performance. WiFi 7 and 10gbit for less money. No need to run a management OS in a VM either.

14 comments

MrDarcy

Reply
enjo  20 hours ago

TP-Link comes with a healthy dose of worry about the Chinese governments surveillance practices tho.

cromka  2 days ago

> TP-Link is less expensive and better performance. WiFi 7 and 10gbit for less money.

Thanks, they really seem like good alternative.

  • zmix  2 days ago

    Not sure, how active development is right now, but EdgeOS got forked from an open source devian based distro after that went commercial.

    But EdgeOS was not the only fork, another one was VyOS (vyos.io). Pretty sure, that EdgeOS has done larger steps forward, especially, since it was bound to the hardware's developer.

alt227  1 day ago

The thread you reference ends with the post saying "it is fixed in the 7.4.140 controller release", so im not sure how you can say it wasnt acknowledged or fixed.

  • MrDarcy  1 day ago

    Are you a ubiquiti employee who can see internal posts perhaps? There’s no such post on the public discussion. The only two instances of “fixed” on the page are people expressing hope it will someday be fixed.

    • alt227  4 hours ago

      As the other poster said, you actually need to read the thread and then a Ubiquiti employee links to a post where it says it was fixed. Considering you were so affected and frustrated by this issue, Im surprised you didnt actually read the thread you posted to realise this issue was fixed 2 years ago.

zamalek  1 day ago

> Their security issues in the past.

That's why I moved off as well. Maybe some day SDN (at least so far as the ubiquity experience goes) will become an OpenWRT priority.

orion138  1 day ago

I’m curious to know more about your setup! Which switches do you prefer? What hardware are you using for proxmox? And what does your network look like?

Cheers!

  • MrDarcy  1 day ago

    For the switches I'm considering replacing them all with 2.5gbit but don't see the need to yet. Currently I have a TL-SG1016DE as the core switch. The main proxmox servers are 3 used Dell 1U servers I bought from Ebay. Each has 256GB ECC ram, 2x 8 core CPUs, 4x Gbit intel nics. I flashed the PERC card to be a plain SCSI controller so ZFS in Proxmox has direct access to the disks. If I were to buy them today I'd look for R720's or newer. I got mine for about $800 USD each. They're overkill, but provide a lot of capacity. They're also unnecessary, you can ignore them and only consider the rest of this comment. They're the most expensive, hottest, and loudest devices on the network.

    I have a separate tower that's a old 9th gen intel that provides the large ~50TB ZFS NFS server. It used to be an intel Atom, but that finally died after 10 years so I moved the drives to a gaming PC I had lying around. Over the years, nicest thing about ZFS and Proxmox is the drives are fully independent of the hardware and the software OS they're attached to. Now, I just pass the devices through Proxmox to a Debian VM and they come up just like they did before.

    Regarding the rest of the network, let's move from the edge in toward the 3x 1U servers and NFS storage box. I have 1gig symmetric fiber from Ziply. The ONT has cat5 running to 1 of the 4 gig ports in an Intel Atom C2758. The other 3 ports are bridged together in Proxmox to act as a switch. It kind of looks like an EdgeRouter-4 if you squint at the ports. This C2758 only runs a single VM, OpenWRT. The nice thing is I can take snapshots before upgrades, and upgrade or replace the hardware easily.

    The OpenWRT VM is the most critical thing in the whole network. I try to manage it simply, I have some shell scripts that copy the /etc/config files into place and restart services for a simple IaC setup.

    The main services OpenWRT provides are:

    1. WAN DHCP client, my ISP doesn't offer static IPs. 2. One minute cron job that makes sure the A record for home.example.com is correct. *.home.example.com is a CNAME to home.example.com, this simplifies configuration and TLS cert management. 3. HAProxy runs on OpenWRT listening on 0.0.0.0:80 and 0.0.0.0:443 Extremely valuable for SNI routing of TLS connections. I moved the LuCI web UI to alternate ports, which is simple to do via config. 4. dnsmasq provides dhcp and dns for the main and guest VLANs. 5. OpenWRT is configured as a WireGuard server. Each wireguard client device is allocated an dedicated IP in a separate 192.168.x/24 subnet. This has been great for source based IP access control which I'll cover below. Wireguard clients connect to home.example.com.

    That's it for OpenWRT. The key lesson I learned is it's been incredibly valuable to run haproxy on OpenWRT. All L4 connections terminate to it, but crucially it does not handle TLS certificates. It only forwards TCP connections based on the SNI in the client hello. HAProxy is also configured to use the PROXY protocol to preserve source IP addresses, which has been great for access control.

    Most TLS connections are forwarded to a single node Talos VM running on another Proxmox host. This VM runs Cilium, Istio, and the Gateway API. The istio envoy gateway is configured to accepts PROXY protocol connections, which means AuthorizationPolicy resources work as expected. By default, only connections coming from the local subnets, or the wireguard subnet are allowed. OpenWRT does hairpin NAT, so this works just fine, all sources connect to the WAN IP regardless if they're internal or external.

    I don't do much with Kube yet, most of the traffic is forwarded on to another VM running Portainer. Most of my backend services are in Portainer. The Kube VM does handle Certificate and AuthorizationPolicy resources though, using cert-manager and Istio. This has been nice, I don't need to configure each service for TLS or access control in bespoke way, it's all in one place.

    The only other thing to note is the Dell 1U servers have 3 of their 4 gig nics aggregated into LACP bonds. Similar to the Atom router, they're configured as a bridge in Proxmox and I use them for the Ceph data plane. 9 of the 16 ports in that TL-SG1016DE are just for Ceph and I'm able to get close to 600 MiB/sec reads (yes megabytes) which is pretty neat given 1gbit interfaces.

    That's about it. Overall I'm trying to eliminate VLAN's, but it still makes sense to have them for Ceph and for a Guest wifi network.

    Edit: Lastly I've maintained a home lab for 25 years and this is the best iteration yet. All of the trade-offs feel "right" to me.

close04  1 day ago

Their software updates are also very flakey. The past few releases for the Console and occasionally the Network were pulled right after being published for having blocking bugs. Again and again they publish the update and then do QA on their users. If you have an IT department you probably have some sort of process in place to deal with this and deploy when you're satisfied. A home user will probably have auto-updates enabled and bite the bullet again and again.

A while ago one update automatically enabled PMF (set to required, I believe) on all Wi-Fi networks. That didn't go great for me when half of my IoT devices stopped connecting and I wasn't available to fix.