[Guide] What equipment to buy for Sonic 10 Gigabits Fiber

[tarzxf]

Found an open box R86s-G2 on ebay, ordered some 10GBASE-T SFP+ adapters from Amazon, and have a refurb M2 Studio on the way to complete the first stage of the home 10G rollout. Everything else will have to live with 2.5 and 1 gigabit.

Planning to stick with pfSense, the current J4125/2.5G config has been solid enough.

[dwheet]

Found an open box R86s-G2 on ebay, ordered some 10GBASE-T SFP+ adapters from Amazon, and have a refurb M2 Studio on the way to complete the first stage of the home 10G rollout. Everything else will have to live with 2.5 and 1 gigabit.

Planning to stick with pfSense, the current J4125/2.5G config has been solid enough.

I hope the j4125 is powerful enough to do pfsense for what you have planned. I'm leery on my n305 being powerful enough for what I have planned. although I think it also comes down to how many connections your gonna be pushing through your 10gig interface I think 10 or 20 connections doing 10gig probably doesn't require the beefy boxes although I'm not sure. My current firewall is a qotom Intel(R) Core(TM) i5-7200U CPU @ 2.50GHz on a 1gig down 30mb up cable connection. I never see the firewall with everything turned on going above 2% cpu. I do download stuff at full gig speeds and again dont see the dial moving at all so I think cpu wise comes down to your use case. in an office environment with couple hundred users all pushing hard on it, thats where you might need a beefy box to keep up. just my thoughts.

[tarzxf]

Found an open box R86s-G2 on ebay, ordered some 10GBASE-T SFP+ adapters from Amazon, and have a refurb M2 Studio on the way to complete the first stage of the home 10G rollout. Everything else will have to live with 2.5 and 1 gigabit.

Planning to stick with pfSense, the current J4125/2.5G config has been solid enough.

I hope the j4125 is powerful enough to do pfsense for what you have planned. I'm leery on my n305 being powerful enough for what I have planned. although I think it also comes down to how many connections your gonna be pushing through your 10gig interface I think 10 or 20 connections doing 10gig probably doesn't require the beefy boxes although I'm not sure. My current firewall is a qotom Intel(R) Core(TM) i5-7200U CPU @ 2.50GHz on a 1gig down 30mb up cable connection. I never see the firewall with everything turned on going above 2% cpu. I do download stuff at full gig speeds and again dont see the dial moving at all so I think cpu wise comes down to your use case. in an office environment with couple hundred users all pushing hard on it, thats where you might need a beefy box to keep up. just my thoughts.

The J4125 has handled 2.5Gig traffic both ways with ease as well. The R86S is an N5015 system, and I just got a Mac Studio to see how far it goes towards the 10Gig max.

I'd love to have a cisco nexus switch fall off the office, but the power, heat and NOISE from that wouldn't work well for home use.

[dwheet]

Found an open box R86s-G2 on ebay, ordered some 10GBASE-T SFP+ adapters from Amazon, and have a refurb M2 Studio on the way to complete the first stage of the home 10G rollout. Everything else will have to live with 2.5 and 1 gigabit.

Planning to stick with pfSense, the current J4125/2.5G config has been solid enough.

I hope the j4125 is powerful enough to do pfsense for what you have planned. I'm leery on my n305 being powerful enough for what I have planned. although I think it also comes down to how many connections your gonna be pushing through your 10gig interface I think 10 or 20 connections doing 10gig probably doesn't require the beefy boxes although I'm not sure. My current firewall is a qotom Intel(R) Core(TM) i5-7200U CPU @ 2.50GHz on a 1gig down 30mb up cable connection. I never see the firewall with everything turned on going above 2% cpu. I do download stuff at full gig speeds and again dont see the dial moving at all so I think cpu wise comes down to your use case. in an office environment with couple hundred users all pushing hard on it, thats where you might need a beefy box to keep up. just my thoughts.

The J4125 has handled 2.5Gig traffic both ways with ease as well. The R86S is an N5015 system, and I just got a Mac Studio to see how far it goes towards the 10Gig max.

I'd love to have a cisco nexus switch fall off the office, but the power, heat and NOISE from that wouldn't work well for home use.

Thats why I picked up a CRS317-1G-16S+RM. I've seen a number of content creators talk about it and how quiet it is. It'll be sitting in the same room as my office and I can't have what sounds like dual turbines (cisco nexus etc) going off while I'm on a zoom call lol

[tjm]

I'm using a pair of Ubiquiti Switch Flex XG 10GbE four port switches for connecting my 10GbE devices. For routing I'm using a Asus ROG AXE-16000 router. It has a 10GbE WAN port and 10GbE LAN port both 10G-base-T and connected to the Switch Flex XG. The 622v ONT also connects the Switch Flex XG. It gets connected to the WAN port via the use of a dedicated VLAN on the Unifi config. I have a SNAP QNA-T310G1T that connects my MacBook Pro to the 10GB fabric. With this setup I am able to get speedtest of 8.2Gbit down and 7Gbit up. I was able to download 100GB of data from GCS in just a few minutes, which was awesome. The only issue I have with this setup is that the AXE-16000 is a bit flakey, it's speed degrades over time, a reboot recovers things, but I have to reboot every few days to keep things stable. After a week it can slow down to below 1Mbit/s which become pretty unusable. The last firmware update from ASUS was in April, prior to then firmware was being updated every one to two months. I would like to find a router than can support full 10GbE and is stable. Haven't seen much that is in the $500 price range.

[artakamoose]

I'm using a pair of Ubiquiti Switch Flex XG 10GbE four port switches for connecting my 10GbE devices. For routing I'm using a Asus ROG AXE-16000 router. It has a 10GbE WAN port and 10GbE LAN port both 10G-base-T and connected to the Switch Flex XG. The 622v ONT also connects the Switch Flex XG. It gets connected to the WAN port via the use of a dedicated VLAN on the Unifi config. I have a SNAP QNA-T310G1T that connects my MacBook Pro to the 10GB fabric. With this setup I am able to get speedtest of 8.2Gbit down and 7Gbit up. I was able to download 100GB of data from GCS in just a few minutes, which was awesome. The only issue I have with this setup is that the AXE-16000 is a bit flakey, it's speed degrades over time, a reboot recovers things, but I have to reboot every few days to keep things stable. After a week it can slow down to below 1Mbit/s which become pretty unusable. The last firmware update from ASUS was in April, prior to then firmware was being updated every one to two months. I would like to find a router than can support full 10GbE and is stable. Haven't seen much that is in the $500 price range.

Why do you have the Switch Flex XG connected in between the router and ONT? That shouldn't be necessary and isn't really recommended for a number of reasons. I'd pull that out of there and connect the ONT directly to the Asus.

If you still have problems after that and want to ditch the Asus (and since you're already running Unifi hardware), just go with the UDM Pro or UDM SE depending on your needs. You'll need a couple of SFP+ to RJ45 modules (Ubiquiti UACC-CM-RJ45-MG or an equivalent FS.com module), one each for WAN and LAN. If you don't run IPS and IDS, either model should push line rate which is about what you're seeing.

[joeyyung911]

Seems like only a handful of folks are building their own 10Gb router. Looking to build mine with a spare computer, what do y'all think? I'm leaning towards OPNSense.

AMD Ryzen 2400G (4-cores/8-threads)
Asrock Fatal1ty B450 Gaming-ITX (1x Gb port, 1x PCIe 3.0 x16)
32GB RAM
Mellanox 10Gb/40Gb Dual Port Network Adapter (SFP+ and QSFP ports), MHZH29B-XTR
(Yes, I'll need transceivers)
If this card doesn't work with OPNSense, I'm looking at Connect-X 3 and newer SFP+ cards. (Eyeing a dual port SFP28 card)

[joeyyung911]

I guess no one's in the DIY router game. Played with OPNSense, but leaning towards OpenWRT...so much simpler. Couldn't get any of my QSPF ports on my NICs to work in either software and got a Mellanox MCX4121A-ACAT ConnectX-4 Dual Port SFP28 25GbE NIC. Just had to install kmod-mlx5-core before it'll detect it. Did an iperf test for 5 minutes (utilizing all cores) and the router was only at 82% idle, so plenty of power for it.

[daniel15]

I guess no one's in the DIY router game. Played with OPNSense, but leaning towards OpenWRT...so much simpler. Couldn't get any of my QSPF ports on my NICs to work in either software and got a Mellanox MCX4121A-ACAT ConnectX-4 Dual Port SFP28 25GbE NIC. Just had to install kmod-mlx5-core before it'll detect it. Did an iperf test for 5 minutes (utilizing all cores) and the router was only at 82% idle, so plenty of power for it.

I'm trying out a software router and it's working very well!

I'm using a spare SFF PC I had, with an Intel Core i5-9500 and an Intel X540-T2 dual-port 10Gbps network card. These network cards are very old (close to 10 years old now I think) but they still work well and you can find them on eBay for ~$30-40 each. They're RJ45 rather than SFP+, which uses more power, but both the ONT and the rest of my network uses Ethernet anyways. Note that they predate NBase-T so you can only use them at 10Gbps or 1Gbps speed.

I tried Opnsense but couldn't get speeds higher than around 3.1Gbps even after a lot of tweaking (including disabling mitigations for Spectre and Meltdown CPU bugs, increasing a bunch of buffers, and various other tweaks). Opnsense was using 100% of one core, so I think something (NAT, maybe?) is single-threaded. Maybe a CPU with a higher turbo speed could have handled it properly - I'm not sure.


I switched to OpenWrt and I could immediately get full speed (~8.4Gbps) with no tweaking. The out-of-the-box configuration worked fine. It was only using ~14% of all six cores. I turned on software flow offloading and now it's using even less CPU.


most of it for processing interrupts from the network card:




I'm running OpenWrt in a VM on Proxmox, passing through the network card. I tried Opnsense both as a VM and on bare metal.

I've read that there's known issues with BSD-based routers and very fast connections. For now, Linux-based systems like OpenWrt and VyOS are better. VyOS is supposed to be very good, but it's currently command-line-only. I'm not a network administrator and like having a web UI.

The Core i3-N305 has roughly the same power - It's about 20% lower in single-core performance, but higher multi-core performance due to the extra two cores, so I believe that a mini PC with a Core i3-N305 and two 10Gbps ports (like the newer Gowin R86S ones) should be able to easily achieve 10Gbps throughput.

[bobfff]

Wonder what switches are being used for a 10Gb lan? Anyone have experience with QNAP QNAP QSW-2104-2T? Other reasonable options? Unmanaged ok.

Thanks!

Bob