Firewalla & Tailscale: Performance Bottleneck
Hi,
I'm genuinely interested in learning a bit more about firewalla, and this issue seems to be a good place to start.
Issue: There's a big (57%) overhead on the local lan between 2 machines when they go through firewalla using tailscale. This overhead doesn't exist for non-tailscale traffic, or when the 2 machines are connected directly via an unmanaged switch without firewalla in between. It's as though firewalla is picking on tailscale traffic between the 2 machines, but not other traffic.
Here's a bit more detail on the setup and testing:
Machine 1 (M1): 1Gbps network, AMD Ryzen 5 2400G, Unraid 6.12.6
Machine 2 (M2): 2.5Gbps network, AMD Ryzen 7 5600G, Pop!_OS 22.04
Firewalla: Gold SE, V1.977
Switch: Netgear 5x 1Gbps unmanaged.
Tailscale: 1.56.1
Local subnet: 10.*.*.*/24
Tailscale Subnet: 100.*.*.*/32
Here are the two configs:
Config 1 (slow):
- M1 connected to switch. Switch connected to 1Gbps port 2 on firewalla. 1Gbps link speed.
- M2 connected to 2.5Gbps port 1 on firewalla. 2.5Gbps link speed.
Config 2 (fast)
- M1 and M2 connected to switch. Switch connected to 1GBps port 2 on firewalla. 1Gbps link speed everywhere.
And here are the speed test results between the machines using iperf3:
1. Config 1 + No Tailscale: 943 Mbps
2. Config 1 + Tailscale: 346 Mbps (63% overhead)
3. Config 2 + No Tailscale: 941 Mbps
4. Config 2 + Tailscale 817 Mbps (13% overhead)
I would expect that for a local lan connection physically going through firewalla that there would be little difference between any of the above scenarios, but clearly the firewalla is doing something with tailscale traffic on the local lan.
Why does this matter? 346 Mbps is still fast.
- It's unexpected and I'm curious and wanna learn why. :-)
- No, I don't need to use Tailscale VPN between these two local machines. But it was a testbed to isolate things. The actual issue is that both my friend and I have 1.7Gbps internet connections and want to connect our Unraids for mutual backup and large file transfers. In this case, it has to go through firewalla and it'll be limited to 344 mbps instead of closer to what should be 2-3 times faster.
I'm wondering:
- Is it to do with routing between 10.x and 100.x, even though tailscale is actually directly connected and not going through a relay. Shouldn't be... firewalla should just be seeing 10.x traffic, even if it's routed over tailscale.
- Is it to do with some kind of packet inspection, even for the local lan connections? Even then, the Gold SE is rated higher than this and why doesn't it do it for non-tailscale lan traffic?
- It can't really be because of the physical connection difference since it gets near 1Gbps without tailscale.
- ....?
Thanks in advance,
Trev.
-
You will need to flash https://help.firewalla.com/hc/en-us/articles/19523706861843-Firewalla-Gold-SE-How-to-Flash-Installer-Image
This image has the pppoe optimization
-
Hydralein, I would bet that the 2 hosts on your LAN running tailscale aren't able to form a peer to peer connection resulting in the traffic being sent out to the internet and back when you are testing. You should be able to see the dynamic tunnel being formed and maintained using wireshark or perhaps just in the firewalla logs, it may be not allowed. Apologies that I don't have any Firewalla specific knowledge as I am just here browsing for a firewall.
-
Hello,
I have problems with wireguard not with tailscale. The guys starting this topic using tailscale. But Tailscale is built on top of WireGuard so same problem here.
I also tried to modify my MTU but nothing helps. For informations when I change my router back to my "old" one I get fullspeed with wireguard. It have to be something with the firewalla.
Please sign in to leave a comment.
Comments
41 comments