Firewalla Gold Frustrating Intermittent Connection Loss
Howdy,
One of my two Firewalla Golds has been acting strangely since I purchased it two months ago. LAN clients occasionally lose connection to the WAN/internet. Typically, Chrome displays the error "DNS_PROBE_FINISHED_NO_INTERNET". I've had a support case open for weeks, however Firewalla support has been useless.
Topography and Symptoms:
Frontier Fiber ONT - Firewalla Gold in Router Mode - 24 port switch - wired computers and APs
Pings and traces to the WAN from wired PCs behind Gold occasionally fail, however most of the issues seem to impact wireless clients. Pings and communications to/from wireless clients fail about 40% of the time. I've replaced all APs, the 24-port switch, tested all cables, and had the ISP replace their ONT and test the incoming WAN cable. I'm not using any sort of private DNS or VPN on LAN devices. This issue doesn't impact my other Firewalla Gold (different network entirely). Also, when I replace the problematic Firewalla Gold with the ISP's router, the problem completely disappears.
Firewalla insists that there is nothing wrong with this Firewalla Gold, however troubleshooting steps clearly point to the Firewalla Gold being defective here. Has anyone experienced a problem like this with their Firewalla? Do you have any additional troubleshooting recommendations? I suspect some kind of hardware fault with this Firewalla Gold.
Thanks in advance for your help!
-
Joining in here. I've been using a Firewalla Gold for the past year or so, with no issues, until about a week ago. Now, have been experiencing loss of WAN one to two times daily. Wireless/ LAN seem to remain intact.
I called Spectrum first & they ran a diagnostic on the modem, which showed no anomalies (of course). I was able to access the Firewalla during one of the more recent outages, via App & it appeared to have a valid WAN IP, but was showing DNS Failures. Upon physical inspection, the Spectrum cable modem shows ONLINE (LED status anyway) & a reboot of the FW brought everything back. As I have a power controller monitoring connection status (pinging internet or FW IP), it typically handles the reboots "behind the scenes", so I suspect this is what has been restoring service during the other drops.
Contacted support, got a ticket, and was trying to work through it, but last response was saying my ISP/WAN connection is unstable. Not sure why rebooting the FW seems to rectify if that were the case. I am going to install my old router to see whether the issue persists. Will keep you posted.
My system consists of:
Spectrum (Arris) Cable Modem -> Firewalla Gold (Router Mode) -> 16 Port POE (Core) Switch -> Ruckus R550 WAPs (Running Unleashed).
-
@scott, have you looked at the network events? and the network diagnostics? See https://help.firewalla.com/hc/en-us/articles/360053534593-How-do-I-debug-network-connectivity-issues#h_01FD7C39EEZWDGQ7D64Z71E9ZB
- If you see a DNS failure, what is the DNS server showing up? is it your own or from the ISP?
- If you look at the network events, it should detect events, if you see any events, tap into them and see what it is displaying
The reason when you unplug things works is because, a simple action of turning off the network will trigger many things to reset. (yes, even on your ISP side).
-
Sorry, I wasn't clear because I couldn't recall how to get to the screen, but Network Diagnostics IS where I saw the failures. I got there when I accessed the Firewalla locally (by clicking "Diagnostics" in the "Internet Connection Lost" banner). While I should have done a screen grab, it pretty much mirrored the second image from the internet connectivity debug page of the link you sent.
More precisely, Obtain IP Address/ WAN = Successful, Ping Test = Failed, DNS Lookup = Failed.
Again, this was only one of the recent instances. While I'm not looking forward to future instances, I'm prepared to dig a little deeper, assuming I'm home if/when it happens. Any other tips on where to look/ intel to gather would be appreciated. Otherwise, I can swap it out & see if the disconnects continue.
Thanks for the quick reply!
Scott
-
Hi all. Found this thread trying to figure out how to fix my Firewalla. Same issue. After about 3 fantastic years running my Firewalla Gold SE continuously, the last 2 weeks it has been killing my internet access straight across the board. It has been dropping us 4-5 times an hour for 1-3mins duration. I just switched from coaxial to fiber 3-4 weeks prior, so, obviously, I figured it was the ISP. To be clear, I had to remove Firewalla from my system and I am only able to type this online because my old router is in its revered place.
-
Try this article https://help.firewalla.com/hc/en-us/articles/360053534593-How-do-I-debug-network-connectivity-issues it has some tips how how to look at connectivity problems. If you just changed ISP, I'd focus on the WAN side.
The most common problem we see are
1. smart queue rules that's made for the old ISP
2. WAN health checks that uses old ISP end points
3. DNS pointing to the old ISP.
-
This is still a problem. It is very annoying. I used Firewalla Gold without any issues for over an year and then this started happening. I disconnect Firewalla during the working days (and business hours) so that I and members of my family can work. I can only diagnose this during off business hours.
My Setup
[ISP Modem] <-> Firewalla Gold (Router Mode/DHCP on) <--> Router (Serving as AP only)
My Diagnosis
It seems the Firewalla Gold will disconnect if it is not able to reach your cloud services.
The errors show ESOCKETTIMEDOUT when trying to connect to Firewalla's cloud services. This indicates that connection attempts to firewalla.encipher.io are timing out rather than being rejected or returning errors.
ERROR DestIPFoundHook: Failed to load intel, err: Error: HTTP failed after 1 attempt(s) POST https://firewalla.encipher.io/bone/api/v3/intel/host/*/check
This could be due to the fact that due to some strange reason once Firewalla fails the diagnostics (Ping and DNS lookup), it just disconnects. This is strange because this problem does not occur if I remove Firewalla from the setup and connect my router to the ISP modem. (YES! I have checked the cables). If the argument is that DNS lookup should succeed to resolve firewalla.encipher.io - then does the Firewalla device not cache that address for longer than 30 minutes.
Apologies for rambling on, but Firewalla has become a timesink for me.
-
What is the DNS settings on your system? See this article https://help.firewalla.com/hc/en-us/articles/24559271667603-Firewalla-s-cloud-domain-is-blocked-by-OpenDNS
Can you run https://diag.firewalla.com and scroll down to "
Services Used by Firewalla"See what it saysIn general, even firewalla domains fail to load, your firewalla should still be up. So unless the firewalla domains are explicitly blocked, likely your ISP may have issues with DNS or your DNS service is broken -
Firewalla Gold worked great for a few months. Then, for no apparent reason, I'd get intermittent dropouts throughout the day which would disconnect Zoom calls, kill a stream that doesn't cache sufficiently (NPR), etc. Opened a ticket and never heard back from support. I've been suffering this issue for so long. Definitely not cable connections, the device is not overheating, replaced the WiFi AP to no avail. Definitely makes purchasing another Firewalla device undesirable.
-
cloatus
the only ticket I see you is May 17th 2023, we did reply to you and suggested you look at
"Have you checked the network events ? was there anything related to this?" You did not reply back
You can refer to this article, or create a new ticket. https://help.firewalla.com/hc/en-us/articles/360053534593-How-do-I-debug-network-connectivity-issues
-
I'm having similar problems currently; they began few weeks back when I updated my ISP to a faster connection, and it came with a new router. Before the update, the box had been running pretty smoothly for three years. The only constant I have discovered is that the box becomes unreachable when it's doing an internet quality test, and I have to reboot it. I have tried two configurations with new ISP router. IP passthrough mode and DMZ to Firewalla WAN with double NAT. Same problem with both modes. I have failover via SD-WiFi to a other ISP, but it's no use because the box just gets stuck.
-
@CosmicSisu have you looked at the diagnostics article? For example, when your network is having problems, we will need to determine if it is LAN or WAN first, and if it LAN, need to know if it is ethernet or WiFi. If both are dead, it is a completely different issue https://help.firewalla.com/hc/en-us/articles/360053534593-How-do-I-debug-network-connectivity-issues
-
Firewalla Like I said, the box gets stuck and I cannot use the Firewalla box in that situation. Before I reboot (do a power on/off), the internet connection is fine in that situation because it works through the ISP router's own Wi-Fi. I have tried different cables and did a full factory reset, but had no help. Currently, I have connection quality test off, and there's no crashing whatsoever.
-
Hi all, I would like to share the source of my connectivity problems last year in case it may be helpful to others. It turns out that many usb-c laptop docking stations can cause "pause frame" spamming over wired ethernet that in turn leads to major network connectivity issues with some switches/routers. The issue occurs when the laptop is powered off (or goes to sleep) and the docking station remains powered and connected via wired ethernet.
For me, the solution was to stop using wired ethernet on my Dell laptop docking station. I have had zero firewalla network connectivity issues since. The dell forum posting below describes the problem in a little more detail and some potential solutions other than unplugging the wired ethernet that worked for others, such as a managed switch that disables pause frames or specific configurations of the client's network adapter. I would be curious if anyone else has been running into network connectivity issues with laptop docking stations.
https://www.dell.com/community/en/conversations/latitude/wd19-dock-kills-local-network-when-laptop-is-turned-off/647f8a41f4ccf8a8dea13a80?page=3 -
CosmicSisu A few questions
1. When your network is down, if you have a PC/MAC/Linux, can you ping the Firewalla gateway? (via ethernet and wifi)? this will test out your LAN.
2. When your network is down, if you take your phone to LTE, and try to connect to the firewalla app, does it work? this will test out your WAN.
3. Do you mean, you turn off health check and your network is good? if you do, let me know, I will create a case for you
-
Had connection crashes again this evening, so here goes:
1. My workstation runs on win11. Sometimes i can ping and sometimes not. Tonight gateway responded from lan.
2. Tried this tonight when my connection died once again. I can access internet via LTE wifi with phone, but firewalla app said: Unable to resolve host firewalla.encipher.io: No address associated with hostname. Also firewallas wan address is not reponding to LTE router.
.3. Yes, i'm not running firewalla health checks currently, so my network wont crash so often.
Bonus, i checked firewallas kernel logs when my connection died tonight, im pretty sure there's something wrong with my wan interface.
"Jun 18 20:31:29 localhost kernel: [ 2614.115102] igc 0000:01:00.0 eth0: NIC Link is Down
Jun 18 20:31:40 localhost kernel: [ 2625.203006] igc 0000:01:00.0 eth0: NIC Link is Up 2500 Mbps Full Duplex, Flow Control: RX
Jun 18 20:32:09 localhost kernel: [ 2654.887029] igc 0000:01:00.0 eth0: NIC Link is Down
Jun 18 20:32:14 localhost kernel: [ 2659.275011] igc 0000:01:00.0 eth0: NIC Link is Up 2500 Mbps Full Duplex, Flow Control: RX
Jun 18 20:32:22 localhost kernel: [ 2667.218871] igc 0000:01:00.0 eth0: NIC Link is Down
Jun 18 20:32:53 localhost kernel: [ 2698.491216] igc 0000:01:00.0 eth0: NIC Link is Up 2500 Mbps Full Duplex, Flow Control: RX/TX" -
Are you using any DNS server that may actively block things? "Unable to resolve host firewalla.encipher.io", this usually indicate there is a DNS block.
If you have port 4 as your WAN, this means, it is going down. Very likely your modem is rebooting or powering off and on ... this you will need to look and see why your modem is rebooting.
I also created a ticket for you, we can take a look at more logs
Please sign in to leave a comment.
Comments
51 comments