Gold - Slow Performance DNS?
Hello All,
As of a couple days ago, our network has really been having some slowness issues. The issues seem to be very close to the issue reported here:
https://help.firewalla.com/hc/en-us/community/posts/360000704174-DNS-queries-taking-way-too-long
It appears the initial load times of websites take forever or sometimes time out (saying host not found). Email connectivity was failing (failed to connect to host imap.[domain].com) However after a 2nd or 3rd time, the sites start to respond better. I haven't done the level of troubleshooting the other poster did, this is just anecdotal.
Network is:
Dual WAN into a MikroTik router.
Mikrotik to Firewalla Gold
Firewalla to POE Switch
Switch to 3 Ruckus r510's wifi APs (1 for each floor and basement)
However today, unplugging the firewall and bypassing it, all is back to normal. Any ideas of what this could be?
-
I am also experiencing this same issue all of a sudden with both a Firewalla Gold AND a Firewalla Red, at two different sites, on two different ISP's. I have not changed any custom settings since the installation of the Firewalla devices months ago.
If any type of Monitoring is turned on, both networks slow to a crawl or outside stop passing traffic. Both devices starting having this issue within about a week of each other too. -
So we re-connected the Firewalla gold and now again we're getting sporadic slowdowns and connectivity issues. We didn't have any issues with it taken out of the network. My computer (at the time of this writing) is having issues connecting and it's currently in Emergency Access mode. So it's not rules based.
-
Ok I've done some testing, and the thing that is happening with the firewalla in between my router and our Wifi APs is that we sporadically get http 502 errors. This causes the initial handshake to fail and the website not to load or authorization to fail. After getting the 502, if I hit F5 or refresh the connection it works. It only fails on the intial load.
Here is a pertinent log from fiddler accessing 3 webites and my email client. All failed initially.
Accessing Greddy.com
40 502 HTTP www.greddy.com /products/exhausts/gpp-rs-ti/ 512 "no-cache, must-revalidate" text/html; charset=UTF-8 firefox:16096
41 304 HTTP crl.identrust.com /DSTROOTCAX3CRL.crl 0 "max-age=3600; Expires: Wed, 01 Sep 2021 20:16:02 GMT" application/pkix-crl svchost:3160
42 200 HTTP www.greddy.com /favicon.ico 894 image/vnd.microsoft.icon firefox:16096Accesssing Autoanything.com
1 502 HTTP Tunnel to www.autoanything.com:443 512 "no-cache, must-revalidate" text/html; charset=UTF-8 firefox:16096
2 200 HTTP Tunnel to calendar-a.wbx2.com:443 0 ciscocollabhost:8380Accessing summitracing.com
1 502 HTTP Tunnel to www.summitracing.com:443 512 "no-cache, must-revalidate" text/html; charset=UTF-8 firefox:16096Accessing my email client (thunderbird which failed imap auth to yahoo.com)
16 200 HTTP Tunnel to start.thunderbird.net:443 0 thunderbird:7700
17 200 HTTP ocsp.pki.goog /gts1o1core 471 "public, max-age=86400" application/ocsp-response thunderbird:7700
18 200 HTTP ocsp.pki.goog /gts1o1core 471 "public, max-age=86400" application/ocsp-response thunderbird:7700
19 200 HTTP Tunnel to start.thunderbird.net:443 0 thunderbird:7700
20 200 HTTP Tunnel to start.thunderbird.net:443 0 thunderbird:7700
21 502 HTTP Tunnel to api.login.yahoo.com:443 512 "no-cache, must-revalidate" text/html; charset=UTF-8 thunderbird:7700
22 200 HTTP Tunnel to start.thunderbird.net:443 0 thunderbird:7700
23 200 HTTP Tunnel to start.thunderbird.net:443 0 thunderbird:7700
24 200 HTTP Tunnel to start.thunderbird.net:443 0 thunderbird:7700 -
Can you please send an email to help@firewalla.com so that we can do further troubleshooting?
-
I am experiencing similar issues as well. I am not running any custom rules on either my Firewalla Red or Gold. I am not running any other types of DNS devices like Pi Hole or similar.
Only thing I have done with both Firewalla units is turn on Monitoring, and AD blocking, with no other custom settings set up on either device. I have not created any custom rules, nor am I using DNS over https. I have tried monitoring in both simple mode, and experimental simple mode.
If monitoring is turned on at all, my network slows to a crawl, or outsight stops passing traffic out to the internet.Both devices were working fine for months on end (two different sites, on two different ISP's) but then they both just started having these issues all of a sudden.
-
None of those commands failed. It seems to be random.
Microsoft Windows [Version 10.0.19043.1165]
(c) Microsoft Corporation. All rights reserved.C:\WINDOWS\system32>ping fire.walla
Pinging fire.walla [ XXX.XXX.XXX.XXX] with 32 bytes of data:
Reply from XXX.XXX.XXX.XXX: bytes=32 time=1ms TTL=64
Reply from XXX.XXX.XXX.XXX: bytes=32 time=1ms TTL=64
Reply from XXX.XXX.XXX.XXX: bytes=32 time=2ms TTL=64
Reply from XXX.XXX.XXX.XXX: bytes=32 time=2ms TTL=64Ping statistics for XXX.XXX.XXX.XXX:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 1ms, Maximum = 2ms, Average = 1msC:\WINDOWS\system32>ping 1.1.1.1
Pinging 1.1.1.1 with 32 bytes of data:
Reply from 1.1.1.1: bytes=32 time=17ms TTL=57
Reply from 1.1.1.1: bytes=32 time=20ms TTL=57
Reply from 1.1.1.1: bytes=32 time=22ms TTL=57
Reply from 1.1.1.1: bytes=32 time=14ms TTL=57Ping statistics for 1.1.1.1:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 14ms, Maximum = 22ms, Average = 18msC:\WINDOWS\system32>nslookup firewalla.com
Server: firewalla.inc.lan
Address: XXX.XXX.XXX.XXXNon-authoritative answer:
Name: firewalla.com
Address: 23.227.38.32 -
Not sure it carries over to everyone else's issue. But I was able to solve my issue with moving the Firewalla Gold into bridge mode (transparent monitoring). The problem initially showed itself in Simple mode (I wish there was a better explanation of what Simple mode is). And then after switching to router mode, there were no improvements.
Moving the firewalla into bridge mode fixed the issues. Seems it took a bit longer for blocking rules to take effect, but it all works now.
Still wish I had a better understanding from Firewalla as to why this issue just randomly showed up after running clean without any issues for a few months. Then all of the sudden, dns timeouts causing handshake failures out of nowhere, and the only fix is a change in my "configuration" that was working fine before ? Seems there should be a reason somewhere. -
@Firewalla my issue is resolved (kind of) as I already had a ticket open. The support team has been very responsive (save for the time differential). The problem is we never found the culprit. Moving to "bridge mode" fixed the issue (even though support never suggested that change).
But here is an example of the connection issues. Failed the first 2 times, and worked on the 3rd.belacyrf@DF_Home:/mnt/c/Users/David$ curl -w "%{http_code}" -o /dev/null -vvv www.workwheelsusa.com
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- 0:00:09 --:--:-- 0* Could not resolve host: www.workwheelsusa.com
* Closing connection 0
curl: (6) Could not resolve host: www.workwheelsusa.combelacyrf@DF_Home:/mnt/c/Users/David$ curl -w "%{http_code}" -o /dev/null -vvv www.workwheelsusa.com
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- 0:00:09 --:--:-- 0* Could not resolve host: www.workwheelsusa.com
* Closing connection 0
curl: (6) Could not resolve host: www.workwheelsusa.combelacyrf@DF_Home:/mnt/c/Users/David$ curl -w "%{http_code}" -o /dev/null -vvv www.workwheelsusa.com
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 192.254.236.68:80...
* TCP_NODELAY set
* Connected to www.workwheelsusa.com(192.254.236.68) port 80 (#0)
GET / HTTP/1.1
Host: www.workwheelsusa.com
User-Agent: curl/7.68.0
Accept: */*0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Mark bundle as not supporting multiuse
HTTP/1.1 503 Service Unavailable
Date: Fri, 03 Sep 2021 12:28:03 GMT
Server: Apache
Retry-After: 60
Upgrade: h2,h2c
Connection: Upgrade, close
Vary: Accept-Encoding
Content-Length: 188
Content-Type: text/html{ [188 bytes data]
100 188 100 188 0 0 350 0 --:--:-- --:--:-- --:--:-- 350
* Closing connection 0
Please sign in to leave a comment.
Comments
14 comments