DNS queries taking way too long

DNS queries take way too long via the Firewalla device.  Over the last week I've noticed significant  slowdowns when surfing web pages, and all seem to be related to performing DNS queries whenever accessing a website (or web page components) that require a DNS look up.

To check this, I did some comparisons via dig (note that dns lookups going through the firewalla are consistently taking several seconds unless cached from previous lookups, and once that cache expires, it's back to several seconds, while bypassing the firewalla and digging directly through google DNS is consistently in the sub 500ms range)

bladernr@galactica:~/$ time dig www.time.com

; <<>> DiG 9.10.3-P4-Ubuntu <<>> www.time.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 36521
;; flags: qr rd ra; QUERY: 1, ANSWER: 9, AUTHORITY: 8, ADDITIONAL: 8

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;www.time.com. IN A

;; ANSWER SECTION:
www.time.com. 43200 IN CNAME time.com.
time.com. 58 IN A 52.85.131.157
time.com. 58 IN A 52.85.131.149
time.com. 58 IN A 52.85.131.101
time.com. 58 IN A 52.85.131.80
time.com. 58 IN A 52.85.131.104
time.com. 58 IN A 52.85.131.225
time.com. 58 IN A 52.85.131.15
time.com. 58 IN A 52.85.131.35

;; AUTHORITY SECTION:
time.com. 86400 IN NS ns3.p07.dynect.net.
time.com. 86400 IN NS ns1.p07.dynect.net.
time.com. 86400 IN NS ns4.p07.dynect.net.
time.com. 86400 IN NS ns-1235.awsdns-26.org.
time.com. 86400 IN NS ns2.p07.dynect.net.
time.com. 86400 IN NS ns-641.awsdns-16.net.
time.com. 86400 IN NS ns-1978.awsdns-55.co.uk.
time.com. 86400 IN NS ns-179.awsdns-22.com.

;; ADDITIONAL SECTION:
ns1.p07.dynect.net. 86396 IN A 208.78.70.7
ns1.p07.dynect.net. 296 IN AAAA 2001:500:90:1::7
ns2.p07.dynect.net. 86396 IN A 204.13.250.7
ns3.p07.dynect.net. 86396 IN A 208.78.71.7
ns3.p07.dynect.net. 296 IN AAAA 2001:500:94:1::7
ns4.p07.dynect.net. 86396 IN A 204.13.251.7
ns-179.awsdns-22.com. 172792 IN A 205.251.192.179

;; Query time: 2443 msec
;; SERVER: 127.0.1.1#53(127.0.1.1)
;; WHEN: Mon Feb 12 18:22:24 EST 2018
;; MSG SIZE rcvd: 539

real 0m12.471s
user 0m0.008s
sys 0m0.015s
bladernr@galactica:~/development/git/plainbox-provider-checkbox/bin$ time dig @8.8.8.8 www.time.com

; <<>> DiG 9.10.3-P4-Ubuntu <<>> @8.8.8.8 www.time.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 33957
;; flags: qr rd ra; QUERY: 1, ANSWER: 9, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;www.time.com. IN A

;; ANSWER SECTION:
www.time.com. 43182 IN CNAME time.com.
time.com. 40 IN A 52.85.131.157
time.com. 40 IN A 52.85.131.15
time.com. 40 IN A 52.85.131.35
time.com. 40 IN A 52.85.131.149
time.com. 40 IN A 52.85.131.80
time.com. 40 IN A 52.85.131.104
time.com. 40 IN A 52.85.131.225
time.com. 40 IN A 52.85.131.101

;; Query time: 0 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Mon Feb 12 18:22:42 EST 2018
;; MSG SIZE rcvd: 191

real 0m0.023s
user 0m0.015s
sys 0m0.007s

Note the Firewalla took a full 12 seconds to return an IP for time.com, where going directly to google's DNS servers took a fraction of a second. Some more:

bladernr@galactica:~/$ time dig firewalla.com

; <<>> DiG 9.10.3-P4-Ubuntu <<>> firewalla.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 58921
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;firewalla.com. IN A

;; ANSWER SECTION:
firewalla.com. 299 IN A 52.34.29.236

;; Query time: 87 msec
;; SERVER: 127.0.1.1#53(127.0.1.1)
;; WHEN: Mon Feb 12 18:28:49 EST 2018
;; MSG SIZE rcvd: 58

real 0m5.092s
user 0m0.006s
sys 0m0.000s

bladernr@galactica:~/$ time dig @8.8.8.8 firewalla.com

; <<>> DiG 9.10.3-P4-Ubuntu <<>> @8.8.8.8 firewalla.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 60442
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;firewalla.com. IN A

;; ANSWER SECTION:
firewalla.com. 299 IN A 52.34.29.236

;; Query time: 72 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Mon Feb 12 18:29:05 EST 2018
;; MSG SIZE rcvd: 58

real 0m0.094s
user 0m0.007s
sys 0m0.016s

And just to be sure there's no weird caching going on, I did them in the opposite order:

bladernr@galactica:~$ time dig @8.8.8.8 www.suse.com

; <<>> DiG 9.10.3-P4-Ubuntu <<>> @8.8.8.8 www.suse.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 26461
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;www.suse.com. IN A

;; ANSWER SECTION:
www.suse.com. 20788 IN A 130.57.66.10

;; Query time: 37 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Mon Feb 12 18:31:12 EST 2018
;; MSG SIZE rcvd: 57

real 0m0.042s
user 0m0.000s
sys 0m0.005s
bladernr@galactica:~$ time dig www.suse.com

; <<>> DiG 9.10.3-P4-Ubuntu <<>> www.suse.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 4175
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;www.suse.com. IN A

;; Query time: 4935 msec
;; SERVER: 127.0.1.1#53(127.0.1.1)
;; WHEN: Mon Feb 12 18:31:27 EST 2018
;; MSG SIZE rcvd: 41

real 0m4.957s
user 0m0.018s
sys 0m0.005s

bladernr@galactica:~$ time dig @8.8.8.8 homehotel.com.tw

; <<>> DiG 9.10.3-P4-Ubuntu <<>> @8.8.8.8 homehotel.com.tw
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 47740
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;homehotel.com.tw. IN A

;; ANSWER SECTION:
homehotel.com.tw. 21599 IN A 114.34.178.46

;; Query time: 235 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Mon Feb 12 18:33:22 EST 2018
;; MSG SIZE rcvd: 61

real 0m0.253s
user 0m0.009s
sys 0m0.009s

bladernr@galactica:~$ time dig homehotel.com.tw

; <<>> DiG 9.10.3-P4-Ubuntu <<>> homehotel.com.tw
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 45305
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 11, ADDITIONAL: 22

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;homehotel.com.tw. IN A

;; ANSWER SECTION:
homehotel.com.tw. 21599 IN A 114.34.178.46

;; AUTHORITY SECTION:
tw. 161065 IN NS anytld.apnic.net.
tw. 161065 IN NS h.dns.tw.
tw. 161065 IN NS b.dns.tw.
tw. 161065 IN NS g.dns.tw.
tw. 161065 IN NS ns.twnic.net.
tw. 161065 IN NS a.dns.tw.
tw. 161065 IN NS e.dns.tw.
tw. 161065 IN NS i.dns.tw.
tw. 161065 IN NS f.dns.tw.
tw. 161065 IN NS d.dns.tw.
tw. 161065 IN NS c.dns.tw.

;; ADDITIONAL SECTION:
a.dns.tw. 161065 IN A 203.73.24.25
a.dns.tw. 161065 IN AAAA 2001:45b1:0:5::25
b.dns.tw. 161065 IN A 210.201.138.58
b.dns.tw. 161065 IN AAAA 2404:0:10a0::58
c.dns.tw. 161065 IN A 203.66.87.201
c.dns.tw. 161065 IN AAAA 2001:b020:0:77::1
d.dns.tw. 161065 IN A 60.199.165.186
d.dns.tw. 161065 IN AAAA 2001:4541:9010:7::186
e.dns.tw. 161065 IN A 211.20.231.11
e.dns.tw. 161065 IN AAAA 2001:b000:1e0:c000::11
f.dns.tw. 161065 IN A 163.28.1.10
g.dns.tw. 161065 IN A 220.229.225.195
g.dns.tw. 161065 IN AAAA 2001:cd8:400::195
h.dns.tw. 161065 IN A 204.61.216.119
h.dns.tw. 161065 IN AAAA 2001:500:14:6119:ad::1
i.dns.tw. 161065 IN A 194.0.11.105
i.dns.tw. 161065 IN AAAA 2001:678:e:105::53
ns.twnic.net. 161065 IN A 192.83.166.11
ns.twnic.net. 161065 IN AAAA 2001:288:1:1006::11
anytld.apnic.net. 161065 IN A 203.119.43.53
anytld.apnic.net. 161065 IN AAAA 2001:dd8:12::53

;; Query time: 247 msec
;; SERVER: 127.0.1.1#53(127.0.1.1)
;; WHEN: Mon Feb 12 18:33:06 EST 2018
;; MSG SIZE rcvd: 718

real 0m5.270s
user 0m0.015s
sys 0m0.007s