DNS forwarding for local domain stopped working with latest update
I have a config for dnsmasq which forwards queries for my local domain to another DNS server, this has been working perfectly for the last 6+ months until the latest update. I know get a timeout on dns requests to the local domain
Config
~/.firewalla/config/dnsmasq_local/mydomain
server=/my.domain.com/192.168.47.2
server=/my.domain.com/192.168.47.2$family_protect
DNS query:
nslookup myserver.my.domain.com
Server: firewalla.inc.my.domain.com
Address: 192.168.47.1
DNS request timed out.
timeout was 2 seconds.
DNS request timed out.
timeout was 2 seconds.
DNS request timed out.
timeout was 2 seconds.
DNS request timed out.
timeout was 2 seconds.
*** Request to firewalla.inc.my.domain.com timed-out
Box version: 1.975
Firewalla Gold
-
After a bit more investigation I found this file
/home/pi/.firewalla/config/dnsmasq/local_domain_block.conf
which contains
server-uhigh=/my.domain.com/127.0.0.1#33333
If I update my config to also use "server-uhigh" dns resolution starts working again as expected. Where did this "local_domain_block.conf" come from? Is there a feature in the app to enable/disable this?
-
This new "feature" in 1.975 also caused me significant pain. I run a mail server in my network, and this new configuration blocked the mail server from making DNS MX queries for my domain, which broke my email server.
I reported it to Firewalla, but they did not see this as a bug.
In the end, I had to change my local domain name to "home.mydomain.com" and then set the search domain on all my networks to "mydomain.com". This way, only queries for "home.mydomain.com" are blocked, and queries for "mydomain.com" are forwarded to the upstream DNS servers as required by my mail server.
I really don't understand why Firewalla thinks it is a good idea to block DNS queries for the local domain itself. I understand why they'd want to block "host.mydomain.com" queries, but not "mydomain.com" queries.
-
Did you mean
server-high=/my.domain.com/127.0.0.1#33333
this overrides other upstream DNS server.
On iOS there is currently what I consider a bug using the new app.
Say you have a server, "pigpen". You have always been able to use host.lan by default so "pigpen.lan" to find the device. But you can't just call the device. You can edit the dnsmasq to use just the hostname and it works fine, but the iOS app does not allow this. I am told Android does. Hopefully Firewalla will fix this.
I think the reason Firewalla doesn't send local DNS upstream is two-fold:- Not to share internal DNS information with upstream providers.
- What value would there be in sending a query upstream for a local host that you already have the information for?
-
What value would there be in sending a query upstream for a local host that you already have the information for?
I have a real, registered domain, that I'd like to use as my "local domain" in Firewalla. That way, I can secure services on my network with real SSL certificates.
Suppose my registered domain is "mydomain.com".
I agree that there is no value in sending requests for Firewalla clients upstream, like "host1.mydomain.com" or "host2.mydomain.com".
My problem is Firewalla is also blocking DNS requests for the local domain itself: "mydomain.com". It should be possible for Firewalla clients to do a DNS query for "mydomain.com" to, for example, find out the MX server for the domain. Firewalla itself has absolutely no information about "mydomain.com"; it only knows about its clients. So, it makes no sense to me not to forward DNS queries for "mydomain.com" upstream. Not doing so breaks my SMTP (sendmail) server.
Perhaps Firewalla believes that the local domain will not be a real domain, but something like "lan" instead, and that is why they are not forwarding those requests, but this decision breaks things if the local domain is a real domain, like it is for me.
It should at least be an option to forward those requests, even if that option is available only through SSH.
-
Did you mean
server-high=/my.domain.com/127.0.0.1#33333
I tried to use "server-high" but this didn't work, changing to "server-uhigh" (the same as in local_domain_block.conf) did manage to override the block.
What value would there be in sending a query upstream for a local host that you already have the information for?
I'm not really trying to send the query upstream, I'm trying to send it to another server on the lan. I am running a domain controller which hosts it's own DNS server containing all the records required for active directory to work, so I need queries for my domain to be handled by that DNS server rather than Firewalla
-
@Mike So I think that may be what prevents Firewalla from forwarding on local requests. Basically you are nullifying that by allowing it.
For what you are describing, I think there are a few ways to do this but the one I would start with is:
Point FW LAN DNS to your local DNS server and have that send upstream DNS where you want it. Anything not found in the local DNS would go upstream. This is similar to how you can run pihole https://help.firewalla.com/hc/en-us/articles/360051625034-Guide-How-to-install-Pi-Hole-on-Gold-Purple-Beta-
-
@David,
So you are hosting your mail server internall...It should be possible for Firewalla clients to do a DNS query for "mydomain.com" to, for example, find out the MX server for the domain.
Don't you have a public MX record? Why does this DNS need to be local? I am missing something here. What clients are local that have no public DNS record?
-
@Mike,
Yes, I host an email server locally, and email for some of my public email addresses is delivered there directly. (I use an SMTP forwarding service, since my ISP blocks inbound SMTP traffic.)
I do have a public MX record for my domain (e.g., "mydomain.com"). When my email server receives an email for "user@mydomain.com", it tries to do DNS queries (AAAA and A) for "mydomain.com". The problem is that Firewalla is sending all queries for "mydomain.com" (as well as "host.mydomain.com") to the internal DNS server, and refusing to forward it externally. The internal DNS server doesn't know about "mydomain.com" (only "host.mydomain.com" and "host2.mydomain.com"), so the DNS query made by the email server fails, and the email is rejected.
I could turn off DNS Booster for the email server, but then it can't resolve the names of any Firewalla clients, which breaks other things, requiring me to maintain an /etc/hosts file on the server.
-
@David,
So I tried this:
server=/foo.mydomain.net/192.168.0.19
# pass along to the upstream dns
server=/private.mydomain.net/#
# kill
address=/kill.mydomain.net/#Results
ping foo.mydomain.net = 192.168.0.19 (as expected)
ping private.mydomain.net = my public DNS record (as expected)
ping kill.mydomain.net > no response (as expected)
Now in my case, the domain (A record) is externally hosted and there is a CNAME for private.mydomain.net to my external IP. foo and kill do not exist and I was able to control them all independently just fine.
It is clear (I think) from the dnsmasq that you can treat subdomains independent of the parent domain.
Maybe try: (edited)
# pass along to the upstream DNS
server=/mydomain.com/#
#use local DNS IP
server=/*.mydomain.com/serverIP
# OR
address=/host1.mydomain.com/host1IP
address=/host2.mydomain.com/host2IP -
@Mike,
For what you are describing, I think there are a few ways to do this but the one I would start with is:
Point FW LAN DNS to your local DNS server and have that send upstream DNS where you want it. Anything not found in the local DNS would go upstream. This is similar to how you can run pihole
There are a couple of reasons I didn't want to take this approach:
- The upstream from the AD server would always be the firewalla so that I can still benefit from all the ad blocking and other DNS based features the firewalla offers, so this adds an additional hop to the majority of DNS lookups
- This causes problems when using VLANs, for example in order for something on the guest VLAN to lookup an address on mydomain.com it requires the guest VLAN to have direct access to the AD server, whereas by setting up the forwarding rule on the firewalla machines on the guest VLAN have no knowledge of the AD servers existence
-
Hi @Mike,
- Having used the approach when I tried pihole using Firewalla's directions I can say that I found no observable difference in speed.
- You could limit access to port 53 only from the guest VLAN to minimize access.
I don't know of another way to do this. The instructions for the dnsmaq say that it is not officially supported and is not guaranteed to work long term. I'm glad to see Firewalla adding some of this to the UI and it doesn't surprise me that it may change some of the behavior of how the old instructions worked.
-
Been troubleshooting for a week now why my traefik dashboard stopped resolving. Using custom local DNS record in pihole pointed to traefik and this worked previously but now getting 'DNS PROBE POSSIBLE' and unable to resolve traefik from my box. Luckily i found this thread after i found out that adding in a custom DNS record into FWG under custom DNS worked. I was asking myself "why do i need this record here instead of pihole?".
also under ~/.firewalla/config/dnsmasq_local/ i see:
traefik unifi
cat traefik
address=/traefik/172.18.0.2My local DNS record in pihole = traefik-dashboard-internal.local.mydomain.com
I can instead create a file under here called mydomain and add the following?
address=/traefik-dashboard-internal.local.mydomain.com/172.18.0.2
I wonder if my record 'traefik' found here was the culprit all along. Does this record here need to exist if pihole being used? -
do i need to change my FWG LAN domain from mydomain.com to local.mydomain.com?
I'm not sure that I completely understand your problem, but if your Traefik instance thinks its name is traefik-dashboard-internal.local.mydomain.com, you might want to add a file under config/dnsmasq_local to assign that name to the right IP address.
It would be helpful to understand the failure in more detail. What happens when you try to access your Traefik dashboard now?
-
Basically trying to figure out why pihole local DNS records (previosly working for traefik) no longer work. pihole running in FWG docker container. I can only get traefik dashboard to load now only after entering into dnsmasq_local. FWG LAN is pointed to pihole DNS 172.16.0.2 so shouldnt pihole records be used? This problem looks to have happened after recent FW update.
If i remove the entry under dnsmasq_local, and add the same record into pihole local DNS records, i can't resolve from my machine. pihole is working fine otherwise. Cloudflare upstream DNS is used in pihole DNS settings.
I have one registered domain 'mydomain.com' and only record (A record) pointed to my WAN IP. -
Is the Firewalla receiving the request? Try running tcpdump on the Firewalla, like this:
sudo tcpdump -vv -i any 'port 53'
You might want to change the "any" to the interface for the LAN to which your client is connected. "any" will catch DNS requests on any interface (including your WANs, if they are using regular DNS and DoH).
-
when i remove the dns entry under .firewalla/config/dnsmasq_local/ and then configure the same record in pihole local DNS for traefik-dashboard-internal.local.<mydomain>.com, when i nslookup from my machine:
nslookup traefik-dashboard-internal.local.mydomain.com
i see timeout from command prompt.
What i see in tcpdump for br0 interface for port 53:
20:44:18.737631 IP (tos 0x0, ttl 128, id 17636, offset 0, flags [none], proto UDP (17), length 94)
192.168.169.153.54849 > 192.168.169.1.domain: [udp sum ok] 64313+ A? traefik-dashboard-internal.local.mydomain.com. (66)
20:44:20.118900 IP (tos 0x0, ttl 128, id 17649, offset 0, flags [none], proto UDP (17), length 69)
192.168.169.153.64978 > 192.168.169.1.domain: [udp sum ok] 1+ PTR? 2.0.16.172.in-addr.arpa. (41)
20:44:20.128587 IP (tos 0x0, ttl 64, id 58732, offset 0, flags [DF], proto UDP (17), length 69)
172.16.0.2.domain > 192.168.169.153.64978: [bad udp cksum 0x1697 -> 0x7bc9!] 1 NXDomain* q: PTR? 2.0.16.172.in-addr.arpa. 0/0/0 (41)
If i nslookup 172.18.0.2:
20:50:53.133529 IP (tos 0x0, ttl 128, id 19269, offset 0, flags [none], proto UDP (17), length 69)
192.168.169.153.61498 > 192.168.169.1.domain: [udp sum ok] 1+ PTR? 2.0.16.172.in-addr.arpa. (41)
20:50:53.140425 IP (tos 0x0, ttl 64, id 42179, offset 0, flags [DF], proto UDP (17), length 69)
172.16.0.2.domain > 192.168.169.153.61498: [bad udp cksum 0x1697 -> 0x8961!] 1 NXDomain* q: PTR? 2.0.16.172.in-addr.arpa. 0/0/0 (41)
20:50:53.143988 IP (tos 0x0, ttl 128, id 19270, offset 0, flags [none], proto UDP (17), length 69)
192.168.169.153.61499 > 192.168.169.1.domain: [udp sum ok] 2+ PTR? 2.0.18.172.in-addr.arpa. (41)
20:50:53.151032 IP (tos 0x0, ttl 64, id 42181, offset 0, flags [DF], proto UDP (17), length 69)
172.16.0.2.domain > 192.168.169.153.61499: [bad udp cksum 0x1697 -> 0x875f!] 2 NXDomain* q: PTR? 2.0.18.172.in-addr.arpa. 0/0/0 (41)
192.168.169.153 is my machine172.16.0.2 is pihole
local record in pihole for traefik-dashboard-internal.local.mydomain.com > 172.18.0.2 -
pi@firewalla:~ (Firewalla) $ sudo docker network list
NETWORK ID NAME DRIVER SCOPE
cb1f30336485 bridge bridge local
0a1ebbb11363 host host local
258e4dfaef53 none null local
e467245b67f1 pi-hole_default bridge local
b4811b1e7816 proxy bridge local
afd3cc7283aa unifi_default bridge local
pi@firewalla:~ (Firewalla) $ route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default cpe-*-*-*- 0.0.0.0 UG 1 0 0 eth0
*.*.*.* 0.0.0.0 255.255.248.0 U 0 0 0 eth0
172.16.0.0 0.0.0.0 255.255.255.0 U 0 0 0 br-e467245b67f1
172.16.1.0 0.0.0.0 255.255.255.0 U 0 0 0 br-afd3cc7283aa
172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0
172.18.0.0 0.0.0.0 255.255.0.0 U 0 0 0 br-b4811b1e7816
192.168.169.0 0.0.0.0 255.255.255.0 U 0 0 0 br0
192.168.237.0 0.0.0.0 255.255.255.0 U 0 0 0 br1
dns-cac-lb-01.r cpe-98-150-200- 255.255.255.255 UGH 1 0 0 eth0
dns-cac-lb-02.r cpe-98-150-200- 255.255.255.255 UGH 1 0 0 eth0
pi@firewalla:~ (Firewalla) $ nslookup 172.18.0.2
** server can't find 2.0.18.172.in-addr.arpa: NXDOMAINpi@firewalla:~ (Firewalla) $ nslookup traefik-dashboard-internal.local.mydomain.com
Server: 209.18.47.61
Address: 209.18.47.61#53** server can't find traefik-dashboard-internal.local.mydomain.com: NXDOMAIN
-
so i think my issue is related to local domain vs search domain?
if i do a local record on my pihole for my nas like 'nas.mydomain.com' > 192.168.169.161, i can resolve fine from my machine. No record under dnsmasq_local.
traefik is using 172.18.0.2, and no record configured on pihole works. -
am i doing something clearly wrong here or likely do to the recent update? it was working fine as i expected when pointed to pihole DNS IP few weeks ago and local records worked as expected as well.
now i need these local records under dnsmasq_local, not in pihole and don't understand why. -
Does the pihole get the requests if you turn off the DNS Booster for the device making the query?
If so, you might try creating a file
/home/pi/.firewalla/config/dnsmasq_local/pihole
with the contents
server-uhigh=/mydomain.com/XX.XX.XX.XX
where XX.XX.XX.XX is the IP address of your pihole.
After you make that file, do
sudo systemctl restart firerouter_dns.service
The issue I found, as well as the OP, is that a recent change causes Firewalla not to forward DNS queries for the local domain to upstream DNS servers.
-
I was talking about the DNS Booster on the FWG.
I'm out of ideas. If DNS Booster is off, your clients should just be talking to the LAN DNS server, which I assume you have set to your pihole's IP address. The dnsmasq_local files shouldn't have any effect, since those are only used by the DNS Booster.
You might want to reach out to help@firewalla.com to see if they can help you.
Please sign in to leave a comment.
Comments
31 comments