Wifi calling packet storm
Hi,
I have a mystery I am trying to debug.
My network setup is this:
Verizon 5G Home Internet Gateway --> Firewalla Purple --> Switch --> Primary Eero & wired devices --> Switches --> Secondary Eeros + wired devices
The Verizon Gateway is acting as a router.
Firewalla is in Bridge Mode, WAN port to the Verizon Gateway and LAN port to the rest of the network.
Secondary Eeros are downstream of the Primary Eero. The Eero app confirms all 3 are connected by wired ethernet.
The switches are simple unmanged switches, no VLANs are in use.
There are no ethernet loops. Only loop I could think of is if Eero is internally using Wifi between Eeros as well as wired, but the app reports fully wired.
Background on this setup is here: https://help.firewalla.com/hc/en-us/community/posts/9511434250131/comments/19682624008979
Problem:
Verizon Wifi calling on iPhone mobile devices connected via Eero can trigger a packet storm which saturates the WAN uplink, all aimed at the Verizon Wifi calling server but sent to what I think is a multicast ethernet address to help spread the pain over the LAN.
I captured over 200k identical packets in ~30 seconds. Wifi becomes inoperable during the storm, and the storm typically lasts around 5 minutes. Turning off Wifi on the phone reported as having a high upload bitrate appears to end the storm early.
Workaround:
Block access to wo.vzwwo.com to all devices such that the devices do not enable Wifi calling. I don't know if other carriers Wifi calling could trigger this.
Debugging tried so far:
- Rebooting everything (Verizon Gateway, Firewalla, All Eeros) - no change to behavior
- Disabling "Block traffic from internet" rule in Firewalla, and/or Re-enable IPV4 firewall 'standard security' setting in the Verizon Gateway. No change to behavior.
- This problem appears to be recent. Either this change has arisen from the new firmware on the Verizon Gateway or from using Firewalla in bridge mode instead of router mode. I can not rollback the gateway firmware.
I have not tested Firewalla in router mode and do not wish to use router mode as per linked article above. I can not discount Eero firmware updates. Too many variables have changed.
- I have not tried removing Firewalla, it would be hard to tell (except for periods of unreliable wifi) if the problem were happening since I do not have another means to do the packet capture or measure live bitrate.
- tcpdump captures from Firewalla - details are below
Related information:
Internet searching for "Wifi calling packet storm" will get a number of hits. None of them have a documented explanation, many of them are not debugged in detail, and many of them without common hardware. But it does sound like the same problem.
Next steps:
I am looking for inspiration on how to pin this down further, especially from anyone who is familiar with Wifi calling and/or IPSEC.
TCP Dump:
The command to catpture, run on Firewalla from an ssh session is:
sudo tcpdump -i br0 -c 100000 -n -e -vvv "net 141.207/16 || port 4500
141.207 captures Verizon wifi calling service, and perhaps more of Verizon.
Port 4500 (IPSEC) is used by Verizon for Wifi calling, perhaps others.
There are multiple devices on this network:
PHONE_1_MAC - I replaced the MAC address of phone 1 with this string in the dump below
PHONE_2_MAC
PHONE_3_MAC
GATEWAY_MAC - The Verizon Gateway LAN port.
DYSON_FAN_MAC - The MAC address of an unrelated appliance (a wifi connected fan). Sometimes I've seen others appear in place or as well, such as a fridge.
After I started tcpdump, I removed the block for wo.vzwwo.com, and the devices started connecting. It didn't take long for a storm to start.
Looks to me that some chatter goes on between the devices and Verizon. Then at 05:50:16.382362 PHONE_2_MAC sends a packet with IP address of the Verizon server but to a multi-cast (01:00) ethernet MAC address, not the gateway. That seems odd. The DYSON_FAN_MAC device (a wifi connected fan) then repeates the packet, same IP information, guess it felt the need to rebroadcast! Then at 05:50:19.036295 PHONE_2_MAC, or perhaps the Eero system on its behalf, starts a storm of identical packets. I limited the tcpdump to 200k packets to avoid filling up the Firewalla 'disk'.
05:49:53.718963 PHONE_3_MAC > GATEWAY_MAC, ethertype IPv4 (0x0800), length 338: (tos 0x0, ttl 64, id 8734, offset 0, flags [none], proto UDP (17), length 324)
192.168.0.183.500 > 141.207.169.233.500: [udp sum ok] isakmp 2.0 msgid 00000000 cookie 7c58e55f453b12bc->0000000000000000: parent_sa ikev2_init[I]:
(sa: len=44
(p: #1 protoid=isakmp transform=4 len=44
(t: #1 type=encr id=aes (type=keylen value=0100))
(t: #2 type=prf id=hmac-sha )
(t: #3 type=integ id=hmac-sha )
(t: #4 type=dh id=modp1024 )))
(v2ke: len=128 group=modp1024 ...)
(nonce: len=16 nonce=(ebb1ef5e3ed7426eae5499361c01b926) )
(n: prot_id=#0 type=16388(nat_detection_source_ip))
(n: prot_id=#0 type=16389(nat_detection_destination_ip))
(n: prot_id=#0 type=16430(status))
05:49:54.725050 PHONE_3_MAC > GATEWAY_MAC, ethertype IPv4 (0x0800), length 338: (tos 0x0, ttl 64, id 7929, offset 0, flags [none], proto UDP (17), length 324)
192.168.0.183.500 > 141.207.169.233.500: [udp sum ok] isakmp 2.0 msgid 00000000 cookie 7c58e55f453b12bc->0000000000000000: parent_sa ikev2_init[I]:
(sa: len=44
(p: #1 protoid=isakmp transform=4 len=44
(t: #1 type=encr id=aes (type=keylen value=0100))
(t: #2 type=prf id=hmac-sha )
(t: #3 type=integ id=hmac-sha )
(t: #4 type=dh id=modp1024 )))
(v2ke: len=128 group=modp1024 ...
(nonce: len=16 nonce=(ebb1ef5e3ed7426eae5499361c01b926) )
(n: prot_id=#0 type=16388(nat_detection_source_ip))
(n: prot_id=#0 type=16389(nat_detection_destination_ip))
(n: prot_id=#0 type=16430(status))
05:49:56.723882 PHONE_3_MAC > GATEWAY_MAC, ethertype IPv4 (0x0800), length 338: (tos 0x0, ttl 64, id 64229, offset 0, flags [none], proto UDP (17), length 324)
192.168.0.183.500 > 141.207.169.233.500: [udp sum ok] isakmp 2.0 msgid 00000000 cookie 7c58e55f453b12bc->0000000000000000: parent_sa ikev2_init[I]:
(sa: len=44
(p: #1 protoid=isakmp transform=4 len=44
(t: #1 type=encr id=aes (type=keylen value=0100))
(t: #2 type=prf id=hmac-sha )
(t: #3 type=integ id=hmac-sha )
(t: #4 type=dh id=modp1024 )))
(v2ke: len=128 group=modp1024 ...
(nonce: len=16 nonce=(ebb1ef5e3ed7426eae5499361c01b926) )
(n: prot_id=#0 type=16388(nat_detection_source_ip))
(n: prot_id=#0 type=16389(nat_detection_destination_ip))
(n: prot_id=#0 type=16430(status))
05:50:00.731011 PHONE_3_MAC > GATEWAY_MAC, ethertype IPv4 (0x0800), length 338: (tos 0x0, ttl 64, id 25887, offset 0, flags [none], proto UDP (17), length 324)
192.168.0.183.500 > 141.207.169.233.500: [udp sum ok] isakmp 2.0 msgid 00000000 cookie 7c58e55f453b12bc->0000000000000000: parent_sa ikev2_init[I]:
(sa: len=44
(p: #1 protoid=isakmp transform=4 len=44
(t: #1 type=encr id=aes (type=keylen value=0100))
(t: #2 type=prf id=hmac-sha )
(t: #3 type=integ id=hmac-sha )
(t: #4 type=dh id=modp1024 )))
(v2ke: len=128 group=modp1024 ...
(nonce: len=16 nonce=(ebb1ef5e3ed7426eae5499361c01b926) )
(n: prot_id=#0 type=16388(nat_detection_source_ip))
(n: prot_id=#0 type=16389(nat_detection_destination_ip))
(n: prot_id=#0 type=16430(status))
05:50:08.733982 PHONE_3_MAC > GATEWAY_MAC, ethertype IPv4 (0x0800), length 338: (tos 0x0, ttl 64, id 62416, offset 0, flags [none], proto UDP (17), length 324)
192.168.0.183.500 > 141.207.169.233.500: [udp sum ok] isakmp 2.0 msgid 00000000 cookie 7c58e55f453b12bc->0000000000000000: parent_sa ikev2_init[I]:
(sa: len=44
(p: #1 protoid=isakmp transform=4 len=44
(t: #1 type=encr id=aes (type=keylen value=0100))
(t: #2 type=prf id=hmac-sha )
(t: #3 type=integ id=hmac-sha )
(t: #4 type=dh id=modp1024 )))
(v2ke: len=128 group=modp1024 ...
(nonce: len=16 nonce=(ebb1ef5e3ed7426eae5499361c01b926) )
(n: prot_id=#0 type=16388(nat_detection_source_ip))
(n: prot_id=#0 type=16389(nat_detection_destination_ip))
(n: prot_id=#0 type=16430(status))
05:50:15.266164 PHONE_2_MAC > GATEWAY_MAC, ethertype IPv4 (0x0800), length 338: (tos 0x0, ttl 64, id 65504, offset 0, flags [none], proto UDP (17), length 324)
192.168.0.227.500 > 141.207.197.233.500: [udp sum ok] isakmp 2.0 msgid 00000000 cookie 9bf72201546b1e93->0000000000000000: parent_sa ikev2_init[I]:
(sa: len=44
(p: #1 protoid=isakmp transform=4 len=44
(t: #1 type=encr id=aes (type=keylen value=0100))
(t: #2 type=prf id=hmac-sha )
(t: #3 type=integ id=hmac-sha )
(t: #4 type=dh id=modp1024 )))
(v2ke: len=128 group=modp1024 ...
(nonce: len=16 nonce=(9f76d9e5dd436c3ffccb83b57a1145b9) )
(n: prot_id=#0 type=16388(nat_detection_source_ip))
(n: prot_id=#0 type=16389(nat_detection_destination_ip))
(n: prot_id=#0 type=16430(status))
05:50:15.300070 GATEWAY_MAC > PHONE_2_MAC, ethertype IPv4 (0x0800), length 354: (tos 0x0, ttl 60, id 8287, offset 0, flags [DF], proto UDP (17), length 340)
141.207.197.233.500 > 192.168.0.227.500: [udp sum ok] isakmp 2.0 msgid 00000000 cookie 9bf72201546b1e93->00414314292599f7: parent_sa ikev2_init[R]:
(sa: len=44
(p: #1 protoid=isakmp transform=4 len=44
(t: #1 type=encr id=aes (type=keylen value=0100))
(t: #2 type=prf id=hmac-sha )
(t: #3 type=integ id=hmac-sha )
(t: #4 type=dh id=modp1024 )))
(v2ke: len=128 group=modp1024 ...
(nonce: len=32 nonce=(c984d400f1b0f57430b0dd7dd024522fa01c04a42b2bb3fd0c6d9df99f1b1215) )
(n: prot_id=#0 type=16388(nat_detection_source_ip))
(n: prot_id=#0 type=16389(nat_detection_destination_ip))
(n: prot_id=#0 type=16430(status))
05:50:15.323654 PHONE_2_MAC > GATEWAY_MAC, ethertype IPv4 (0x0800), length 410: (tos 0x0, ttl 64, id 18723, offset 0, flags [none], proto UDP (17), length 396)
192.168.0.227.4500 > 141.207.197.233.4500: [udp sum ok] NONESP-encap: isakmp 2.0 msgid 00000001 cookie 9bf72201546b1e93->00414314292599f7: child_sa ikev2_auth[I]:
(v2e: len=332 ...)
05:50:15.366360 GATEWAY_MAC > PHONE_2_MAC, ethertype IPv4 (0x0800), length 1150: (tos 0x0, ttl 60, id 8290, offset 0, flags [DF], proto UDP (17), length 1136)
141.207.197.233.4500 > 192.168.0.227.4500: [udp sum ok] NONESP-encap: isakmp 2.0 msgid 00000001 cookie 9bf72201546b1e93->00414314292599f7: child_sa ikev2_auth[R]:
(#53) [|v2IDr]
05:50:15.371626 GATEWAY_MAC > PHONE_2_MAC, ethertype IPv4 (0x0800), length 798: (tos 0x0, ttl 60, id 8292, offset 0, flags [DF], proto UDP (17), length 784)
141.207.197.233.4500 > 192.168.0.227.4500: [udp sum ok] NONESP-encap: isakmp 2.0 msgid 00000001 cookie 9bf72201546b1e93->00414314292599f7: child_sa ikev2_auth[R]:
(#53)
05:50:15.371633 GATEWAY_MAC > PHONE_2_MAC, ethertype IPv4 (0x0800), length 1150: (tos 0x0, ttl 60, id 8291, offset 0, flags [DF], proto UDP (17), length 1136)
141.207.197.233.4500 > 192.168.0.227.4500: [udp sum ok] NONESP-encap: isakmp 2.0 msgid 00000001 cookie 9bf72201546b1e93->00414314292599f7: child_sa ikev2_auth[R]:
(#53)
05:50:15.396268 PHONE_2_MAC > GATEWAY_MAC, ethertype IPv4 (0x0800), length 170: (tos 0x0, ttl 64, id 48173, offset 0, flags [none], proto UDP (17), length 156)
192.168.0.227.4500 > 141.207.197.233.4500: [udp sum ok] NONESP-encap: isakmp 2.0 msgid 00000002 cookie 9bf72201546b1e93->00414314292599f7: child_sa ikev2_auth[I]:
(v2e: len=92 ...)
05:50:15.655619 GATEWAY_MAC > PHONE_2_MAC, ethertype IPv4 (0x0800), length 186: (tos 0x0, ttl 60, id 8576, offset 0, flags [DF], proto UDP (17), length 172)
141.207.197.233.4500 > 192.168.0.227.4500: [udp sum ok] NONESP-encap: isakmp 2.0 msgid 00000002 cookie 9bf72201546b1e93->00414314292599f7: child_sa ikev2_auth[R]:
(v2e: len=108 ...)
05:50:15.716284 PHONE_2_MAC > GATEWAY_MAC, ethertype IPv4 (0x0800), length 154: (tos 0x0, ttl 64, id 3586, offset 0, flags [none], proto UDP (17), length 140)
192.168.0.227.4500 > 141.207.197.233.4500: [udp sum ok] NONESP-encap: isakmp 2.0 msgid 00000003 cookie 9bf72201546b1e93->00414314292599f7: child_sa ikev2_auth[I]:
(v2e: len=76 ...)
05:50:15.925981 GATEWAY_MAC > PHONE_2_MAC, ethertype IPv4 (0x0800), length 122: (tos 0x0, ttl 60, id 8772, offset 0, flags [DF], proto UDP (17), length 108)
141.207.197.233.4500 > 192.168.0.227.4500: [udp sum ok] NONESP-encap: isakmp 2.0 msgid 00000003 cookie 9bf72201546b1e93->00414314292599f7: child_sa ikev2_auth[R]:
(v2e: len=44 ...)
05:50:15.931293 PHONE_2_MAC > GATEWAY_MAC, ethertype IPv4 (0x0800), length 138: (tos 0x0, ttl 64, id 59600, offset 0, flags [none], proto UDP (17), length 124)
192.168.0.227.4500 > 141.207.197.233.4500: [udp sum ok] NONESP-encap: isakmp 2.0 msgid 00000004 cookie 9bf72201546b1e93->00414314292599f7: child_sa ikev2_auth[I]:
(v2e: len=60 ...)
05:50:16.283904 GATEWAY_MAC > PHONE_2_MAC, ethertype IPv4 (0x0800), length 426: (tos 0x0, ttl 60, id 9031, offset 0, flags [DF], proto UDP (17), length 412)
141.207.197.233.4500 > 192.168.0.227.4500: [udp sum ok] NONESP-encap: isakmp 2.0 msgid 00000004 cookie 9bf72201546b1e93->00414314292599f7: child_sa ikev2_auth[R]:
(v2e: len=348 ...)
05:50:16.382362 PHONE_2_MAC > 01:00:5e:28:00:01, ethertype IPv4 (0x0800), length 158: (tos 0x0, ttl 64, id 20635, offset 0, flags [none], proto UDP (17), length 144)
192.168.0.227.4500 > 141.207.197.233.4500: [no cksum] UDP-encap: ESP(spi=0xb6416169,seq=0x1), length 116
05:50:16.396189 PHONE_2_MAC > GATEWAY_MAC, ethertype IPv4 (0x0800), length 1246: (tos 0x50, ttl 64, id 64934, offset 0, flags [none], proto UDP (17), length 1232)
192.168.0.227.4500 > 141.207.197.233.4500: [no cksum] UDP-encap: ESP(spi=0xb6416169,seq=0x2), length 1204
05:50:16.452748 DYSON_FAN_MAC > GATEWAY_MAC, ethertype IPv4 (0x0800), length 158: (tos 0x0, ttl 63, id 20635, offset 0, flags [none], proto UDP (17), length 144)
192.168.0.227.4500 > 141.207.197.233.4500: [no cksum] UDP-encap: ESP(spi=0xb6416169,seq=0x1), length 116
05:50:16.577556 GATEWAY_MAC > PHONE_2_MAC, ethertype IPv4 (0x0800), length 638: (tos 0x80, ttl 236, id 0, offset 0, flags [none], proto UDP (17), length 624)
141.207.197.233.4500 > 192.168.0.227.4500: [no cksum] UDP-encap: ESP(spi=0x00ae30f1,seq=0x1), length 596
05:50:16.624355 PHONE_2_MAC > GATEWAY_MAC, ethertype IPv4 (0x0800), length 1246: (tos 0x50, ttl 64, id 18612, offset 0, flags [none], proto UDP (17), length 1232)
192.168.0.227.4500 > 141.207.197.233.4500: [no cksum] UDP-encap: ESP(spi=0xb6416169,seq=0x3), length 1204
05:50:16.753906 GATEWAY_MAC > PHONE_2_MAC, ethertype IPv4 (0x0800), length 878: (tos 0x80, ttl 236, id 0, offset 0, flags [none], proto UDP (17), length 864)
141.207.197.233.4500 > 192.168.0.227.4500: [no cksum] UDP-encap: ESP(spi=0x00ae30f1,seq=0x2), length 836
05:50:17.631379 PHONE_1_MAC > GATEWAY_MAC, ethertype IPv4 (0x0800), length 338: (tos 0x0, ttl 64, id 42011, offset 0, flags [none], proto UDP (17), length 324)
192.168.0.245.500 > 141.207.151.233.500: [udp sum ok] isakmp 2.0 msgid 00000000 cookie 738f2fc910409573->0000000000000000: parent_sa ikev2_init[I]:
(sa: len=44
(p: #1 protoid=isakmp transform=4 len=44
(t: #1 type=encr id=aes (type=keylen value=0100))
(t: #2 type=prf id=hmac-sha )
(t: #3 type=integ id=hmac-sha )
(t: #4 type=dh id=modp1024 )))
(v2ke: len=128 group=modp1024 ...)
(nonce: len=16 nonce=(1997949bc71225161beb184ea59dbd74) )
(n: prot_id=#0 type=16388(nat_detection_source_ip))
(n: prot_id=#0 type=16389(nat_detection_destination_ip))
(n: prot_id=#0 type=16430(status))
05:50:17.697930 GATEWAY_MAC > PHONE_1_MAC, ethertype IPv4 (0x0800), length 354: (tos 0x0, ttl 53, id 55805, offset 0, flags [DF], proto UDP (17), length 340)
141.207.151.233.500 > 192.168.0.245.500: [udp sum ok] isakmp 2.0 msgid 00000000 cookie 738f2fc910409573->001fc33ec9fda0ae: parent_sa ikev2_init[R]:
(sa: len=44
(p: #1 protoid=isakmp transform=4 len=44
(t: #1 type=encr id=aes (type=keylen value=0100))
(t: #2 type=prf id=hmac-sha )
(t: #3 type=integ id=hmac-sha )
(t: #4 type=dh id=modp1024 )))
(v2ke: len=128 group=modp1024 ...)
(nonce: len=32 nonce=(ffaf9efee806acb395148cfa587de23b820e53e3088058e3a29ea7d2ac09a71a) )
(n: prot_id=#0 type=16388(nat_detection_source_ip))
(n: prot_id=#0 type=16389(nat_detection_destination_ip))
(n: prot_id=#0 type=16430(status))
05:50:17.730569 PHONE_1_MAC > GATEWAY_MAC, ethertype IPv4 (0x0800), length 394: (tos 0x0, ttl 64, id 41753, offset 0, flags [none], proto UDP (17), length 380)
192.168.0.245.4500 > 141.207.151.233.4500: [udp sum ok] NONESP-encap: isakmp 2.0 msgid 00000001 cookie 738f2fc910409573->001fc33ec9fda0ae: child_sa ikev2_auth[I]:
(v2e: len=316 ...)
05:50:17.797994 GATEWAY_MAC > PHONE_1_MAC, ethertype IPv4 (0x0800), length 1150: (tos 0x0, ttl 53, id 55878, offset 0, flags [DF], proto UDP (17), length 1136)
141.207.151.233.4500 > 192.168.0.245.4500: [udp sum ok] NONESP-encap: isakmp 2.0 msgid 00000001 cookie 738f2fc910409573->001fc33ec9fda0ae: child_sa ikev2_auth[R]:
(#53) [|v2IDr]
05:50:17.797997 GATEWAY_MAC > PHONE_1_MAC, ethertype IPv4 (0x0800), length 814: (tos 0x0, ttl 53, id 55880, offset 0, flags [DF], proto UDP (17), length 800)
141.207.151.233.4500 > 192.168.0.245.4500: [udp sum ok] NONESP-encap: isakmp 2.0 msgid 00000001 cookie 738f2fc910409573->001fc33ec9fda0ae: child_sa ikev2_auth[R]:
(#53)
\05:50:17.803591 GATEWAY_MAC > PHONE_1_MAC, ethertype IPv4 (0x0800), length 1150: (tos 0x0, ttl 53, id 55879, offset 0, flags [DF], proto UDP (17), length 1136)
141.207.151.233.4500 > 192.168.0.245.4500: [udp sum ok] NONESP-encap: isakmp 2.0 msgid 00000001 cookie 738f2fc910409573->001fc33ec9fda0ae: child_sa ikev2_auth[R]:
(#53)
05:50:17.828267 PHONE_1_MAC > GATEWAY_MAC, ethertype IPv4 (0x0800), length 170: (tos 0x0, ttl 64, id 8876, offset 0, flags [none], proto UDP (17), length 156)
192.168.0.245.4500 > 141.207.151.233.4500: [udp sum ok] NONESP-encap: isakmp 2.0 msgid 00000002 cookie 738f2fc910409573->001fc33ec9fda0ae: child_sa ikev2_auth[I]:
(v2e: len=92 ...)
05:50:18.202421 GATEWAY_MAC > PHONE_1_MAC, ethertype IPv4 (0x0800), length 186: (tos 0x0, ttl 54, id 56073, offset 0, flags [DF], proto UDP (17), length 172)
141.207.151.233.4500 > 192.168.0.245.4500: [udp sum ok] NONESP-encap: isakmp 2.0 msgid 00000002 cookie 738f2fc910409573->001fc33ec9fda0ae: child_sa ikev2_auth[R]:
(v2e: len=108 ...)
05:50:18.328642 PHONE_1_MAC > GATEWAY_MAC, ethertype IPv4 (0x0800), length 154: (tos 0x0, ttl 64, id 37828, offset 0, flags [none], proto UDP (17), length 140)
192.168.0.245.4500 > 141.207.151.233.4500: [udp sum ok] NONESP-encap: isakmp 2.0 msgid 00000003 cookie 738f2fc910409573->001fc33ec9fda0ae: child_sa ikev2_auth[I]:
(v2e: len=76 ...)
05:50:18.610930 GATEWAY_MAC > PHONE_1_MAC, ethertype IPv4 (0x0800), length 122: (tos 0x0, ttl 54, id 56434, offset 0, flags [DF], proto UDP (17), length 108)
141.207.151.233.4500 > 192.168.0.245.4500: [udp sum ok] NONESP-encap: isakmp 2.0 msgid 00000003 cookie 738f2fc910409573->001fc33ec9fda0ae: child_sa ikev2_auth[R]:
(v2e: len=44 ...)
05:50:18.615728 PHONE_1_MAC > GATEWAY_MAC, ethertype IPv4 (0x0800), length 138: (tos 0x0, ttl 64, id 50168, offset 0, flags [none], proto UDP (17), length 124)
192.168.0.245.4500 > 141.207.151.233.4500: [udp sum ok] NONESP-encap: isakmp 2.0 msgid 00000004 cookie 738f2fc910409573->001fc33ec9fda0ae: child_sa ikev2_auth[I]:
(v2e: len=60 ...)
05:50:19.036295 PHONE_2_MAC > 01:00:5e:28:00:01, ethertype IPv4 (0x0800), length 158: (tos 0x0, ttl 64, id 45956, offset 0, flags [none], proto UDP (17), length 144)
192.168.0.227.4500 > 141.207.197.233.4500: [no cksum] UDP-encap: ESP(spi=0xb6416169,seq=0x4), length 116
05:50:19.036302 PHONE_2_MAC > 01:00:5e:28:00:01, ethertype IPv4 (0x0800), length 158: (tos 0x0, ttl 64, id 45956, offset 0, flags [none], proto UDP (17), length 144)
192.168.0.227.4500 > 141.207.197.233.4500: [no cksum] UDP-encap: ESP(spi=0xb6416169,seq=0x4), length 116
05:50:19.036305 PHONE_2_MAC > 01:00:5e:28:00:01, ethertype IPv4 (0x0800), length 158: (tos 0x0, ttl 64, id 45956, offset 0, flags [none], proto UDP (17), length 144)
192.168.0.227.4500 > 141.207.197.233.4500: [no cksum] UDP-encap: ESP(spi=0xb6416169,seq=0x4), length 116
05:50:19.036308 PHONE_2_MAC > 01:00:5e:28:00:01, ethertype IPv4 (0x0800), length 158: (tos 0x0, ttl 64, id 45956, offset 0, flags [none], proto UDP (17), length 144)
192.168.0.227.4500 > 141.207.197.233.4500: [no cksum] UDP-encap: ESP(spi=0xb6416169,seq=0x4), length 116
05:50:19.036421 PHONE_2_MAC > 01:00:5e:28:00:01, ethertype IPv4 (0x0800), length 158: (tos 0x0, ttl 64, id 45956, offset 0, flags [none], proto UDP (17), length 144)
192.168.0.227.4500 > 141.207.197.233.4500: [no cksum] UDP-encap: ESP(spi=0xb6416169,seq=0x4), length 116
05:50:19.036425 PHONE_2_MAC > 01:00:5e:28:00:01, ethertype IPv4 (0x0800), length 158: (tos 0x0, ttl 64, id 45956, offset 0, flags [none], proto UDP (17), length 144)
192.168.0.227.4500 > 141.207.197.233.4500: [no cksum] UDP-encap: ESP(spi=0xb6416169,seq=0x4), length 116
05:50:19.036527 PHONE_2_MAC > 01:00:5e:28:00:01, ethertype IPv4 (0x0800), length 158: (tos 0x0, ttl 64, id 45956, offset 0, flags [none], proto UDP (17), length 144)
192.168.0.227.4500 > 141.207.197.233.4500: [no cksum] UDP-encap: ESP(spi=0xb6416169,seq=0x4), length 116
05:50:19.036530 PHONE_2_MAC > 01:00:5e:28:00:01, ethertype IPv4 (0x0800), length 158: (tos 0x0, ttl 64, id 45956, offset 0, flags [none], proto UDP (17), length 144)
192.168.0.227.4500 > 141.207.197.233.4500: [no cksum] UDP-encap: ESP(spi=0xb6416169,seq=0x4), length 116
05:50:19.036727 PHONE_2_MAC > 01:00:5e:28:00:01, ethertype IPv4 (0x0800), length 158: (tos 0x0, ttl 64, id 45956, offset 0, flags [none], proto UDP (17), length 144)
192.168.0.227.4500 > 141.207.197.233.4500: [no cksum] UDP-encap: ESP(spi=0xb6416169,seq=0x4), length 116
05:50:19.036730 PHONE_2_MAC > 01:00:5e:28:00:01, ethertype IPv4 (0x0800), length 158: (tos 0x0, ttl 64, id 45956, offset 0, flags [none], proto UDP (17), length 144)
192.168.0.227.4500 > 141.207.197.233.4500: [no cksum] UDP-encap: ESP(spi=0xb6416169,seq=0x4), length 116
05:50:19.036832 PHONE_2_MAC > 01:00:5e:28:00:01, ethertype IPv4 (0x0800), length 158: (tos 0x0, ttl 64, id 45956, offset 0, flags [none], proto UDP (17), length 144)
192.168.0.227.4500 > 141.207.197.233.4500: [no cksum] UDP-encap: ESP(spi=0xb6416169,seq=0x4), length 116
05:50:19.036835 PHONE_2_MAC > 01:00:5e:28:00:01, ethertype IPv4 (0x0800), length 158: (tos 0x0, ttl 64, id 45956, offset 0, flags [none], proto UDP (17), length 144)
192.168.0.227.4500 > 141.207.197.233.4500: [no cksum] UDP-encap: ESP(spi=0xb6416169,seq=0x4), length 116
05:50:19.036936 PHONE_2_MAC > 01:00:5e:28:00:01, ethertype IPv4 (0x0800), length 158: (tos 0x0, ttl 64, id 45956, offset 0, flags [none], proto UDP (17), length 144)
192.168.0.227.4500 > 141.207.197.233.4500: [no cksum] UDP-encap: ESP(spi=0xb6416169,seq=0x4), length 116
05:50:19.036939 PHONE_2_MAC > 01:00:5e:28:00:01, ethertype IPv4 (0x0800), length 158: (tos 0x0, ttl 64, id 45956, offset 0, flags [none], proto UDP (17), length 144)
192.168.0.227.4500 > 141.207.197.233.4500: [no cksum] UDP-encap: ESP(spi=0xb6416169,seq=0x4), length 116
05:50:19.037137 PHONE_2_MAC > 01:00:5e:28:00:01, ethertype IPv4 (0x0800), length 158: (tos 0x0, ttl 64, id 45956, offset 0, flags [none], proto UDP (17), length 144)
192.168.0.227.4500 > 141.207.197.233.4500: [no cksum] UDP-encap: ESP(spi=0xb6416169,seq=0x4), length 116
05:50:19.037140 PHONE_2_MAC > 01:00:5e:28:00:01, ethertype IPv4 (0x0800), length 158: (tos 0x0, ttl 64, id 45956, offset 0, flags [none], proto UDP (17), length 144)
192.168.0.227.4500 > 141.207.197.233.4500: [no cksum] UDP-encap: ESP(spi=0xb6416169,seq=0x4), length 116
05:50:19.037241 PHONE_2_MAC > 01:00:5e:28:00:01, ethertype IPv4 (0x0800), length 158: (tos 0x0, ttl 64, id 45956, offset 0, flags [none], proto UDP (17), length 144)
192.168.0.227.4500 > 141.207.197.233.4500: [no cksum] UDP-encap: ESP(spi=0xb6416169,seq=0x4), length 116
05:50:19.037244 PHONE_2_MAC > 01:00:5e:28:00:01, ethertype IPv4 (0x0800), length 158: (tos 0x0, ttl 64, id 45956, offset 0, flags [none], proto UDP (17), length 144)
192.168.0.227.4500 > 141.207.197.233.4500: [no cksum] UDP-encap: ESP(spi=0xb6416169,seq=0x4), length 116
05:50:19.037346 PHONE_2_MAC > 01:00:5e:28:00:01, ethertype IPv4 (0x0800), length 158: (tos 0x0, ttl 64, id 45956, offset 0, flags [none], proto UDP (17), length 144)
192.168.0.227.4500 > 141.207.197.233.4500: [no cksum] UDP-encap: ESP(spi=0xb6416169,seq=0x4), length 116
05:50:19.037349 PHONE_2_MAC > 01:00:5e:28:00:01, ethertype IPv4 (0x0800), length 158: (tos 0x0, ttl 64, id 45956, offset 0, flags [none], proto UDP (17), length 144)
192.168.0.227.4500 > 141.207.197.233.4500: [no cksum] UDP-encap: ESP(spi=0xb6416169,seq=0x4), length 116
05:50:19.037450 PHONE_2_MAC > 01:00:5e:28:00:01, ethertype IPv4 (0x0800), length 158: (tos 0x0, ttl 64, id 45956, offset 0, flags [none], proto UDP (17), length 144)
192.168.0.227.4500 > 141.207.197.233.4500: [no cksum] UDP-encap: ESP(spi=0xb6416169,seq=0x4), length 116
05:50:19.037551 PHONE_2_MAC > 01:00:5e:28:00:01, ethertype IPv4 (0x0800), length 158: (tos 0x0, ttl 64, id 45956, offset 0, flags [none], proto UDP (17), length 144)
192.168.0.227.4500 > 141.207.197.233.4500: [no cksum] UDP-encap: ESP(spi=0xb6416169,seq=0x4), length 116
05:50:19.037653 PHONE_2_MAC > 01:00:5e:28:00:01, ethertype IPv4 (0x0800), length 158: (tos 0x0, ttl 64, id 45956, offset 0, flags [none], proto UDP (17), length 144)
192.168.0.227.4500 > 141.207.197.233.4500: [no cksum] UDP-encap: ESP(spi=0xb6416169,seq=0x4), length 116
05:50:19.037656 PHONE_2_MAC > 01:00:5e:28:00:01, ethertype IPv4 (0x0800), length 158: (tos 0x0, ttl 64, id 45956, offset 0, flags [none], proto UDP (17), length 144)
192.168.0.227.4500 > 141.207.197.233.4500: [no cksum] UDP-encap: ESP(spi=0xb6416169,seq=0x4), length 116
05:50:19.037757 PHONE_2_MAC > 01:00:5e:28:00:01, ethertype IPv4 (0x0800), length 158: (tos 0x0, ttl 64, id 45956, offset 0, flags [none], proto UDP (17), length 144)
192.168.0.227.4500 > 141.207.197.233.4500: [no cksum] UDP-encap: ESP(spi=0xb6416169,seq=0x4), length 116
05:50:19.037760 PHONE_2_MAC > 01:00:5e:28:00:01, ethertype IPv4 (0x0800), length 158: (tos 0x0, ttl 64, id 45956, offset 0, flags [none], proto UDP (17), length 144)
192.168.0.227.4500 > 141.207.197.233.4500: [no cksum] UDP-encap: ESP(spi=0xb6416169,seq=0x4), length 116
05:50:19.037862 PHONE_2_MAC > 01:00:5e:28:00:01, ethertype IPv4 (0x0800), length 158: (tos 0x0, ttl 64, id 45956, offset 0, flags [none], proto UDP (17), length 144)
192.168.0.227.4500 > 141.207.197.233.4500: [no cksum] UDP-encap: ESP(spi=0xb6416169,seq=0x4), length 116
05:50:19.037865 PHONE_2_MAC > 01:00:5e:28:00:01, ethertype IPv4 (0x0800), length 158: (tos 0x0, ttl 64, id 45956, offset 0, flags [none], proto UDP (17), length 144)
192.168.0.227.4500 > 141.207.197.233.4500: [no cksum] UDP-encap: ESP(spi=0xb6416169,seq=0x4), length 116
05:50:19.037967 PHONE_2_MAC > 01:00:5e:28:00:01, ethertype IPv4 (0x0800), length 158: (tos 0x0, ttl 64, id 45956, offset 0, flags [none], proto UDP (17), length 144)
192.168.0.227.4500 > 141.207.197.233.4500: [no cksum] UDP-encap: ESP(spi=0xb6416169,seq=0x4), length 116
05:50:19.037970 PHONE_2_MAC > 01:00:5e:28:00:01, ethertype IPv4 (0x0800), length 158: (tos 0x0, ttl 64, id 45956, offset 0, flags [none], proto UDP (17), length 144)
192.168.0.227.4500 > 141.207.197.233.4500: [no cksum] UDP-encap: ESP(spi=0xb6416169,seq=0x4), length 116
05:50:19.038167 PHONE_2_MAC > 01:00:5e:28:00:01, ethertype IPv4 (0x0800), length 158: (tos 0x0, ttl 64, id 45956, offset 0, flags [none], proto UDP (17), length 144)
192.168.0.227.4500 > 141.207.197.233.4500: [no cksum] UDP-encap: ESP(spi=0xb6416169,seq=0x4), length 116
05:50:19.038170 PHONE_2_MAC > 01:00:5e:28:00:01, ethertype IPv4 (0x0800), length 158: (tos 0x0, ttl 64, id 45956, offset 0, flags [none], proto UDP (17), length 144)
192.168.0.227.4500 > 141.207.197.233.4500: [no cksum] UDP-encap: ESP(spi=0xb6416169,seq=0x4), length 116
05:50:19.038271 PHONE_2_MAC > 01:00:5e:28:00:01, ethertype IPv4 (0x0800), length 158: (tos 0x0, ttl 64, id 45956, offset 0, flags [none], proto UDP (17), length 144)
192.168.0.227.4500 > 141.207.197.233.4500: [no cksum] UDP-encap: ESP(spi=0xb6416169,seq=0x4), length 116
05:50:19.038275 PHONE_2_MAC > 01:00:5e:28:00:01, ethertype IPv4 (0x0800), length 158: (tos 0x0, ttl 64, id 45956, offset 0, flags [none], proto UDP (17), length 144)
192.168.0.227.4500 > 141.207.197.233.4500: [no cksum] UDP-encap: ESP(spi=0xb6416169,seq=0x4), length 116
05:50:19.038376 PHONE_2_MAC > 01:00:5e:28:00:01, ethertype IPv4 (0x0800), length 158: (tos 0x0, ttl 64, id 45956, offset 0, flags [none], proto UDP (17), length 144)
192.168.0.227.4500 > 141.207.197.233.4500: [no cksum] UDP-encap: ESP(spi=0xb6416169,seq=0x4), length 116
05:50:19.038476 PHONE_2_MAC > 01:00:5e:28:00:01, ethertype IPv4 (0x0800), length 158: (tos 0x0, ttl 64, id 45956, offset 0, flags [none], proto UDP (17), length 144)
192.168.0.227.4500 > 141.207.197.233.4500: [no cksum] UDP-encap: ESP(spi=0xb6416169,seq=0x4), length 116
05:50:19.038579 PHONE_2_MAC > 01:00:5e:28:00:01, ethertype IPv4 (0x0800), length 158: (tos 0x0, ttl 64, id 45956, offset 0, flags [none], proto UDP (17), length 144)
192.168.0.227.4500 > 141.207.197.233.4500: [no cksum] UDP-encap: ESP(spi=0xb6416169,seq=0x4), length 116
05:50:19.038582 PHONE_2_MAC > 01:00:5e:28:00:01, ethertype IPv4 (0x0800), length 158: (tos 0x0, ttl 64, id 45956, offset 0, flags [none], proto UDP (17), length 144)
192.168.0.227.4500 > 141.207.197.233.4500: [no cksum] UDP-encap: ESP(spi=0xb6416169,seq=0x4), length 116
05:50:19.038683 PHONE_2_MAC > 01:00:5e:28:00:01, ethertype IPv4 (0x0800), length 158: (tos 0x0, ttl 64, id 45956, offset 0, flags [none], proto UDP (17), length 144)
192.168.0.227.4500 > 141.207.197.233.4500: [no cksum] UDP-encap: ESP(spi=0xb6416169,seq=0x4), length 116
05:50:19.038686 PHONE_2_MAC > 01:00:5e:28:00:01, ethertype IPv4 (0x0800), length 158: (tos 0x0, ttl 64, id 45956, offset 0, flags [none], proto UDP (17), length 144)
192.168.0.227.4500 > 141.207.197.233.4500: [no cksum] UDP-encap: ESP(spi=0xb6416169,seq=0x4), length 116
05:50:19.038787 PHONE_2_MAC > 01:00:5e:28:00:01, ethertype IPv4 (0x0800), length 158: (tos 0x0, ttl 64, id 45956, offset 0, flags [none], proto UDP (17), length 144)
192.168.0.227.4500 > 141.207.197.233.4500: [no cksum] UDP-encap: ESP(spi=0xb6416169,seq=0x4), length 116
05:50:19.038791 PHONE_2_MAC > 01:00:5e:28:00:01, ethertype IPv4 (0x0800), length 158: (tos 0x0, ttl 64, id 45956, offset 0, flags [none], proto UDP (17), length 144)
192.168.0.227.4500 > 141.207.197.233.4500: [no cksum] UDP-encap: ESP(spi=0xb6416169,seq=0x4), length 116
05:50:19.038892 PHONE_2_MAC > 01:00:5e:28:00:01, ethertype IPv4 (0x0800), length 158: (tos 0x0, ttl 64, id 45956, offset 0, flags [none], proto UDP (17), length 144)
192.168.0.227.4500 > 141.207.197.233.4500: [no cksum] UDP-encap: ESP(spi=0xb6416169,seq=0x4), length 116
05:50:19.038895 PHONE_2_MAC > 01:00:5e:28:00:01, ethertype IPv4 (0x0800), length 158: (tos 0x0, ttl 64, id 45956, offset 0, flags [none], proto UDP (17), length 144)
192.168.0.227.4500 > 141.207.197.233.4500: [no cksum] UDP-encap: ESP(spi=0xb6416169,seq=0x4), length 116
05:50:19.039094 PHONE_2_MAC > 01:00:5e:28:00:01, ethertype IPv4 (0x0800), length 158: (tos 0x0, ttl 64, id 45956, offset 0, flags [none], proto UDP (17), length 144)
192.168.0.227.4500 > 141.207.197.233.4500: [no cksum] UDP-encap: ESP(spi=0xb6416169,seq=0x4), length 116
05:50:19.039098 PHONE_2_MAC > 01:00:5e:28:00:01, ethertype IPv4 (0x0800), length 158: (tos 0x0, ttl 64, id 45956, offset 0, flags [none], proto UDP (17), length 144)
192.168.0.227.4500 > 141.207.197.233.4500: [no cksum] UDP-encap: ESP(spi=0xb6416169,seq=0x4), length 116
05:50:19.039199 PHONE_2_MAC > 01:00:5e:28:00:01, ethertype IPv4 (0x0800), length 158: (tos 0x0, ttl 64, id 45956, offset 0, flags [none], proto UDP (17), length 144)
192.168.0.227.4500 > 141.207.197.233.4500: [no cksum] UDP-encap: ESP(spi=0xb6416169,seq=0x4), length 116
05:50:19.039202 PHONE_2_MAC > 01:00:5e:28:00:01, ethertype IPv4 (0x0800), length 158: (tos 0x0, ttl 64, id 45956, offset 0, flags [none], proto UDP (17), length 144)
192.168.0.227.4500 > 141.207.197.233.4500: [no cksum] UDP-encap: ESP(spi=0xb6416169,seq=0x4), length 116
05:50:19.039304 PHONE_2_MAC > 01:00:5e:28:00:01, ethertype IPv4 (0x0800), length 158: (tos 0x0, ttl 64, id 45956, offset 0, flags [none], proto UDP (17), length 144)
192.168.0.227.4500 > 141.207.197.233.4500: [no cksum] UDP-encap: ESP(spi=0xb6416169,seq=0x4), length 116
05:50:19.039307 PHONE_2_MAC > 01:00:5e:28:00:01, ethertype IPv4 (0x0800), length 158: (tos 0x0, ttl 64, id 45956, offset 0, flags [none], proto UDP (17), length 144)
192.168.0.227.4500 > 141.207.197.233.4500: [no cksum] UDP-encap: ESP(spi=0xb6416169,seq=0x4), length 116
05:50:19.039408 PHONE_2_MAC > 01:00:5e:28:00:01, ethertype IPv4 (0x0800), length 158: (tos 0x0, ttl 64, id 45956, offset 0, flags [none], proto UDP (17), length 144)
192.168.0.227.4500 > 141.207.197.233.4500: [no cksum] UDP-encap: ESP(spi=0xb6416169,seq=0x4), length 116
05:50:19.039508 PHONE_2_MAC > 01:00:5e:28:00:01, ethertype IPv4 (0x0800), length 158: (tos 0x0, ttl 64, id 45956, offset 0, flags [none], proto UDP (17), length 144)
192.168.0.227.4500 > 141.207.197.233.4500: [no cksum] UDP-encap: ESP(spi=0xb6416169,seq=0x4), length 116
05:50:19.039609 PHONE_2_MAC > 01:00:5e:28:00:01, ethertype IPv4 (0x0800), length 158: (tos 0x0, ttl 64, id 45956, offset 0, flags [none], proto UDP (17), length 144)
192.168.0.227.4500 > 141.207.197.233.4500: [no cksum] UDP-encap: ESP(spi=0xb6416169,seq=0x4), length 116
05:50:19.039613 PHONE_2_MAC > 01:00:5e:28:00:01, ethertype IPv4 (0x0800), length 158: (tos 0x0, ttl 64, id 45956, offset 0, flags [none], proto UDP (17), length 144)
192.168.0.227.4500 > 141.207.197.233.4500: [no cksum] UDP-encap: ESP(spi=0xb6416169,seq=0x4), length 116
05:50:19.039714 PHONE_2_MAC > 01:00:5e:28:00:01, ethertype IPv4 (0x0800), length 158: (tos 0x0, ttl 64, id 45956, offset 0, flags [none], proto UDP (17), length 144)
192.168.0.227.4500 > 141.207.197.233.4500: [no cksum] UDP-encap: ESP(spi=0xb6416169,seq=0x4), length 116
05:50:19.039717 PHONE_2_MAC > 01:00:5e:28:00:01, ethertype IPv4 (0x0800), length 158: (tos 0x0, ttl 64, id 45956, offset 0, flags [none], proto UDP (17), length 144)
192.168.0.227.4500 > 141.207.197.233.4500: [no cksum] UDP-encap: ESP(spi=0xb6416169,seq=0x4), length 116
05:50:19.039820 PHONE_2_MAC > 01:00:5e:28:00:01, ethertype IPv4 (0x0800), length 158: (tos 0x0, ttl 64, id 45956, offset 0, flags [none], proto UDP (17), length 144)
192.168.0.227.4500 > 141.207.197.233.4500: [no cksum] UDP-encap: ESP(spi=0xb6416169,seq=0x4), length 116
05:50:19.039823 PHONE_2_MAC > 01:00:5e:28:00:01, ethertype IPv4 (0x0800), length 158: (tos 0x0, ttl 64, id 45956, offset 0, flags [none], proto UDP (17), length 144)
192.168.0.227.4500 > 141.207.197.233.4500: [no cksum] UDP-encap: ESP(spi=0xb6416169,seq=0x4), length 116
05:50:19.039925 PHONE_2_MAC > 01:00:5e:28:00:01, ethertype IPv4 (0x0800), length 158: (tos 0x0, ttl 64, id 45956, offset 0, flags [none], proto UDP (17), length 144)
192.168.0.227.4500 > 141.207.197.233.4500: [no cksum] UDP-encap: ESP(spi=0xb6416169,seq=0x4), length 116
05:50:19.040025 PHONE_2_MAC > 01:00:5e:28:00:01, ethertype IPv4 (0x0800), length 158: (tos 0x0, ttl 64, id 45956, offset 0, flags [none], proto UDP (17), length 144)
192.168.0.227.4500 > 141.207.197.233.4500: [no cksum] UDP-encap: ESP(spi=0xb6416169,seq=0x4), length 116
05:50:19.040125 PHONE_2_MAC > 01:00:5e:28:00:01, ethertype IPv4 (0x0800), length 158: (tos 0x0, ttl 64, id 45956, offset 0, flags [none], proto UDP (17), length 144)
192.168.0.227.4500 > 141.207.197.233.4500: [no cksum] UDP-encap: ESP(spi=0xb6416169,seq=0x4), length 116
05:50:19.040128 PHONE_2_MAC > 01:00:5e:28:00:01, ethertype IPv4 (0x0800), length 158: (tos 0x0, ttl 64, id 45956, offset 0, flags [none], proto UDP (17), length 144)
192.168.0.227.4500 > 141.207.197.233.4500: [no cksum] UDP-encap: ESP(spi=0xb6416169,seq=0x4), length 116
-
NB There isn't any option like "Preserve IKE Port for Pass Through Connections" on the Verizon Gateway or equivalent. The Gateway is from Verizon, and it's their Wifi calling, so you'd think they'd have that figured out. Besides, Wifi calling does actually work, it's just a packet storm shows up now and again!
-
Further findings:
- I can not reproduce the problem with 1 or 2 Eeros. But easy to reproduce with 3.
- Whilst the problem is live, I can see it in Firewalla live data rate of ~18mbit/s. Meanwhile the SysInfo app on my phone, which Firewalla reports is sending 18mbit/s, shows close to zero wifi usage. The total data sent since boot on wifi shown in the phone (SysInfo app again) does not increase enough for the amount of bits sent, or for the flow data usage reported afterwards in firewalla. This suggests to me that the phone is not sending all these packets, but perhaps it sends one and Eero then keeps repeating it.
This leads me to suspect that Eero is the culprit here. I wonder if there is a bug in their routing/spanning tree between using wired and wireless connection between the Eeros with multicast packets. Again I confirmed Eero reports using wired connections and used an ethernet cable tester to be sure no obvious problems.
Why I've not seen this for the last year I don't know, but I wonder if it's triggered by the fact Eero can now see the Verizon Router and it gets IPV6 (since Firewalla now in bridge mode and not router mode), as per my linked comment earlier. Just a wild guess, but not much else has changed. There's no IPV6 in any of these problem packets though.
I should probably contact Eero support, but this problem sounds pretty deep to present to a consumer product support. Might just leave Wifi calling off, or see if I can make do with 1 less Eero, assuming that continues to avoid the issue.
-
Main Eero -> switch -> other Eeros, as per original posting. Note there are other wired devices both before and after that main Eero.
One more update: Even with 3 Eeros, if I disconnect the Ethernet from the secondary Eeros, the problem does not seem to be reproducible, even testing with the phone using each of the 3 Eeros.
-
What’s also interesting is that it can take several attempts of enabling Wi-Fi calling on a device to trigger this. So it’s not reproducible every time. I have packet dumps where I see one packet with the multicast Ethernet adresss but then no ensuing storm.
I speculate there is some kind of race condition going on somewhere.
-
Further findings:
- Need more than one iPhone (or presumably other Apple device) on the network attempting to talk to the Verizon Wifi calling server. I can not reproduce the problem if only one device has wifi calling turned on. The MAC multicast packet is still transmitted but no storm results.
My current theory:
- The problem started happening recently. The Verizon 5G gateway firmware had a major upgrade recent.
- Needs multiple devices. It's well known that Wifi calling needs IPSEC support in NAT to work. Without it Wifi calling won't work at all.
- The packets storms are on port 4500, which is IPSEC NAT traversal.
- I speculate that something has changed (or is broken) in the Verizon gateway firmware IPSEC NAT handling. This triggers something different in the behavior of the phone/Eero which then somehow triggers Eero to produce a packet storm (remember it does have wired and wireless connections between units).
This is obviously speculation but it sort of fits all the facts I've work through above. Verizon would likely tell me I'm not supposed to use someone else's Wifi, and Eero would likely tell me they don't support use with 5G gateways.
It also looks like Verizon Wifi calling will only use IPV4 (at least through their 5G gateway), just blocking IPV4 address range blocks Wif calling completely, no switch to IPV6 takes place.
So the solution I am going to stick with is to block wifi calling in Firewalla. Block all traffic to UDP port 4500 will disable all 3 US networks Wifi calling. I can live without it. A bonus is that once the phone learns that port 4500 doesn't work, it doesn't keep trying too often.
-
Beta tester with eero since 2017. Your topology is wrong... it seems like it shouldn't be, but it is with eeros...
Verizon 5G Home Internet Gateway --> Firewalla Purple --> Primary Eero --> Switches --> & wired devices & Secondary Eeros + wired devices
Is the ONLY thing that will work. Again, for those of us with networking backgrounds it doesn't SEEM like it will cause an issue, but does and has been documented for over 6 years. I run a similar topology with a FWG+ to you in place of your purple. You WILL have problems if you don't have that eero in bridge mode ahead of that first switch. -
Note I do already have the secondary Eero's 'downstream' of the Primary one already, i.e. connected to the 2nd port on the Primary Eero which is what everyone is usually pointing out is required. The difference between your topology and mine is I currently have some additional wired devices between Firewalla and the first Eero, although none of them are obviously taking part in the packet storm as far as I could see.
Anyway, I will test this and see what happens.
-
Yeah, I was thinking that too. I'll see if I can test without purple in place, the catch is then it's hard to observe anything other than internet becomes unusably slow from wifi devices. Removing purple isn't part of any solution I want anyway, better just to disable wifi calling by blocking it.
I took many tcpdump logs (no filter), and did some stuff with sed/sort/uniq to look see if there were many duplicate packets with the same 'seconds' in the time stamp, didn't find anything too alarming, so it's just this Wifi calling use. Don't have any other IPSEC use though to test that.
When I had purple in router mode I never hit this problem, but that was also with older Verizon firmware - so two variables. I will try purple in router mode again sometime and see what happens, but again I'd rather just disable the wifi calling and be done, since except for this one snag everything is working just nicely, and getting things to work just nicely with that Verizon box has been some work...
-
One thing for example which would be different between FW in router mode and bridge mode, is that there is VLAN10 STP packets coming from the Verizon box. I don't have any VLANs configured, so that's a little interesting. I wonder if something like that could be having some impact on Eero.
22:41:53.406314 4c:ab:f8:94:6c:70 > 01:80:c2:00:00:00, ethertype 802.1Q (0x8100), length 60: vlan 10, p 0, LLC, dsap STP (0x42) Individual, ssap STP (0x42) Command, ctrl 0x03: STP 802.1d, Config, Flags [none], bridge-id 7fff.4c:ab:f8:94:6c:70.8001, length 39
Also stuff like this is coming out of the Verizon box:
22:41:52.986490 4c:ab:f8:94:6c:70 > 01:80:c2:00:00:13, ethertype IEEE1905.1 (0x893a), length 247:
IEEE19501 appears to be related to load balancing over concurrent connections.
I guess this is happening because the Verizon box is designed to work with their wifi mesh solution.
I wonder if this stuff is interfering with Eero in some way. None of it would make it to Eero with FW in router mode.
This looks to be the only multicast ethernet traffic coming out of the Verizon box. How can I stop Purple's bridge from forwarding to the LAN port, assuming it is doing that? Even temporarily with some linux commands so I can test.
-
I did a quick test of "sudo ifconfig br0 -multicast". I thought I got lucky at first but I still managed to see the packet storm. Actually that means I have a misunderstanding as I still captured 200k plus ethernet multicast packets on an interface which showed "br0: flags=4163<UP,BROADCAST,RUNNING> mtu 1500.
-
"sudo ifconfig eth0 -multicast" as expected stops any multicast showing up from the Verizon box. I was able to enable wifi calling 20 times in a row, without any packet storm. That's previously unheard of.
Turning off all multicast on eth0 sounds a little drastic, I don't know if something else might break. Question for 'firewalla', I'd like to block the IEEE1905.1 traffic coming from eth0 as my next test. (a) what commands to use (b) if it works which file to put that in so it runs each time firewalla boots up.
-
You may try with this.
sudo ebtables -A FORWARD -i eth0 -p 0x893a -j DROP
ethertype: 0x893a ==> IEEE1905.1
Disclaimer: It is untested.
If it works, you may add the script to /home/pi/.firewalla/config/post_main.d/ , it will auto launch when firewalla boots up.
More info here: https://help.firewalla.com/hc/en-us/articles/360054056754-Customized-Scripting-
-
This doesn't work alone. Of course the earlier trick of blocking all multicast might not stop the packet storm internally, but it does stop it going through the bridge (and being counted therefore I assume) and stops it reaching the Verizon box.
I have some further experiments to try yet.
-
The problem is reproducible without Firewalla with this arrangement. 5G Gateway --> Eero (bridge mode) --> Switches & more Eeros. Set a 0.5s interval ping going on the phone (Net Analyzer app), wait to check its not dropping anything, force wifi calling to come on from the phone settings, the pings mostly get dropped with maybe 1 in 10 getting through.
It also maybe looks like the phone needs to be reported by the Eero app as using a non-primary Eero at the time. I say maybe as I'm not 100% sure about this, but it does seem that I can trigger it every time that way.
The problem is not reproducible connecting one phone directly to the 5G Gateway wifi.
I've checked that all switches are dumb (Netgear basic GS108 model), no ethernet loops, and used a cable tester on all patch cables and through house wiring.
Clearly my next stop would be Eero support, but thanks to having Firewalla I can block port 4500 and workaround the issue, and can capture packet dumps to know enough to find that issue.
-
Eero ticket has not really progressed but they have recorded it. But I found something else which is interesting:
If on Firewalla I add an allow rule for "traffic from internet, bidirectional, to/from wo.vzwwo.com", that is to permit incoming traffic from the wifi calling server, then this seems to avoid the packet storms, and wifi calling works on multiple devices (all are Verizon). Note that the 5G gateway also has its firewall turned off. This arrangement has been working for several days. My guess, but I've not had the time to debug in any detail, is that the setup sequence between the server and the devices differs a little, since some additional 'return' packets get through, and the handshake somehow avoids the failure case. Note also that when I tested earlier without firewalla present at all (see earlier comment), the 5G gateway firewall was switched back on first. Multiple variables but a better solution, and just posting it here in case it should ever help anyone else, unlikely as that seems.
Please sign in to leave a comment.
Comments
24 comments