Overlapping LAN subnet on WAN network after WAN failure

Comments

10 comments

  • Avatar
    Firewalla

    Check your modem and see what it is giving to your WAN side. After, also check if you have another router upstream (in case you haven't turned off the previous one) that may running in parallel.

    0
    Comment actions Permalink
  • Avatar
    DB

    Almost sounds like he accidently plugged the WRONG cable into the WAN port. Can easily happen if they are the same color.

    But then, it could be something else.

    0
    Comment actions Permalink
  • Avatar
    Bilal

    Thanks for your responses.
    The WAN cable I was working on (which is normally working) was on the modem end to a wall socket (which then goes off to another cabinet where the firewalla box is), so there was no possibility of confusion.
    The overlapping subnet is the same one that the firewalla box is supplying on one of the LAN networks.
    There are no other routers in the network.
    Perhaps I could try to replicate the issue by simply unplugging the uplink and connecting it again.
    Last time I did it, the active light on the GPON box didn’t light up until way later, even though the Firewalla reported receiving a DHCP address (but was failing ping tests, etc, and then got all confused with subnets).
    Sorry if this comment is all messed up. Typing from a mobile and between a couple of things.

    0
    Comment actions Permalink
  • Avatar
    DB

    Do you have mDNS running? (just wondering). 
    Based on your explanation, I would check the cables in your environment to ensure they are CAT6 quality, I have a cable TDR tester. Then I'd recheck the termination quality inside your boxes.
    Only after the physical layer checks out, I would address the modem ability to properly work after disconnects by using a laptop.
    Then I would proceed into making a paper map of what the network is logically supposed to be vs. the physical layout. I've seen IT install network loops by accident and even with STP the network would crash.
    Finally, if all checks out, start with the logical setup of each network.
    Somewhere something in your network is amiss.
    I have a new Gold+ and disconnected the WAN port a few times but it came up quickly each time, although I'm not running a Vlan right now.
    Sorry, you don't get a quick easy answer 🙁, even as a professional IT networking guy with 35 years experience I need to be methodical on problems you describe.
    Love to hear how you fix it.

    0
    Comment actions Permalink
  • Avatar
    Bilal

    Hi DB,

    Appreciate your input.

    Before I respond to your suggestions: I just tested a few unplug / replugs, and the app or system hasn't exhibited the same behaviour of a few days ago. **helpless shrug**. Was watching dhclient.log and all seemed to be in order. Took about 30s from an igc Link up message in syslog to a "bound to" dhclient message. The app took a bit longer - around a minute or two to demonstrate the link being back up. It's a busy little box!

    RE mDNS: I have the reflector switched off on both LANs. I don't want cross-network stuff happening.

    RE cable health check, don't think I will be able to get my hands on a proper TDR. The most I could probably do is use one of the regular Fluke cable length, continuity and connectivity tester. Might do that tomorrow if I can borrow the Fluke from work.

    The network is very simple: GPON box -> Firewalla WAN -> Two Ethernet ports, one for each LAN, which includes a few physically conected clients (downstream unmanaged switches), a couple of wifi mesh routers each, all in AP mode.

    I wouldn't rule out the wifi mesh routers doing some funny business, although they shouldn't...

    The subnet numbers I've chosen are "suprise me" ones (paraphrasing), they're unlikely to be anything that other services would ever provide, including the mesh APs.

    Thanks to all who have attempted to assist. I'll try to report any further findings I might come across.

    Regards,
    Bilal

     

    0
    Comment actions Permalink
  • Avatar
    Bilal

    Hi Everyone,

    I was having issues with on of my mesh APs the other day and discovered that the LANs were crossed, so potentially DHCP leakage could occur between the LAN networks. D'oH. Kudos to DB's recommendation to check (all!) the topology.

    Things are much more stable now.

    Still doesn't really explain how or why the WAN link could've suffered, though. I'm sure it's related to this crossing of LANs. Makes me worry if the box is capable of being resistant against rogue DHCP server attacks!

    Regards,
    Bilal

     

    0
    Comment actions Permalink
  • Avatar
    Firewalla

    What do you mean by LAN's are crossed? is it physically done by having them on two networks? if it is, likely you will get storms or STP issues ... (and DHCP)

    0
    Comment actions Permalink
  • Avatar
    Bilal

    @Firewall:

    This is the target configuration:

    Firewalla Box
    - Eth1: LAN1 <-- Eth/Switch --> Mesh1 AP1 <-- Eth --> Mesh1 AP2
    - Eth2: LAN2 <-- Eth --> Mesh2 AP1 <-- WiFi --> Mesh2 AP2 <-- WiFi --> Mesh2 AP3
    - Eth4: <-- Eth --> WAN/ISP

    LAN1 and LAN2 have, let's say 192.168.21.0/24 and 192.168.22.0/24, respectively. These aren't the real ones, but just an example.

    At some point, I would've incorrectly hooked up either LAN2's first Mesh2 AP into the switch for LAN1, OR the other way around, resulting in LAN1 and LAN2 being connected to each other.

    No messing about with Eth4, ever. That was a simple ISP NTU connection.

    The result was erratic / inconsistent behaviour on the WAN port, getting an IP address from Firewalla's own DHCP server.

    I could imagine LAN1 and LAN2 clients tripping over, but not the WAN interface being compromised by DHCP packets from elsewhere.

    Thanks

    0
    Comment actions Permalink
  • Avatar
    Firewalla

    If you are crossing networks, it is highly the behavior is unknown. (or highly unpredictable) I don't think Firewalla can change the WAN side, but if it does, I'd suggest you check the circuits.

     

    0
    Comment actions Permalink
  • Avatar
    Bilal

    It was a terrible / silly mistake, but I think it exposed a weakness or problem in the system - that the app and / or the router couldn’t handle.
    Granted - it’s complicated.
    This isn’t even a rogue DHCP issue. The server (itself) would be trusted, but NOT on its WAN interface, especially when nothing is physically affecting that side.

    One day when I have time, I hope to see if I can confuse the system again, but more likely I probably won’t be getting around it. Always more important things to do.

    0
    Comment actions Permalink

Please sign in to leave a comment.