Purple major issues since the Jun 6th update?
I have one customer using the purple and it has been completely unstable since Jun 6th, last internet test was completed at Jun 6 at 2:40 am and the firmware was updated at 7:40 and for the last 5 days it has not been able to complete a test, the log shows nothing but high packet losses and occasionally ISP drops. Customer isn't happy and while I'd like to think this is an ISP problem the coincidence that you guys pushed an update right when this started doesn't bode well for this fw release. Now I have to make a 2 hour drive to replace this device and as I recall the last purple I installed for another customer had to be replaced too but I've never had issues with the Gold versions I have installed. Is there a known issue being worked on as I type or will I have make this long drive which will waste an entire day?
-
I have two purples and they are both experiencing the same network performance issue. My gold and gold plus devices are all fine.
In one case I wiped the box and started with a fresh configuration and it did not fix the issue.
I have confirmed that my ISP is not a problem by using a Gold device - it shows normal performance. When the Purple is connected performance is horrible.
There appears to be a software issue with the latest release. I have tried to use the Beta version but it has not resolved it. Can we get an update on when this will be resolved with a new release or get the option to go back?
-
@Jen, did I? Maybe it was your comment? But then again, your reply and other post proved my point and yes, tech is a hobby, and I didn't make it to this level by resetting everything and starting over. When I started my first tech job 26 years ago, that was the answer to everything on the help desk... just format and start over because I have no clue how to actually resolve your issue.
FWIW, my ISP has been flawless going on 3 years now. I can't even remember any downtime that ever impacted me and, my network along with everything connected is solid as a rock. Since I've never had an issue with my FW, I didn't even start looking there until I checked everything else first. Overheating? really? This many people at the same time? I stand by my statement.
This is only one thread of many... Anyway, I'm not here to waste more time.
I opened a ticket as well and now they are asking me to take pictures and do other waste of time activities. I'm really shocked that this is what I have to look forward to working with! This has not been edited!
"Can you please upload several screenshoot about the previous and now testing result on latency and upstream speed. We need your cooperation to do troubleshooting. Whether you have tested your IPS and internet or not.
Here is a guide : Network Performance and Quality Monitoring, which can provide you a favor." -
Few thoughts on this:
Don't really believe it to be hardware/chipset - would have manifested earlier. I have had my box a while with no issues and this seems to be a 'point-in-time' issue across a number of devices.
Do think it's only manifesting against particular Purple/ISP equipment combinations and the aggressiveness of the policing policy of the ISP on the upstream link. There are going to be combos across the globe that Firewalla would be unable to test in development in their labs before a software roll-out. In Australia where I am located, my NBN 'broadband link' is an asymmetric 50Mbits (d) /20Mbits (u) to the HFC modem. The modem normally connects to the Firewalla WAN NIC operation directly at 1Gbit/s. At these NIC speeds, the downstream direction traffic to the Firewalla can never be oversubscribed, but in the upstream direction with a 1Gbit/s NIC pushing data into a 20Mbit/s link, if the Firewalla does not shape the traffic flow properly and in particular, bursty traffic, there is a real danger of oversubscribing the ISPs link. This will result in packet loss; how much will depend on your ISP policing/ISP equipment. You'll note that it's the Firewalla making all the decisions on how to deal with the upstream link as you are not required to configure your device to tell it what bandwidth of internet service you have. Setting the WAN interface to 100Mbit/s is forcing the Firewalla to shape the traffic more aggressively which will more likely fit within the ISPs allowed parameters. BTW your standard ISP Service Desk person has limited diagnostics, to them they will just see that the link to their device is running at its contracted speeds and looks fine.
Don't know what others have in terms of internet connections but for someone like Jen Fox who practically has a line speed 1GBit/s fibre internet connection this type of problem would never appear. A link like that can't be oversubscripted by a Purple with a 1Gbit/s WAN interface. But within this thread I am picking up an number of possible 'regional' issues like mine - and would like to know what sort of internet services others have.
You'll note that in the last release notes, Firewalla have been tinkering with the Purple to implement a CAKE function which would be linked with the management of traffic flows in the upstream direction. I don't really care whether it's only really enabled as an 'early access' feature; it's a sign of code changes in the upstream traffic flow. I am still running on June 6th code and not using early access features.
@Firewalla - Given the wide diversity of regional (i.e. global) ISP set-ups and equipments with which your devices need to operate and which you can't possibly have access to for testing, this is surely another reason to offer your customers a rollback feature even if it's a manual install option done via your Support Desk.
-
Hi all. I had been using the USB port on my router to power my firewalla unit and had experiencing issues just after the latest update. I switched over to a power supply and the packet loss issues seemed to have disappeared. So not sure if that may be related or things just started to work. So not sure if that helps to try out as well and see
-
I too am using the USB for power but haven't switched it. I had to leave Friday afternoon for a weekend trip with the family and just got home a couple hours ago (Sunday), and my network appears to be a bit more stable.
The latest speed test from the FW is better than it was before I left, but not even close to what it has been since I installed the FW just over a year ago. I still have an open ticket and now that I'm back for a few days, hopefully I can get a resolution and root cause.
-
I’ve had a reply from the support team and they want me to send my Purple back to San Jose for investigation.
This is going to be expensive (I’m in the UK) and seems excessive for what’s clearly a software problem. Before I reply, has anybody had an alternative solution from the support team?
-
Hi @Nathan Courtney, so far the work around proposed by @Wolfgang Hollaus has worked for me:
- SSH login to the Firewalla box
- Set network ports to 100 MBit/s
pi@Firewalla:~ (Firewalla) $ sudo ethtool -s eth1 speed 100 duplex full autoneg on
pi@Firewalla:~ (Firewalla) $ sudo ethtool -s eth0 speed 100 duplex full autoneg onfor me it works because I have 100mbps
firewalla support connected to my Purple but were unable to find anything meaninful
-
@Firewalla, USB power is the only option I have, so yes, it's very stable. Everything in my office is designed to stay online should the power go out. So far, I've made it over an hour until power was restored. None of my equipment experiences a blip and is protected from power spikes.
Besides, thousands of purple owners having identical issues in the same month as an update, is not a coincidence. I want to know where our support is coming from before I let anyone on my FW without me present.
-
@Robert Obando
I can confirm that this mitigates the issue for me. It's repeatable:
1. Reboot Purple - around 25% packet loss.
2. sudo ethtool -s eth0 speed 100 duplex full autoneg on (I found I only needed to do the WAN port) - packet loss gone.
This isn't ideal because my internet connection is more than 100Mb, which is why I bought a 1Gb firewall. Support are shipping me a replacement firewall, which is very generous of them. When it arrives I'm going to set it up from scratch and see how it performs because I remain sceptical this is a hardware issue. -
@Firewalla: It would be very good to know what the next step is now! Is anyone working on a solution? When will this solution be ready? It has been 3 weeks since we started having these problems.
In the meantime I received some newsletters that I should buy the "Firewalla Gold SE". That's nice. But not exactly the solution now ;) -
@wolfang, if you are having issues, the best way is to contact help@firewalla.com with the details. We can help you there.
-
I received my replacement Purple today. I've set it up as a fresh install and it's been working perfectly for the last couple of hours.
I'm still not convinced the problems aren't related to the software patch on the 6th June as the timings line up too well, but I can't really pester support any more as the replacement hardware has fixed the issue. -
Mine just showed up and sitting on my desk. I'm very disappointed in everything I've found out about this company and garbage support. They know they are at fault and refuse to admit it, for now.
I will be looking for alternate hardware that preferably doesn't force updates with no way to roll back on our own.
-
My new hardware has not arrived yet. So I can't say if it is solved now. The problem also exists in other (open source) projects that have the same LAN connector/driver and are based on ARM. With Intel there doesn't seem to be these problems (Firewalla Gold...).
So it will not be Firewalla specific. From there already a thank you to Firewalla that they have exchanged the hardware free of charge.
-
I'm facing the same issue, I have been noticing the network performance was degraded and this weekend I lost control of my entire home when I was traveling, my cameras went offline.
Tried all the troubleshoot steps and clearly the problem was the FWP, thanks to the proposed workaround provided by @wolfgang I was able to once again have internet access and smart home controls working again.
Obviously this is not a solution because I'm going from 600mbs to 100mbs.
I also have a ticket with all the logs and screenshots, I hope to hear something soon.
-
More ppl reporting the issue all over the place on Reddit, not sure if is exactly the same issue but if you see the posts most of them started around a month ago or so.
https://www.reddit.com/r/firewalla/comments/14jdjj6/firewalla_purple_recall/
Please sign in to leave a comment.
Comments
55 comments