Log files?
Hello,
does "Firewalla Gold" logs it's actions, etc. in log files on the file system.
I would like to collect them and send it to a Splunk system for monitoring, analysis.
-
After a few early hiccups, I'm pretty impressed with how the Firewalla Gold is running, THANK YOU! I'd like to add my name to the list of who would like to see events (alarms, blocks, flows, etc...) written to logs on the file system for forwarding to ES, Splunk, etc... Having the ability to send that data elsewhere would improve flexibility for those of us who would like to implement different levels of alerting and monitoring.
Along the same lines, anything blocked should be notified somewhere. I personally don't like the options of anything deemed really bad (by someone?) being silently blocked or monitoring turned off altogether.
Thanks!
-
Wouldn't having the log files (and different log levels selectable) make chasing down customer issues easier and more efficient? As an example, the IGMP issue that apparently is seen with the Blue beta release, but not the production release (even though no changes were made in the IGMP-related code) might be tracked down with a debug-level log. Normal requests, such as "what rule caused this site to be blocked?" would be easily seen with normal-level logs.
Just my $0.02.
-Bob
-
In case anyone is still having issues with this, here's how I'm sending my Firewalla Gold Zeek logs to a remote syslog server (at least until the functionality exists within Firewalla). Disclaimer: I make no claims that I know WTF I am doing. Do this or similar at your own risk. I welcome comments/concerns from Firewalla or anyone else.
First, assuming the logs were rotated within the last minute, this will send the zeek logs to port 514 on the remote syslog server:
find /log/blog -mmin -1 -type f -exec zcat {} \; | nc -q 5 destination_server 514
I took mine a step further by appending the firewalla log name to the end like this:
for l in `find /log/blog/ -type f -mmin -1 `;do zcat $l | sed "s|\}|,\"firewalla_log\":\"$l\"}|g" | nc -q 5 destination_server 514;done
Next, I added that last line to a new script in /home/pi and set it to run every minute in cron.
Hope this helps.
-
I have to agree with Aaron, I am a security professional and want to be able to send the log information (alarms, blocks, flows, etc...) to a file and either send it to SIEM or something that can parse the logs ( heck could a txt file) and forward them on to where ever we want.
Thanks
-
a) Thank you @Aaron for your tip/hack on how to send logs to external host... working well for me as well :)
Anyone figure out how to push out URL's visited? I am seeing the DNS requests but actual web requests would be nice.
b) @firewalla admins.... PLEASE expedite building this into your next release. Offloading logs off of this box would make an already good platform EVEN BETTER.... keep up the good work (and pretty please!!) :D
-
@Firewalla not sure what you mean. Expensive in what way? Why not just make access to the logs a simple documented process. Let anyone who wants to use them deal with the data issues.
Maybe just have a 25 hour rotating log file where you dump everything and let users deal with processing.
-
@Quakewalla - No problem - Glad it worked! I did change one thing recently, as I noticed that I would at times have issues waiting for it to quit after sending, so I tweaked the last command to below to use the -w switch rather that -q, and it seems to be working better (for me, anyway).
...| nc -u destination_server 514 -w0;done
Aaron
-
Built a Windows server on my Proxmox, downloaded kiwi free syslog server and configured the input as the default gateway.
Sent this command as stated above:
find /log/blog -mmin -1 -type f -exec zcat {} \; | nc -q 5 Kiwi server 514
But nothing has appeared on the server, any ides what I've done wrong. Checked the pid on the widows server and it matches the UDP netstat output.
Ive turned any firewall rule on the kiwi server off, but still nothing.
Update: I get this after a while but not every time:
find: ‘zcat’ terminated by signal 13
Thanks for any help
Andy
-
for l in `find /log/blog/ -type f -mmin -1 `;do zcat $l | sed "s|\}|,\"firewalla_log\":\"$l\"}|g" | nc -q 5 destination_server 514;done
I also don't know what I'm doing however the { } are used for grouping in SED, so how does starting with } and ending with } work in the above expression that you have built.
Still unpicking the commands to try and see where mine doesn't work.
Thanks.
-
for l in `find /log/blog/ -type f -mmin -1 `;do zcat $l | sed "s|\}|,\"firewalla_log\":\"$l\"}|g" | nc -q 5 destination_server 514;done
Ok here is my attempt at explaining the above command. Someone may want to jump in to do a better job. I'm doing this so I can learn and try and find the issue in my system as others have got it to work.
- for I in = Start for each entry and then
- Find /log/blog = Find in this directory
- type f = Regular expression
- mmin -1 = time 1 min
- do Zcat $I = Look at each compressed file
- | = transfer standard output to
- sed "s|\}|,\"firewalla_log\":\"$l\"}|g" = Manipulate the name (someone needs to explain this, I got lost)
- | = transfer standard output to
- nc -q 5 = netcat, send packets every 5 seconds
- destination_server = the IP of the SYSLOG server
- 514 = the UDP port No
Again thanks for any input
Andy
-
Consider getting a small Linux box (a $15 Raspberry Pi Zero would do the trick) to play with and build Linux experience. This way when you accidentally “sudo rm /* -fr” you don’t wipe out your FW. Also, refer to the Linux man pages for details on any command in Linux.
“ transfer standard output to” you should say “pipe” instead.The “for I” is going to loop through all the files in “log/blog”. In the command the file names can be referenced with “$I”.
The find command is listing all files modified in the past minute. mmin
type f - tells find to return only files and not other items like directories
nc -q 5 is actually waiting five seconds for the EOF on stdin and then quitting.
sed "s|\}|,\"firewalla_log\":\"$l\"}|g"
sed - use the stream editor command.
"s| - s/regexp/replacement/ Attempt to match regexp against the pattern space. If successful, replace that portion matched with replacement. Normal convention is to use the slash but for some reason the author is using the vertical bar.
\} - look for ever occurrence of “}”. The backslash in this case is the escape character saying “the next character should not be interpreted as a command”.
| - the separator of the regex text from the replacement text.
,\"firewalla_log\":\"$l\"} Replace the regex with ,"firewalla_log":name_of_the_current_file_referenced_by_$I”}
| - end of the replacement string.
g - make the replacement globally - everywhere the regex has a matched.
" - end of the sed command string.
Looking at a typical file shows they are json. So the sed command is adding a new key:value pair to the data with a key of firewalla_log and a value of what log (file name) these other key:values are sourced. It’s actually really well done. @Aaron did a great job here. He does explain a lot of this in his original post{"ts":1639399543.274395,"uid":"CRdqSo2icEkjDNRpuj","id.orig_h":"192.168.55.116","id.orig_p":64472,"id.resp_h":"199.794.186.154","id.resp_p":443,"version":"TLSv12","cipher":"TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384","curve":"x25519","server_name":"agents.dem.prismaaccess.com","resumed":false,"established":false,"cert_chain_fuids":["FtxTnhdbdGlXlNqr1ek","FGV2Mr1mOC3x8SIHdhdhd","FVrcLzhdycthdb"],"client_cert_chain_fuids":[],"validation_status":"unable to get local issuer certificate"}
Escaping (outside character classes)There are several characters that need to be escaped to be taken literally (at least outside char classes):
Brackets: []
Parentheses: ()
Curly braces: {}
Operators: *, +, ?, |
Anchors: ^, $
Others: ., \
In order to use a literal ^ at the start or a literal $ at the end of a regex, the character must be escaped.
Some flavors only use ^ and $ as metacharacters when they are at the start or end of the regex respectively. In those flavors, no additional escaping is necessary. It's usually just best to escape them anyway. -
Thanks for taking the time, it doesn't matter how much I read about SED command and expressions, I end up getting lost.
@Aaron he did do an amazing job, just its a shame that I cant get mine to send any logs to my server. Its in my crontab and runs every min as suggested, but no output as yet. I'll continue to debug to see what/where the issue is.
If I hold on long enough maybe the Firewalla team will incorporate it into there next release.. :-)
-
Sorry folks - Been out of pocket the past few days. Thanks for breaking that down so well, @Chris. @Andy - Depending on your setup, there could be a number of issues. First, I would make sure 'for l in `find /log/blog/ -type f -mmin -1' is showing results. Assuming you're seeing log files as expected, running the full command *should* send the content of the files over. Is port 514 open on the Windows host? If you run Wireshark and filter on the Firewalla IP, do you see the data making it to the host? Good luck!
-
@Aaron,
So, we need to look at the SSL logs to see any HTTPS traffic, while the port 80 stuff is in HTTP logs
DNS and X509 also contain some good intel. Wish it was just all under one "hit" but hey, we'll take it for now.
@Firwealla hope u are listening :) Also, as someone else pointed out elsewhere on the forums, we need the "deny/drop" logs too please :)
Anyone try feeding into ELK? I tried Wazuh (fork) and I think it will need a decoder :(
-
Got it working…😁
I added the -u for UDP port as my syslog server only had UDP configured. Plus the specified interface on the syslog config was changed to 0.0.0.0 instead of the source IP of Firewalla I had originally.
find /log/blog -mmin -1 -type f -exec zcat {} \; | nc -q 5 -u destination_server 514
Thanks for the help and detailed explanation from everyone
Andy
Today I learnt something new, so I can tick it off as a success 😄
Please sign in to leave a comment.
Comments
42 comments