Seeking Clarification on MSP API Retention Limits and Bulk Data Export Workarounds.

Follow

Ilco Louis

• June 30, 2026 09:36

Hi everyone,

Over the past 30 days, my Firewalla Gold has logged approximately 3.5 million network flows. For deeper security auditing, pattern analysis, and local archiving, I am looking to ingest this entire historical dataset into my on-premise data server.

However, after extensive testing and developing a custom integration against the Firewalla MSP environment, I have encountered a few structural constraints that seem to severely limit automated data ownership. I would like to verify if my findings are correct, or if I am misinterpreting the platform's architecture:

1. The Cloud API Retention Window: While the MSP web portal beautifully retains 30 to 90 days of history, my programmatic requests to the /v2/flows endpoint appear to hit a hard filtering ceiling. Any queries utilizing timestamps older than 24 hours return an empty dataset (0 results). Is the API gateway intentionally restricted to a rolling 24-hour cache, regardless of the MSP tier retention settings?

2. Inconsistent API Synchronization: During high-frequency polling, the API responses exhibit symptoms of eventual consistency across the cloud backend (distributed server clusters). We frequently observe fluctuating pagination behaviors, where successive requests alternatively yield chunks of data or empty results before catching up. This makes a reliable data pipeline highly unpredictable.

3. Manual CSV Export Constraints: The manual export utility via the MSP UI is capped at 50,000 rows per file, combined with a strict rate limit of 20 downloads per 48 hours. For a volume of 3.5 million rows, extracting this data manually would turn into a tedious, week-long administrative task.

I might entirely be missing a vital piece of documentation or an alternative workflow.

Are my assumptions regarding these API and export limits accurate? Are there any official, streamlined endpoints or asynchronous bulk-export methods (e.g., a /v2/exports request) designed for full data ingestion that I have overlooked?

Any insights, workarounds, or official clarifications would be greatly appreciated!

Best regards,
Ilco Louis

0

• 

  Firewalla

  • June 30, 2026 10:05

  Do you have any sample code how you are encountering the issues? For example, for flows, you need to page through https://github.com/firewalla/msp-api-examples/blob/main/flow-pagination/index.js

  I am not understanding high frequency polling, but I do know, there is a rate limit, your code need to detect 429 and then back down.

   

   

  0

  Comment actions Permalink
• 

  Ilco Louis

  • June 30, 2026 11:08

  Thank you for the quick response!

  To clarify, my integration does not utilize high-frequency polling or aggressive scraping. In fact, quite the opposite: the script is explicitly designed to operate safely and conservatively well within standard operational thresholds. I firmly believe that good etiquette in API consumption means pacing requests deliberately from the outset, rather than adopting a brute-force approach that continuously rams the server until it is forced to scream an HTTP 429 error to back down. Think of it as ringing a doorbell once, rather than using a battering ram to get inside.

  The client-side code itself is functioning perfectly and is completely stable. It handles pagination seamlessly via the response cursors, pulling data in orderly chronological blocks and committing them to our database without any structural errors or network drops.

  The issue is not the execution of the pagination logic, but the hard boundary of the returned dataset itself. Our script successfully processes every single page the API offers, but the dataset simply runs dry and yields 0 results once it reaches approximately 24 hours into the past. The remaining 29 days of history—which are fully visible within the MSP web portal—seem to be completely missing from the API's reach.

  What I am primarily looking for is a definitive architectural confirmation: Is the /v2/flows API gateway strictly limited to a rolling 24-hour retention window, or is there an official method to page further back into the 30-day history?

  If a 24-hour window is indeed the hard limitation of the API platform, that is perfectly fine to know. It simply means that continuing with "Plan API" is a dead end for historical auditing, and I can gracefully pivot my development efforts toward alternative local ingestion methods like Syslog.

  Looking forward to your clarification on this specific boundary!

  Grtz,
  Ilco

  0

  Comment actions Permalink
• 

  FirewallaSupportDesk

  • July 01, 2026 03:13

  By default, it queries the last 24 hours flows if time range is not set. Could you share some sample code so we can check for you?

  0

  Comment actions Permalink

Please sign in to leave a comment.