What happened: Bright Data (the network provider our menu-refresh pipeline runs through) has an automated security system that hunts for stolen accounts. It mistook our own Cloudflare servers for suspicious traffic and silently blocked the IP addresses our pipeline uses.
Impact: Website menus stopped refreshing for roughly 30 hours across all 54 stores. Ordering kept working, but stock info on the site went stale - some customers saw "out of stock" on items that were in stock, and a few checkouts were blocked.
Fix: Root cause pinpointed on July 30. Within about 5 hours the blocks were cleared, a permanent 33-entry IP allowlist was installed in Bright Data (their auto-blocker cannot touch allowlisted addresses), and an automated self-heal now re-fixes it without human involvement if it ever recurs. All 54 store menus were fully refreshed the same day.
Every store's live menu on sessions.ca comes from an automated pipeline: a Cloudflare worker fetches each store's menu from the POS system every half hour and caches it for the website. Because the POS blocks ordinary datacenter traffic, those fetches travel through Bright Data, a commercial network provider. This same pipeline is what makes brand links, the shop grid, and promo pages show accurate, per-store stock.
Bright Data operates an automated defense called lum_auth_monitor. Its job is to detect leaked or stolen account credentials by watching where traffic comes from. Cloudflare workers (where our pipeline lives) send traffic from Cloudflare's shared IP ranges - which, to that system, looked like our account being used from "too many unknown places." It responded by automatically adding Cloudflare address blocks to our account's IP denylist. No notification is sent when this happens; requests simply start failing with an error (HTTP 520) that looks like a generic network fault.
Small partial blocks appeared as early as July 20, 26 and 27 (intermittent, self-masked by retries). On July 29 the blocks accumulated enough to cover our whole egress path, and menu refreshes stopped fleet-wide.
| When | What |
|---|---|
| Jul 29, ~morning | Auto-blocks reach full coverage; menu refreshes silently begin failing on every run. |
| Jul 30, midday | Staleness detected while QA-testing a marketing link that showed missing products. Investigation traced it from the website back through the cache to the refresh pipeline. |
| early afternoon | Root cause identified: 7 Cloudflare ranges sitting in Bright Data's "Blocked IPs" list, added by their automated system. Blocks cleared manually in the control panel. |
| ~12:00 PM | Permanent allowlist rollout begins. Brief 5-minute self-lockout while the list was partially entered (expected and planned around - a partial allowlist blocks everything not yet on it). |
| ~1:30 PM | Full 33-entry allowlist confirmed in place, covering all 15 of Cloudflare's published IP ranges (their console rejects large blocks, so four had to be split into 22 smaller /16 ranges). |
| afternoon | Forced refresh of every stale store begins (sequential, paced to avoid tripping the same defense); pipeline hardening deployed in parallel. |
| ~4:40 PM | Fleet fully refreshed. Refresher cron green on every run since (8 stores per run, 0 failures). |
Our monitoring actually caught this. A "catalog stall" alert fired correctly and repeatedly. Two things kept it from reaching a human: (1) the alert-email key was never configured on that worker, so every alert sat unsent in the database; and (2) when the alert was reviewed, it was checked against the wrong internal data table (a second, healthy pipeline) and dismissed as a false alarm. Both failure modes are now fixed: the daily automated health check audits every fired alert and treats undelivered ones as unseen incidents, and it is hard-coded to verify each alert against its own pipeline's data. The email key is being installed as the last step.
No data was lost, no orders were mischarged, and no systems were compromised - this was an availability issue in menu freshness only. The measurable costs were ~30 hours of stale menus, a handful of blocked checkouts, one support ticket, and one marketing link that briefly looked broken. The lasting benefit: the entire class of "our own provider silently blocks us" failures is now prevented, self-healing, and independently monitored.