The setup: performance falling with no traffic spike
A subscription SaaS platform, serving multiple clients on custom domains (e.g. subscription.example.com) with backend APIs centralised at apis.example.com behind an AWS Application Load Balancer, began degrading for no obvious reason.
Backend servers showed unexpected CPU spikes, API response times slowed, and performance dropped, yet user traffic stayed within normal levels. The degradation was intermittent, appearing only in certain time windows, which made it hard to pin down. Something was consuming server resources that wasn't legitimate user load.
The investigation
A methodical elimination. Each step: what we checked, and what we found.
Ruling out the frontend
Reviewed the React app's configuration (process.env.PUBLIC_URL, the hardcoded API base URL) and confirmed all API calls hit the correct backend. Verified DNS: client domains resolved only to frontend hosting (S3, Amplify, CloudFront), and the backend domain mapped solely to the ALB. No frontend misconfiguration.
Logging request headers
Instrumented the FastAPI backend to log all incoming headers on suspicious requests. This surfaced the anomaly: many requests arrived with Host: apis.example.com and unusual User-Agent strings (e.g. Bloomberg-Gecko and generic bot identifiers). Critically, they carried a Referer header but were consistently missing the Origin header, which real browser-initiated cross-site requests always include. These weren't browsers.
Correlating with metrics
Matched the invalid requests to server metrics. The backend was raising an exception for every invalid request and generating noisy logs, and that processing was eating CPU and slowing responses for real users. The bot surge was the direct cause of the spikes.
Diagram: why bad traffic reached the backend
The ALB forwarded all traffic to the backend, so validation happened after compute was already consumed. Bots could degrade the system just by flooding it with bad requests.
Bots / scrapers — no Origin header
Legitimate traffic — valid Origin
- Public ALB — forwards all traffic to ECS, with no validation of its own.
- ECS / FastAPI — validates the request only after compute is already spent.
- 403 + noisy logs — every bad request raises an exception, burning CPU and burying real events.
The ALB forwarded everything. Validation happened at the backend, after the damage was done.
Root cause
Bots overwhelming the API
Automated crawlers and search-engine bots (e.g. bingbot, custom scrapers) routinely probe for subdomains like api.*, admin.*, and test.*. They don't execute frontend JavaScript or respect browser security models, so they never send an Origin header, and they bypassed the frontend validation entirely and hit the backend directly.
Direct API probing through the public ALB
Because apis.example.com was exposed via a public ALB, scrapers could discover it and send traffic straight to it. The ALB forwarded all traffic to ECS before any domain validation, so requests missing an Origin, carrying spoofed Host headers, or sent via curl/wget all reached the application.
Backend validation as the bottleneck
Every invalid request triggered FastAPI exception handling, generated error logs, and burned ECS compute. At peak, invalid requests were numerous enough to slow legitimate traffic and bury real events in log noise. Backend-first validation meant bots could degrade performance simply by flooding the API.
Diagram: the fix — validation moved to the edge
Bad requests are now rejected at the edge, before they ever reach the backend. AWS WAF on the ALB filters traffic against four rules, and blocked traffic is diverted to WAF logs.
Backend logs stay clean and focused on real user activity.
- AWS WAF on the ALB — four rules filter every request before ECS.
- Clean only — solely authorised, well-formed requests reach FastAPI.
- WAF logs — blocked traffic is diverted to CloudWatch and S3, not the app logs.
AWS WAF filters traffic at the ALB, so only clean, authorised requests reach the backend.
The solution we engineered
Edge-level blocking with AWS WAF
Attached to the ALB, so traffic is filtered before ECS.
Block requests with no Origin header
Any request missing Origin is rejected with 403 at the edge, stopping script and crawler traffic before it consumes resources.
Allowlist authorised client domains
Only requests whose Origin matches *.example.com or client domains stored in the database are permitted, so legitimate frontends get through and direct API probing is blocked.
Block known bad bots
WAF rules reject suspicious User-Agent strings (bingbot, Bloomberg-Gecko, and other automated clients), cutting log noise and repeat access attempts.
Rate limiting
Rate-based rules block IPs sending more than 1,000 requests per 5 minutes, guarding against scraping and DDoS-style floods.
Backend safeguards (defence in depth)
- FastAPI middleware now returns 403 immediately for unauthorised requests, without raising exceptions or heavy logging.
- Blocked requests are skipped in logging, cutting noise and storage cost.
- ECS tasks handle only clean, authorised traffic, improving CPU usage and performance.
Infrastructure and observability
- Bot and blocked-request traffic is redirected to WAF logs in CloudWatch and S3, keeping application logs focused on real user activity.
- Filtering at the edge reduced ECS load, giving more consistent response times.
Outcome
CPU spikes and server slowness disappeared
With unauthorised requests rejected at the edge, the backend stopped processing needless exceptions and noisy logs, and ECS now serves only valid requests, giving faster, more consistent API responses under load.
Clean, actionable logs
Application logs now contain only business-relevant events, making monitoring and debugging far easier.
Scalable traffic control
The WAF allowlist can be updated dynamically to authorise new client domains without redeploying backend code, a future-proof mechanism as the platform grows.
Key takeaways for CTOs and chief engineers
Stop malicious traffic at the edge
Filter at the ALB/WAF layer, not just in backend logic. The Origin header is a simple, reliable gatekeeper; combine allowlists, rate limiting, and bot signatures for layered protection.
Optimise logging and monitoring
Don't heavily log invalid requests; it slows the system and hides real events. Keep security logs separate from application logs.
Design for scalability
For multi-tenant SaaS, use dynamic WAF allowlists to authorise new client domains without redeploying backend code.