Stopping Bots Before They Hit the Backend: How Edge-Level Filtering Restored API Performance
Background and Problem Statement
A sudden performance degradation was observed on one of our subscription-based SaaS platform, which serves multiple clients under custom domains (e.g., subscription.example.com) with backend APIs centralized at apis.example.com behind an AWS Application Load Balancer (ALB). During normal operations, the backend servers began showing unexpected CPU spikes, API response times slowed noticeably, and overall application performance degraded, even though user traffic remained within expected levels. The problem appeared intermittently during certain time windows, making it difficult to immediately pinpoint the cause, and prompted a closer investigation into the traffic reaching our backend APIs to determine why server resources were being consumed and what was causing the intermittent slowdowns.
Our Investigative Steps
Eliminating frontend misconfiguration
The first step was to rule out any issues in our frontend builds. Our team at ellocent carefully reviewed the React application’s configuration, including environment variables such as process.env.PUBLIC_URL and the hardcoded API base URL. We confirmed that all API calls were correctly directed to the backend URL and not to any unintended endpoint. DNS records were also checked to verify that client domains were resolved only to our frontend hosting services (S3, Amplify, CloudFront), while the backend domain was solely mapped to the ALB. These checks confirmed that there was no misconfiguration on the frontend side.
Checking backend logs and request headers
Next, we instrumented our FastAPI backend to capture and log all incoming headers on suspicious requests. This deeper logging quickly revealed anomalies. Several requests were coming in with Host: apis.example.com, accompanied by unusual User-Agent strings such as Bloomberg-Gecko and generic bot identifiers. Interestingly, many of these requests included a Referer header but were consistently missing the Origin header — a critical detail, since legitimate browser-initiated cross-site requests always include the Origin. This strongly suggested that the requests were not from real browsers but from automated crawlers or scripts.
Tracing the root cause of performance degradation
Once the pattern became clear, we correlated these invalid requests with server metrics. We found that the backend was raising exceptions for every invalid request, which in turn generated noisy logs. This unnecessary processing consumed CPU cycles and slowed down response handling for genuine users. The analysis confirmed that the sudden increase in bot and crawler activity was the direct cause of our observed CPU spikes and server slowness.
Root Cause Analysis
Bots and crawlers overwhelming the API
Our investigation revealed that much of the unauthorized traffic was being generated by automated crawlers and search engine bots. Common examples included bingbot and custom scrapers that routinely probe for subdomains such as api.*, admin.*, and test.*. These entities do not behave like real browsers. In particular, they never send an Origin header because they do not execute frontend JavaScript or respect browser security models. As a result, every crawler request bypassed the frontend validation logic and landed directly on our backend APIs.
Direct API probing through the public ALB
Because our backend API (apis.example.com) was exposed via a public Application Load Balancer (ALB), attackers and scrapers could easily discover the endpoint and send traffic directly to it. The ALB, by design, forwarded all traffic to the ECS service before any domain validation occurred. This meant that even completely invalid requests — those missing an Origin, those with spoofed Host headers, or those sent by bots using curl/wget — reached our backend application. While our backend logic correctly rejected these requests, the validation happened late in the chain, after resources had already been consumed.
Backend validation becoming a bottleneck
The real performance issue from the fact that invalid traffic was being processed at the backend layer. Every unauthorized request triggered exception handling in FastAPI, generated error logs, and consumed ECS compute cycles. During traffic spikes, the number of invalid requests grew large enough to slow down legitimate request handling. Logs became cluttered with repetitive error messages, making it difficult to isolate real issues. Ultimately, this backend-first validation strategy meant that bots could degrade overall system performance simply by flooding the API with bad traffic.
Solution We Implemented
Edge-Level Blocking with AWS WAF
Migrating to a clean traffic-filtering layer
To prevent bots and unauthorized clients from overwhelming our backend, we attached AWS WAF to the Application Load Balancer (ALB). This ensured that all traffic was filtered before reaching ECS, protecting backend resources from unnecessary load.
Blocking requests without an Origin header
Any request missing the Origin header was immediately rejected with a 403 Forbidden response. This simple check prevented requests coming from scripts or crawlers from consuming server resources. Example WAF condition:
Allowlisting authorized client domains
Only requests with Origin headers matching *.example.com or other client domains stored in our database were permitted. This ensured that traffic originating from legitimate frontend applications could reach the backend, while unauthorized or direct API requests were blocked at the edge.
Blocking known bad bots
WAF rules were configured to reject requests containing suspicious User-Agent strings, such as bingbot, Bloomberg-Gecko, and other automated clients. This reduced noise in logs and prevented repeated bot access attempts.
Rate limiting for high-volume requests
Rate-based rules were implemented to block IPs sending more than 1,000 requests per 5 minutes, protecting the backend from brute-force scraping and potential DDoS activity.
Backend Safeguards
Strengthening server-side validation
Even with WAF in place, we enhanced backend security by updating FastAPI middleware:
- Unauthorized requests received a 403 response immediately, without raising exceptions or performing heavy logging.
- Blocked requests were skipped in logging, reducing noise and storage costs.
- ECS tasks now handled only clean, authorized traffic, improving CPU usage and overall performance.
Infrastructure Improvements
Centralizing logs and monitoring traffic
- All bot traffic and blocked requests were redirected to WAF logs stored in CloudWatch and S3. This centralized approach kept backend logs clean and focused only on legitimate user requests.
- By filtering traffic at the edge, ECS tasks experienced less load, leading to more consistent response times and reduced backend slowness
Bot traffic blocked at the edge
With WAF attached to the ALB, invalid requests were filtered before reaching ECS. This prevented automated crawlers, scripts, and spoofed requests from consuming backend resources, ensuring that only legitimate traffic reached the application.
Improved backend performance
CPU spikes and server slowness disappeared. By rejecting unauthorized requests at the edge, the backend no longer processed unnecessary exceptions or generated noisy logs. ECS tasks now focus solely on serving valid client requests, resulting in faster API responses and more consistent performance under load.
Clean and actionable logs
Because malicious and invalid requests were handled by WAF, application logs now contain only business-relevant events. This has made monitoring and debugging far easier, allowing our team to detect real issues quickly without sifting through bot-induced noise.
Scalable traffic control
The WAF allowlist can be updated dynamically to include new client domains without redeploying backend code. This gives the platform a scalable, future-proof mechanism to authorize new clients while keeping security tight.
Key Takeaways for CTOs and Chief Engineers
Stop malicious traffic at the edge
Filter unwanted requests at the ALB/WAF level instead of relying solely on backend logic. Use the Origin header as a simple, reliable gatekeeper, and combine allowlists, rate limiting, and bot signatures for layered protection.
Optimize logging and monitoring
Avoid excessive logging of invalid requests, which can slow the system and hide meaningful events. Keep security logs separate from application logs to maintain clarity and actionable insights.
Design for scalability
For subscription-based platforms, use dynamic allowlists in WAF to authorize new client domains without redeploying backend code, ensuring security and operational efficiency as the platform grows.
At Ellocent Labs, by implementing edge-level traffic filtering with AWS WAF, enhancing backend safeguards, and centralizing logs, technical leaders can prevent unauthorized requests from degrading system performance, maintain clean application logs, and ensure reliable API responsiveness across all client domains.
Explore More Solutions
Ready to transform your SaaS business? Let’s innovate together!
Schedule a 15-Minutes call
Let’s make things happen and take the first step toward success!
Got Ideas? We’ve Got The Skills.
Let’s Team Up!
Let’s Team Up!
What Happens Next?
We review your request, contact you, and sign an NDA for confidentiality.
We analyze your needs and create a project proposal with scope, team, time, and cost details.
We schedule a meeting to discuss the offer and finalize the details.
The contract is signed, and we start working on your project immediately.