Business logic of the scrapper: why the attacking profiles of users
Before you build a scraping detection, it is worth to understand the motivation. Without this, it is impossible to prioritize protective measures and explain to business why "just reading profiles" requires a budget. More details - in our detailed analysis of bola idor vulnerability.
Enriching bases for credential stuffing. The scraper collects email, username, registration date, tied social networks. This data enriches existing password leaks: if the attacker has a pair email
assword from another service, and the scraped profile confirms that email is active on your platform - Credential stuffing (T1110.004, Credential Access) becomes an order of magnitude more effective.
Targeted phishing. Username, email, rating, last login date - ready context for personalized writing. “Congratulations on reaching the 1800 rating – confirm the account” looks more convincing than mass mailing.
Resale and competitive exploration. The databases of profiles are sold on shadow areas. For services with a paid subscription, the data about the tier status of users have direct commercial value.
By classification OWASP API Security Top 10 (2023) Massive scraping falls at once at two risks:
API4:2023 - Unrestricted Resource Consumption : API accepts unlimited requests without limits.
API6:2023 - Unrestricted Access to Sensitive Business Flows : The business process of viewing the profile is available for automated abuse.
Legal context for the Russian Federation: since 2024, fines for leakage of personal data are negotiable. If the regulator determines that the data has leaked through the API without the basic rate limiting, the “we didn’t know” position will not work.
Kill chain scrupper by MITRE ATT&CK
Scraping is not a one-way. A typical chain for mass collection of user data is folded over MITRE ATT&CK:
Stage MITRE ATT&CK Technique What does the attacker do
Intelligence Search Open Websites/Domains (T1593, Reconnaissance) Learns Swagger/OpenAPI specifications, searches for endpoints through Google dorks type site:target.com inurl:api
Scanning Active Scanning (T1595, Reconnaissance) Searches endpoints, tests limits, determines the format of answers
Initial access Exploit Public-Facing Application (T1190, Initial Access) Uses a public API without authentication or with a minimum API key
Collection Data harvesting via HTTP GET Consistent requests for /api/user/{id}with ID increment or overkill of the dictionary of users
Data output Automated Exfiltration (T1020, Exfiltration) Automatic uploading and aggregation of responses to local database
If the API requires authentication, the attacker takes the purchased or stolen API keys - option with Cloud Accounts (T107.004, Initial Access). In the case of Chess.com, authentication for reading public profiles was not required at all.
Key point for SOC: Each individual request in the chain is legitimate. Alert must be triggered on the pattern, not a single event. According to Cequence, HTTP GET requests make up up to 99% of all transactions of a typical domain, and it is through them that scraping goes. Bot's mind-based tools, sharpened by the analysis of POST requests, this vector simply does not see.
Scraping Detective: From Logs to SIEM Rules
Anomalies in API Traffic: A Script from Practice
Monday, 9:15 a.m. In Grafana-dashboard on API-metrics I see an anomaly: endpoint /api/v2/users/{id} over the weekend, I processed 40 times more requests than the average week. RPS from one /24-subset - stable 12 requests per second, exactly 2 below the threshold rate-limiter. Someone carefully picked up the speed. User-Agent - Mozilla/5.0 (Windows NT 10.0; Win64; x64) without a single variation in 48 hours. The time between requests is 83+-2 ms, ideal frequency, impossible for a person.
What the scrupper gives in the logs:
Constant interval between requests. A person clicks with a spread of 0.5-30 seconds. Script on requests+ time.sleep(0.08)gives a spread in milliseconds. This flat "saw" is visible to the naked eye on the graph.
One User-Agent without variation. The real browser is updated, changing the minor version. The scrapper uses a slunted line.
Consistent overkill of ID. Requests to /api/user/1001, /api/user/1002, /api/user/1003- increment on 1, or overkill by alphabetical dictionary.
Lack of navigational context. The normal user goes to the main, clicks in the profile, goes to another section. The scrupper hits only one endpoint like a woodpecker.
IP from hosting subnets. 95% of requests with DigitalOcean, Hetzner, OVH - not from resident subnets.
404-response share above 30%. The selection of non-existent IDs or users generates a mass of 404. With normal use, this does not happen.
Baseline of normal API traffic (DE.AE-01 by NIST CSF v2.0) - a mandatory starting point. Without it, any threshold of the alert will be either too noisy or useless.
Compromised legitimate accounts: blind SOC zone
A separate headache is scraping through compromised legitimate accounts or affiliate APIs that exceed the stated scope. When the attacker uses the employee’s real API key, the IP rate-limiter and reputation listings are useless. It's the same with an insider that unloads data through full-time access.
It is possible to detect such cases only through anomaly detection on a specific account: if affiliate integration, which usually makes 200 requests per day, suddenly begins to generate 20 000 - this is a signal. The SIEM rule should consider the historical baseline of each API key, not just global thresholds. In practice, this baseline is few who lead - and in vain.
Sigma Rules for Correlation in SIEM
To detect automated data collection, you need rules that work on aggregation:
YAML:
title: API Scraping - High Volume Profile Enumeration
description: Detects high-volume requests to user profile endpoint
logsource:
category: webserver
product: nginx
detection:
selection:
cs-uri-stem|contains: '/api/user/'
timeframe: 5m
condition: selection | count(cs-uri-stem) by c-ip > 200
level: high
tags:
- attack.collection
- attack.t1020
Basic rule: more than 200 requests to the profile endpoint with one IP in 5 minutes. The threshold is selected under your baseline - 200 can be and little, and a lot. It is worthwhile to supplement with a correlation on the distribution of ID in queries: if the difference between consecutive IDs is stable is 1 - it is a scraping with an increment, there is nothing to think about. Additional signals for correlation: the ratio of unique requested paths to the total number of requests > 0.9 (each request is a new ID), no static requests (CSS, JS, images) from the same source.
Honeypot-endpoints and canary data
Honeypot is one of the most underrated methods for preventing mass scraping. The protection of personal data of API users here works ahead: you do not just block, but identify the attacker.
Three appointments that actually work:
Hidden endpoint through robots.txt. Add Disallow: /api/v2/internal/users-export. Legitimate crowners will miss it, and the scrupper, who reads robots.txt as a treasure map, will send a request. Any access to this URL is an immediate alert (RS.AN-01 by NIST CSF v2.0).
Canare records in API responses. Build fake profiles with unique email addresses on the trap domain. If such an email pops up in a leaked database, confirmation of scraping and a point for investigation. Simple but elegant trick.
CSS-hidden links. Element with display:none, containing a link to the API-endpoint: a person will not see it, but the bot that parries the DOM will pass.
Request to honeypot-endpoint = immediate blocking of source IP, fingerprint and related sessions.
API Abnormal Activity Monitoring: Checklist for SOC
Checking The instrument Priority
Baseline RPS for each API Grafana/Prometheus, ELK Critical
Alert on an abnormal RPS from one IP/subnet SIEM (Sigma-rule above) Critical
Differentiated rate limiting by endpoint type nginx/Kong/Cloudflare Critical
ID predictability check (IDOR audit) DAST scanner, hand pentest Critical
Incoming traffic monitoring JA3/JA4 hashes nginx + ja3 module, WAF High
200/404-response ratio by IP ELK/Splunk High
Audit of public APIs (Swagger/OpenAPI) Hand review High
Baseline queries for each API SIEM / custom dashboard High
Honeypot-endpoint in robots.txt Custom handler + SIEM Average
Canary records in API responses Custom logic in backend Average
The checklist covers DE.AE-01 on NIST CSF v2.0 (baseline installation and anomaly monitoring) and RS.AN-01 (investigation of alerts from detection systems). Start with critical points - they close 80% of the vector at minimal cost.
Mass scruffing is the only category of API attacks that most SOC teams still do not monitor at all. Credential stuffing detects, DDoS beat back, SQ injectionL WAF catches. And when the bot quietly pulls out the profiles at a speed of 5 RPS - it is below the radar, because formally nothing breaks. The consequences appear later: these months lie on shadowy platforms, then return to credential stuffing attacks against you. And when the regulator asks why the API with personal data worked without rate limiting at the endpoint level - "we didn't consider it a threat" sounds like a recognition of negligence.
Before you build a scraping detection, it is worth to understand the motivation. Without this, it is impossible to prioritize protective measures and explain to business why "just reading profiles" requires a budget. More details - in our detailed analysis of bola idor vulnerability.
Enriching bases for credential stuffing. The scraper collects email, username, registration date, tied social networks. This data enriches existing password leaks: if the attacker has a pair email
assword from another service, and the scraped profile confirms that email is active on your platform - Credential stuffing (T1110.004, Credential Access) becomes an order of magnitude more effective.Targeted phishing. Username, email, rating, last login date - ready context for personalized writing. “Congratulations on reaching the 1800 rating – confirm the account” looks more convincing than mass mailing.
Resale and competitive exploration. The databases of profiles are sold on shadow areas. For services with a paid subscription, the data about the tier status of users have direct commercial value.
By classification OWASP API Security Top 10 (2023) Massive scraping falls at once at two risks:
API4:2023 - Unrestricted Resource Consumption : API accepts unlimited requests without limits.
API6:2023 - Unrestricted Access to Sensitive Business Flows : The business process of viewing the profile is available for automated abuse.
Legal context for the Russian Federation: since 2024, fines for leakage of personal data are negotiable. If the regulator determines that the data has leaked through the API without the basic rate limiting, the “we didn’t know” position will not work.
Kill chain scrupper by MITRE ATT&CK
Scraping is not a one-way. A typical chain for mass collection of user data is folded over MITRE ATT&CK:
Stage MITRE ATT&CK Technique What does the attacker do
Intelligence Search Open Websites/Domains (T1593, Reconnaissance) Learns Swagger/OpenAPI specifications, searches for endpoints through Google dorks type site:target.com inurl:api
Scanning Active Scanning (T1595, Reconnaissance) Searches endpoints, tests limits, determines the format of answers
Initial access Exploit Public-Facing Application (T1190, Initial Access) Uses a public API without authentication or with a minimum API key
Collection Data harvesting via HTTP GET Consistent requests for /api/user/{id}with ID increment or overkill of the dictionary of users
Data output Automated Exfiltration (T1020, Exfiltration) Automatic uploading and aggregation of responses to local database
If the API requires authentication, the attacker takes the purchased or stolen API keys - option with Cloud Accounts (T107.004, Initial Access). In the case of Chess.com, authentication for reading public profiles was not required at all.
Key point for SOC: Each individual request in the chain is legitimate. Alert must be triggered on the pattern, not a single event. According to Cequence, HTTP GET requests make up up to 99% of all transactions of a typical domain, and it is through them that scraping goes. Bot's mind-based tools, sharpened by the analysis of POST requests, this vector simply does not see.
Scraping Detective: From Logs to SIEM Rules
Anomalies in API Traffic: A Script from Practice
Monday, 9:15 a.m. In Grafana-dashboard on API-metrics I see an anomaly: endpoint /api/v2/users/{id} over the weekend, I processed 40 times more requests than the average week. RPS from one /24-subset - stable 12 requests per second, exactly 2 below the threshold rate-limiter. Someone carefully picked up the speed. User-Agent - Mozilla/5.0 (Windows NT 10.0; Win64; x64) without a single variation in 48 hours. The time between requests is 83+-2 ms, ideal frequency, impossible for a person.
What the scrupper gives in the logs:
Constant interval between requests. A person clicks with a spread of 0.5-30 seconds. Script on requests+ time.sleep(0.08)gives a spread in milliseconds. This flat "saw" is visible to the naked eye on the graph.
One User-Agent without variation. The real browser is updated, changing the minor version. The scrapper uses a slunted line.
Consistent overkill of ID. Requests to /api/user/1001, /api/user/1002, /api/user/1003- increment on 1, or overkill by alphabetical dictionary.
Lack of navigational context. The normal user goes to the main, clicks in the profile, goes to another section. The scrupper hits only one endpoint like a woodpecker.
IP from hosting subnets. 95% of requests with DigitalOcean, Hetzner, OVH - not from resident subnets.
404-response share above 30%. The selection of non-existent IDs or users generates a mass of 404. With normal use, this does not happen.
Baseline of normal API traffic (DE.AE-01 by NIST CSF v2.0) - a mandatory starting point. Without it, any threshold of the alert will be either too noisy or useless.
Compromised legitimate accounts: blind SOC zone
A separate headache is scraping through compromised legitimate accounts or affiliate APIs that exceed the stated scope. When the attacker uses the employee’s real API key, the IP rate-limiter and reputation listings are useless. It's the same with an insider that unloads data through full-time access.
It is possible to detect such cases only through anomaly detection on a specific account: if affiliate integration, which usually makes 200 requests per day, suddenly begins to generate 20 000 - this is a signal. The SIEM rule should consider the historical baseline of each API key, not just global thresholds. In practice, this baseline is few who lead - and in vain.
Sigma Rules for Correlation in SIEM
To detect automated data collection, you need rules that work on aggregation:
YAML:
title: API Scraping - High Volume Profile Enumeration
description: Detects high-volume requests to user profile endpoint
logsource:
category: webserver
product: nginx
detection:
selection:
cs-uri-stem|contains: '/api/user/'
timeframe: 5m
condition: selection | count(cs-uri-stem) by c-ip > 200
level: high
tags:
- attack.collection
- attack.t1020
Basic rule: more than 200 requests to the profile endpoint with one IP in 5 minutes. The threshold is selected under your baseline - 200 can be and little, and a lot. It is worthwhile to supplement with a correlation on the distribution of ID in queries: if the difference between consecutive IDs is stable is 1 - it is a scraping with an increment, there is nothing to think about. Additional signals for correlation: the ratio of unique requested paths to the total number of requests > 0.9 (each request is a new ID), no static requests (CSS, JS, images) from the same source.
Honeypot-endpoints and canary data
Honeypot is one of the most underrated methods for preventing mass scraping. The protection of personal data of API users here works ahead: you do not just block, but identify the attacker.
Three appointments that actually work:
Hidden endpoint through robots.txt. Add Disallow: /api/v2/internal/users-export. Legitimate crowners will miss it, and the scrupper, who reads robots.txt as a treasure map, will send a request. Any access to this URL is an immediate alert (RS.AN-01 by NIST CSF v2.0).
Canare records in API responses. Build fake profiles with unique email addresses on the trap domain. If such an email pops up in a leaked database, confirmation of scraping and a point for investigation. Simple but elegant trick.
CSS-hidden links. Element with display:none, containing a link to the API-endpoint: a person will not see it, but the bot that parries the DOM will pass.
Request to honeypot-endpoint = immediate blocking of source IP, fingerprint and related sessions.
API Abnormal Activity Monitoring: Checklist for SOC
Checking The instrument Priority
Baseline RPS for each API Grafana/Prometheus, ELK Critical
Alert on an abnormal RPS from one IP/subnet SIEM (Sigma-rule above) Critical
Differentiated rate limiting by endpoint type nginx/Kong/Cloudflare Critical
ID predictability check (IDOR audit) DAST scanner, hand pentest Critical
Incoming traffic monitoring JA3/JA4 hashes nginx + ja3 module, WAF High
200/404-response ratio by IP ELK/Splunk High
Audit of public APIs (Swagger/OpenAPI) Hand review High
Baseline queries for each API SIEM / custom dashboard High
Honeypot-endpoint in robots.txt Custom handler + SIEM Average
Canary records in API responses Custom logic in backend Average
The checklist covers DE.AE-01 on NIST CSF v2.0 (baseline installation and anomaly monitoring) and RS.AN-01 (investigation of alerts from detection systems). Start with critical points - they close 80% of the vector at minimal cost.
Mass scruffing is the only category of API attacks that most SOC teams still do not monitor at all. Credential stuffing detects, DDoS beat back, SQ injectionL WAF catches. And when the bot quietly pulls out the profiles at a speed of 5 RPS - it is below the radar, because formally nothing breaks. The consequences appear later: these months lie on shadowy platforms, then return to credential stuffing attacks against you. And when the regulator asks why the API with personal data worked without rate limiting at the endpoint level - "we didn't consider it a threat" sounds like a recognition of negligence.
