Crawler telemetry
Observe crawler behavior instead of assuming it
Crawler policy says what we permit. Server logs tell us what actually happened. Identity verification tells us how much confidence to place in the user-agent label.
Current first-party tool
php tools/crawl-report.php /path/to/access.log reports recognized bot families, status codes, requested paths, and discovery-resource requests. Its JSON output explicitly records whether network identity verification was performed.
What to watch after deployment
- 200/304 versus 3xx/4xx/5xx response mix for crawlers;
- requests for
/robots.txt,/sitemap.xml, feeds, corpus, metadata, proposals, and canonical articles; - whether canonical research pages are reached after discovery files;
- bot-protection/WAF failures that contradict permissive robots rules;
- request-rate anomalies and spoofed crawler user agents.
Host-specific extensions
The telemetry research proposes Nginx conditional logging, APCu, SQLite/WAL and provider range verification. Those are useful implementation options only if the production host supports them. The deploy baseline remains portable PHP and ordinary server logs.