See how AI crawlers actually request your site.
Classify crawler requests at the server, then find the pages machines reach, miss, or revisit.
Move from server-log noise to a useful map of machine access.
Requests grouped by crawler family, with the routes each one reaches.
Routes ranked by crawler requests in the period, so the ones with little or none stand out.
Response statuses crawlers received, so redirects, blocks and errors stand out by route.
Crawler requests to the same routes before and after the date of a change.
Crawler activity happens without running your browser analytics.
Most crawlers request HTML without executing browser JavaScript, so Virent captures requests in the server, edge, middleware, reverse proxy, or plugin hook instead. Server secrets stay in server-side environment variables.
Core measurements
Crawler classification
Group requests using maintained user-agent and crawler-family rules, with the reported agent preserved for inspection. A classification is not proof of cryptographic identity.
Request volume and timing
Review counts and trends by family, route, section, method, and time range. A spike is a prompt to inspect, not automatically harmful.
Coverage
Compare the pages crawlers requested with the inventory that matters. Coverage describes observed requests, not guaranteed indexing or model use.
Response status
See the status returned to crawlers and find repeated redirects, blocked routes, missing pages, or server failures.
User-agent: GPTBot
Allow: /docs/
Disallow: /account/
# changed 2 days ago
Rules and change context
Compare behavior around robots, middleware, proxy, or routing changes to validate the observed effect rather than the intent.
Being accessible, being selected, and being cited are different states.
A request proves that a machine reached an observed layer. It does not prove that a provider indexed the page or cited it in an answer. Virent makes the gap measurable:
- 01
inspect crawler coverage and response behavior for the pages that matter;
- 02
compare them with citation URLs from repeatable prompt runs;
- 03
separate access problems from relevance and source-selection questions;
- 04
repeat the comparison after a change.
Workflow
- 01
Install at the request layer
Use the recommended hook for your framework. Exclude API routes, static assets, favicons, robots files, and sitemaps unless the measurement plan needs them.
- 02
Verify with a controlled request
Send a request with a known crawler user agent and confirm the event appears. This tests the integration, not the external crawler's network identity.
- 03
Establish coverage
Group routes into business-relevant sections and compare observed requests with the expected page inventory.
- 04
Investigate changes
Use status, route, timing, and citation evidence to decide whether the next step is technical, editorial, or simply more observation.
For teams
SEO teams
Connect crawl behavior to visibility and citations without reducing it to a single score.
Developers and platform teams
Inspect concrete routes and responses, keep secrets server-side, and validate changes with controlled requests.
Content teams
Find pages that receive crawler attention but never appear as sources in monitored answers.
Questions
Frequently asked questions
Why not use the browser script for crawler analytics?
Many crawlers do not run the browser script. Capture has to happen before browser JavaScript, at the server or another request-interception layer.
Does Virent verify that every request came from the named company?
Virent classifies observed signals such as user agents using its maintained taxonomy, and surfaces stronger verification separately where it is available. A classification alone is not cryptographic identity proof.
Does a successful crawl mean the page can appear in an AI answer?
No. Access is one prerequisite. Indexing, retrieval, source selection, and answer generation are controlled by external providers.
Can Virent change our robots rules?
No. Virent measures and supports decisions. It does not push robots, canonical, middleware, or content changes to your site.
Manage crawler access with observed requests.
See which machines reach which pages, how the site responds, and whether a change produced the behavior you expected.