Quick Answer
robots.txt is a file at the site root with instructions for crawlers and it may declare a sitemap. It can make some technical choices explicit without requiring every crawler to behave the same way.
Why it may deserve attention
A 404 means the file was not found at the expected path. A different HTTP status or an unobserved Sitemap directive needs context before it becomes an action to take.
What is observed
The check requests the file at /robots.txt and observes its HTTP availability, presence, and a Sitemap: directive in the acquired body. It is not a complete parser of Allow/Disallow rules or a test of actual crawling.
How to read the result
Finding
robots.txt not found: the robots_txt.not_found finding is produced only when the HTTP response is 404.
Observations
A Sitemap directive can be observed, or not observed when the body is conclusive. An HTTP status other than 404, including 410, 5xx, or 429, is an unexpected outcome to investigate: it does not automatically mean the file is missing.
Inconclusive evidence
An unavailable fetch, request failure, or truncated body makes the evidence inconclusive. In particular, a truncated body cannot support a conclusion that a Sitemap directive is absent.
What it can help you review
An observed file and Sitemap directive can be useful technical facts to review in site configuration. Their presence does not prove crawling, indexing, ranking, or a crawler's real behavior.
Recommended action
- HTTP 404: if the site intends to publish robots.txt, restore or publish it at the correct root location.
- Sitemap directive not observed: add it only when appropriate for the site's configuration and when the absence was assessed with a conclusive body.
- Unexpected status or failure: review server, proxy, WAF, or access rules with the owner or host; do not treat it as a missing file without further evidence.
What this check cannot conclude
TrustyRunner cannot confirm actual crawling, indexing, ranking, citation, every crawler's behavior, or the completeness and semantic quality of the file. File presence does not automatically make a site crawlable.
What a new scan can verify
A new scan can check again the presence and reachability of /robots.txt and observable Sitemap: presence within its new coverage. It cannot confirm search-engine behavior.