Robots.txt Tester — Is a URL Blocked for Google?

Free — no credit card. Sign in with Google (free) to keep checking and save your results; Lifetime unlocks the full lists.

What you get

Verdict per bot

Allowed or blocked for 10 crawlers

Matching rule

The exact line that decides

Syntax issues

Lines crawlers will ignore

Sitemaps

Sitemap URLs declared in the file

About the Robots.txt Tester & Validator

robots.txt is the first file every crawler reads. One wrong line — a stray 'Disallow: /' after a redesign — can stop Google from crawling an entire site. AI crawlers like GPTBot and ClaudeBot follow it too, which decides whether AI assistants can read and cite your pages.

Nuqti downloads the site's robots.txt, validates every line, and tests your URL for 10 major crawlers using Google's matching rules (longest match wins, Allow beats Disallow on a tie, * and $ wildcards).

How to use it

1Enter a URLAny public page, e.g. https://example.com/page.
2Watch the check run liveEach step appears the moment it finishes — no waiting on a blank screen.
3Sign in free with GoogleSave your results and keep using every Nuqti tool on the free plan.

Frequently asked questions

Does robots.txt stop a page from appearing in search results?

No — it stops crawling, not indexing. A blocked URL can still appear (without a snippet) if other sites link to it. Use a noindex tag to keep a page out.

Should I block AI crawlers?

It's your choice. Blocking GPTBot/ClaudeBot keeps content out of their training and answers, but also makes AI assistants less likely to cite you.

Where must robots.txt be?

At the root of each host: https://example.com/robots.txt. Sub-domains need their own file.

What happens if robots.txt returns 5xx?

Google treats a server error as 'don't crawl anything' for a while. A 404 means 'crawl everything'.

Related free tools

XML Sitemap ValidatorIs your sitemap valid and full of live pages? URL counts, freshness, errors — and a status check on a sample.Canonical Tag CheckerWhich URL does this page tell Google to index? Check the canonical tag, conflicts and the target's status.Bulk HTTP Status Code CheckerPaste up to 20 URLs and see each one's status code, final destination and response time — in seconds.Free Backlink CheckerSee who links to any website — referring domains, anchors and link quality, explained in plain words.Website Authority CheckerA 0–100 score of how strong a site's backlink profile is — compared against millions of real websites.Free Keyword GeneratorTurn one seed word into hundreds of keyword ideas — with search volume and difficulty — for any country.

See all 49 free SEO tools →

Want the full picture? Every Nuqti tool is free to start.

Site Explorer, Keywords Explorer, Site Audit, Rank Tracker and AI visibility — one sign-in with Google, no card.

Start free with Google