Build your robots.txt
Choose your platform, review the suggestions, then copy your draft.
Everything runs in your browser; this tool does not read or change your website.Robots.txt controls crawling, not guaranteed indexing or privacy. Blocked URLs can still appear in search. Use authentication for private content and crawlable
noindexpages for search exclusion. Keep CSS, JavaScript, images and public content crawlable.
One directive per line. Supported: Allow, Disallow, User-agent, Crawl-delay and # comments. Rules before a custom User-agent apply to *. A named crawler group replaces the wildcard rules for that crawler; repeat any needed restrictions. Blank Disallow means no restriction. Google ignores Crawl-delay. Noindex and Host are not supported here.
Your robots.txt draft
How to publish your draft
Back up and review your existing rules first. For WordPress, use your SEO plugin’s robots.txt editor or your host’s file manager. A physical file can override WordPress’s virtual file. Serve the file at your domain root as /robots.txt, even if WordPress lives in a subdirectory. Confirm Reading settings allow search engine visibility.
For a general site, upload the UTF-8 plain-text file to the domain root. For Shopify, use the theme code editor’s Templates folder to create or edit robots.txt.liquid, preserve existing customizations, and compare the rendered output with your backup.
After publishing, open https://your-domain.com/robots.txt, check the response is 200 with plain-text content, and review affected URLs in Search Console. Each hostname and protocol has its own robots.txt scope.
Reach out to Horhei for a robots.txt review before changing important crawling rules.
Exploring the World of Robots.txt: Your Guide to Best Practices
Have you ever wondered how search engines like Google know which pages of your website to crawl and index? Well, they rely on a little file called robots.txt to navigate through your site. It's like giving directions to a friendly robot explorer! In this guide, we'll...
Standard Robots.txt disallows for WordPress
Here are some common URLs in WordPress that you might want to disallow from search engines in your robots.txt file: wp-admin: Disallow access to the WordPress admin area.Disallow: /wp-admin/ wp-includes: Disallow access to core WordPress files.Disallow: /wp-includes/...