Written by Admin on 2025-05-06

Downloading Robots.txt for WordPress

If you're running a WordPress website, you may be familiar with the term "robots.txt". This file tells search engine bots which pages of your website can be crawled and indexed, and which pages should be excluded.

By default, WordPress includes a basic robots.txt file that allows search engines to crawl your website's content. However, if you want to customize this file, you'll need to create and download your own robots.txt file.

Here's how to download a custom robots.txt file for your WordPress website:

1. Log in to your WordPress dashboard.
2. Go to Settings > Reading.
3. Scroll down to the "Search Engine Visibility" section.
4. Verify that the option "Discourage search engines from indexing this site" is unchecked.
5. Save changes.
6. Open a text editor (like Notepad or TextEdit) on your computer.
7. Copy and paste the following text into the text editor:

User-agent: * Disallow: /wp-admin/ Disallow: /wp-includes/

8. Customize the "Disallow" lines to exclude any directories or files you don't want search engines to crawl. For example, if you have a membership site, you may want to exclude the directory where your members-only content is stored.
9. Save the file as "robots.txt".
10. Navigate to your website's root directory via FTP or your web hosting control panel. This is usually the "public_html" or "www" folder.
11. Upload the "robots.txt" file to the root directory of your website.

Congratulations! You've successfully downloaded and uploaded a custom robots.txt file for your WordPress website. Now, search engines will better understand which pages to crawl and index, which can improve your website's search engine rankings.