Hello friends, welcome to the techparimal site, today we will discuss an important topic. Do you have a WordPress site? But do you know what should be the right or perfect robots txt for that site? Today I will give you a big and complete robots txt for WordPress. Before that, you need to know some basic ideas about the robots.txt file.
What is robots txt for WordPress?
However, your blogger site or WordPress site must submit the robots.txt file to your site. But what exactly is this robots.txt file? Search engines such as Google, being, or Yandex use a type of program that collects some important information stored on the Internet. And traveling from one website to another, this system is called bots, crawlers, robots, or spiders.
At the beginning of the Internet, some website owners were annoyed with these crawlers because the crawler visited the website repeatedly and as a result, the server of the website became very slow. As a result, the website did not reach the visitors, and the website’s resources were depleted.
To solve this problem, crawlers create robots.txt files, which allow the website owner to crawl certain files on his site or not allow crawl of certain files.
The robots.txt file is a text format that resides in the root folder of the website. The crawler will search by linking to /robots.txt after a site link, if no robots find the file, they will crawl all the files on that site. Otherwise, if the crawler receives the robots.txt file, it crawls the site’s file with the owner’s permission.
What should robots.txt be for your site?
Learn what robots.txt is and how it works from the paragraph above. Let’s see what will be the correct robots.txt file for your site? However, if you have a Blogger site, use the default robots file or generate it from the Blogger robots.txt generator site.
But if you have a WordPress site, you must be using either an SEO plugin, be it rank math or Yoast. How to set up your robots.txt file for these two plugins? In the robots txt for WordPress, what do you use the allow tag for? What other disallow tags to use? Look at that carefully. Let me show you now, first download the robots.txt file.
How to Submit robots.txt on Rank Math?
1. Go to your WordPress admin panel,
2. Click on General setting under Rankmath option,
3. Then click on edit robots.txt,
4. Now you can get the default robots to file in a text box, replace this code with the downloaded robots file,
5. Save changes.
How to Submit robots.txt on Yoast SEO?
1. Go to WordPress admin panel,
2. Click on the SEO option,
3. Then click tools option,
4. Open file editor,
5. Click on create robots.txt
6. Replace old robots file with new robots file,
7. Save changes to robots.txt
Note 1:- The robots.txt file you downloaded is mine. So the code contains my site URL, change my URL here and set your site URL.
Note 2:- After setting up a new robots file, an error will appear in your search console, which is “blocked by robots.txt”. Don’t worry, it’s not an error. Google wants to crawl URLs you’ve blocked with robots files.
Robots txt for WordPress in Hindi Video
Conclusion
If this post helps you, please share it with your blogging friends. If the new robots.txt brings up an error for your WordPress, please comment and I will reply. To follow my site, click on the join to telegram option in the right sidebar.
