How to generate robot.txt for any website
Most people preferred google indexing all page while some don’t, there is a saying that goes that “ones man meat is another’s man poison“
Your blog robot.txt file sends message to web crawlers which enable Google bot to determined which pages to accept or ignore,this function is optional and can be used in different varsities.
This function can be used direct Google to block a particular page from getting indexed on search engines ,It can be used for in many ways to improve pages
Step 1: log on to google search console, navigation to Crawl, click robot.txt column on the left side
Step 2: When clicked you will be taking to robots.txt root from there full editing can be carry out .
Step 3: once you are done with the “txt generator” editing with your desire looks on how you want your robots.txt click Submit
Comments