Now, Create 'robots.Txt' document at your root directory. Replica above textual content and paste into the textual content document.
Robots.Txt Generator generates a record this is very a lot contrary of the sitemap which shows the pages to be blanketed, therefore, robots.Txt syntax is of awesome significance for any internet site. Each time a search engine crawls any website, it always first seems for the robots.Txt file this is located on the domain root level. When identified, crawler will examine the report, and then discover the files and directories that may be blocked.