Robots.txt is a standard content document for speaking with web Crawler/Spider/bots and train them to which region of site or site page is crept or not. Robots.txt is a Publicly accessible document and anybody can see effectively with part or url’s of site crawler output or not. As a matter of course Search Engine Crawler creep all that they conceivable.
How to Create robots.txt ?
There are numerous online instruments for making robots.txt record furthermore you will make this document manually.Before making robots.txt record you will see a few guidelines and regulation. I have disclosed here how to make a robots.txt record
In the robots.txt document USERAGENT line recognizes the web crawler and DISALLOW: line characterizes which some portion of the webpage is refused
 Here is the Basic robots.txt document
Disallow : /
In the Above assertion “*” demonstrate the all crawler/arachnid/bots and “/” characterize every one of the pages are prohibited. We can say it at the end of the day, that the entire destinations are refused for all crawlers.
 on the off chance that you need to prohibit any particular web crawler or insect not slither your website, then the robots.txt record will be
In above illustration I utilized Yahoobot and you will utilize that crawler which you not have any desire to creep you site.
 If you need to diallow all crawler for particular envelope or website pages then
Client specialists: *
 If you need to forbid all crawler for entire site with the exception of any particular crawler for permitted slithering then
 in next case you can utilize refuse to transform into permit standard by not entering any quality or/after semicolon (:) .