The pages of each site are regularly crawled by search robots, which process the information on the site so that it is available at the request of users in search engines. Visits of visitors from search engines have a positive effect on site traffic and stimulate the development of the resource. But, if there are pages on the site, the information on which is not intended for the Internet audience (for example, the personal pages of users or the registration page), then the transition from search engines to such pages is not needed and it is better to prohibit this information from indexing.
Necessary
- - have your own website
- - to know the links to the pages, the indexing of which should be prohibited.
- - have at least basic knowledge of HTML
Instructions
Step 1
In order to prevent the search robot from indexing certain pages of the site, you need to make changes to the robots.txt file, if it is already on your site, or create this file.
Step 2
If the robots.txt file is already uploaded to the site and it is possible to modify this file directly from the resource control panel, then open robots.txt through the web service for editing.
Step 3
If there is no way to make changes to the desired file directly from the control panel, then open robots.txt in a browser by clicking on the link to this file. Copy all the data from the page that opens and paste it onto the sheet in Notepad.
Step 4
If there is no robots.txt on the site, open the "Notepad" program, in the new document of which you will need to enter the necessary code.
Step 5
On a blank sheet of text document, write the first required line:
User-agent: *
Step 6
Then go to the next line by pressing the Enter key and type the following code:
Disallow: / index / page3Disallow: indicates no indexing, and / index / page3 is a link to a page that does not need to be indexed. Instead of the link / index / page3, insert a link to the page of your site that you want to block for processing by robots.
Step 7
Note that the domain name itself is missing at the beginning of the link; its presence in the entry implies the forward slash. When crawling a site, a search robot automatically substitutes a domain name in such links. Therefore, write links to the necessary pages in this way.
Step 8
After Disallow: you can place not only a link to a specific page, but also a link to any section or directory of the site. Similarly, you can prohibit indexing of the desired part of the site.
Step 9
If it is necessary to prohibit processing by robots of several different sections of the resource, write a link to each section or page on a new line, at the beginning of which do not forget to put the Disallow: entry.
Step 10
To complete changing robots.txt, click the "Save" button while editing the file in the control panel, or save a text document with entries under the name robots.txt and upload this file to the site.
