Almost every webmaster tries to optimize the pages of his site in such a way as to get as many indexed elements as possible. But in some cases, on the contrary, you need to hide some pages, for example, a form for entering registration data, recovering passwords, etc.
It is necessary
Editing the Robots.txt file
Instructions
Step 1
The easiest way to close the indexing of several sections or a specific entire site is to make the appropriate entries in the robots.txt file. He is used as a guide for search engines. When entering the pages of the site, the search robot refers to this file, if it is not there, it indexes the entire site, otherwise it follows the recommendations specified in this file.
Step 2
If you created your site yourself, and you do not have such an application, in the form of a text document indicating the prohibitions, it is easy to create it yourself. Open a new text document, for example, using the Notepad program. Place the following lines in the body of the document: User-agent: * Disallow: /file.htmlDisallow: / directory /
Step 3
Now let's figure out what you asked the search engine as a task. The "*" sign opposite the User-agent means that this instruction must be executed by absolutely all file systems. For example, if you write the value “Yandex” instead of “*”, this appeal will be read only by the robot of the search engine Yandex. Disallow statements prohibit indexing of a specific section (directory) or file (.html file).
Step 4
After compiling the robots.txt file, you can go directly to the texts on the site pages. As you know, search engines don't like having more than 3 external active links on your pages. What if you need a nosebleed but post 5 or more links. In these cases, you can use an attribute or a tag.
Step 5
To close links from indexing by the Google search engine, you need to bring the link into the following form: anchor links. The body of the attribute must be inserted into any part of the link, not forgetting about the separation by a space, i.e. the attribute can be placed between "a" and "href" or between the link and its anchor.
Step 6
To close links from indexing by the Yandex search engine, you need to bring the link into the following form: anchor link. The tag is placed at the beginning and at the end of the link. Do not forget that this tag is paired: the second part, unlike the first, comes with a "/" sign.
