Control Google Bot With The New X-Robots-Tag
Google has extended its support for Google Bot restriction by giving us web developers a new tool to stick in our belt. It was announced today on the Google Blog that you can now control access to your non-HTML files on your website with a simple header. The header X-Robots-Tag will allow you to do everything the normal Robots Meta tag will, but now you can do it for the PDF, Word, Image, and any other document you can think of that is served via HTTP. They also announced on the same post a new type of exclusion cause that lets you set when the document will be unavailable, see below for more information on this new feature as well as currently supported ones for use with X-Robots-Tag: ...