A sitemap and robots.txt
A sitemap is a list of addresses you ask Google to crawl. robots.txt says what not to crawl. A common mistake is blocking the site by accident, or submitting a map while the pages are not linked.
Two files, two jobs
sitemap.xml usually lives at /sitemap.xml. It is not a ranking. It is a discovery request. A page missing from the map but linked in the menu can still be crawled. A page in the map but blocked in robots will not be crawled.
In robots, Disallow only what should not be in Google, such as an admin area. Not / and not the design files. Blocking the design makes it harder for Google to see the page as a visitor does.
| File | It says | Mistake |
|---|---|---|
| sitemap.xml | These are the public URLs | An admin URL or a thank-you page |
| robots.txt | Do not enter here | Disallow: / on the whole site |
| A line in the map | A crawl request | Expecting a rank the same day |