# https://help.ubuntu.com/robots.txt # # Notes: # # 2022.04.20: Because we always forget, with the last edit we put in # several years worth of Disallow directives. Now we need # some more. # # 2016.11.23: We, the docs team, do not actually have access to the # web server access logs, which would help in improving this # robots.txt file. Some things we do herein are based on experiences # from other web servers, where we do have access to the access logs. # Always keep in mind that crawlers, and google in particular, seem # to keep looking for deleted content for a very very long time after # it is gone. # # 2016.11.23: Canonical is moving the CloudDocs elsewhere. # A disallow lts/clouddocs is being added in order to assist # web crawlers to realize that they should delete that content. # It should be left for at least a year, probably two. # For this type of server, a crawl delay of 20 is too long, # changing to 5, which might still be a little long. User-agent: * Crawl-delay: 5 Disallow: /img/ Disallow: /libs/ Disallow: /14.04/ Disallow: /16.04/ Disallow: /16.10/ Disallow: /17.04/ Disallow: /17.10/ Disallow: /18.04/ Disallow: /18.10/ Disallow: /19.04/ Disallow: /19.10/ Disallow: /20.04/ Disallow: /20.10/ Disallow: /21.04/ Disallow: /21.10/ Disallow: /22.04/ Disallow: /22.10/ Disallow: /23.04/ Disallow: /23.10/ Disallow: /24.04/ Disallow: /24.10/ Disallow: /25.04/ Disallow: /25.10/ Disallow: /26.04/ Disallow: /26.10/ Disallow: /lts/ubuntu-help/ Disallow: /stable/installation-guide/ Disallow: /stable/serverguide/ Disallow: /dev/ # Leave the following two lines at least until 2017.12.01, perferrably 2018.12.01 # 2022.04.20 left these Disallow: /stable/clouddocs/ Disallow: /lts/clouddocs/ Disallow: /community/*?action= # as of 20.04 the serverguide has moved, and bots should find and index it from its new location. Disallow: /lts/serverguide/