I am organising a trip up to London with some friends in September. We’ve decided to save some money and use the MegaBus to travel from Southampton. Due to my love of all thing organising, I was put in charge of sorting out the costs etc. So first thing I did was to Google MegaBus. Now I was in a rush and my typing went awry and I accidentally typed ‘Meaga bus’. Without even really looking at the list of results, I clicked on the first link:

And this is what I was presented with:

So I had the quite reasonable impression that their site was down for maintenance. As an alternative I logged onto the National Express website to see how much their coaches were.

Once I had my information from the National Express I emailed my friends to tell them that I couldn’t get any information from MegaBus, but that the National Express seemed pretty reasonable and that we should just book with them. It was only then that my friend said she’d also tried the MegaBus site and was able to access it without any problems. So I went back to check and realised the error I’d made.

Even if you type ‘MegaBus’ correctly into Google, the maintenance page is still the third result. It would be interesting to look at their analytics to see how many visits this page gets; I bet it’s shockingly high.

So what can they do to fix this? It’s simple, just add http://uk.megabus.com/Maintenance.aspx to their robots.txt file.

What is a robots.txt file?

A robots.txt is a file you can create in which you can list all the pages (URLs) of your website that you don’t want the search engines to index. There are many reasons why you wouldn’t want certain pages to show in the search results. This MegaBus maintenance page is just one such example. Other examples include:

1. Random pages such as your thank you or error pages
2. If you are in the process of building your website or new pages and don’t want your unfinished work to appear in the search results
3. If your website includes information that you don’t want people to find whilst searching online (such as member only information that isn’t hidden behind a password)
4. Any duplicate content you may have. For example if you had content in both a web page and in PDF format you would want to add one of those to your robot.txt file to avoid being penalised

The file itself is a simple text file, which can be created in Notepad. It must be saved to the root directory of your site – which is the directory where your home page or index page is.

You can find out more about robots.txt files and how to create them here http://www.robotstxt.org

Have you ever found any pages in the search engine results that you shouldn’t have? Let us know in the comments!