Filter by Crawl Result and Get a List of Your Broken Pages

Did you know that with Majestic you can analyse your pages within your own site? It’s true! Just by using the Pages tab within Site Explorer, you can see how many links are linking to each page along with the Crawl Result. See if any of your pages has issues, blocks or redirects. Using this data properly, you can manage your website more efficiently while improving your visitor experience.

How Do I Find My Broken Pages?

A broken page is usually identified by a HTTP 404 crawl result.  HTTP 404 page is returned to the user when they try to reach a specific page that either doesn’t exist anymore, has been misspelled or moved. Usually 404 pages can be easily fixed by yourself or your Webmaster. However if you prefer to keep this in place, a good 404 page should provide visitors with information on where they should go next. We have all been there, we have hit a dead page with no help whatsoever and so we get frustrated and move onto another search or website.

Why Is This Data Important?

If the website contains a large number of 404 pages, it may be perceived by crawlers as a low quality domain with low engagement, as it doesn’t show any care for a user’s navigation around the web or a proper user experience. Therefore, you should proceed with caution:

• Firstly, make sure the page returns a correct 404 status to be properly indexed by bots like Google or Majestic.
• Create a good looking, funny page that keeps the users engaged and direct them elsewhere on your site.
• You may be losing link juice and trust with links pointing to a dead page. Contact sources and request they change their links to not lose any brownie points with Search Engines.
• Provide useful navigation. Look at your 404 pages as an opportunity to inform users of other content  on your site that could be suitable. You can always add ‘domain search box’ to make it even more convenient and quick.
• Redirect these pages to sources that would be a good match and makes sense,

What Are The Other Results?

Not only has Majestic given you the ability to filter your broken pages, but you can also check all redirects, non-crawled results or blocked pages.

Redirecting is the process of forwarding one URL to a different address. Redirecting links should help point both users and Search Engines to the right URL. Don’t redirect all of your broken links to your homepage, it may simply be too annoying for your visitors.

It can take Search Engines a little bit time to recognize and credit the new sources, so the quicker you fix your redirects the more time you are allowing bots to register this. There are two types of redirects, HTTP 301 and HTTP 302. HTTP 301 Redirect is a permanent change and Search Engine’s will interpret it as a page moving its’ content to a new address with a new URL. HTTP 302 Redirect on the other hand is for temporary changes, usually during maintenance or redesign. Using Majestic to check your live redirected pages are placed properly, the type of redirect is correct and the URL you are sending the user to is correct.

If you see the result ‘Not Crawled’, this means that that our crawlers has not yet fully crawled it but the URL is known, usually because a link has been found pointing to it. We just haven’t yet got round to visiting it ourselves. If you wish for this page to get crawled as soon as possible, try submitting the page via our URL Submitter. Please allow up to 72 hours for our bot to visit these pages and update our indexes.

Results such as HTTP 403/406 or Blocked by Robots.txt equals blocked access. While robots.txt is a file usually something you have control over, it is possible that your hosting company blocks good and bad bots in order to save on bandwidth usage or security – mainly resulting in HTTP 406/403 errors. As a consumer you have a right to decide whether you want the bots to crawl your pages. If not, that’s OK, however it can affect your domain with decreased traffic, missing some opportunities to exchange good backlinks, not being shown in the results of some Search Engines; as well as not being  properly assessed by Majestic. If you discover that such a block is in place, contact your server or hosting provider and request they remove all blocks for the MJ12 Bot and to white-list this.

3 Simple Clicks, Is That All It Takes?

The answer is yes.

All you need to do is to type in domain’s name into the Site Explorer and go to the Pages tab:

Then click the Export Data button:

This should allow you to see this small pop-up:

While scrolling the exports drop-down options, you can see a full menu:

Depending on your subscription level, how many of your website’s pages you can download may be limited. For example on a Silver subscription, you would be able to export and filter up to 5,000 results. With the Gold plan, up to 15,000, and up to 30,000 when using a Platinum plan.

How do you plan to use our new filter options? Do you know any other techniques to convert dead HTTP 404 pages into a great navigation point? We would love to hear your opinions, so please comment below.

Back