Majestic is happy to announce that the new Historic Index has not only arrived, but it is working better than before. It has been a few months in the making, and as we know you like transparency, here are the top line statistics:
The new Historic Index shows an EXTRA 369 BILLION URLs found WITH OVER 100 Billion extra crawled. Below is the change table.
The top line statistics:
Date range: 30 Jun 2012 to 30 Jan 2018
Last updated: 05 Mar 2018
| Old | New | Increase | |
| Unique URLS Found | 6,659,283,985,220 | 7,028,327,426,799 | 5.54% |
| Unique URLs Crawled | 1,413,052,059,901 | 1,532,667,950,732 | 8.47% |
| Percentage Crawled | 21.2% | 21.8% |
What changed and why did it take so long?
Our last Historic Index update was on the 2nd November last year. We usually update once a month, so when we knew it was taking longer than normal, we shared what we could through infrastructure improvements. Now that we are live, we are really happy to see that our changes resulted not only in a larger index, but also, we are able to have crawled a slightly higher percentage of all URLs seen.
The delay was partly changes to hardware – but the real gain was a software upgrade on our back end servers which helps us process everything a little faster. Crawling the Historic index takes… wait for it… 5 YEARS! Yes – it is over 5 years’ worth of crawl data, so when you change the way in which you process the data, it is a huge rollout which nobody was ever meant to see.
Some old school SEOs will remember Google trying something similar when they rolled out “Caffeine” in 2009 where they tried to change their indexing system without impacting regular users. They knew that power users would notice, though, so they asked for feedback from the SEO community. Majestic, by contrast, ONLY has power users! So we really did NOT want our stats to change just because we were making changes to our indexing system.
We thought we had achieved that, and we had, but one way or another, we built the entire Historic Index up with a slightly different setting on the new index. This means that we COULD have launched the Historic Index last month, but it would have effectively undone a change we made in the late summer of 2016 which caused more than a bit of discussion at the time. If we had released it as we did, then the flow metrics would have varied dramatically, and worse… a month later… they would have all reset again when we corrected the setting. Rather than upset our users twice, we chose rerun the Index build – and launch it right.
So here is it… over three months of infrastructure and Index improvements have led to a bigger index and a better overall coverage of the web, and we very much hope our users like it.
How to Enjoy the New Historic Index Data Today
“Pro” account and legacy account users have the ability to see the Historic index. If you are on a free or a “Lite” plan, you can upgrade here. The difference between Fresh and Historic is huge… about 4X the number of URLS found and 8X the number of URLs crawled than in the Fresh Index… so right there is some significant difference, but it is also valuable to understand the difference in the underlying data. The Fresh index DOES contain old links!
In fact, it will show all links that we see over a 90 day crawl cycle. So even if the link was 10 years old, if it is a good link and we are allowed to crawl it, we’ll pick it up much more often than once every 90 days… so it never falls out of the Fresh index. If it is DELETED or REMOVED for whatever reason, we will flag it as such, and remove it from the Fresh index after it has been flagged as removed for 90 days. This gives you a chance to spot lost links. The Historic Index is different, because the crawl cycle on which the entire index is based moves from 90 days to over 5 years. That is 20X the lifecycle. As you can imagine, this can give a different perspective. It is almost like the “archive.org” of links. You can see links that were deleted up to five years ago.
We hope you are happy to see the Historic Index back and larger. If you would like to delve in, but need to upgrade Pro accounts start at US $99.99 / UK £79.99 / Euro €94.99 monthly equivalent when purchased quarterly.
- Majestic Historic Index – normalising builds. - March 5, 2020
- Majestic Historic Index – Reflecting on 2019 & looking forward to 2020. - November 11, 2019
- Introducing Related Sites. Find similar websites using Majestic - October 25, 2019
it is nice article keep it up
March 9, 2018 at 5:31 amWell, you need it because Google likes to rank mobile-friendly website higher, and also because they are slowly switching toward the mobile-first indexing.
March 12, 2018 at 8:07 pm