Historic Index

Saturday 25th April sees a new release of the Historic Index, just 15 days since the last update. We are now confident our new Historic build cluster is now fully operational. This post showcases recent achievements and explains how we seek to communicate Historic builds in future.

Historic Index is the most comprehensive Index available in Majestic. The Majestic Index contains five and a half years of crawl data, making it invaluable for competitor intelligence and deep insight into ongoing and legacy linkbuilding campaigns.

Why do you need to build an index?

An index build works out how all of an index’s URLs relate to one another then runs and re-runs iterative processes to calculate our Flow Metrics for each one. Finally, it prepares the result data in a way that gives you near-instant results when you search for any one of those URLs.

As the most recent Historic build has 9.2 trillion URLs, you can probably start to imagine the processing power and time needed to build a stable and reliable index.

The new cluster, designed to deliver more up-to-date data, more often, went online on March 5th.

New Historic Cluster in Numbers

As you can see from the data below – the new cluster is performing well. Despite an ongoing increase in the amount of data, the build time has reduced from nineteen to fifteen days over this period.

25th April

Build Time: 15 days
Unique URLs crawled: 2,382,856,824,596 (2.38 trillion)
Unique URLs found: 9,203,341,592,652 (9.20 trillion)
Date range: 09 September 2014 to 10 April 2020

10th April

Build Time: 17 days
Unique URLs crawled: 2,371,111,956,878 (2.37 trillion)
Unique URLs found: 9,179,565,120,114 (9.18 trillion)
Date range: 24 August 2014 to 25 March 2020

24th March

Build Time: 19 days
Unique URLs crawled: 2,361,260,434,768 (2.36 trillion)
Unique URLs found: 9,165,400,849,448 (9.17 trillion)
Date range: 07 August 2014 to 08 March 2020

What this means for you

We plan to keep up this pace of Historic Index updates. While it is possible that timescales may slip on a given build during hardware maintenance or software upgrades, two updates each month appears achievable

A knock-on effect of a regular build and the frequency of update is that Historic Index updates will become a little less news-worthy. We plan to no longer release regular Historic Update notices on this blog.

We do recognise that the change to Historic Index notifications may concern some customers who will time link research to releases of a new Historic Index. We are investigating a number of options to keep you notified of releases to Historic and we are interested in your views and ideas on how we can support you in this process.

You can always find information on our index updates and build dates on the homepage of majestic.com, and in the footer of all pages.

How to get Historic

Historic Index data is available on our Pro and API plans. If you subscribe to our Lite plan, upgrades are available today.

What next?

We are delighted with the performance of the new Historic Cluster, but will not rest on our laurels. We plan to continue to develop our link mapping technology, and have a number of exciting developments planned over the next weeks and months.

As always, the easiest ways to follow all minor and major Majestic updates are to follow @Majestic on Twitter or join the Majestic Community on LinkedIn.

Back