New 15-year-plus Historic Index

The Majestic Historic Index has been upgraded to contain over FIFTEEN years of data. With links dating back to June 2006, you can now search through some of the rarest commercially-available backlinks on the planet.

From Five Years to Fifteen

When we started to crawl the web, we had to decide how best to slice a huge amount of data. 

Our Fresh Index is a view on the contemporary web, containing links discovered in the past few months.

The Historic Index obviously had to be bigger and longer. Due to the processing challenge of compiling the index, we decided to cap it at the longest manageable time, which was around 5 years.

We are delighted to announce that following our recent infrastructure investment, we have maximised the timescale of the Historic Index to cover the entire 15-and-a-half years of our backlink-recording history. 

That’s almost every link that we have found since we started looking on June 6th 2006.  

Comparing Indexes

Even with (only) five years of depth, our Historic Index was a bit of a behemoth. Indeed, in backlink comparisons (that are almost always a bad idea), it is usually omitted from comparison studies, with the 90% smaller Fresh Index used in its place.

With today’s data boost, we’re delighted to share that the Historic Index is now TEN YEARS older, and contains 7.5 TRILLION additional URLs that were not in the old index. For link audits, research, and general losing-yourself-down-a-rabbithole, that’s an astonishing depth of Link Intelligence.

And 15.5 years is the youngest that the Historic will be. The Index will now age at the same rate as the rest of us, with a fixed start point of June 2006.

Watch the Web evolve

To give you a feel for how long we’ve been collecting data, take a look at this cumulative Referring Domains graph for some (current day) social websites.

While TikTok feels ubiquitous in 2021, it’s only really been picking up significant numbers of web links since early 2020. Its link count is dwarfed by soundcloud, who have an ten extra years of backlink collection. With this update, you can now view collected data for around four years before even Soundcloud started to collect a volume of links.

Here is another look at cumulative Referring Domains, this time for old search engines. You can see that most domains followed the same trajectory. Altavista and Lycos did well in early 2007, but most never really gained traction post-2010.

However, look at those clear spikes in Ask.com. They may be linked to purchases of about.com (2012) and askfm.com (2014).

A Nostalgic Look at Old Links

This week in the virtual Majestic office, we’ve enjoyed browsing through old data, particularly the older members of the team.

Here are some of our favourite data items that are only now available with the new and boosted Historic Index.  

Note: To make it easier to see the link, we have removed page titles from these examples. Later in the blog post, we explain why.

Crypto.com

In 2006, crypto.com was a policy and resource source for encryption.  

This link, on June 7^th 2006, is from the first full day of link discovery for the new Historic Index.

Airbed & Breakfast

It has taken the rolling 15 years of data into our Historic Index for most of your Majestic team to discover the etymology of AirBnB. It’s Airbed & Breakfast, not Air Bed and Breakfast. Here’s their first recorded links, from October 2007.

The birth of a TLD

Jumping forward to 2008, you can witness the beginnings of a new TLD, with some of the first links to the registrar for Montenegro’s new (at the time) .me domain.

T.co

In late 2010, Twitter launched their in-house link shortening service. Our earliest links are from October of that year.

Facebook and fb.com

Later, in 2011, Facebook bought fb.com for a rumoured $8.5m. 

The bigger Historic Index has links from way back in 2006, when the American Farmers Bureau owned the domain.

Become a patron

May 2013 has our first direct link to Patreon (note the possible ‘typo’ in the Anchor text, from before the brand name was established!)

Subtleties to keep in mind.

While it’s been great to look through some old links, we did notice a few things that we’d like to share with you.

Bear in mind that the web is a big place, and an archive of how 17 trillion URLs evolve over 15 years is bound to throw up some oddities.

Link indicator flags

For instance, a lot of backlink statuses have spent a few years in ‘cold storage’ in our hard drive archive. This means that they haven’t been scanned in the years since they disappeared from the Historic Index. Link indicator flags, like deleted or redirects will show the link status from the last time we recorded them, and a lot may have changed in the meantime.

Page titles are contemporary, not historic

When we fetch page titles for domains or URLs, we fetch the latest title for that entity that we have in the relevant index.  So, for a 15-year-old index, it’s likely that you will discover instances where it looks like an old link points to a new entity. 

For instance, in the screenshot of one of our examples (above), we increased readability by removing the ‘Facebook – Log In or Sign Up’ page title. 

For transparency, here is that link, but with the page title reinstated.

You can see how “weird” page titles may initially look confusing, but once you realise that these are the latest available titles for these target pages, or domains, and NOT the target site page title at the time of the link, it may be easier to work out what’s going on.

Watch out for redirects

While the domain name was registered in August 2008, AirBnB officially launched in 2013… so how can they have this link from 2006?

If you look closely at this backlink (screenshot below), you may be able to figure out what’s going on.

It looks like we found a link on August 28^th 2006, from silvercity.org to walnutcreekcottages.com. 

At some point in the future, walnutcreekcottages.com was 301 Permanently Redirected to an AirBnB product listing page. This redirect makes it look like this link points to a site that didn’t exist at the time.

Watch out for redirects if you’re puzzled by an old link coupling.

Seven trillion new URLs is a LOT of new links, and ten years is a long time. So, we’re sure that some of you may find some examples where data looks odd to you. If you find anything that you want to flag up, please get in touch with our Customer Service team. They are very nice.

How to find the oldest links

Finding the oldest links to a site is easy, but maybe not obvious.

First, choose a domain in Site Explorer.

Move to the Backlinks (or Context) tab and switch to the Historic Index. After the page refreshes, in the filter bar, choose ‘Order by’ [First Indexed] and [Ascending]

Your results page will have the oldest links to any domain or URL of your choosing.

All this, at no extra cost

To show the new depth of data, this blog post has taken a nostalgic look at links from the past.  But for many SEOs, particularly those who deal with domain name reselling, toxic link recovery, link audit and entrepreneurial data analysis, this new information has the potential to be an exciting treasure trove that you won’t find anywhere else.

If you would like to check out the new data, the Historic Index is available on our Pro and API plans.  And we are delighted to announce that all of this new data is included in existing Pro and API plans.  That’s an additional TEN YEARS of backlink data rolled into in your existing subscription.

The Majestic Historic Index is truly unique in size and legacy, and we don’t believe that any other backlink tool shares this type of Link Intelligence at this volume and timescale.  Pro and API plan prices start from $99.99 / €99.99 / £79.99 per month. There’s no contract and you can cancel rebilling at any time. Lite customers can upgrade today.

Back