Biggest Link Index Hits New Highs with Historic Update

By Dixon Jones |

Majestic’s Historic Link Index has just been released and it is the most comprehensive link index ever! Although Majestic does not report the overall volume of links known (and may not even know!), they do report on several some vital statistics. For example:

The Number of pages CRAWLED in Historic Index

This has rocketed 13.5 percent from 1.0 TRILLION to 1.14 TRILLION pages. For the purists, the new number is :

1,136,685,742,504

Rumours that Majestic might be falling behind are somewhat exaggerated. Now other Indexes are aging in a similar manner to ours, a cursory look at the other index numbers revealed intereting numbers

This number is 700% larger than the numbers quoted by our nearest rival!

700 percent equates to a staggering difference, lathough the performance enhancement is not entirely down to an increase in crawling capacity (which has not especially chaged. If anything, the actual crawling rates have reduced. However, Majestic’s crawling has increased efficiency over speed. More about this on Thursday, but in short we are able to better surface new URLs which – whilst seen – have not yet been crawled.

Next, the following increment is even more impressive:

The Number of URLs in the Index

The number of URLs Majestic has seen and recorded in the Historic Index has increased this month by a whopping 27 Percent to over 6 TRILLION URLs! Again for the purists, the new number is:

6,043,450,503,470

The Cartographer’s Paradox

Many customers of Majestic take the issue of size extremely seriously. This writer has had long conversations with Googler engineers, Bing Engineers and other big web crawlers over the years and eventually all crawlers understand the limitation of such a quest. Majestic’s head of development talks about the cartographer’s paradox. When trying to map the world, you could get more and more and more detailed… but at SOME point, to map absolutely everything, you would have to create a whole new world, identical to the one you are trying to map. On Thursday, we will talk more about this and how it applies to Majestic’s crawlers. In the meantime, please enjoy our new – biggest ever – Historic Index.

Majestic aims to document historic index numbers in this feed.

 

Dixon Jones
Latest posts by Dixon Jones (see all)

Posted In: Updates

Comments

Comments are closed.