Our new index is live and we have just set a few new records. We now have knowledge of over 3 trillion URLs and we have crawled over 300 billion of them. What’s more, we have dramatically improved our recrawl rate:

Index Statistics

307,174,759,713
Pages crawled

3,003,219,049,123
Unique URLs

The data actually includes a few links crawled in December, but because these few days might obscure December’s graph (which would have been extrapolated in December based on just a day or so of information) we have put set the charts to map to the end of November.

Faster Crawl Rate

Our last record is a new crawl rate. You might have noticed in November we announced a new intelligent crawl methodology. We can now report that of the half a billion URLs that we are crawling EVERY DAY, a whopping 300,000,000 are RE-crawls. This means that links on busy pages will now get updated and checked much more between updates.

Dixon Jones
Latest posts by Dixon Jones (see all)

Comments

Comments are closed.

THANK YOU!
If you have any questions in the meantime, please contact help@majestic.com
You have successfully registered for a Majestic Demo. A Customer Advisor will contact you shortly to schedule a suitable time to connect.