Our new index is live and we have just set a few new records. We now have knowledge of over 3 trillion URLs and we have crawled over 300 billion of them. What’s more, we have dramatically improved our recrawl rate:
The data actually includes a few links crawled in December, but because these few days might obscure December’s graph (which would have been extrapolated in December based on just a day or so of information) we have put set the charts to map to the end of November.
Faster Crawl Rate
Our last record is a new crawl rate. You might have noticed in November we announced a new intelligent crawl methodology. We can now report that of the half a billion URLs that we are crawling EVERY DAY, a whopping 300,000,000 are RE-crawls. This means that links on busy pages will now get updated and checked much more between updates.
Latest posts by Dixon Jones (see all)
- A New Approach to Blogging. Expect a Better Standard. - February 7, 2017
- Fresh Index Doubles (Historic Index Passes new Milestone) - January 26, 2017
- Alexa top 1 Million sites is retired. Here’s the Majestic Million. - November 22, 2016