A quick look at the Fresh Index stats over the weekend showed (at the time) a Fresh Index with 188,575,337,745 unique URLs. Since the index is updating most hours these days, the numbers will be different by the time this blog hits your screen.
We are not quite at the 200 billion, but the improvements in index size are more than skin deep. Now we have better insight into which URLs are important, we have been able to improve the crawler controller.
This change in our technology – whilst having very little outward signs -should iteratively improve URL discovery and ultimately our Flow Metrics data.
The Real News
The real news is that the way we have optimised the crawler, we will be more likely to see an increase in the referring domains count than anything else. We are predicting this figure will increase by 20-25% when the weekend data has filtered into the system.
Here is Majestic’s bump in Referring Domains in the last few days, as taken on Sunday night.
- How Important will Backlinks be in 2023? - February 20, 2023
- What is in a Link? - October 25, 2022
- An Interview with… Ash Nallawalla - August 23, 2022
I have another optimisation for you – don’t crawl any page that has “bieber” in the URL, and avoid about 5billion pages 🙂
October 22, 2012 at 10:01 am> wise suggestion!!!
October 24, 2012 at 12:26 am