A quick look at the Fresh Index stats over the weekend showed (at the time) a Fresh Index with 188,575,337,745 unique URLs. Since the index is updating most hours these days, the numbers will be different by the time this blog hits your screen.
We are not quite at the 200 billion, but the improvements in index size are more than skin deep. Now we have better insight into which URLs are important, we have been able to improve the crawler controller.
This change in our technology – whilst having very little outward signs -should iteratively improve URL discovery and ultimately our Flow Metrics data.
The Real News
The real news is that the way we have optimised the crawler, we will be more likely to see an increase in the referring domains count than anything else. We are predicting this figure will increase by 20-25% when the weekend data has filtered into the system.
Here is Majestic’s bump in Referring Domains in the last few days, as taken on Sunday night.