Whilst we cannot speak for our competitors, here at Majestic SEO we know our customers have been asking for fresher data, which could provide feedback on what is happening to your clients’ link profiles right now. So, we have taken a big step towards “Real Time”, and have created a publicly available data set that we will aim to update on a daily basis. You could say that latency is to SEO what location is to Real Estate Agents.
We call this daily update the “Fresh Index”, and believe it marks an evolutionary step in the provision of backlinks intelligence. By limiting the data set to a rolling 30 day period ( approximate ), and investing heavily in new equipment we have been able to generate an index of about 40 billion unique URLs. This builds on our recent work on intelligent crawling, and the development of our toolsets – in recent months we have introduced our co-citation tool “Clique Hunter“, and “Site Explorer” – our backlink investigation tool.
We have made the Fresh Index available on all of our tools for subscribers with a Silver subscription or higher. That’s Site Explorer, Clique Hunter, Advanced reports and Standard Reports all available with up-to-date backlinks data, in addition to our existing comprehensive industry leading data set – which we have renamed “Historic” – due to its industry leading strength for historical backlinks analysis. Users on the free or bronze plan will be able to get a glimpse of the Fresh Index data on site explorer and on the backlink history screen.
How Fresh is Fresh?
In order to demonstrate how fresh our fresh index is, we looked hard to find a recent, verifiable “good news” story to track. A quick read of the British online tech publication – The Register – furnished us with an example able to bring a smile to many web developers faces – an article from March 4th about Microsoft launching a fresh campaign to eradicate Internet Explorer 6 from the web, in the shape of The IE6 countdown website. So how good are our crawlers at detecting this – we will let our backlink graph do the talking:
Clicking on the image should reveal an image with sufficient resolution for you to see the daily resolution of link accumulation – this graph works best with new sites and has some characteristics we will return to later. For now – Lets get a feel for the enhanced resolution this new Fresh dataset provides by comparing this graph with the historic graph for the same domain:
The “Fresh Index” graphs work best with new domains due to a subtle charactoristic of the Fresh Index – as it is rolling data, old links will get forgotten after about 30 days, leading them to be “rediscovered”, however the value of data for new site owners is such that we didn’t want to hold up the release of this important toolset – now, with the Majestic SEO index, feedback can be provided to clients during the sensitive “post launch” period.
All data that feeds into the Fresh Index is also added to the historic index which we currently release twice a month, as it takes approximately 8 days to build on our teraflop class cluster due to the massive – some would say majestic – amount of data being processed.
Is this onsite only?
This post would not be complete without our platinum customers who have access to our Backlink API – we are in the process of updating our developer docs at http://developer-support.majesticseo.com, and are happy to report that the Fresh Index will be made available by adding the parameter “Datasource”, with a value of “Fresh” to any of the existing API commands.
Its difficult to express our excitement about the Fresh index without breaking into superlatives. Whilst Dixon is manning our Expo stand at Search Engine Strategies in New York, our tech and support teams will be celebrating the release from the Majestic HQ in Birmingham, England. Rest assured, however, that we will not be resting on our laurels and are working hard on new features and developments. We see the release of the Fresh Index as just the starting point in a new wave of tools and services to drive Majestic fowards. 2011 is becoming an exciting year for link intelligence.
I'm the Chief cat herder and hirer for our development team. I work with Dixon and Alex to contribute to the design and oversee the development of software which is developed by our talented team of software engineers.