Whilst we cannot speak for our competitors, here at Majestic SEO we know our customers have been asking for fresher data, which could provide feedback on what is happening to your clients’ link profiles right now. So, we have taken a big step towards “Real Time”, and have created a publicly available data set that we will aim to update on a daily basis. You could say that latency is to SEO what location is to Real Estate Agents.
We call this daily update the “Fresh Index”, and believe it marks an evolutionary step in the provision of backlinks intelligence. By limiting the data set to a rolling 30 day period ( approximate ), and investing heavily in new equipment we have been able to generate an index of about 40 billion unique URLs. This builds on our recent work on intelligent crawling, and the development of our toolsets – in recent months we have introduced our co-citation tool “Clique Hunter“, and “Site Explorer” – our backlink investigation tool.
We have made the Fresh Index available on all of our tools for subscribers with a Silver subscription or higher. That’s Site Explorer, Clique Hunter, Advanced reports and Standard Reports all available with up-to-date backlinks data, in addition to our existing comprehensive industry leading data set – which we have renamed “Historic” – due to its industry leading strength for historical backlinks analysis. Users on the free or bronze plan will be able to get a glimpse of the Fresh Index data on site explorer and on the backlink history screen.
How Fresh is Fresh?
In order to demonstrate how fresh our fresh index is, we looked hard to find a recent, verifiable “good news” story to track. A quick read of the British online tech publication – The Register – furnished us with an example able to bring a smile to many web developers faces – an article from March 4th about Microsoft launching a fresh campaign to eradicate Internet Explorer 6 from the web, in the shape of The IE6 countdown website. So how good are our crawlers at detecting this – we will let our backlink graph do the talking:
Clicking on the image should reveal an image with sufficient resolution for you to see the daily resolution of link accumulation – this graph works best with new sites and has some characteristics we will return to later. For now – Lets get a feel for the enhanced resolution this new Fresh dataset provides by comparing this graph with the historic graph for the same domain:
The “Fresh Index” graphs work best with new domains due to a subtle charactoristic of the Fresh Index – as it is rolling data, old links will get forgotten after about 30 days, leading them to be “rediscovered”, however the value of data for new site owners is such that we didn’t want to hold up the release of this important toolset – now, with the Majestic SEO index, feedback can be provided to clients during the sensitive “post launch” period.
All data that feeds into the Fresh Index is also added to the historic index which we currently release twice a month, as it takes approximately 8 days to build on our teraflop class cluster due to the massive – some would say majestic – amount of data being processed.
Is this onsite only?
This post would not be complete without our platinum customers who have access to our Backlink API – we are in the process of updating our developer docs at http://developer-support.majesticseo.com, and are happy to report that the Fresh Index will be made available by adding the parameter “Datasource”, with a value of “Fresh” to any of the existing API commands.
In Conclusion.
Its difficult to express our excitement about the Fresh index without breaking into superlatives. Whilst Dixon is manning our Expo stand at Search Engine Strategies in New York, our tech and support teams will be celebrating the release from the Majestic HQ in Birmingham, England. Rest assured, however, that we will not be resting on our laurels and are working hard on new features and developments. We see the release of the Fresh Index as just the starting point in a new wave of tools and services to drive Majestic fowards. 2011 is becoming an exciting year for link intelligence.
Steve has been engaged with the Birmingham tech scene for years, co-organising the grassroots open-source conference YAPC:: Europe 2008, and founding the West Midlands Java User Group in 2014.
- Introducing Duplicate Link Detection - August 27, 2021
- Python – A practical introduction - February 25, 2020
- Get a list of pages on your site with links from other sites. - February 7, 2020
This is the next big wonderful thing. I can’t express my pleasant surprise when I just checked one site and my jaw dropped in awe !
March 22, 2011 at 8:46 amNice one Alex! I do love majestic and I love its flexability… and with fresh date – it just gets better and better 🙂
March 22, 2011 at 10:51 amNice seeing you making progress!
March 22, 2011 at 11:50 amGreat work Alex thanks. Majestic is a fantastic tool ad should be at the heart of any SEO as it provides up to date content and is always getting better. Keep up the great work
March 22, 2011 at 11:54 amSo for the less technical folks like myself, would you say the most apparent use of the “freshness index” is monitoring successful campaigns within YOUR vertical?
We monitor the top ten competitor sites rankings, try to reverse engineer any that see’s bumps up in rankings, and we have no idea of freshness of links until now. Is this precisely why you made this?
Thanks!
March 22, 2011 at 6:34 pmHi Brent,
Fresh index is there to achieve 2 main goals:
1) show found backlinks much quicker
2) only use recently found backlinks to provide cleaner data with more live URLs
Specifically the charts can help detect increase in backlinks faster – possibly in matter of days rather than weeks as historic index takes more time.
March 23, 2011 at 11:28 amAlex
This is great. I recently became aware of MajesticSEO (I know, where have I been?) and have already decided to replace a few other lesser quality tools with this one. Especially important when you are a one-man show and time management is critical. Keep up the good work.
March 23, 2011 at 5:33 pmThanks for making this, it sounds interesting.
I gather that the results are more recent, and more likely to show active, juice passing backlinks, but I still find the differences between the two, and how to use both, a confusing issue.
For example, I want to add Majestic SEO link information manually to an excel file every week or two for all my sites, and I can’t decide whether to use the fresh or historic index.
I’m looking for the most accurate figures for active, juice passing links. You suggest the fresh index might offer this, but surely if it crawls fewer pages, it’s going to miss a lot of older pages that are relevant?
Would you suggest that I note down the figures for the fresh index every week, or the historic index every two weeks?
Thanks.
April 4, 2011 at 4:55 pmHi Daniel,
I’d recommend looking at two indexes – fresh is designed to provide data quicker and have only recently found links in it, thus providing cleaner data. Historical index on the other hand contains all data. Crawling rate isn’t a big factor since all data included in fresh index also gets included into historic index, it just takes more time for this to happen.
Alex
April 4, 2011 at 5:10 pmThis is truly a great service. With the addition of fresh date it enhances the user experience that much more. Thanks!
April 6, 2011 at 5:32 amthis is a really good service. Oh and BTW – I just love that great picture of the fruit – it’s so sharp and clear.
April 11, 2011 at 10:10 amHi Alex. I’m still trying to figure out a couple points about this. As I understand it, after 1 month the links drop off the fresh index “only to be rediscovered”. Does this mean when the site is crawled again, if the link is still there it is added back in to the fresh index? If the site is crawled more than once in a month, how many “links” will be added to the fresh index? Does it avoid duplicates?
April 11, 2011 at 11:30 amFresh index includes data crawled in the last 30 days – most of this data is on our automatic recrawl meaning that the same pages will be checked again and again: some of them are checked hourly, others checked weekly or even monthly.
We avoid counting duplicate links from the same pages: say if there is a link from our homepage and we crawl that page every day then we won’t count same link 30 times every day.
I hope that explains.
Alex
April 11, 2011 at 8:06 pmWe’ve got a good handle on our on-page optimization process and this new combo w/ Majestic and Market Samurai seems to address the more complex off-page optimization tactics needed to find those high quality links that we all seek to secure. Maybe a MoDo video is in the offing showing how this new fresh and historical feed works best?
April 11, 2011 at 2:09 pm