Majestic SEO are excited to announce that we are  working on a new version of the fresh crawl data to be launched in January 2011.

A number of users have noted that the “fresh crawl” column in the advanced reports is not showing new links each day. When we first introduced this data, back in an upgrade in 2008, it was with a very small user base. we were therefore able to have the crawlers check the urls at the root level against a very small list of domains being watched by users in our system. Nowadays, however, there are tens of thousands of domains being analyzed by users at any one time and the system simply will not scale in its current form. However – as you may have seen in other news – MajesticSEO has now started to roll out a much more intelligent form of crawling. the new crawl will help us build a much more effective fresh index, which will be able to not only give you data at the root domain level, but also at the URL level. The challenge, though, is that we needed to get the new crawler rolled out to our crawler network first, before we can build a new fresh index program – and the new crawl can’t handle the antiquated version of this code.

We know that some of our customers really liked the daily updates and apologize that the system is now just too large to cope with them in their current form, but while the bad news is that you will not get daily updates for a few months, this is already mitigated by giving you and entire index update every two weeks through the improved indexing methodology and is further mitigated by the promise of daily updates in the new year which we feel will be worth the wait.

Comments

  • Deep Ripples Bill

    ***sigh of relief*** Exactly what I wanted to hear, well, almost. I’d rather hear the daily updates were coming tomorrow. But I’ll gladly wait. Great work!

    December 2, 2010 at 9:11 pm
  • John Fairley

    It’s January! Still 21 days remaining in the month. Will we get this highly anticipated update this January? Fingers and toes crossed. Thanks for all the hard work.

    January 11, 2011 at 5:42 am
  • Dixon

    Right! Never make promises on a blog post. Alex told me that… :)

    We should be able to at least bring back the old daily updates style by the end of the month – or there abouts – which will then give us a bit more room to get to a better permanent solution.

    January 11, 2011 at 2:11 pm
  • Andrew Biggs

    C’mon guys halfway thru Feb, still no daily updates since October 22!!!

    At least give us an update please?

    February 16, 2011 at 11:03 am
  • Ash Nallawalla

    I’m not worried about daily updates, but I wouldn’t mind if dead backlinks were purged. By way of an example, a site sponsorship back in 2008 might still show as hundreds of links in the data, but those links haven’t existed for perhaps two years now. (I am viewing via Raven, not directly)

    March 21, 2011 at 2:11 pm
  • Alex Chudnovsky

    Hi Ash,

    We’ve just released Fresh Index that Raven as well as our other partners will be using – http://blog.majesticseo.com/development/site-explorer-fresh-index/ – this index only contains backlinks found in the last 30 days, and we are updating it every day!

    March 21, 2011 at 4:01 pm
  • Ash Nallawalla

    I am not clear whether that post means that older, invalid data won’t be displayed or how literally your “just released” means. :) A Backlink Explorer check done 15 hours ago containing 19,714 rows begins to show 404s from row 88xx onwards (not all of them, but many seem to be sitewide links where the page or domain itself doesn’t exist anymore). I was checking bnz.co.nz (no www prefix).

    I have used up 39 out of 40 permitted searches per month and the last search was of a minor site, so the bulk of my analysis has to wait another week. :(

    March 21, 2011 at 8:10 pm
    • Dixon

      Hi Ash,
      older data is dispayed, but you will need to select “historical” reports now, instead of “Fresh” to get it. However – Site Explorer data is limited to X uses per hour – so you don’t have to wait for full reports to get useful backlinks. (But we are glad you are using it so heavily!)

      March 21, 2011 at 10:17 pm
  • Ash Nallawalla

    I am working on the client to subscribe directly to Majestic SEO. In Raventools, there isn’t any option to select Historical vs Fresh.

    There is just one field in Raven to input the URL. I heard just now that Market Samurai can also access Majestic SEO data, so I’ll have to check that from the home PC later.

    While this puts Raven in a bad light, it also rubs off on you when the partners offer a nobbled version and don’t warn you.

    March 22, 2011 at 12:36 am
  • Dixon Jones

    Hi Ash,
    You are a hard man to please Ash! The daily updates went live on Monday and I know for a fact that Raven had a stand at SES in NY then, so even if they started reprogramming with the new functionality straight away, they still have to get through the Expo, back on a plane to Memphis and start talking with the develoPnent team :)

    Give then a few weeks. I’m sure they’ll integrate the new functionality.

    March 26, 2011 at 1:04 pm

Comments are closed.