Fresh Site Explorer – The story develops

On April the 5th we blogged about the online coverage in terms of link growth reflected in our Fresh Index of a story in a UK newspaper – The Guardian questioning where Larry Page will lead Google, now, three days after the article was published we would like to review our progress.

We have just performed todays update – the third in three days, and are now in a position to reflect on the detected link growth for the article in question:

Because we have developed our own crawling infrastructure, and control our crawling we have the ability to continue to improve the intelligence of our recrawl. As we explained in our last post, this has enabled our crawlers to work more like those of major full text search engines. Rather than simply trawl through a list of URLs, or wandering randomly over the web, we have a orchestrated crawl strategy. This has a number of advantages:

• We aim to scan more important sites regularly, picking up new backlinks fast
• We have a progressive crawl which discovers new links running simultaneously with the recrawl
• By revisiting pages we detect deleted links – even on a resolution as tight as the fresh index – On the 5th we had detected that some short term links had been deleted. ( If indexing this seems odd to you, pause to consider the value of knowing someone has just pulled one of your links after a short period of time )

Let’s compare todays totals from our fresh index for the Guardian article with some link counts from other sources:

We admit that it isn’t a totally fair comparison – the article was selected to play to our strengths, and all of these companies offer more than just link intelligence services ( for some, backlink data provision may appear more of a distraction than a core business ). We do hope though, that the table above illustrates why we consider our position as a market leader in link intelligence is well earned.

Where next for Majestic?

It’s a great feeling to be able to test drive a new product when its out of development and find it works as well as expected in the real world as we hoped. The union of Site explorer, the Fresh index and our intelligent recrawls serve as an example of the importance Majestic place in strategic planning – the launches and way they interact are far from coincidental and are representative of Majestic’s development philosophy and direction.

We are determined to unpick the mysteries of full-text search engine ranking piece by piece and make this valuable intelligence available for the ever increasing range of professions interested in this data. We have a number of exciting projects in development and on our roadmap – including new metrics, new tools and enhanced presentation. Its going to be an exciting journey – why not register and see what we do?

• Author
• Recent Posts

Steve Pitchford

Steve has a been in tech since the late 90's, learning the ropes with a UK e-retailer through much of the first dotcom boom after developing a passion for the web during his BSc in Computer Science.

Steve has been engaged with the Birmingham tech scene for years, co-organising the grassroots open-source conference YAPC:: Europe 2008, and founding the West Midlands Java User Group in 2014.

Latest posts by Steve Pitchford (see all)

• Introducing Duplicate Link Detection - August 27, 2021
• Python – A practical introduction - February 25, 2020
• Get a list of pages on your site with links from other sites. - February 7, 2020

Back