Fresh Links Now Updated (MUCH) Faster

By Dixon Jones |

We just made another milestone.

We have moved from updating every day to updating every hour or so! That means that if a new page goes viral, we will be able to report stats about it to you throughout the day.

We know that our competition is envious of our Fresh Index with its frequent update cycle – but we also know that some indexes (including Google) update more frequently.

We not only want to have the largest Link Map on the planet… we also want the fastest and the freshest. We always felt we had, but in terms of absolutely getting the links to you more quickly than anyone else, there was just one milestone we felt the need to overcome. We think we now have after some really intense efforts from the team.

What has changed?

Our Historic Index (The one with 4 trillion URLs, give or take) will still update every month. The Fresh Index (The one with 191 billion URLs, give or take) used to update daily – but now is getting new data added most hours!

What are the caveats?

Right now, these updated numbers are being introduced into the site explorer, and we will be “rippling” the new technology out across our service as we can integrate it. The newest links will soon also be be available in the API, and our Reports in due course as well. The incremental nature of the update, is because we have to balance the demands of a large number of agencies, professionals and resellers who rely on our data, and those who need the most up to date data 24/7 – so dumping our database and starting again isn’t an option for us.

Due to the truly massive amount of processing required for our award winning metrics, we may not be able to give you as much accuracy about the Trust Flow and Citation Flow for some of the newest URLs and Domains until a snapshot has been taken (still carried out daily or so). Flow Metrics ® are dependent on every other URL that influences it over many iterations of links away – so it would be impossible to accurately assess one URL’s Flow Metrics without understanding the entire universe. This equation cannot be calculated incrementally.

When will this be live?

It’s live now!

What Technology Feats of Engineering were Involved?

Have you noticed the increase in rate of new features released by Majestic recently? A number of different projects have come together to facilitate this development, making the new data just the tip of the Iceberg. This year, we started many things over, increasing the size of our development team and investing in deliverable focussed processes, resulting in multiple, optimised development methodologies across different engineering teams. From a systems perspective, our hardware upgrades are becoming a continuous process which is resulting in more and more processing power becoming available to deliver this, and more future features.

I think it’s fair to say that there is enough material to cover from an engineering perspective to fill a whole series of blog posts. But maybe our engineers time would be better spent developing new features than boasting about our past successes – we will leave it to you to decide!

Update 16 October 2012 14:07

We are now live, detecting new and lost links to new content. As an example, look at this link on the BBC News website:

http://www.bbc.co.uk/news/business-19957779 ( A story about Yahoo hiring Henrique de Castro for $58 Million )

And we are already registering new links to the page:

That’s not all – we are not just reporting NEW links for it, BUT REPORTING DELETED LINKS!!! That’s links that have been removed since the link was first discovered

Our headline stats at 14:14 16th October for the BBC “Henrique de Castro” article:

Referring Domains: 336
External Backlinks: 2,287

Dixon Jones
Latest posts by Dixon Jones (see all)

Posted In: Development

Comments

Comments are closed.