Over the last few days, we quietly launched our largest Historic Index ever with nearly 4.2 trillion URLs. It’s impressive, but we have also decided that we should maybe get a little more memory. For that reason, we need to let warn our customers that we may have a few short maintenance upgrades over the next week as our Network Administrator and his team start to take Pallet loads of new computers out of their boxes and into some data racks. Here is one delivery:

Computers by the Pallet

Computers by the Pallet

When the lorry arrived he was somewhat surprised to see a bunch of young faces with an assortment of trolleys. His first question was “What, no forklift?”.

Better Indexing Means More Memory Need

Over the last few months, Majestic has worked hard to improve what we call “Discovery”. We want to find new links, but not at the cost of revisiting and re-crawling good pages which may have changed. The usual trade off for increasing the speed of discovery of URLs is that you have to take away from recrawl time – but we now have implemented several ideas to improve discovery without compromising the re-crawl data. Some of this we are going to keep as part of our “secret sauce” but other ideas – like our bulk URL submitter – we are happy to shout about.

We have built our entire system for scale and thanks to your support we are able to now build on that design.

We’ll try not to disrupt your service – but…

Obviously – installing computers by the pallet-load is not a simple operation, especially in a live environment. We believe we have a way to do this with the minimum of disruption – but hope that you will give us leeway if you find short outages whilst server reboot over the next seven days or so.

What’s the Payback to our customers?

Bigger, Faster and Fresher data – with no increase in prices, of course!

 

Comments

  • Alex

    Wow, what DELL servers you are using?

    April 9, 2013 at 10:38 am
  • Jamie Knop

    Must post a pic of your new data room once its all set up (a little less blurry though!).

    April 9, 2013 at 11:20 am
    • Dixon Jones

      > Actually – the data “room” is a secret too. We do have a “room” onsight where we showed off a machine way back in June 2010 http://blog.majesticseo.com/research/how-large-does-a-computer-need-to-be-for-good-seo/ but I think we are beyond a “room” now :) – I don’t know, though, as it’s a secret :)

      April 9, 2013 at 1:16 pm
  • mohsin

    Good Luck to the team :)

    April 9, 2013 at 11:57 am
  • Vampire Joe

    Do you run all your servers in your own office/datacenter or do you also rent machines from hosters or in the cloud ?

    April 9, 2013 at 1:18 pm
  • Net66

    That’s a serious amount of hardware.
    Air conditioning must cost you a Bomb! Is your secret “room” just a giant fridge?

    April 10, 2013 at 4:42 pm
    • Dixon Jones

      Actually – we have an exceptional “Miniaturization” technology, which reduces all the servers into a mini fridge which we keep in the back of a VW Camper Van. ;)

      April 10, 2013 at 5:09 pm
  • blog bayu vai

    I am a happy MajesticSEO customer for past few years and being in the SEO industry I keep an eye on every upcoming tool. But in my opinion there is no worthy competitor yet. It is my personal opinion but no other backlink data provider comes near the level of MajesticSEO.
    Alex very well concluded the post “regardless of who sits at top spot at any given moment in this market, without constant innovation, it won’t be for long.” I agree with you.
    Regards

    April 19, 2013 at 3:37 pm

Comments are closed.