Over the last few days, we quietly launched our largest Historic Index ever with nearly 4.2 trillion URLs. It’s impressive, but we have also decided that we should maybe get a little more memory. For that reason, we need to let warn our customers that we may have a few short maintenance upgrades over the next week as our Network Administrator and his team start to take Pallet loads of new computers out of their boxes and into some data racks. Here is one delivery:
When the lorry arrived he was somewhat surprised to see a bunch of young faces with an assortment of trolleys. His first question was “What, no forklift?”.
Better Indexing Means More Memory Need
Over the last few months, Majestic has worked hard to improve what we call “Discovery”. We want to find new links, but not at the cost of revisiting and re-crawling good pages which may have changed. The usual trade off for increasing the speed of discovery of URLs is that you have to take away from recrawl time – but we now have implemented several ideas to improve discovery without compromising the re-crawl data. Some of this we are going to keep as part of our “secret sauce” but other ideas – like our bulk URL submitter – we are happy to shout about.
We have built our entire system for scale and thanks to your support we are able to now build on that design.
We’ll try not to disrupt your service – but…
Obviously – installing computers by the pallet-load is not a simple operation, especially in a live environment. We believe we have a way to do this with the minimum of disruption – but hope that you will give us leeway if you find short outages whilst server reboot over the next seven days or so.
What’s the Payback to our customers?
Bigger, Faster and Fresher data – with no increase in prices, of course!
- How Important will Backlinks be in 2023? - February 20, 2023
- What is in a Link? - October 25, 2022
- An Interview with… Ash Nallawalla - August 23, 2022
Wow, what DELL servers you are using?
April 9, 2013 at 10:38 amIt’s a secret…
April 9, 2013 at 11:59 amMust post a pic of your new data room once its all set up (a little less blurry though!).
April 9, 2013 at 11:20 am> Actually – the data “room” is a secret too. We do have a “room” onsight where we showed off a machine way back in June 2010 http://blog.majesticseo.com/research/how-large-does-a-computer-need-to-be-for-good-seo/ but I think we are beyond a “room” now 🙂 – I don’t know, though, as it’s a secret 🙂
April 9, 2013 at 1:16 pmGood Luck to the team 🙂
April 9, 2013 at 11:57 amDo you run all your servers in your own office/datacenter or do you also rent machines from hosters or in the cloud ?
April 9, 2013 at 1:18 pmWe use all kinds of processing models.
April 9, 2013 at 2:35 pmThat’s a serious amount of hardware.
April 10, 2013 at 4:42 pmAir conditioning must cost you a Bomb! Is your secret “room” just a giant fridge?
Actually – we have an exceptional “Miniaturization” technology, which reduces all the servers into a mini fridge which we keep in the back of a VW Camper Van. 😉
April 10, 2013 at 5:09 pmI am a happy MajesticSEO customer for past few years and being in the SEO industry I keep an eye on every upcoming tool. But in my opinion there is no worthy competitor yet. It is my personal opinion but no other backlink data provider comes near the level of MajesticSEO.
April 19, 2013 at 3:37 pmAlex very well concluded the post “regardless of who sits at top spot at any given moment in this market, without constant innovation, it won’t be for long.” I agree with you.
Regards