Over the last few days, we quietly launched our largest Historic Index ever with nearly 4.2 trillion URLs. It’s impressive, but we have also decided that we should maybe get a little more memory. For that reason, we need to let warn our customers that we may have a few short maintenance upgrades over the next week as our Network Administrator and his team start to take Pallet loads of new computers out of their boxes and into some data racks. Here is one delivery:
When the lorry arrived he was somewhat surprised to see a bunch of young faces with an assortment of trolleys. His first question was “What, no forklift?”.
Better Indexing Means More Memory Need
Over the last few months, Majestic has worked hard to improve what we call “Discovery”. We want to find new links, but not at the cost of revisiting and re-crawling good pages which may have changed. The usual trade off for increasing the speed of discovery of URLs is that you have to take away from recrawl time – but we now have implemented several ideas to improve discovery without compromising the re-crawl data. Some of this we are going to keep as part of our “secret sauce” but other ideas – like our bulk URL submitter – we are happy to shout about.
We have built our entire system for scale and thanks to your support we are able to now build on that design.
We’ll try not to disrupt your service – but…
Obviously – installing computers by the pallet-load is not a simple operation, especially in a live environment. We believe we have a way to do this with the minimum of disruption – but hope that you will give us leeway if you find short outages whilst server reboot over the next seven days or so.
What’s the Payback to our customers?
Bigger, Faster and Fresher data – with no increase in prices, of course!
Latest posts by Dixon Jones (see all)
- Crawling Smarter on the Infinite Web - March 16, 2017
- Biggest Link Index Hits New Highs with Historic Update - March 14, 2017
- A New Approach to Blogging. Expect a Better Standard. - February 7, 2017