Majestic Historic Index – Reflecting on 2019 & looking forward to 2020.

We have recieved queries from a number of our customers asking about our plans for Historic Index. We would like to assure our valued users that we recognise the value of the Historic Index, and want to share our plans for enhancing the turn-around of Historic Index Updates.

The Majestic Historic index is a feat of software engineering. A huge volume of data, accessible via the web and a high speed API. Over 8 Trillion URLs have been identified by it, with the results of a crawl of over 2 Trillion URLs available for query.

Historic Index updates in 2019

Until the beginning of 2019, we had an enviable reputation for updating the index – most calendar months heralding a release of new data. Majestic has continued to invest in Historic – both by adding new features, and increasing the amount of data present.

Maintaining this huge database has been a significant endeavour. A huge amount of work has gone on behind the scenes to support what is an amazingly complex bespoke system.

Unfortunately, 2019 has seen a “perfect storm” of a number of events come together to interrupt the consistency of update of the Historic Index.

Increasing volumes of data in Historic has pushed the system to it’s limits. The results of this – halted builds, delayed releases and a lack of ability for the team to commit to a regular schedule of updates.

While we continue to work hard to develop the software to manage larger and larger volumes of data, we have experienced difficulties sourcing hardware needed to enhance the platform on which the software runs.

Lead times on various components which would have previously taken days have extended to weeks and months, disrupting our ability to perform the routine, ongoing maintenance necessary to maintain regular updates to the Historic Index.

We recognise that it is our responsibility to update the Historic Index, and share the above only to be honest about influences outside of our control which have complicated operations.

We had very much hoped to be in a position to update the Index in October. We regrettably announce that our attempts to build an index to this deadline has failed. We had hoped that a number of manual interventions could have resulted in the release of new data on old hardware. However, in final rounds of testing, we were unable to satisfy ourselves that the interventions had not created further anomalies. We therefore took the difficult decision to restart the historic build – a process that takes close to a calendar month to run due to the huge volume of data processed and depth of analysis performed. We have performed an incremental upgrade in order to accommodate this build and are optimistic for success.

We are aiming to release at least one further Historic Index Update in 2019.

2020 and beyond ( two months time )

Early 2019 saw significant investment into our software development – maintaining build times in Fresh, and adding a range of new features including Link Context. We have wanted to deliver more, but this has been complicated by the disruption described above.

Our team is working hard to prepare for the arrival of remaining hardware necessary to upgrade the historic build system. The final order of hardware for this upgrade cycle is due to arrive soon. This will facilitate the building of a greatly enhanced Historic Build Cluster, boasting more memory, more storage, and faster processing.

This cluster will free us from some of the bottlenecks impacting the current historic build. Our objective is to build a platform on which we can optimise our software to ensure a greater turnaround of data.

Our immediate priority is to return to previous Historic update levels as soon as possible. When we have achieved this, we will balance the benefits of upgrading Historic build systems with adding Link Context data. Our aim is to create an update schedule for Historic that is faster, and more reliable than ever before.

We would like to thank all of our customers for their ongoing support through this difficult period. We do recognise the difficulty delays in historic must have caused and hope that our significant program of investment begins to benefit you soon.

Back