Recrawling 300,000,000 urls every day and we STILL increased the index size to an all time high.
Recrawling 300,000,000 urls every day and we STILL increased the index size to an all time high.
On Saturday, the team hit “Birmingham Wheels” Go-karting for their Christmas team event.
Our last Webinar covered using Clique Hunter and Advanced report filters to help identify which
Majestic SEO are restricting bots from scraping the main content, but opening up avenues for these bots through the API system.
We think we have won the “mine is bigger than yours” debate. Now we want to not just crawl faster, but also smarter.