Majestic SEO are restricting bots from scraping the main content, but opening up avenues for these bots through the API system.
Posts by Dixon Jones
New Intelligent Crawling
We think we have won the “mine is bigger than yours” debate. Now we want to not just crawl faster, but also smarter.
Where’s the fresh crawl data?
we are working on a new version of the fresh crawl data to be launched in January 2011.
New Developer Platform Launched
Sneaking it on on a Friday is unusual for us – but we’ve been itching
Introducing Clique Hunter
Inspired by Aaron Wall’s Hubfinder tool, we felt that with Yahoo’s API looking a little