Our January Historic Index is now live.
Here are the new numbers for the record.
Historic Index |
|
|---|---|
| Unique URLs crawled: | 884,184,043,798 |
| Unique URLs found: | 3,583,740,663,828 |
| Date range: | 25 Apr 2010 to 25 Nov 2015 |
| Last updated: | 11 Jan 2016 |
Founder of DHJ Ventures and Founding CEO of Inlinks, Dixon is also the Global Brand Ambassador at Majestic.
Latest posts by Dixon Jones (see all)
- How Important will Backlinks be in 2023? - February 20, 2023
- What is in a Link? - October 25, 2022
- An Interview with… Ash Nallawalla - August 23, 2022
Great, thank you !
January 20, 2016 at 6:30 amHow can you crawl the back links of [Mysite URL removed]
January 27, 2016 at 8:58 amWe crawl the whole web where we are allowed to. The site you linked to blocks our bot at the server level (406 response). This does not stop us seeing links TO your site, it only stop us properly interpreting the pages and links OUT of your site. We do not recommend using 400 responses if you want to block us, because our bot may start hitting your server much harder trying to find the Robots.txt file where we expect the instruction to be. We do respect Robots.txt, including the "crawl delay" command.
January 27, 2016 at 3:23 pm