At Thinkvisibilty last weekend, I was demonstrating Majestic SEO and Richard Baxter (SEO Gadget) said that he had not realized that we were showing newly found links in between index updates. He suggested that I should tell people, because it is data that is not available elsewhere.
How to see “Fresh Crawl” information
You can start tracking fresh crawl data only after you have requested an advanced report for a root domain. (We support fresh data crawls for full root domain reports, not individual URL reports). After you have analysed the domain for the first time, you do not have to wait until a full update to be able to see links that we find from that day on. You can see them and drill down to them straight from the control panel, in the Advanced reports list:
From here you can see that although our last index was announced on the third of September, almost all of the sites in the control panel have new data in the fresh crawl section. You can then drill down on either of the fresh crawl columns. The “Last Fresh Crawl” column shows you the actual links that we found on the date in question, to reveal the link, anchor text and source flags (whether it is nofollowed, or in an image for example)
In addition, you can see all the newly found links, day by day, by selecting the “Fresh Crawl” drill down, rather than the date. This then lists the number of links that we find, day by day, and again you can drill down to see them in detail or export to CSV.
How powerful is that?
You need to bear in mind that the date we FIND a new link may not bear any relation to the day the link was actually created. However, the very fact that we can report as we go, so to speak, may well please dedicated link builders who would like to be able to generate independent reports on links found. A rapid increase here may either suggest something going viral or could suggest the presence of a paid link campaign starting. Alternitively, if our crawlers find a new site-wide link, then the numbers may suddenly be inflated on a day as we crawl the new pages. For this reason, we switch on a toggle called “domain clustering” so that on the detail, by default, you see the strongest link from a given domain, and we also tell you the number of other links from that site. You can switch domain clustering off if you choose.
- How Important will Backlinks be in 2023? - February 20, 2023
- What is in a Link? - October 25, 2022
- An Interview with… Ash Nallawalla - August 23, 2022
Great stuff! This should be very useful, thanks.
September 14, 2010 at 12:10 pmBut how can I filter out deleted backlinks? I still get a lot of outdated backlinks: Majestic SEO reports them, but they are not there. Google counts only alive backlinks.
September 14, 2010 at 10:13 pmHi Bence, Google and Yahoo show non-existent links when you use link: command, I don’t think the answer is that clear cut as to whether Google counts only alive links – they certainly show non-alive ones.
In our case we focus on historical links that span back 4 years – this helps uncover business relationships that you otherwise won’t see if you just look at current links, but if you must have those then you can use filtering parameters in our reports to show only recently found links – we’ll be introducing more tools to help do that.
Alex
September 15, 2010 at 1:36 pmGreat question, Bence. I will look to cover this – not necessarily in this first webinar, but once I have the basics of learning myself how to use the webinar system, I will get a bit more adventurous with the syllabus.
In the meantime – you need an answer! With advanced reports, we mark urls as “deleted” when we recrawl a link whichisn’t there. The challenge is that the index is SO large, that not all URLs are recrawled every index. This means that we are likely to pick up (say) a paid link campaign, but may be slower to show that the campaign is over. The result in that case would be as you described.
My suggestion, if you find this happening in your reports alot (and whether it does depends a bit on your vertical) would be to make use of the source URL “DATE” options in the options panel of the advanced reports. Tick the “enable date range filtering” rule and select a from date of (say) three or six months ago. This will then only bring back links which were crawled within the last six months. Most of the dead ones were so old that crawling them again may take time.
Whilst you are in those options, also check that the “Exclude deleted” flag is also on, so you do not end up with links we checked in the last six months and we know are removed from the web.
Hope that helps.
September 15, 2010 at 1:43 pmThanks for the update. Didn’t know this info.
September 16, 2010 at 3:22 pmSo far only relied on Google time-stamp for the crawling/indexing info of pages.
I used to check indexed pages and individual pages showed a time-stamp. It was just URL specific though and worked only with authority sites !
This is the first time I use advanced report. But I can only see latest backlinks in July though I know I have lots of new links in Aug and Sept.
The system told me “3 Sep 2010 New index update released – please ‘force new analysis’ on your Advanced Reports, to get them updated with new data”, I did it, but still no “fresh” data shown.
Any clue?
September 17, 2010 at 9:16 pmAwesome tool! I think we are going to give this a try. We’ve tried other tools and were greatly disappointed.
September 19, 2010 at 12:14 amHi Billy,
I’ll email you and find out more. The index update on 3rd September may have missed your August links, but they should start getting seen in the fresh data, when we crawl them.
Dixon.
September 19, 2010 at 9:28 pmHi Again Billy.
Thanks for sending that through.
I cannot see individual account data, but according to the database team, you only added your site to the system a few days ago. That means that we can only start recording “Fresh link” data after you add your site. As it says in the post:
You can start tracking fresh crawl data only after you have requested an advanced report for a root domain… After you have analysed the domain for the first time, you do not have to wait until a full update to be able to see links that we find from that day on.
Now – between our last index update and the date you have selected an advanced report, there is indeed going to be a gap in your knowledge. This data will mostly be filled in when the index updates AFTER you have created the domain in your advanced report panel.
September 20, 2010 at 1:54 pm