Link Building Blog, News and Articles - Textlinkbrokers.com/blog

Blog Homepage : Category - Advanced SEO

Yesterday I started a series on competitive intelligence gathering and how you can use it to assess your competitors and the market space. 

In this article I continue with this theme:  Introducing you to a couple more tools I like to use on a regular basis.  One is web based while the other is a freely downloadable application.

Yahoo! Site Explorer

The first tool I like is a relatively new one.  It’s called the Yahoo! Site Explorer and, as I’m sure you’ve guessed, it was released by Yahoo!

As the name implies you can explore a URL – either an entire site, or specific pages within it.

Simply inputting the URL into the search box returns a list of either pages as well as inlinks (inbound links, or backlinks).  From here you can navigate through individual pages, as well as view the links pointing to those pages.

Through this review of deep backlinks you may be able to find those hidden gems that are relevant and valuable links from which you too can request a link.

Further, if your competitor’s site is composed of subdomains you can also restrict your search to an individual domain (ie. site.com) or a subdomain (ie. subdomain.site.com).  From there you can again refine your query to backlinks for the domain, or select a page to find backlinks to that page.

It is really quite a handy tool once you get into it.  You can learn a lot about competitor’s sites by looking at the URLs, and reviewing the backlinks to the individual pages.

You can also see the title tag the pages are using, as well as get the file size which can be useful as well.

On a final note, you can also export the results to a tab separated file.  Then you can use a program like Microsoft Excel to open the TSV file and store the pages somewhere for review later.

Also, be sure to note the total number of pages indexed by Yahoo! As well as the linkage numbers, because they become very informative as you get into my next favorite tool.

Xenu

This is a free tool which you can download from this website.  While you would probably want to use this mostly for your own site, it too is a great competitive intelligence gathering tool.

It is essentially a crawler – it can go and follow links and prepare a report for you based on all the links it finds.

First, lets look at what it can do for your own site.

Upon installing Xenu and opening it for the first time you will see that it has a pretty plain interface.

If you select “Check URL” from the “File” menu you will be able to add the full URL (including the “http://”) to the box and select “OK.” From here Xenu begins to crawl the site, finding links and requesting them.  Depending on the size of your site it could take as little as a few minutes to as long as a few hours.

A word of advice here – if your site relies on session id’s in the URL string, you may not want to use this tool.  I’ve found that it tends to index multiple versions of the same page because it gets different session id’s each time it requests pages.

When it is done crawling it will offer to prepare a report for you.  Before I go into the report, lets look at what Xenu pulls back.

On the bottom of the screen you will see the total number of URLs it found.  On the screen you should see the page you started on, as well as all the links it found.  These links include all the images, CSS and other spiderable links including email addresses and links to outside websites.

You will also see the current file size and any errors generated by Xenu in the retrieval of the links.

If you scroll over you will see a “Level” field.  This is a useful field.  This tells you how far away any page is from the entry point (usually the home page).  In other words, this is how many clicks it can take to get from the entry to a page.

So if you have a large site and the level is something like “20” that means it probably takes 20 clicks to get to that page from the home page.

Now I probably don’t have to tell you, but if it takes that many clicks to get to some really good content, there’s little chance of a crawler getting to it, or if it does it will assign it a low link popularity because of how buried it is in the site.  A way to improve the level the page is found at is to link to it from a page that is shallower.

In other words, if your page is level 20, you could move it to level 3 by linking to it from a page found in level 2.  This will help move the page up on the search engine’s radar making it more important to them.  It doesn’t change the function or look of your site, but makes that page appear more relevant.

Beside the level you will also see the links in and out to the page(s).  This tells you just that – how many pages link to this one and how many it links to.

Another great option this tool has is the ability to help you manage broken links.

If you press “Control B” it will show you all the broken links on the site.  That means, the program has found links that are no longer working.  From there you can right click on the link and select properties to see which pages link to it.

If its valid – that means a page that should exist – then it needs to be fixed. However if it’s truly expired then you can simply remove the link from the page(s) linking to it.

Pretty simple huh?

Finally, you can export the whole thing to an HTML report or a CSV to open in Excel for further analysis.  The HTML report gives you an idea of the broken links and so on and even gives you a spiderable sitemap which you could then copy to your site, if you don’t already have one.

Overall a very handy FREE program to help you manage your site.

So let’s look at how you can use this cool little program to gather information about your competitors.

First, you can look at the same things you would look at if it was your own site – number of pages, broken pages and so on.

But you can also see some of their deep content as well as get an idea of how their site is organized.

If they are beating you in the rankings, perhaps one of the elements affecting that is the level.  If much of your content is found below level 10 while much of theirs is found above level 10 then that is contributing to their success.

Their content is found closer to the home page, making it more relevant.  Plus the indirect link inheritance to the home page is higher because there is less points in between the deepest page and the home page taking a share of that link popularity.

In other words, every page that is between any other page and the home page inherits part of that link popularity.  In the simplest of terms, if there are 9 pages between the home page and a deep page, then each of those pages (including the home page) inherits 1/10 of the link popularity of that deep page.

If the same page is only found 3 clicks away however, then the 4 pages involved (home page plus the three in between) inherit ¼ of the link popularity.

Now before you go commenting about how link popularity is much more complex than that, I know.  I’m really only simplifying it to this level to make a point.  And that is deep content can ultimately hurt your rankings.

Back to the competitive analysis stuff:

So, you’ve run Xenu on your competitor, what else should you look for?

Well, look at the number’s of incoming and outgoing links per page - these too are affecting the site’s link popularity and inheritance within the site.  If a page only has one incoming link and hundreds of outgoing links, its not helping the overall link popularity much.

If, however, you find that a page has tons of incoming links and only a few outgoing links, then it’s likely going to be a highly performing page.  You should make note of the page(s) that are like this and look at them individually.

Check their optimization and I think you will find that (if they aren’t buried too deep in the site) chances are they rank competitively.

Tying it all together

Now that you’ve ran the competitors site through both the Yahoo! Site Explorer and Xenu, you should begin to develop a pretty good idea of what pages are considered important just by looking at their level as found by Xenu as well as their incoming and outgoing links represented in both Site Explorer and Xenu.

From here you can begin to develop a strategy to target these highly performing individual pages by improving link popularity and optimization on your site to meet or exceed the values set by the competing site.

This is, of course, after you’ve taken care of the broader tactics – sitewide interlinking, building relevant links to the site overall, optimization of all the pages and so on.

The strategy I’m talking about here could be considered a page specific strategy.  Finding those individual competitors pages which are beating yours by a few positions and then optimizing the content and links on your page(s) to compete.

One final note.  Don’t expect this tactic to return instant results.  We should all know by now that the search engines rarely broadcast their update schedules.  So while you may see your changes being cached in the engine’s index, don’t automatically assume that will translate into rankings right away.

And always be aware that as you are making these minute changes, your competitors are also likely doing something similar – analyzing their next largest competitor.

So if you are competing for a keyword and just can’t figure out why a competitor is consistently beating you on a certain page, perhaps the strategy outlined above, using these 2 tools can help.