The Importance of Spider Analysis in a Well-Rounded Search Engine Marketing Plan
Give Us Your Feedback

Editor's Note - 03.26.02





You may call it the Mark of the Beast, but Nancy Nisbet thinks of it as an investigation into the relationship between humans and technology. In October of last year she had a microchip implanted into the back of her left hand. This February she had another implanted in her right hand. The chips, normally used to identify pets, livestock, and wildlife, contain a numerical ID that is broadcast via a low frequency radio signal.

Her aim is to learn about the impact of technology on identity. How is technology effecting the way we think of our selves? What is the potential for abuse with this type of technology? What are its limits. These are the kinds of question Nisbet hopes to answer.

Some might think she's going a little far in having the chips physically implanted, but Nisbet says, ""the electronic ID chips embedded in my hands will heighten the connection between my virtual identity(ies) and my biological/physical identity." The way she sees it, the integration of man and machine is inevitable, so she wants to see the possibilities and the pitfalls before the technology is in widespread use.

Check out the text of Nancy Nisbet's recent presentation to the Vancouver Art Gallery’s Symposium here: http://www.finearts.ubc.ca/faculty/artworks/nisbet/

Today's issue features the first part of a two-part article on search engine spider analysis. Today you'll learn the fundamentals, and the next issue will teach you how to create a robots.txt to round out your search engine marketing plan.



send comments/feedback

The Importance of Spider Analysis in a Well-Rounded Search Engine Marketing Plan

WebTrends' "10 Reports Every Webmaster Lives For"
Want to know which critical indicators webmasters are watching?
Need to know how to put the analysis to work for your organization today?
Get our free guide, "10 Reports Every Webmaster Lives For."
This guide will tell you where to find valuable information, how to analyze it
and best of all, how to reap immediate benefits.
You’ll learn how to improve your web site’s performance immediately and how WebTrends’ products from NetIQ can help get the comprehensive information you need to master your web site.

By Robin Nobles

The field of search engine optimization is a constantly evolving industry with important changes being made almost daily.

In the beginning, search engine optimizers focused on one thing only: rankings. Tracking of actual sales or conversion rates was almost unheard of.

Slowly, search engine marketers began to realize that all of the 1's in the world won't help if they don't convert to traffic, and all the traffic in the world won't help if it doesn't convert to sales.

So, search engine experts began analyzing their log files and tracking sales. With today's technology, Web site owners can now tell which of their marketing campaigns are truly effective and which need to be replaced by a different campaign. In other words, through log analysis, you can analyze your human traffic and their visits to your Web site.

But what about your "spider" traffic?

One part of log analysis that has remained surprisingly lacking, both in terms of content and accuracy, is spider analysis.

Though traffic analysis programs may look at spider activity, the information often isn't detailed enough or presented in a format to do you much good. Also, spider and robot analysis is acknowledged as being a main culprit for inaccurate log analysis measurements. internetnews.com/IAR/article/0,,_908361,00.html

Therefore, the need for detailed spider analysis has begun entering the minds of search engine marketers.

So, what is "spider analysis"?

You know that when you submit a Web page to an engine for indexing, the engine sends a spider to your site to index the contents of the page. "Spider analysis" is simply analyzing the search engine spider visits to your site.

Through effective spider analysis, you can learn the following information about your site in a concise, easy-to-read format.

* Has your site been spidered? * If so, by which engines? * When did the spiders visit? * Which directories and pages did they visit? * Are certain pages getting respidered more often, signaling their importance to the search engines? * Are certain pages not getting spidered at all? * Are the spiders indexing inappropriate content? * Are the spiders getting everything they want and need, or are they receiving error messages? * Was your site spidered within the specified time agreed upon in the pay inclusion programs you're participating in? * Is your site getting respidered on a regular basis, as agreed upon in your participating pay inclusion programs?

Another important issue in spider analysis is robots.txt files.

Though the object of search engine marketing is to help spiders find all the pages on your Web site, there will be times when you want to keep spiders out of certain pages. You can do this with a robots.txt file.

What is a robots.txt file?

A robots.txt file is a text file that is placed on your server that instructs the search engine spiders not to crawl or index certain sections or pages of your site.

But why would you want to keep the spiders out?

Let's say that you're creating a new page for your site, and you've placed the page online while you tweak and edit it. The page isn't ready for visitors, so you wouldn't want it indexed yet. Or, let's say you've placed some employee guidelines on your site. The guidelines are of interest only to your employees, and there's no reason for the public to view them.

Using a robots.txt file, you can keep the spiders out of those pages.

Therefore, when working with spiders or robots, you want to be able to:

* Create a robots.txt file quickly and easily; * Use a robots.txt file to present optimized pages to specific engines. For example, using a robots.txt file, you can focus English language robots onto the relevant pages and direct robots from international search engines to the localized content areas of your site; *Send e-mail harvesting programs away from your site to keep your e-mail spam down; * View highlighted pages requested in error by the spiders; and, * Direct search engine spiders to relevant areas of your Web site.

Don't miss the second part of this article in the next issue of WebSiteNotes.

About the Author:
This article was written by Robin Nobles, a professional freelance writer and the Director of Training of the Academy of Web Specialists (http://www.academywebspecialists.com/more_info), where she has trained several thousand people in her online courses in search engine marketing strategies. She also teaches onsite search engine marketing workshops (http://www.searchengineworkshops.com) with John Alexander, and she has written three books that can be ordered through Amazon.


News Headlines



Get Free Email Newsletters

Get WebsiteNotes!

Enter Email


Resource Links:
http://www.finearts.ubc.ca/faculty/artworks/nisbet
internetnews.com/IAR/article/0,,_908361,00.html

-- WebsiteNotes is an iEntry, Inc. publication --
© 2002 iEntry, Inc.  All Rights Reserved  Privacy Policy  Legal

Click For Advertising Info iEntry Sitemap News Headlines Your Subscription Print Version Archives More Newsletters News Headlines Click For Advertising Info