Analyzing Your Log Files to Improve Your Website

If you know what to look for, the log files provided by your host can give you valuable feedback and tell you how effective your website is in converting visitors into customers.

What is a Log File?

Any good webhost keeps track of everything that happens on your website - every request for a page or part of a page, every error, demographic data about every visitor, etc... and the log file is where that data is stored.

The data in log file can give quick answers to questions like:
- how many visitors per day / month
- the most popular pages on your website
- what was the website your visitor came from
- how much time a visitor spends on your website
- how much time the visitor spends on each page
- what was the path a visitor followed during his visit
- what was the page he entered on and exited from
- what pages search engine spiders are crawling.

In short, your server log files can be a goldmine of helpful information. They can help you to understand the traffic patterns of your visitors, and can help you to see your website "Through Their Eyes"... This can give you some insight into ways to improve their experience, which could increase your conversion rate.

There are far too many variables from one host to another for me to get into any specifics, but the following will give you an idea of what you can expect from the log files your host provides.

Stats Worth Watching

To get the most benefit from your analysis, it may help to focus on some specific questions:
- What percentage of visitors actually end up buying?
- How many times do they visit before they buy?
- Which pages do they look at?
- What was the click-through rate from a particular ad?
- What paths do buyers -vs- non-buyers follow?
- Are non-buyers all bailing out on the same page?

Your log will show how many visitors enter through your homepage, but never go beyond it. Knowing that statistic can help you to measure the effectiveness of the homepage - and decide if you should make changes.

You can also track and measure the click-through rates of other popular entry pages, and perhaps enhance them so people visit more pages.

You may learn that a lot of people exit your website from a product detail page without buying. There could be many reasons for that, but it may indicate that a stronger 'call to action' is needed, or that some essential customer question has not been addressed.

One of the most important stats to track is the average time-on-site and page views per visitor. Monitoring these stats can help you identify potential usability and navigation problems.

You may find that your average visitor spends less than a minute on a page and only visits a total of two or three pages...

That's because the average person is surfing with a goal in mind and they are attempting to accomplish a task in the shortest time possible. They will quickly scan a page and determine if it contains (or links to) the information that they are looking for.

If your content can convince them that they have found what they need then it is doing the job - but if scanning the page does not convince your visitor that he's on the right track he will click away, and a return visit is unlikely.

Another important stat to track is what search words and phrases people are using to find your website.

You may learn that people find your website after searching for keywords that are on your page, but that you aren't optimizing for. That may be all you need to know to modify your meta tags and headlines, just a little, and attract even more search engine traffic.

On the other hand, you may learn that the keywords you are optimizing for aren't bringing in much traffic, or that they are visiting but leaving right away. If that is happening, you may be using the wrong words and keywords to describe your website and it may be time to re-assess them.

Who Are Your Friends?

Your log files will also show you who your top referring websites are... this can help you find hidden opportunities for increasing traffic. Contact the top referring websites and see if you can negotiate a better placement for your link. If their average visitor is your target audience, this can be priceless.

Error Correction & Hotlinking

Something else your log files can show you are the Errors: broken links, missing pages or images, etc. This gives you the chance to correct the problem and improve the experience for your visitors.

A careful analysis of your log files can also show you all the websites that are hot-linking to your files or images. This is a big problem for my partner over at abbys-good-stuff.com - people all over the world are linking to her images.

These guys are stealing bandwidth, and if you can identify them you can take steps to block their access (via the .htaccess file) or change the image name so their link is broken, (or change the image to something rude) but the first thing to do is email them and demand they undo it.

Visitors Browsers & Operating Systems

Cross-browser compatibility is an ongoing issue for webmasters. On large, complex websites, addressing these issues can take a lot of time and energy. Log files tell you about what browsers and platforms your visitors are using.

If you know the top four or five browsers and platforms that your visitors use you can do some testing and tweaking to make sure that they are seeing the website the way you want them to.

What Log Files Cannot Do

Log Files Have Limitations. Some of the data in log files is inaccurate, incomplete and/or misleading. For one example, they can't explain a visitor's thought process or his motivation for visiting, or why he left from the page he did.

When you are analyzing your log data, keep the following in mind:

The log counts all visits from all sources the same. So a search engine 'bot or a real person looks much the same, and this will skew your statistics. Also, if someone has visited your site before and has a stored version of your page in the computer cache, the page view probably won't be logged.

If your website is accessed from a public library, the log will show all visits as returning visits, because they all have the same IP - even though they are all different people.

On the other hand, if someone is using an ISP that generates a unique IP address on every log-in, the log will show visitors coming from different IP addresses, even though it's a returning visitor.

One more thing: "Hits". What can you do with the "Hit" statistic? Pretty much nothing, actually. "Hits" only refers to the number of browser requests for a file that your web server receives.

To illustrate - if your webpage has 10 images, two menu files, header and footer files, plus a counter - every time that page is accessed the log will count 15 hits. But it's only one visitor.

Tools That Can Help

If your host does not provide webalizer or some sort of analog summary of your log files (most do) then you have to deal with a giant page showing line after line after line of data...

Fortunately, there are a few programs that can take this raw data and turn it into something you can read and manipulate.
Webalizer, The Webalizer is a fast, free web server log file analysis program. It produces highly detailed reports in HTML format, for viewing with a standard web browser. It's a pain in the butt to setup and get started, but it's worth the effort.
SuperStats, With SuperStats standard, you get minimal reports on number of visitors, search engines and keywords that are driving visitors to your site -- with SuperStats premium you get hundreds of reports including unique tools and detailed HTML analysis, starting at $99/year and going all the way up to $ ridiculous.

Conclusion

Analyzing your logs will help you determine what the actual usage patterns are for your website. Then you can make decisions about your content, and maybe make some changes that improve your overall success.

It's not a fix-all - it's just another tool you can use to get the best return on your investment of time and resources, and as such it can be very useful.

To Your Success!
Tim

By Tim Brown © 2006. About the author: Tim is the webmaster at http://BLT-Web.com, where webmasters can find free tools, advice, tips and other useful resources designed to help them build a successful website.

---

Webmasters & Publishers:
You are welcome to publish this article free of charge on your website, newsletter, or e-zine, provided:
* You don't change the article in any way and all links remain intact.
* You include the entire article, including the "about the author" box.
* In doing so you agree to indemnify the author, blt-web.com and its directors, officers, employees and agents from and against all losses, claims, damages and liabilities which arise out of its use
* I would appreciate it if you send a courtesy copy of your publication to