Search Engine/Directory Submission:
Automated, Manual or Not at All?

by

Peter Lavin

April 29, 2004

Overview

If you want your site to be found on the internet very few people would debate the wisdom of submitting it to Google or DMOZ. It obviously makes sense to be listed on the premier search engine and in the top directory. But what about all those other directories and engines out there? Is it worthwhile being listed and if so is it better to make your submission manually or using an automated software tool?

This article will examine these questions and in the process try to determine what to look for if you decide to use an automated tool for submission.

Introduction

Some search engines and directories make it virtually impossible to be included except by manual submission. The two best examples are perhaps Google and DMOZ. While no sites actually seem to encourage automated submission it is certainly actively discouraged at a number of sites. This does not mean that there is something inherently wrong with automated submission. After all it's just an easy way to fill out a form. While Google themselves do not allow automated submissions they implicitly admit the legitimacy of doing so by providing an “autofill” button on their plug-in toolbar.

The aversion to automated submissions has varied causes but some obvious reasons can be identified. Automated submissions can lead to server overload and vulnerability to denial of service attacks. Inappropriate submissions, for example submitting a general interest URL to a specialty engine, can happen much more easily when submission is automated. Finally, manually maintained directories are particularly sensitive to inappropriate categorisation and trivial requests for inclusion. For them manual submissions separate the wheat from the chaff.

If we accept that automated submissions are legitimate the question still remains as to whether it is worthwhile. Even with the recent changes at Yahoo, the dominance of Google tends to overshadow all other search engine. While estimates of Google's importance vary, a significant number of searches do originate with other engines. Certainly there can be no denying the usefulness of specialised search engines.

With these issues aside, let's get to the heart of the matter. Automated submissions can be done in a couple of ways. You can purchase a software package or find a website that performs this function. We are primarily concerned with the former since this has more appeal for the web developer because it provides an inexpensive way of making multiple submissions for different sites. These packages may be dedicated engine-submission tools or part of an integrated suite of Search Engine Optimisation (SEO) tools. What I propose to do here is develop some general guidelines to help you determine what to look for if you are in the market for this kind of software.

User Friendly Interface

While it is a truism to say that a user-friendly interface is important it is especially critical for automated submission software because this is a task that the user can choose to do manually. Most people won't use an automated tool if significant savings in time are not immediately evident and if using software is more confusing than doing it manually.

On the other hand, if the software allows you to set up your site and begin submissions very quickly then it probably will not do a good job for you. Different engines/directories have different requirements from the most basic that want only an URL, to those that want a very specialised category selection. In particular, submissions for directories require careful choices regarding categories and subcategories. When setting up a site for automated submission be prepared for a fair bit of data entry. Automation can relieve some of the tedium involved but proper set-up will still require patience.

Organisation

In order to be really useful engines and directories need to be sorted in a variety of ways. First off, engines should be categorized according to their status. Certainly major and minor engines should be distinguished.

While there are various ways of categorising information it is important that the user have enough information to make a sensible judgment about whether or not to submit to a specific engine. For instance, in some cases you will find the “Femina” Engine listed as a general search engine. By their own definition Femina consider themselves a specialized site. Submitting a general site to this engine is probably a waste of time for all concerned.

Submissions should probably also be sorted by both country and language. Some packages seem fairly sloppy in this regard. It is not very helpful when a site such as “Canadopedia” is designated as a U.S. site especially when it is self-described as “made in Canada by Canadians for all Canadians”.

Good organisation will also help you keep track of your submissions and thus avoid multiple submissions to the same site. Some sites will assume you are spamming them if you submit more than once and you could end up achieving the exact opposite of what you intend.

In any case, I think we can conclude that the better organised and the more information that a software tool supplies the more useful it is to the web developer.

What is CozyCabin.com?

Beware of claims such as, "We'll submit your website to 1000+ major search engines”. At best this statement is misleading. There aren't a thousand major search engines out there.

You should check out some of the engines listed for automated submission especially if the software does not provide a link directly to an engine's site. It was with this in mind that I tried to locate a search engine called “Pete's Internet Links”. A search on Google turns up over ten pages of sites with references to this “engine” but the engine itself is nowhere to be found. Perhaps it once existed but there's no reason for it to appear in current versions of autosubmission software other than to bloat numbers and mislead buyers. Where extravagant claims are made about the number of engines submitted to, further investigation will doubtless turn up other “vapour engines”.

For this reason an active link to the home page of an engine is a must-have feature of any submission software. Clicking the link should open the search engine page in your default browser or within the software itself. This will help verify the nature and existence of the site.

Another way of assessing the engines you are submitting to is by checking their popularity. To form some idea of the ranking of different search engines go to “http://www.freewebsubmission.com/”. Here you'll find a listing of search engines and directories ranked by their popularity on Alexa's engine.

Alternately, you can go to the Alexa site itself and check out their ranking of different sites. You will have to sort out the information for yourself though. For instance, directories and engines are not listed together.

And, oh yes, cozycabin.com does exist but have a look at the site before you decide to make a submission.

Confirmation and Updates

In view of what has been said above it is important to ask “What kind of confirmation do you get when you make a submission to a site?” Does the software show you the page returned by the engine when a successful submission has been made? If not you might quite rightly be suspicious that your page has not been properly submitted or that you've made a submission to a “vapour engine”.

For an individual who is doing a one-time submission of his site to search engines, updates are not a major issue. For the web developer who makes submissions on a regular basis for different clients it is quite a different matter. We all know how frequently websites change so if you are purchasing software it is important to know how frequently the database is updated and how long the updates are supplied for free.

The situation is analogous to using anti-virus software. It doesn't matter how good the anti-virus software is if the virus database is outdated. Changes to the database of engines may become necessary for a variety of reasons but primarily because of changes to submission requirements. Most packages allow you to download updates as they become available, again in much the same way as you would update a virus definition file. The better software packages will inform you of the changes made.

Most software packages will allow you to print a report of submissions made. Forwarding this information to your clients is good customer relations but only if the information is reliable and up-to-date.

Word of Caution

As a result of automated submissions you will doubtless receive some unwanted email. I won't dwell on the irony of this situation other than to say “ He who lives by the sword …”.

As part of the submission process many sites require a valid email address. In some cases, a site will not be added if there is no response to a confirming email. An awkward situation can develop if you are adding a listing on behalf of a client and this client starts receiving spam. One solution is to set up a temporary email address that is used for the short term or an address that redirects emails to one of your own accounts. In this way legitimate confirmations can be handled and spam need not reach your client.

Conclusion

For individuals trying to promote their own websites manual submission might well be the best option. Specialty engines can be identified and categorisation can be carefully tailored so that one's site appears exactly where it ought to in a directory. For exactly the same reasons submission for any specialty website might best be handled by manual submission.

On the other hand, for the web professional regularly making submissions on behalf of clients an automated tool is an invaluable assistant. Submissions can be made quickly and be automatically documented. However, careful choice of software is very important.

Look for submission software that documents engines, provides information about them and links to them. Look for software that is regularly updated and that removes engines as well as adds them. Any submission should show the confirmation page generated by a successful submission. Do not look at total number of engines as a sign of quality software. In fact, beware extravagant claims about number of engines – these probably include “vapour engines”.

Submission software should carefully document what submissions have been made and to what sites. In this way you can avoid making multiple submissions. This is something that some sites will interpret as spamming and they will penalise you for it. You need to protect your own and your clients' interests and not achieve the opposite to your desired effect.

Many companies provide free trial or “lite” versions of submission software often as part of an integrated suite of tools. To locate these packages check the DMOZ listing under the category, “Computers: Software: Internet: Website Promotion”. After all, any company in the Search Engine business should be listed here!

Test drive these packages carefully and you'll be able to make the right purchasing decision.

About the Author

Peter Lavin runs a Web Design/Development firm in Toronto, Canada. He has been published in a number of magazines and online sites, including UnixReview.com, php|architect and International PHP Magazine. He is a contributor to the recently published O'Reilly book, PHP Hacks and is also the author of Object Oriented PHP, published by No Starch Press.

Please do not reproduce this article in whole or part, in any form, without obtaining written permission.