Search Engine/Directory Submission:
Automated, Manual or Not at All?
April 29, 2004
If you want your site to be found on the internet very few
people would debate the wisdom of submitting it to Google or
DMOZ. It obviously makes sense to be listed on the premier search
engine and in the top directory. But what about all those other
directories and engines out there? Is it worthwhile being listed
and if so is it better to make your submission manually or using
an automated software tool?
This article will examine these questions and in the process
try to determine what to look for if you decide to use an
automated tool for submission.
Some search engines and directories make it virtually
impossible to be included except by manual submission. The two
best examples are perhaps Google and DMOZ. While no sites
actually seem to encourage automated submission it is certainly
actively discouraged at a number of sites. This does not mean
that there is something inherently wrong with automated
submission. After all it's just an easy way to fill out a
form. While Google themselves do not allow automated submissions
they implicitly admit the legitimacy of doing so by providing an
“autofill” button on their plug-in toolbar.
The aversion to automated submissions has varied causes but
some obvious reasons can be identified. Automated submissions can
lead to server overload and vulnerability to denial of service
attacks. Inappropriate submissions, for example submitting a
general interest URL to a specialty engine, can happen much more
easily when submission is automated. Finally, manually maintained
directories are particularly sensitive to inappropriate
categorisation and trivial requests for inclusion. For them
manual submissions separate the wheat from the chaff.
If we accept that automated submissions are legitimate the
question still remains as to whether it is worthwhile. Even with
the recent changes at Yahoo, the dominance of Google tends to
overshadow all other search engine. While estimates of
Google's importance vary, a significant number of searches
do originate with other engines. Certainly there can be no
denying the usefulness of specialised search engines.
With these issues aside, let's get to the heart of the
matter. Automated submissions can be done in a couple of ways.
You can purchase a software package or find a website that
performs this function. We are primarily concerned with the
former since this has more appeal for the web developer because
it provides an inexpensive way of making multiple submissions for
different sites. These packages may be dedicated
engine-submission tools or part of an integrated suite of Search
Engine Optimisation (SEO) tools. What I propose to do here is
develop some general guidelines to help you determine what to
look for if you are in the market for this kind of software.
User Friendly Interface
While it is a truism to say that a user-friendly interface is
important it is especially critical for automated submission
software because this is a task that the user can choose to do
manually. Most people won't use an automated tool if
significant savings in time are not immediately evident and if
using software is more confusing than doing it manually.
On the other hand, if the software allows you to set up your
site and begin submissions very quickly then it probably will not
do a good job for you. Different engines/directories have
different requirements from the most basic that want only an URL,
to those that want a very specialised category selection. In
particular, submissions for directories require careful choices
regarding categories and subcategories. When setting up a site
for automated submission be prepared for a fair bit of data
entry. Automation can relieve some of the tedium involved but
proper set-up will still require patience.
In order to be really useful engines and directories need to
be sorted in a variety of ways. First off, engines should be
categorized according to their status. Certainly major and minor
engines should be distinguished.
While there are various ways of categorising information it is
important that the user have enough information to make a
sensible judgment about whether or not to submit to a specific
engine. For instance, in some cases you will find the
“Femina” Engine listed as a general search engine. By
their own definition Femina consider themselves a specialized
site. Submitting a general site to this engine is probably a
waste of time for all concerned.
Submissions should probably also be sorted by both country and
language. Some packages seem fairly sloppy in this regard. It is
not very helpful when a site such as “Canadopedia”
is designated as a U.S. site especially when it is
self-described as “made in Canada by Canadians for all
Good organisation will also help you keep track of your
submissions and thus avoid multiple submissions to the same site.
Some sites will assume you are spamming them if you submit more
than once and you could end up achieving the exact opposite of
what you intend.
In any case, I think we can conclude that the better organised
and the more information that a software tool supplies the more
useful it is to the web developer.
What is CozyCabin.com?
Beware of claims such as, "We'll submit your
website to 1000+ major search
engines”. At best this statement is misleading. There
aren't a thousand major search engines out there.
You should check out some of the engines listed for automated
submission especially if the software does not provide a link
directly to an engine's site. It was with this in mind that
I tried to locate a search engine called “Pete's
Internet Links”. A search on Google turns up over ten pages
of sites with references to this “engine” but the
engine itself is nowhere to be found. Perhaps it once existed but
there's no reason for it to appear in current versions of
autosubmission software other than to bloat numbers and mislead
buyers. Where extravagant claims are made about the number of
engines submitted to, further investigation will doubtless turn
up other “vapour engines”.
For this reason an active link to the home page of an engine
is a must-have feature of any submission software. Clicking the
link should open the search engine page in your default browser
or within the software itself. This will help verify the nature
and existence of the site.
Another way of assessing the engines you are submitting to is
by checking their popularity. To form some idea of the ranking of
different search engines go to
you'll find a listing of search engines and directories
ranked by their popularity on Alexa's engine.
Alternately, you can go to the Alexa site itself and check out
their ranking of different sites. You will have to sort out the
information for yourself though. For instance, directories and
engines are not listed together.
And, oh yes, cozycabin.com does exist but have a look at the
site before you decide to make a submission.
Confirmation and Updates
In view of what has been said above it is important to ask
“What kind of confirmation do you get when you make a
submission to a site?” Does the software show you the page
returned by the engine when a successful submission has been
made? If not you might quite rightly be suspicious that your page
has not been properly submitted or that you've made a
submission to a “vapour engine”.
For an individual who is doing a one-time submission of his
site to search engines, updates are not a major issue. For the
web developer who makes submissions on a regular basis for
different clients it is quite a different matter. We all know how
frequently websites change so if you are purchasing software it
is important to know how frequently the database is updated and
how long the updates are supplied for free.
The situation is analogous to using anti-virus software. It
doesn't matter how good the anti-virus software is if the
virus database is outdated. Changes to the database of engines
may become necessary for a variety of reasons but primarily
because of changes to submission requirements. Most packages
allow you to download updates as they become available, again in
much the same way as you would update a virus definition file.
The better software packages will inform you of the changes
Most software packages will allow you to print a report of
submissions made. Forwarding this information to your clients is
good customer relations but only if the information is reliable
Word of Caution
As a result of automated submissions you will doubtless
receive some unwanted email. I won't dwell on the irony of
this situation other than to say “ He who lives by the
As part of the submission process many sites require a valid
email address. In some cases, a site will not be added if there
is no response to a confirming email. An awkward situation can
develop if you are adding a listing on behalf of a client and
this client starts receiving spam. One solution is to set up a
temporary email address that is used for the short term or an
address that redirects emails to one of your own accounts. In
this way legitimate confirmations can be handled and spam need
not reach your client.
For individuals trying to promote their own websites manual
submission might well be the best option. Specialty engines can
be identified and categorisation can be carefully tailored so
that one's site appears exactly where it ought to in a
directory. For exactly the same reasons submission for any
specialty website might best be handled by manual submission.
On the other hand, for the web professional regularly making
submissions on behalf of clients an automated tool is an
invaluable assistant. Submissions can be made quickly and be
automatically documented. However, careful choice of software is
Look for submission software that documents engines, provides
information about them and links to them. Look for software that
is regularly updated and that removes engines as well as adds
them. Any submission should show the confirmation page generated
by a successful submission. Do not look at total number of
engines as a sign of quality software. In fact, beware
extravagant claims about number of engines – these probably
include “vapour engines”.
Submission software should carefully document what submissions
have been made and to what sites. In this way you can avoid
making multiple submissions. This is something that some sites
will interpret as spamming and they will penalise you for it. You
need to protect your own and your clients' interests and
not achieve the opposite to your desired effect.
Many companies provide free trial or “lite”
versions of submission software often as part of an integrated
suite of tools. To locate these packages check the DMOZ listing
under the category, “Computers: Software: Internet: Website
Promotion”. After all, any company in the Search Engine
business should be listed here!
Test drive these packages carefully and you'll be able
to make the right purchasing decision.
About the Author
Peter Lavin runs a Web Design/Development firm in Toronto,
Canada. He has been published in a number of magazines and online sites, including UnixReview.com, php|architect and International PHP Magazine. He is a contributor to the recently published O'Reilly book, PHP Hacks and is also the author of Object Oriented PHP, published by No Starch Press.
Please do not reproduce this article in whole or part, in any form, without obtaining written permission.