Razer Technology Solutions, Inc.

332 Gordon Drive
Exton, PA 19341
610-524-7073

The Myth of Manual Search Engine Submission
by: Ralph Tegtmeier

One of the ongoing myths abounding in the search engine world is that of
manual submission: many search engine optimization agencies make a point
in promoting their "manual submission services", and if you follow discussions
in various forums and newsgroups you will frequently encounter people
advising others to "submit manually only, or else ..."

Well - or else, what? Is there something inherently evil or - at the very least -
inefficient about automatic submission? And if so, where's the problem? To cut
a long story short: there really isn't any, but to understand why not, it's
important to get a grasp on the actual mechanics of search engine submission.
This will help debunk a lot of the mystique currently surrounding the process and
may save you hundreds of hours of unnecessary manual labor.

All important search engines feature special submission pages endowed with HTML
forms where you can enter your web site's URL or, in some cases, additional data
such as your email address, your name, etc. Submission is effected by clicking on
the submit button, whereupon you will typically be directed to a confirmation page.
To avoid spam and oversubmissions, URLs will only be accepted if and when the data is
referred by these same submission pages. Of course, when submitting manually this
is a given, and the user needn't even be aware of it.

The trick with automatic submission is to make believe, i.e. to fool the search engine
submission setup that the URL submitted was actually entered on the submission page
installed for this purpose. So how is this done? Easy: every web browser carries a
variable with it which can be read by any server accessed. In technical terms: it is the
"http_referer" (yes, only one "r"!) variable which tells the search engine from where the
browser accessing the success or confirmation page is actually coming. If you submit a
site manually, your browser will automatically convey this variable to the submission
program (which, by the way, is typically a Perl or PHP script). So what every
automatic submission tool has to do to see the submitted data accepted, is to pose
as a web browser by emulating a browsers identification variable (the "UserAgent"),
configure its http_referer variable to the search engine's submission page's - and bingo!

If this is done right, and all commercial submission programs we are aware of are
intelligent enough to cover this very basic routine, there's really no way a search
engine can tell whether any given URL was submitted by a robot program, i.e. automatically,
or manually.

Of course, there are some telltale hints which may yet either give your automatic submitter
away or provoke a submission denial:

• If your program submits too many pages in too short
  an interval
  
• If your submitter program fails to convey other
  required data beyond the mere URL (e.g. your email
  address, your name, or whatever is required by the
  search engine), as any diligent human submitter
  would
  
• If your program is not up-to-date, referencing an
  obsolete submission page variable (these tend to
  change from time to time as search engines develop)

Again, all commercial programs we know of, whether client or server based, will do all this
quite nicely. Just make sure you are using a current version. To avoid infringement of
submission maximums, submitting no more than 5 pages per domain and engine is generally
considered a safe approach. If submitting multiple URLs to several engines (as most
programs will do), make sure these are submitted to each engine consecutively: first,
you would, for example, submit URL #1 to AltaVista, next to Excite, then to FAST, then
to Google, next to HotBot, next to Northern Light, and so on, till the very last engine on your
list. Only after this list of engines has been submitted to, would you (or, rather, your program)
start submitting URL #2, etc. This ensures that you won't run afault of submission time
limits like you might if you submitted all URLs first to AltaVista in one go, then to Excite, etc.

Ralph Tegtmeier is the co-founder and principal of fantomaster.com Ltd. (UK)
and fantomaster.com GmbH (Belgium), a company specializing in webmasters
software development, industrial-strength cloaking and search engine positioning
services.

He has been a web marketer since 1994 and is editor-in-chief of fantomNews, a free
newsletter focusing on search engine optimization.

You can contact him at Ralph Tegtmeier
(c) copyright 2000 by fantomaster.com