Search engines have robots tha.come to your site and grab everything there is to grab. But becaus.competition is so fierce, there is no way to get in the search engines, unless you pay for ads or hire a SEO (Search Engine Optimization) consultant, right? Wrong!
Even if you pay big money, if your site is not properly seen by the robots used by search engines for indexing, chances are many of your pages will never make it.
In this article I will discuss the importance of having your website structured properly, the importance of using the old fashioned hyperlinks versus the modern Flash menus, scripts and extensions and provide you with a very simple and free tool that will allow you to see your site in a similar fashion most indexing robots do. But first, let"s define some of the concepts.
What is a
robot? A robot is .computer program that automatically reads web pages and goes through every link that it finds.
The first robot was developed by MIT and launched in 1993. It was named the World Wide Web Wander and its initial purpose was of a purely scientific nature, its mission was to measure the growth of the web. The index generated from the experiment's results proved to be an awesome tool and effectively became the first search engine. Most of the online stuff we can"t live without today was born as a side effect of some scientific experiment.
What is a search engine?
Generically, a search engine is a program that searches through a database. In the popular sense, as referred to the web, a search engine is considered to be a system that has a user search form, which can search through a repository of web pages gathered by a robot.
What is a bot? What is a spider? What is a crawler?
Bot is just a shorter, cooler (for some) version of the word robot. Spiders and crawlers are robots, only the names sound more interesting in the press and within metro-geek circles. For reasons of consistency, I will use the term robot throughout this article, when referring to spiders, crawlers and bots.
Are there other... things that crawl out there?
Oh yeah, but these things are way beyond the scope of this article. Well, for the conspiracy theory aficionados, let's see... we have worms - self-replicating programs, webants (or ants) - distributed cooperating robots, autonomous agents, intelligent agents and many other bots and beasties.
How do robots work?
As with all other things technical, I believe that the only way you will utilize a technology to its full potential and to your best advantage is if and when you understand how that technology works. When I say how it works, I don't mean intricate technical details, but fundamental processes, big picture stuff.
Generally, robots are nothing but stripped down versions of web browsers, programmed to automatically browse and record information about web pages. There are some very specialized robots out there, some that look only for blogs, some that index nothing but images. Many (such as Google"s GoogleBot) are based on one of the first popular browsers, called Lynx. Lynx was initially a pure text browser, therefore, in today's internet Lynx would be extremely robust and fast. Basically, if you can program, you can take Lynx, modify it and make a robot.
So how do these things actually work? They get a list of websites, and literally start "browsing" them. The.come to your site and then start reading the pages and following every link, while storing different information, such as page titles, the actual text of the page, etc.
Based on the above, what would happen if instead of your beloved Internet Explorer, Firefox, Opera or whatever browser you are attached to, you go dig on the internet and download a version of the venerable Lynx browser?
I'll tell you what would happen, and some will probably accuse me of giving away one of the secrets the SEO corporat.community does not want you to know:
You will be able to see your site very close to the way a robot sees it. You will be able to look fnavigation menu made in Flash, Java script, PHP, etc. and will not be able to get to your pages, it's as simple as that.
How do I get Lynx?
Lynx first started life as a UNIX application, written by the University of Kansas as part of their campus-wide information system. It then became a gopher application (a pre-web search tool), then a web browser. The official page for Lynx is
lynx.isc.org, however, if you are not a Linux geek, used to play with binary distribution files and used t.compiling your own apps (dont worry about what I just said), you might want to find a version that someone else already made usable for you.computer. For example, if you are a PC user running Windows, you might want to check links to Win3.compiled versions. At the time of writing, one such site is
csant.info/lynx.htm (called a distribution site) where you can download a version that will install onto Windows machines in a fashion that will be familiar to non-geeks. After you install the browser, you might want to read the documentation. To get you going and to alleviate your beginner frustrations, Ill tell you that you must press the G key (as in go), then type th.complete URL of the site you want to browse (starting with http://), then hit Enter. Use the arrows to navigate.
Bottom line, use Lynx to verify that every page of site is accessible and let the robots do all the work for you. You'll save yourself a lot of aggravation and maybe some money that you would waste on advertising your otherwise non-indexable site.
