Lesson 5

One of the most powerful and the most misused tools on the Internet is the key word search engine. These tools are different from the catalog search engines which require humans to intervene and pick out the best Web sites. The key word search engine uses a more brute force approach.

Key word search engines use an automated technology called Web worms, crawlers or Web bots that hunt through the Web following links to find new pages. The worm indexes each page and sends all the information back to a master index. When you search using a key word search engine, you are searching the master index created by the web worms.

While Yahoo is selective and adds only the "best" sites, key word search engines are not selective and try to capture everything. While Yahoo has a small number of sites, they add the sites that are visited most often. Altavista and Google, two popular key word search engines have many obscure and less visited sites. Yahoo is somewhat exclusive requiring a site pass a vetting process. Altavista is all done automatically.

That means the manner is which we use these tools is different. Here's an analogy you can use for thinking about the difference between a key word search engine and a catalog search engine: a catalog search engine is like the table of contents of a book. It lists the chapters and when using the contents, a reader will figure out which chapter is most likely to have the information they are looking for. A key word search engine is like the index at the back of a book where you find the search term you're interested in and it tells you what page to go to.

One is no more effective than the other but rather, one will be faster than the other depending on the nature of your search task. Most people tend, however, to use the key word search engine first. This isn't always effective, though, because the Internet is a book of millions of chapters and billions of pages.

For example: imagine if the Internet were like a biography of Bill Clinton. If you were interested in President Bill Clinton's marital infidelities, you wouldn't look in the index by the term "Clinton" or "Lewinsky". There would be too many references, many of which may refer to events like his impeachment, a ramification of his marital infidelity, but not exactly what you're looking for. Instead, a smart researcher would look for an appropriate chapter heading. Same way with the Internet. You look first for a chapter (catalog search engine) before going to the index (key word search engine).

Key word search engines are best used:

for finding individual Web pages within Web sites

for finding unique information

for finding information that may not be arranged in a way that is useful for what you are searching for

for finding information that is hidden in larger topic.

If you are looking for a particular person and you don't know her occupation, the company she works for, her hobbies or her interests, how do you know where to look? Since a name is somewhat unique, this would be a good use of a key word search engine. Likewise, if you are trying to find whether a particular water pollutant is dangerous, you may not know what agency or organization has studied the environmental impact of the substance. Your best source of informtion might come from a governmental agency, an industry group, a university or an environmental group.

The key is defining the search terms so that they uniquely describe and thus identify your subject. The more unique the description is, the better a key word search engine will work for you. Think of what makes your topic unique. Some items that might add uniqueness to searches for a person are:

email address

social security number

phone number

addresses

name

occupation

company name

organizational affiliation

Notice the items in blue uniquely identify someone - every person that has an email address has their own email address and there is a unique social security number for each person. Email addresses are one of the best way of identifying people on the Web. Always get people's email address.

The items in green are mostly unique: every person has a few phone numbers and addresses and some of these uniquely identify them but others they may share with roomates, family or coworkers. The items in red are not necessarily unique. Many people have the same name and thus name varies from very unique names to very common names. Thus a common name like John Smith must be combined with other terms that make it unique. For example, there might only be one John Smith working for Goodyear Tire in Youngstown, Ohio.

In Lesson 6, we will learn Boolean Logic, the tool we need to make things more unique.