Deep but not unfathomable
If you suspect there is more information on the Internet than search engines retrieve for you, it's time to discover the
Deep Web. The Deep Web is the new name for what used to be called the
invisible Web - so-called because the information stored in there was invisible to Internet search engines. Why so? Much of it is stored in databases that search engines cannot crawl, either because the information is password-protected or because it is dynamically generated in response to queries and doesn't last long enough to be crawled like an HTML page can be. Think of the phone book - a search tool can tell you the
White Pages exist but can't look up a number for you - you need to be on the site itself to run the query. The Deep Web is full of such database-style sites where you need to use the site's search technology to find information. Why bother?
The Deep Web contains some of the most useful information online - databases, directories, statistics, full text legislation and court decisions, full text news archives, image and film databases, lists and rankings and so on. How to fathom it all.? Have a look at
Bright Planet's
Deep Web White Paper at
www.brightplanet.com/technology/deepweb.asp. The paper is also available for download as a PDF (
www.brightplanet.com/pdf/deepwebwhitepaper.pdf). You'll get a good grounding here. The best finding tool for what's in the Deep Web is
DirectSearch (http://www.freepint.com/gary/direct.htm). A Deep Web research blog is at
www.deepwebresearch.info/. Another useful updater is DirectSearch's Gary Price's ResourceShelf blog (
www.resourceshelf.com/)
Posted by belinda at March 11, 2004 11:49 AM