Probing the Deep, Invisible Web

As most of us who are in search know, the web's repository of knowledge is vastly larger than what is accessible to the major commercial search engines. Hundreds of thousands of databases cannot be probed and their documents remain invisible to all but th specialized search tools used to dig them out.

Fortunately, several projects, including one pointed to by Xan Porter (called Qprober - PDF warning) use automatic text extraction followed by intelligent, trained querying of a system to extract as many of the available documents as possible. In the future, this type of intelligent access will provide a gateway for access to much of what is currently 'invisible'.

Today, for example, I used another resource Xan pointed out, called Complete Planet to find an art project that was hidden in the database of the Library of Fine Arts in Dresden, Germany. The search revealed an installation artist's work - Günther Hornig - who has a great eye for the whimsical.