|
Definition: web robots are a class of software programs that traverse network hosts gathering information from and about resources -- lists, information and collections
|
Problems Addressed
-
No pre-defined data set or fields for a document on the Web
-
Manual traversal of the Web is virtually impossible
-
Contents of the resources change as also the hyperlinks
-
Improperly maintained Web sites produce dead links
|
|
Limitation: Information generated Òon-the-flyÓ (e.g. by CGI scripts) cannot be retrieved by Web rebots
|
