Observed Web Robot Behavior on Decaying Web Subsites

Joan A. Smith, Frank McCown, Michael L. Nelson

Research output: Contribution to journalArticlepeer-review

Abstract

We describe the observed crawling patterns of various search engines (including Google, Yahoo and MSN) as they traverse a series of web subsites whose contents decay at predetermined rates. We plot the progress of the crawlers through the subsites, and their behaviors regarding the various file types included in the web subsites. We chose decaying subsites because we were originally interested in tracking the implication of using search engine caches for digital preservation. However, some of the crawling behaviors themselves proved to be interesting and have implications on using a search engine as an interface to a digital library.

Original languageAmerican English
JournalD-Lib Magazine
Volume12
DOIs
StatePublished - Feb 1 2006

Disciplines

  • Computer Sciences

Cite this