A crawler is an agent sent out by a search engine to catalogue websites for indexing purposes.

See Also: Bot, Robot, Spider

This entry was posted on Tuesday, August 25th, 2009 at 12:35 pm and is filed under Search Engines. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.