A web crawler is an agent sent out by a search engine to catalogue websites for indexing purposes.

See Aslo: Bot, Crawler, Robot, Spider,

This entry was posted on Friday, August 28th, 2009 at 11:04 am and is filed under SEO. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.