Also known as a bot (short for robot) or crawler a spider is a software application used by search engines to crawl around the web and gather information about web pages for their index. Although the inference is that the spider actually wanders around the web gathering data, it actually goes nowhere. They reside on their own servers and send out requests to web sites in the same way as a browser does, but instead of downloading pages for viewing, it indexes the content.