Documentation Home |
User Help |
File Types |
FAQ |
Info. for Web Content Providers |
Index Helper |
Custom Style Manager |
Crawler Info
Information About Search Engine Crawlers
The Penn State Search Engine http://search.psu.edu/ uses the following IP addresses and User-Agent HTTP header for crawling new Web sites:
| Search Appliance | Hostname / IP address | User-Agent string |
| Production Google Search Appliance #1, 5.x series |
search-crawler1.aset.psu.edu / 128.118.142.21 |
PennStateSpider (http://aset.its.psu.edu/googledocs/crawler_info.html) |
| Production Google Search Appliance #2, 5.x series |
search-crawler2.aset.psu.edu / 128.118.142.22 |
PennStateSpider (http://aset.its.psu.edu/googledocs/crawler_info.html) |
The Penn State Search Engine is now supported by two load-balanced machines, which both answer to http://search.psu.edu. For more information on the Penn State Search Engine, please review the documentation.
|
