学院首页>网页制作>HTML>搜索引擎的蜘蛛人（ROBOTS）秘密

搜索引擎的蜘蛛人（ROBOTS）秘密

作者：来源：添加时间：2006-5-21 21:13:53

例子:
# robots, scram

User-agent: *
Disallow:/cgi-bin
Disallow:/TRANSCRIPTS
Disallow:/development
Disallow:/third
Disallow:/beta
Disallow:/java
Disallow:/shockwave
Disallow:/JOBS
Disallow:/pr
Disallow:/Interactive
Disallow:/alt_index.html
Disallow:/webmaster_logs
Disallow:/newscenter
Disallow: /virtual
Disallow: /DIGEST
Disallow: /QUICKNEWS
Disallow: /SEARCH

User-agent: Mozilla/3.01 (hotwired-test/0.1)
Disallow:/cgi-bin
Disallow:/TRANSCRIPTS
Disallow:/development
Disallow:/third
Disallow:/beta
Disallow:/java
Disallow:/shockwave
Disallow:/JOBS
Disallow:/pr
Disallow:/Interactive
Disallow:/alt_index.html
Disallow:/webmaster_logs
Disallow:/newscenter
Disallow: /virtual
Disallow: /DIGEST
Disallow: /QUICKNEWS
Disallow: /SEARCH

User-agent: Slurp
Disallow:/cgi-bin
Disallow:/TRANSCRIPTS
Disallow:/development
Disallow:/third
Disallow:/beta
Disallow:/java
Disallow:/shockwave
Disallow:/JOBS
Disallow:/pr
Disallow:/Interactive
Disallow:/alt_index.html
Disallow:/webmaster_logs
Disallow:/newscenter
Disallow: /virtual
Disallow: /DIGEST
Disallow: /QUICKNEWS
Disallow: /SEARCH

第 7 页，共 9 页 [1] [2] [3] [4] [5] [6] [7] [8] [9]

上一篇：样式表全接触（8）
下一篇：“mailto”的六则应用技巧

站内搜索