apaan sih robot.txt?

Posted: 17 June 2008 in Internet
Tags: ,

Gw jg blm tau apaan robot.txt, yang jelas gw nemu ini...
http://www.google.com/robots.txt
User-agent: *
Allow: /searchhistory/ 
Disallow: /news?output=xhtml& 
Allow: /news?output=xhtml 
Disallow: /search 
Disallow: /groups 
Disallow: /images 
Disallow: /catalogs 
Disallow: /catalogues 
Disallow: /news 
Disallow: /nwshp 
Disallow: /? 
Disallow: /addurl/image? 
Disallow: /pagead/ 
Disallow: /relpage/ 
Disallow: /relcontent 
Disallow: /sorry/ 
Disallow: /imgres 
Disallow: /keyword/ 
Disallow: /u/ 
Disallow: /univ/ 
Disallow: /cobrand 
Disallow: /custom 
Disallow: /advanced_group_search 
Disallow: /advanced_search 
Disallow: /googlesite 
Disallow: /preferences 
Disallow: /setprefs 
Disallow: /swr 
Disallow: /url 
Disallow: /default 
Disallow: /m? 
Disallow: /m/? 
Disallow: /m/lcb 
Disallow: /m/search? 
Disallow: /wml? 
Disallow: /wml/? 
Disallow: /wml/search? 
Disallow: /xhtml? 
Disallow: /xhtml/? 
Disallow: /xhtml/search? 
Disallow: /xml? 
Disallow: /imode? 
Disallow: /imode/? 
Disallow: /imode/search? 
Disallow: /jsky? 
Disallow: /jsky/? 
Disallow: /jsky/search? 
Disallow: /pda? 
Disallow: /pda/? 
Disallow: /pda/search? 
Disallow: /sprint_xhtml 
Disallow: /sprint_wml 
Disallow: /pqa 
Disallow: /palm 
Disallow: /gwt/ 
Disallow: /purchases 
Disallow: /hws 
Disallow: /bsd? 
Disallow: /linux? 
Disallow: /mac? 
Disallow: /microsoft? 
Disallow: /unclesam? 
Disallow: /answers/search?q= 
Disallow: /local? 
Disallow: /local_url 
Disallow: /froogle? 
Disallow: /products? 
Disallow: /froogle_ 
Disallow: /product_ 
Disallow: /products_ 
Disallow: /print 
Disallow: /books 
Disallow: /patents? 
Disallow: /scholar? 
Disallow: /complete 
Disallow: /sponsoredlinks 
Disallow: /videosearch? 
Disallow: /videopreview? 
Disallow: /videoprograminfo? 
Disallow: /maps? 
Disallow: /mapstt? 
Disallow: /mapslt? 
Disallow: /maps/stk/ 
Disallow: /mapabcpoi? 
Disallow: /translate? 
Disallow: /ie? 
Disallow: /sms/demo? 
Disallow: /katrina? 
Disallow: /blogsearch? 
Disallow: /blogsearch/ 
Disallow: /blogsearch_feeds 
Disallow: /advanced_blog_search 
Disallow: /reader/ 
Disallow: /uds/ 
Disallow: /chart? 
Disallow: /transit? 
Disallow: /mbd? 
Disallow: /extern_js/ 
Disallow: /calendar/feeds/ 
Disallow: /calendar/ical/ 
Disallow: /cl2/feeds/ 
Disallow: /cl2/ical/ 
Disallow: /coop/directory 
Disallow: /coop/manage 
Disallow: /trends? 
Disallow: /trends/music? 
Disallow: /notebook/search? 
Disallow: /music 
Disallow: /browsersync 
Disallow: /call 
Disallow: /archivesearch? 
Disallow: /archivesearch/url 
Disallow: /archivesearch/advanced_search 
Disallow: /base/search? 
Disallow: /base/reportbadoffer 
Disallow: /base/s2 
Disallow: /urchin_test/ 
Disallow: /movies? 
Disallow: /codesearch? 
Disallow: /codesearch/feeds/search? 
Disallow: /wapsearch? 
Disallow: /safebrowsing 
Disallow: /reviews/search? 
Disallow: /orkut/albums 
Disallow: /jsapi 
Disallow: /views? 
Disallow: /c/ 
Disallow: /cbk Disallow: /recharge/dashboard/car 
Disallow: /recharge/dashboard/static/ 
Disallow: /translate_c? 
Disallow: /s2/profiles/me 
Allow: /s2/profiles 
Disallow: /s2 
Disallow: /transconsole/portal/ 
Disallow: /gcc/ 
Disallow: /aclk 
Disallow: /cse? 
Disallow: /tbproxy/ 
Disallow: /MerchantSearchBeta/ 
Disallow: /ime/
mungkin itu folder" yg dimiliki Google??
gw belum tau jawabannya...
ada yg mau ngasi tau??

 

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s