34 lines
1.9 KiB
Plaintext
34 lines
1.9 KiB
Plaintext
/google desktop|bot|spider|crawler|yahoo|findlinks|
|
|
search|;anonymous\.|alef\/|amagit|
|
|
answerbus|appie |appengine-google|apple(syndication|-pubsub)|ask jeeves|
|
|
barkrowler|berlin-fu-cow|biglotron|binlar|blog(dimension|lines)|
|
|
book(dog|mark)|bubing|butterfly|c4pc|caretcursor|
|
|
charlotte|csci_|ccubee|ce-preload|checker|
|
|
covario|crawl(y| annu)|curl\/|custo |darxi|
|
|
datacha0s|daumoa|dailyperfect|del\.icio\.us\b|depspid|
|
|
developers\.google\.com\/\+\/web\/snippet\/|digger|earthcom|ejupiter|ets v5|
|
|
europarchive|extractor|facebookexternalhit|fast[ mb-]|favorg|
|
|
feed(burner|fetcher)|(\b|c)fetch(\b|p|e)|finder|fleck|flyindex|
|
|
forvohun|funnelback|gigablastopensource|gnomit|goldfire server|
|
|
google-site(maps|-verification)|google-structureddatatestingtool|google (web preview|favicon)|grub|gt::www\/1\.02|
|
|
helix|heritrix|htdig|httping v2.5|httrack|
|
|
ia_archiver|ichiro|ina dlweb|indocrew|integromedb|
|
|
internetseer|interseek|ip\*works|jakarta|java\/|
|
|
l\.webis|larbin|libwww-p|link(dex| check(er)?|aider|fluence)|ltx|
|
|
lwp|mail\.ru|mama (casper|cyber)|mediapartners|metaspinner|
|
|
microsoft data access|minirank|morfeus |\+msie\+6\.0|msnptc|
|
|
net(sprint|vibes )|newsg8|newt activex|ng\/2|nnn\/ttt|
|
|
nutch(-|\d)|\bodp\b|oneriot|ozelot|(page_|slurpy )verifier|
|
|
page(bull|peeker)|panopta\.com|panscient|pflab|^php(\/| v)|
|
|
picmole|pi-?monster|pompos|powermarks|program shareware|
|
|
python|quantcast|qwantify|robozilla|(aide|magpie| )rss( |\/)|
|
|
rssingbot|rtgi|sagool|sbider|scooter|
|
|
scoutjet|scrapy|scrubby|shelob|silk|
|
|
siteexplorer|(sky|synaptic)walker|sohu |survey|sygol|
|
|
synapse|szukaj|t-h-u-n-d-e-r-s-t-o-n-e|tagyu|telnet0.|
|
|
\btest(\b|_)|tfc |theophrastus|thumb|twitturl|
|
|
unicorn|urlbase|vagabondo|validat(o|eu)r|voltron|
|
|
voracious|voyager\/|vse|wauuu|web( ?index|cow|moni|capture|pix|copier|stripper|corp|scan|mastercoffee|-sniffer)|
|
|
wget|wwwster|wysigot|xenu|xirq|
|
|
yacy|yandex|yellowjacket|yeti|yoono|
|
|
zion\/|^(dcs|mozilla\/5\.0|msie 6\.0|teoma|megaindex)$/i |