robots 2
by EngliSea on 2024-11-08
ᐥkeywords to search for in the user agentᐥ
AhrefsBot
Applebot
AwarioBot
Barkrowler
bingbot
Bloglines
CensysInspect
CCBot
ClaudeBot
coccocbot
crawler
curl
CyotekWebCopy
DataForSeoBot
Dataprovider
DotBot
DuckDuckBot
EdgeWatch
Exabot
facebot
FacebookExternalHit
facebookscraper
faqId
feedfetcher
gdnplus
github
Googlebot
Go-http-client
GPTBot
HeadlessChrome
Hello World
ia_archiver
IndeedBot
InternetMeasurement
ipip
Java
LinkedInBot
masscan
Mediatoolkitbot
MegaIndex
MJ12bot
msnbot
NetcraftSurveyAgent
Netcraft Web Server Survey
netsystemsresearch
NiggaBalls
org_bot
paloaltonetworks
Pandalytics
PetalBot
pmoz
Project-Resonance
python
Qwantify
SANDSLab
Scrapy
SeekportBot
SemrushBot
SEOkicks
Slurp
spider
tchelebi
Turnitin
URIBOT
URLSuMaBot
webprosbot
WhatWeb
WhoAreYouBot
WinHttpRequest
woorank
wpbot
YandexBot
Yeti
zgrab