为什么屏蔽垃圾蜘蛛?
屏蔽垃圾蜘蛛主要有以下几个原因:
节省服务器资源:垃圾蜘蛛通常会频繁抓取网站内容,消耗大量的服务器资源,导致正常用户访问速度变慢。
保护网站安全:一些垃圾蜘蛛可能会尝试扫描网站漏洞,增加网站被攻击的风险。
提高SEO效果:垃圾蜘蛛抓取的内容通常不会对网站的SEO有正面影响,反而可能会影响搜索引擎对网站的评价。
减少日志文件大小:垃圾蜘蛛的抓取行为会在服务器日志中留下大量无用的记录,占用存储空间。
通过屏蔽这些垃圾蜘蛛,你可以有效地保护你的网站资源,提升用户体验,并减少潜在的安全风险。
宝塔中提供了付费版和免费版的Nginx防火墙。我们可以利用 User-Agent 来屏蔽掉许多无关紧要的蜘蛛爬虫,并且防止 SQL 注入和菜刀一句话的入侵。
然而,免费版的 nginx 防火墙规则相对较少,种类也不够多样化。因此,我们将分享一些来自付费版的规则,您可以手动逐条添加到免费版的防火墙中。遗憾的是,免费版没有导入功能。
安装Nginx免费防火墙
免费防火墙防御效果
添加过滤垃圾蜘蛛爬虫规则
添加垃圾蜘蛛规则,包括常见的垃圾蜘蛛和AI爬虫
(www\.seokicks\.de|YYSpider|Mattermost|Discord|CCBot|RepoLookoutBot|FBexternalhit|serpstatbot|Pinterestbot|SurdotlyBot|DataForSeoBot|DigExt|HttpClient|MJ12bot|heritrix|Ezooms|FlightDeckReports|LingueeBot|Web-Crawler|WellKnownBot|Yellowbrandprotectionbot|ev-crawler|NE Crawler|Facebot|facebookexternalhit|meta-externalagent|facebookscraper|facebookexternalhit|GrapeshotCrawler|SemrushBot|DotBot|MegaIndex\.ru|MauiBot|AhrefsBot|BLEXBot|HubSpotCrawler|CriteoBot|YaK|Mail\.RU_Bot|Barkrowler|vxiaotou-spider|telegram|dingtalk|DuckDuckGo|applebot|webprosbot|AwarioBot|Amazonbot|AmazonAdBot|YouBot|P/2\.0|YandexBot|Slurp|AdsBot-Google|Googlebot-Image|Googlebot-News|Googlebot-Video|msnbot|Twitterbot|Slackbot|LinkedInBot|NaverBot|Yeti|Daumoa|Teoma|ScoutJet|Exabot|Cliqzbot|SeznamBot|PaperLiBot|SputnikBot|Qwantify|Steeler|Gigabot|Scooter|TurnitinBot|Scrapy|Diffbot|Kozmos|KaloogaBot|LSSRocketCrawler|Nutch|Sphider|Xenu|HTTrack|Wget|Curl|Apache Nutch|Larbin|WebSPHINX|WebCopier|TeleportPro|Offline Explorer|SiteSucker|BlackWidow|DomainCrawler|AspiegelBot|LivelapBot|spbot|DnyzBot|GPTBot|ChatGPT-User|cohere-ai|Google-Extended|omgilibot|anthropic-ai|ClaudeBot|Claude-Web)
您还可以添加其他规则以提升面网站的安全,以下是常见的过滤规则。
关键词过滤 1
(WPScan|HTTrack|antSword|harvest|audit|dirbuster|pangolin|nmap|sqln|hydra|Parser|libwww|BBBike|sqlmap|w3af|owasp|Nikto|fimap|havij|zmeu|BabyKrokodil|netsparker|httperf| SF/)
一句话*屏蔽的关键字*过滤 2
(?:define|eval|file_get_contents|include|require_once|shell_exec|phpinfo|system|passthru|chr|char|preg_w+|execute|echo|print|print_r|var_dump|(fp)open|alert|showmodaldialog|file_put_contents|fopen|urldecode|scandir)(
一句话*屏蔽的关键字*过滤 3
$_(GET|post|cookie|files|session|env|phplib|GLOBALS|SERVER)
SQL 注入过滤 2
selects+.+(from|limit)s+
SQL 注入过滤 3
(?:(union(.*?)select))
SQL 注入过滤 6
benchmark((.*),(.*))
SQL 注入过滤 7
(?:fromW+information_schemaW)
SQL 注入过滤 8
(?:(?:current_)user|database|schema|connection_id)s*(
SQL 报错注入过滤 01
(extractvalue(|concat(0x|user()|substring(|count(*)|substring(hex(|updatexml()
SQL 报错注入过滤 02
(@@version|load_file(|NAME_CONST(|exp(~|floor(rand(|geometrycollection(|multipoint(|polygon(|multipolygon(|linestring(|multilinestring()
SQL 注入过滤 10
(substr()
SQL 注入过滤 1
(ORD(|MID(|IFNULL(|CAST(|CHAR))
SQL 注入过滤 1
(EXISTS(|SELECT#|(SELECT)
菜刀流量过滤
(array_map("ass)
SQL 报错注入过滤 01
(bin(|ascii(|benchmark(|concat_ws(|group_concat(|strcmp(|left(|datadir(|greatest()