james / fedora-infra / ansible

Forked from fedora-infra/ansible 5 years ago
Clone
Source Code
GIT

4a3da58 pagure: drop Bytespider crawler

Authored and Committed by kevin 10 months ago
raw patch tree parent
2 files changed. 6 lines added. 0 lines removed.
roles/pagure/templates/0_pagure.conf
file modified
+4 -0
roles/pagure/templates/robots.txt.j2
file modified
+2 -0 
    pagure: drop Bytespider crawler
    
    This crawler is hitting pagure.io really hard.
    Sent it a 403 and block in robots.txt
    
    Signed-off-by: Kevin Fenzi <kevin@scrye.com>
file modified
+4 -0
file modified
+2 -0
Powered by Pagure 5.14.1
Documentation • File an Issue • About • SSH Hostkey/Fingerprint
© Red Hat, Inc. and others.