To block a spider, edit the spider list in text-acls.conf, upload-acls.conf or common-acls.conf. Then run make && ./deploy frontend.

For more information, see squids.