Wikimore

This page summarizes the available tools against abuse and rate limiting bots/scraping. It is uses GitLab/Abuse and rate limiting as a template.

Malicious traffic pattern ("scraping")

Gerrit is not behind the load balancing infrastructure, so tools like superset are not available to analyze traffic. However there are some Logstash dashboards which can be used to analyze the traffic:

Access logs dashboard

Application logs

It's also useful to look at the nftables tracking table:

sudo nft list set inet filter TRACKINGLIST
sudo nft list set inet filter TRACKINGLIST_V6

Throttling using nftables

Even though Gerrit is not behind the CDN some throttling and rate limiting capabilities were added. In T366882 a new Puppet module profile::firewall::nftables_throttling was implemented which allows basic rate limiting based on a maximum number of parallel TCP connections.

If the threshold is exceeded the offending IP address is blocked for 5 minutes. More information can be found in Firewall#Throttling with nftables.

Metrics for this throttling can be found in Grafana.

Blocked nets

Requestctl has a feature to ban single IPs or IP ranges with the abuse/blocked_nets ipblock. This IPs are dropped automatically on all wmf hosts using firewall rules. This is a simple and effective measure to ban a IP with little work.

For Gerrit this feature is currently not working because of the recent migration to nftables. See T348734. But there is ongoing work to implement this feature also for nftables. As a temporary workaround abusive IPs and IP ranges can be added to: profile::gerrit::gerrit_abusers like in this change: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1113986. Puppet adds this IPs to a nftables block list after the next puppet run.

For emergencies it's possible to drop IPs or ranges using nftables directly on the Gerrit host:

sudo nft add rule inet filter throttling ip saddr <IP_range> drop

This adds a drop entry in the throttling chain. Keep in mind to persist this using the profile::firewall::nftables_throttling::abusers hiera key otherwise puppet reverts this after 30 minutes.

Blocking using Apache config

As seen in this patch, it is also possible to block low traffic incidents through Apache configuration. This method is not recommended for high traffic incident. The recommended method in such context would be to either use throttling or drop traffic.

Emergency access

It is possible to tunnel traffic directly to Gerrit using tunnelencabulator. This tool opens a SSH tunnel and modifies the local /etc/hosts file. In case of traffic related incidents this can be a useful tool to access gerrit (probably in combination with blocking certain/all public traffic, if needed). tunnelcabulator is included in Wmf-laptop.

jelto-wmf@x1:~$ curl -v https://gerrit.wikimedia.org
*   Trying 2620:0:861:2:208:80:154:151:443...
* Connected to gerrit.wikimedia.org (2620:0:861:2:208:80:154:151) port 443 (#0)
...
HTTP/1.1 302 Found
...
jelto-wmf@x1:~$ tunnelencabulator -s # -s is needed for non-CDN-served services

jelto-wmf@x1:~$ curl -v https://gerrit.wikimedia.org
*   Trying 127.149.7.2:443...
* Connected to gerrit.wikimedia.org (127.149.7.2) port 443 (#0)
...
HTTP/1.1 302 Found
...