Anubis (software)


Anubis is an open source software program that adds a proof of work challenge to websites before users can access them in order to deter web scraping. It has been adopted mainly by Git forges and free and open-source software projects.
Anubis was created by Xe Iaso in response to Amazon's web crawler overloading their Git server, as the crawler did not respect the robots.txt exclusion protocol and would work around restrictions. Iaso lists Hashcash as having inspired the project. The application supports inspecting request elements such as headers like the User-Agent header to determine if the request should require proof of work.
The name Anubis is taken from the Ancient Egyptian god of funerals and judgement, who weighs the hearts of the dead to determine if they are allowed passage into the afterlife, whereas the Anubis software "weighs the soul of incoming HTTP requests".
WWW Official Site

Design

Anubis temporarily blocks access to websites until the client completes a cryptographic challenge. This challenge is intended to be a design hurdle and incur a compute cost for web crawlers while minimizing impact on typical visitors. Challenge solutions remain valid for a tunable period of time using EdDSA-signed tokens, which may be stored as HTTP cookies.
The challenge itself is similar to the proof of work algorithms implemented by Hashcash and Bitcoin. Anubis presents a challenge to the client in the form of a random number, to which the client must respond with another number such that when the two numbers are concatenated, the SHA-256 hash of both numbers contains a pre-specified number of leading zeros. SHA-256 is chosen specifically because of its pre-image resistance while simultaneously being cheap to verify. Moreover, tuning the number of required leading zeros controls the difficulty of the challenge: a typical browser can find a solution with four leading zeros in seconds, while one with six can take several minutes.
By default, Anubis challenges any client claiming to be a browser, by checking if the User-Agent header contains "Mozilla", unless the client is requesting "low-harm" content. Addtionally, Anubis exempts Common Crawl "so scrapers have less incentive to scrape".
Although Anubis could be altered to mine cryptocurrency to serve as proof of work, Iaso has rejected this idea: "I don't want to touch cryptocurrency with a 20 foot pole. I realize I'm leaving money on the table by doing this, but I don't want to alienate the kinds of communities I want to protect."

Mascot

The software's loading screen is branded with a commissioned artwork of Anubis as a jackal-eared anime girl by the European artist CELPHASE. The mascot is depicted with a hoodie, skirt and magnifying glass. Before the artwork was ordered, Anubis used an AI-generated placeholder image.
The Anubis mascot is shown to all end users and cannot be altered in the software configuration. The image's feel may clash with websites that have more formal atmospheres, surprising or confusing users of those sites. Altering the branding is an enterprise feature and Iaso has requested that operators not attempt to change it themselves unless they have made financial contributions to the project.
Duke University, which has deployed Anubis for its digital archives, was "hesitant" to use it due to the mascot but has reached an agreement to use the software with custom branding. Jamie Zawinski describes the mascot as "cutesey kawaii bullshit".

Adoption

It has been used by a number of projects, including:

Criticism

Tavis Ormandy noted that the cost of scraping websites protected by Anubis is likely negligible: "I don’t think we reach a single cent per month in compute costs until several million sites have deployed Anubis."