Intelligent crawler
We created an intelligent web crawler. It's loaded with over a million quality assured hashes of known child sexual abuse material, each one having gone through a human assessment by our expert Analysts. The web crawler enables automated detection at scale with an extremely high level of accuracy.
How does using the crawler help our mission?
We use our crawler as one operational tactic in a suite of tools designed to find, remove, and disrupt the availability of child sexual abuse material. Our crawler is deployed in a considered and targeted manner in order to be most effective. In addition, it reduces unnecessary exposure to child sexual abuse imagery for our Analysts.
In 2021, we’ve been using our web crawler to provide an additional service to our Members in the domain registry sector. Now we can automatically scan new sites for known child sexual abuse images after they have been registered and inform our Members if anything is detected.
In 2021 it crawled almost 14 million webpages, and over 66 million images. By comparing each image our crawler finds to the hashes of known child sexual abuse material, it means we can find duplicate child sexual abuse images hidden across the internet.
“Computer vision and machine learning in general are useful tools, and we seek to leverage them where possible to reduce the amount of manual effort our Analysts need to put in to prioritise and classify images. Technology is ever-improving but currently lacks the level of granularity and precision we need for our work. At the IWF the final decision is always based on a human assessment with computer vision being used as a complementary tool.”
Chris Wilson
IWF Head of Software