DuckDuckGo now crawls the web regularly to create a free list of trackers to block
It has become a part time job to navigate and if necessary block the ads and other nuisances that plague practically every website today. DuckDuckGo is hoping to make this easier and more lightweight with its own list of prime offenders that’s kept constantly fresh by its own crawlers. Naturally, it’s free.
In a blog post announcing the new service, called Tracker Radar, DuckDuckGo observes that although ad and tracker blockers are essential software, they’re not designed efficiently or sustainably.
For instance, one of the most popular lists used by blocking software, EasyList, comprises nearly 100,000 rules — URLs or strings it instructs the blocker to watch out for. This is a great resource, but also, owing to the fact that it has been manually curated over several years, a bloated one: Thousands of those rules may no longer be accurate or relevant, and most won’t come into play during the average user’s browsing session hitting a few of the top 100 sites out there.
DuckDuckGo’s approach is to start with a clean slate and use web crawlers — virtual online agents that visit and catalog selected aspects of sites — to build a rolling database of rules that adapts to the latest jukes by trackers and site admins.
Part of the profile of a tracker as recorded by DuckDuckGo’s Tracker Radar.
This database can be both comprehensive and flexible, as it compares the patterns of some 50,000 sites to find rules and associations. Tracking has become highly sophisticated, and methods exist to use multiple sites and services to identify a user who has opted out of cookies and other traditional signals. By comparing behaviors of many sites regularly and profiling the techniques employed across them, the resulting data is rich and up to date.
At least, that’s the pitch. DuckDuckGo has its own suite of privacy-oriented plug-ins and apps it would like people to use, and the data from the Tracker Radar will of course be integrated with those. But it’s also being provided free of charge — not just the data, but the code for collecting and compiling it — for anyone who wants to use it themselves or improve it.
What that means is if you’re currently using a popular blocker like uBlock Origin or the like, you will likely gain the benefit of this new database without having to do anything at all. On the other hand, if you like the idea of a thinned-down tracking solution and want to give DuckDuckGo’s a shot, search for the plugins/extensions/add-ons to your favorite browser (assuming it’s either Chrome, Firefox, or Safari) or on mobile just snag DuckDuckGo’s standalone browser.
It has become a part time job to navigate and if necessary block the ads and other nuisances that plague practically every website today. DuckDuckGo is hoping to make this easier and more lightweight with its own list of prime offenders that’s kept constantly fresh by its own crawlers. Naturally,…
