Friday, July 22, 2022: 4:00 pm (DAC 206): Today there are two dominant approaches to measuring behavior at scale on the web without the cooperation of service providers: there are bot farms, which run automated browsers on infrastructure controlled by the measurer; and there are instrumented extensions that run on the browsers of individuals who have agreed to participate.

Bot farms are bad because it’s hard to measure everything that is interesting to study in a fully automated way; extensions are bad because for them the measurements follow the participant’s use of the service, whereas directly controlling what is measured is often useful in a study (plus, there are privacy risks).

The best way to measure behavior on the web is with a botnet. Botnets are distributed over participant computers, so bots can mix in requests to a human alongside automated measurements. On the other hand, where bots go, and what they ask about, is fully specifiable in a botnet study.

In this talk we’ll see how best to build a measurement botnet: isolating the bot on the participant’s system, deciding when to run, deciding when to ask for human help and how to share achievements with them, and avoiding detection as a bot to improve study validity.

At the end, there will be a discussion about why any of this matters: botnets have always let individuals cooperate to participate in causes they believe in, from fighting COVID-19 with @home, to DDoS as political action, to breaking weak ciphers with distributed.net. That’s true of measurement botnets too. There is little awareness today of actions taken against our interests: botnets can help.

David Sidi