Under manipulations, are some AI models harder to audit?
Abstract
We first prove a constraining result: if a web platform uses models that may fit any data, no audit strategy-whether active or not-can outperform random sampling when estimating properties such as demographic parity. To better understand the conditions under which state-of-the-art auditing techniques may remain competitive, we then relate the manipulability of audits to the capacity of the targeted models, using the Rademacher complexity. We empirically validate these results on popular models of increasing capacities, thus confirming experimentally that large-capacity models, which are commonly used in practice, are particularly hard to audit robustly. These results refine the limits of the auditing problem, and open up enticing questions on the connection between model capacity and the ability of platforms to manipulate audit attempts.
Domains
Computer Science [cs]| Origin | Files produced by the author(s) |
|---|
