NIST’s revised report on mitigating bias in AI emphasizes the importance of societal factors as well as technical ones.
On March 16, 2022, NIST released a report on AI bias, “Towards a Standard for Identifying and Managing Bias in Artificial Intelligence.” This report is a revised version of the draft released with a request for comments in June 2021. It is related to NIST’s development of a voluntary AI Risk Management Framework.
The press release characterizes the revisions to the report as focusing on the broader societal context in which machine learning systems are developed as a source of bias in addition to the sources NIST described in the initial report, the technical processes and data.
“Organizations often default to overly technical solutions for AI bias issues. But these approaches do not adequately capture the societal impact of AI systems,” Reva Schwartz, principal investigator for AI bias at NIST and one of the report’s authors, said in a statement. “The expansion of AI into many aspects of public life requires extending our view to consider AI within the larger social system in which it operates.”
The revised version of the report addresses public comments submitted on the original draft. The most prominent theme in these comments is the original draft’s lack of consideration for societal context when enumerating ways to mitigate bias in AI.
“If the proposal or resulting standards could be read to endorse predominantly technical solutions, that may have the perverse effect of doing practical harm, rather than good, by enabling vendors, developers, and users of algorithmic decision-making tools to remain relatively ignorant of the contextual landscapes in which these tools will be used,” wrote Cynthia Khoo, an Associate at the Center on Privacy and Technology at Georgetown Law, in response to the draft report.
The ACLU wrote in a statement in response to the draft version of the report, “Overall, the publication reflects an overly tech-determinist approach to mitigating bias in AI. While technical characteristics of AI must be addressed, numerous non-technical factors have contributed to the current need to improve trust in AI.”
Other themes of submitted public comments included the draft report’s lack of engagement with the research question of whether AI models should be developed at all in a given context, the lack of emphasis on transparency as a way to address bias, and a misplaced reliance on audits rather than advocating building public trust in AI systems.
The revised version of the report is much more comprehensive than the draft in its overview of the areas that present challenges for addressing bias in AI. In the report, the authors present three categories of AI bias: systemic bias, human bias, and statistical/computational bias. “Understanding AI as a socio-technical system acknowledges that the processes used to develop technology are more than their mathematical and computational constructs,” the report states.
Accordingly, within each of the categories of bias, the report describes areas that present challenges for addressing AI bias: datasets; processes and human factors; and testing and evaluation, validation and verification (TEVV).
The report then delves into bias-related challenges for each of these areas. For example, in the datasets category, availability bias crops up when AI practitioners default to using data that is already available regardless of whether it is best suited for the AI application at hand.
The report cites several challenges for addressing bias in the area of processes and human factors. An AI model may be deployed in a different context than the one for which it was intended to be used. In addition, there is an overreliance on human-in-the-loop processes to prevent adverse model effects based on the misguided belief that a human overseer can provide adequate oversight simply by virtue of being human.
In the TEVV category, one challenge is in validating the performance of an AI model when the ground truth is hazy or labels are noisy. Another challenge for validating models arises from the perception that “AI is magic” and the ways that AI model development deviates from the scientific method in light of models’ lack of interpretability and reproducibility.
In addition to summarizing areas that cause difficulty in addressing AI bias, the report presents guidance for how AI practitioners may mitigate it in these areas. Regarding human factors, the authors advocate for researchers to create impact assessments that state the potential societal risks of the models they submit for publication. The authors also emphasize engaging a variety of stakeholders in the model development process; prioritizing diversity, equity, and inclusion in AI; transparency around systems and procedures, and keeping humans at the center of AI design.
The report also describes substantial guidance for the governance of AI systems.
The response to the final draft published in March seemed largely positive. “Emphasizes that bias from data is just the tip of the iceberg. We have to consider the role of human and systemic institutional and societal biases as well,” tweeted Rebekah Tromble, Director at Institute for Data, Democracy & Politics on March 16.
“This is a huge step. NIST, a US government standards-setting agency, has massive influence across the global AI industry,” Karen Hao, Senior AI Editor at MIT Technology Review, wrote in a tweet on March 18.
NIST will hold a workshop March 29-31 as part of its effort to draft a technical report for addressing AI bias connected with its Risk Management Framework for AI.
Recent posts