With the widespread adoption of AI systems, it’s crucial to explain their decision-making processes, as they can have serious consequences in high-stakes applications, like those in health care or law enforcement.
But what should these explanations be, and what important concepts should they include? And how might one measure the importance of a particular concept?
A Johns Hopkins team has put forward a precise, actionable proposal to produce appropriate AI explanations, as well as rigorous methodology to test a model for its notions of importance by “betting” on whether it “uses” a concept or not in its predictions. With the team’s framework, interested users can now test how important a specific concept is in any given input for a machine learning model’s output.
Jacopo Teneggi, a doctoral student in the Department of Computer Science and an affiliate of the Johns Hopkins Mathematical Institute for Data Science, and his advisor Jeremias Sulam, an assistant professor of biomedical engineering and faculty at MINDS, are presenting the work at the Thirty-Eighth Annual Conference on Neural Information Processing Systems this week in Vancouver, Canada.
Given the complexity of modern machine learning models, it can be difficult to understand which aspects of their input, or the information they’re working with, are important for the results they generate. This becomes even more complicated when you want to measure an abstract concept, like the “cuteness” of a pet picture or the “ruggedness” of a material in an image.
“We propose a formal notion to measure the importance of these types of concepts for modern AI systems’ responses,” Teneggi says.
Supported by Sulam’s NSF CAREER Award, the researchers investigated the extent to which a model’s output depends on specific concepts. To do so, they defined precise notions of variable (or input) importance based on conditional hypothesis testing, but for abstract concepts, allowing them to answer questions like, “Is the size of this object important for the model’s prediction given that the object has wheels, four doors, and is sitting on the road?”
Example results of the proposed method for a popular vision-language model (CLIP:ViT-L/14) on images from a bird classification dataset.
They devised algorithms that could test for this type of hypothesis based on recent sequential testing-by-betting tools, providing answers to questions like the above with associated importance rankings. This allowed the researchers to reject (or fail to reject) a null hypothesis—in this case, the claim that a particular concept is not important—at a given error-tolerance level by “betting” against the claim and rejecting it once enough evidence is accumulated, thus declaring a concept important.
“Our framework gives a precise statistical meaning that allows us to make specific conclusions like, ‘There is no more than a 5% chance that this concept is not important when we are, in fact, reporting it as important,'” says Teneggi. “This level of statistical guarantee is crucial for the rigorous use of these importance metrics and is not present in other prior methodologies.”
The researchers tested the effectiveness and flexibility of their betting framework on synthetic datasets and image classification tasks using vision-language models. They found that even though certain concepts are abstract notions or human constructs, modern machine learning networks can accurately capture useful representations of said concepts.
“By leveraging the representations that these models learn, we can then devise computational procedures to test for precise notions of importance,” says Teneggi.
The researchers are eager to extend their ideas to settings where understanding a model’s predictions is particularly important, such as in medical imaging. They are currently working on a version of these importance measures that will align with the details most valued by clinicians for specific diagnoses.
In the meantime, the team has released an online demo where users can upload their own images—even your own cute pet pics—and concepts of interest to test for their importance.
“We believe this framework can provide formal notions of importance for modern artificial intelligence that can help inform future regulations and decision-making processes,” says Teneggi.