Trustworthiness and Medical Usefulness of Explainability Techniques in ML-Supported Depression Screening within Primary Care
Guilherme Gryschek, Luis Quintero, Alejandro Kuratomi (Stockholm University)
Abstract:
Depression remains a prevalent condition in primary care due to its heterogeneous symptoms. To support early screening, this study presents DepreScan, an interactive web-based clinical decision support system powered by machine learning and explainable artificial intelligence. Trained on a representative health survey dataset, DepreScan uses interpretable models and multiple explanation techniques —including SHAP plots and simplified decision trees— to assist healthcare practitioners in screening depression risk. A mixed-methods user study with 16 clinicians assessed the system’s trustworthiness, usability, and perceived clinical utility. Results indicate moderate to high acceptance and trust, particularly for SHAP-based global feature explanations, with differences in measured trust between two related questionnaires. The study highlights the importance of aligning ML explainability with healthcare professionals' mental models and the findings inform the design of future user-centered XAI tools for mental health decision support in primary care.
Counterfactual Explanations for Time Series Should be Human-Centered and Temporally Coherent in Interventions
Emmanuel Chukwu, Rianne Schouten, Monique Tabak, Mykola Pechenizkiy (Eindhoven University of Technology)
Abstract:
Counterfactual explanations are increasingly proposed as interpretable mechanisms to achieve algorithmic recourse. However, current counterfactual techniques for time series classification are predominantly designed with static data assumptions and focus on generating minimal input perturbations to flip model predictions. This position paper argues that such approaches are fundamentally insufficient in clinical recommendation settings, where interventions unfold over time and must be causally plausible and temporally coherent. We advocate for a shift towards counterfactuals that reflect sustained, goal-directed interventions aligned with clinical reasoning and patient-specific dynamics. We identify critical gaps in existing methods that limit their practical applicability, specifically, temporal blind spots and the lack of user-centered considerations in both method design and evaluation metrics. To support our position, we conduct a robustness analysis of several state-of-the-art methods for time series and show that the generated counterfactuals are highly sensitive to stochastic noise. This finding highlights their limited reliability in real-world clinical settings, where minor measurement variations are inevitable. We conclude by calling for methods and evaluation frameworks that go beyond prediction flips. We emphasize the need for actionable, purpose-driven interventions that are feasible in real-world contexts for the users of such applications.
Data visualization and clustering to judge data quality in interactive sales potential estimation
Lukas Bader (August Rüggeberg GmbH), Dietlind Zühlke (TH Köln) , Ina Terwey-Scheulen (August Rüggeberg GmbH)
Abstract:
This paper investigates the implementation and selection of external data to forecast the volumes of tools used in the metalworking industry in US regional markets. Due to incomplete and noisy internal sales data caused by partial market penetration and gaps in external economic data, traditional time-series forecasting models fall short. To address these challenges, we apply interactive domain-specific data enrichment and feature selection techniques, followed by dimensionality reduction and clustering techniques to group counties and visualize their characteristics based on labor market data. This human-centered approach is crucial given the lack of a clear ground truth for sales potential, so a strong focus is on interpretability and expert collaboration. This method uncovers high-potential sales regions and enhances the feature selection and sampling process, making it ideal for following forecast methods. It is designed to be interpretable and explainable, allowing clear comparison with companies' internal sales data and enabling interactive evaluation with domain experts, directly addressing the need for human-in-the-loop Machine Learning and interactive data visualization in real-world applications.
Interpretable Sample Selection with Exceptional Model Mining
Vincent P. Hoogendam, Kalina R. Bakardzhieva, Luca Mainardi, Luyang Xie, Rianne Schouten (Eindhoven University of Technology)
Abstract:
Active Learning (AL) aims to improve model performance while minimizing labeling effort by selecting informative samples into the training dataset. The importance is particularly clear when humans are involved in the data collection and labeling process, and even more so in the context of human-computer interaction (HCI) where users can provide only few training samples and those samples should be as useful as possible. Traditionally, AL methods iteratively increase the training data set by selecting informative samples from an unlabeled dataset. However, AL methods may be prone to oversampling certain regions of the data space, potentially introducing redundancy in the dataset. To address this issue, we propose an interpretable sample selection approach based on a local pattern mining framework called Exceptional Model Mining (EMM). EMM aims to discover subgroups in the dataset that somehow behave exceptionally. These subgroups are described using an interpretable language of conjunctions of attribute-value pairs. We propose an EMM-based AL approach that discovers, describes and selects informative samples. As such, our method provides an explanation as to why model performance is reduced for certain samples. In addition, our method is an alternative to existing AL methods: given a diverse subgroup set, we create a diverse and representative training set by selecting samples from each subgroup. We evaluate the performance of our approach against a traditional AL baseline and demonstrate that it provides interpretable explanations and has good classification power, especially when labeling budget is low.
Towards the Integration of Interactive Explainable User Interfaces (XUI) into Clinical Practice for Sepsis Mortality Risk Screening
Kent Fredriksdotter (Karolinska Institutet), Alejandro Kuratomi, Lena Mondrejevski, Luis Quintero (Stockholm University)
Abstract:
Machine learning (ML) holds promise for improving clinical decision-making, but its adoption in healthcare remains limited due to challenges in explainability and workflow integration. This paper presents a methodology to guide the development of Explainable User Interfaces (XUIs) for clinical applications combining Design Science Research (DSR) with the frameworks CRISP-ML and FUTURE-AI. We instantiate this methodology in a use case on sepsis risk prediction using the MIMIC-III dataset. Our pipeline includes model training, integration of SHAP explanations, and the design of interactive web-based XUIs with explanatory and exploratory components. These interfaces were developed through iterative prototyping, incorporating principles from ML development, human-computer interaction, and early clinical feedback. The resulting system is a replicable methodology to align ML development with clinical needs, and highlights XUIs as a crucial bridge for making ML tools trustworthy, interpretable, and actionable in high-stakes decision-making environments.
InfoClus: Informative Clustering of High-dimensional Data Embeddings
Fuyin Lai, Edith Heiter, Guillaume Bied, Jefrey Lijffijt (Ghent University)
Abstract:
Developing an understanding of high-dimensional data can be facilitated by visualizing that data using dimensionality reduction. However, the low-dimensional embeddings are often difficult to interpret. To facilitate the exploration and interpretation of low-dimensional embeddings, we introduce a new concept named partitioning with explanations. The idea is to partition the data shown through the embedding into groups, each of which is given a sparse explanation using the original high-dimensional attributes. We introduce an objective function that quantifies how much we can learn through observing the explanations of the data partitioning, using information theory, and also how complex the explanations are. Through parameterization of the complexity, we can tune the solutions towards the desired granularity. We propose InfoClus, which optimizes the partitioning and explanations jointly, through greedy search constrained over a hierarchical clustering. We conduct a qualitative and quantitative analysis of InfoClus on three data sets. We contrast the results on the Cytometry data with published manual analysis results, and compare with two other recent methods for explaining embeddings (RVX and VERA). These comparisons highlight that InfoClus has distinct advantages over existing procedures and methods. We find that InfoClus can automatically create good starting points for the analysis of dimensionality-reduction-based scatter plots.