Interpretable Multimodal Classification with Linear Discriminant Tree Ensembles
06:00 · August 24, 2026 · arXiv cs.AI RSS
Multimodal affect and behaviour classifiers that fuse heterogeneous text, audio, and visual streams must simultaneously achieve competitive accuracy and produce human-understandable explanations of the cues driving their decisions -- a dual objective that current high-capacity models, notably Transformers, only partially address. While Transformers attain strong predictive performance, their distributed representations and deep nonlinearity make it difficult to assign meaningful importance weights to individual multimodal features, limiting their use in trust-sensitive applications such as clinical affect monitoring and educational assessment. We address this gap by developing a framework based on tree-based ensembles that balances accuracy and interpretability. The framework encodes each modality into tokens, extracts and clusters concepts to reduce dimensionality, routes the fused modalities through tree-based ensemble classifiers, and interprets trends using a novel modified feature importance metric. The modified importance reduces the influence of the negative class in binary classification tasks, thereby improving indicator or marker detection. The proposed tree-based ensembles -- Linear Discriminant Tree (LDT), Linear Discriminant Forest (LDF), and Linear Discriminant AdaBoost (LDAB) -- achieve F1-mod gains of 4.3\% over the Multimodal Transformer and accuracy gains of 3.0\% over the primary interpretable multimodal baseline, Interpretable Multimodal Routing (IMR). The proposed multimodal feature importance extracts salient inter-modal concepts with substantially higher human-annotator agreement scores than default feature importance (62.2\% vs.\ 43.2\% on IEMOCAP; 46.7\% vs.\ 32.1\% on CMU-MOSI).
Summary
A new framework addresses the tension between predictive performance and human-understandable explanations in multimodal affect and behaviour classification. It fuses text, audio and visual streams through a pipeline that first encodes each modality into tokens, then extracts and clusters concepts to control dimensionality before feeding the fused representation to tree-based ensembles. The ensembles—Linear Discriminant Tree (LDT), Linear Discriminant Forest (LDF) and Linear Discriminant AdaBoost (LDAB)—insert a Linear Discriminant Analysis projection at every node, providing an oblique split criterion that remains interpretable while improving separation of affective or behavioural classes.
A central element is a modified feature-importance measure that down-weights splits dominated by the negative class. This adjustment yields indicators more aligned with positive-class markers, raising human-annotator agreement from 43.2 % to 62.2 % on IEMOCAP and from 32.1 % to 46.7 % on CMU-MOSI. The same metric supports both global trend analysis and local, sample-level explanations, capabilities that attention-based models rarely deliver without additional post-hoc machinery.
Evaluated on IEMOCAP, CMU-MOSI and a custom Multimodal Mathematics dataset, the ensembles report a 4.3 % gain in the positive-class-oriented F1-mod metric over the Multimodal Transformer and a 3.0 % accuracy improvement over the Interpretable Multimodal Routing baseline. Ablation results indicate that the choice of context-clustering scheme—K-means, eigenvalue-based hierarchical or multi-sense—materially affects both accuracy and the clarity of cross-modal concept associations. The approach therefore supplies competitive accuracy together with transparent, dataset-wide explanations suited to trust-sensitive domains such as clinical monitoring and educational assessment.
Why it matters
This research is highly relevant for Dutch AI researchers and practitioners focusing on ethical and transparent AI, a key priority under the EU AI Act. The proposed interpretable tree ensembles offer a viable, explainable alternative to black-box Transformers for trust-sensitive applications like clinical monitoring and educational assessment.
