Exploring Embedding Interpretability by Correspondences Between Topic Models and Text Embeddings
DOI:
https://doi.org/10.54195/irrj.23703Keywords:
Embedding Interpretability, Language Model Explanability, Topic ModellingAbstract
Text embeddings have become essential for representing documents in Information Retrieval (IR), yet their high-dimensional nature often limits interpretability. To bridge this gap, we introduce a novel mapping framework that aligns embedding dimensions with topics derived from both probabilistic and neural models. Using three standard collections and three embedding methods, we demonstrate that embedding features consistently map to a subset of coherent topics, even as the total number of topics varies. We further quantify this correspondence with a Mean Mapping Specificity Improvement Rate, showing that mapped topics exhibit significantly higher specificity than the global topic set if the embedding dimensions are set properly. A stability analysis over varying embedding dimensions confirms the stability of the mapping across random feature samples. Our contributions are three-fold: A general-purpose mapping method that visualizes and formalizes correspondences between embedding features and topic representations; Empirical evidence that text embeddings and topic models are not independent descriptors but can mutually validate each other’s semantic structures; A numeric indicator that captures the degree to which embedding features correspond to high-quality topics, providing a new tool for evaluating embedding interpretability and guiding dimensionality reduction choices. These findings suggest that topic-embedding mapping can serve both as a diagnostic for embedding quality and as a means to visualise embedding dimensions more human-interpretable, advancing the practice of collection description in IR.
Downloads
References
D. Achlioptas. Database-friendly random projections: Johnson–Lindenstrauss with binary coins. Jour. of Computer and System Sciences, 66(4):671–687, 2003. doi: 10.1016/S0022-0000(03)00025-4.
E. Bingham and H. Mannila. Random projection in dimensionality reduction: Applications to image and text data. In Proc. ACM SIGKDD Int.Conf. on Knowledge Discovery and Data Mining, pages 245–250, 2001. doi: 10.1145/502512.502546.
D. M. Blei, A. Y. Ng, and M. I. Jordan. Latent Dirichlet allocation. Jour. of Machine Learning Research, 3:993–1022, 2003. doi: 10.1162/jmlr.2003.3.4-5.993.
Y. Bu, M. Li, W. Gu, and W. Huang. Topic diversity: A discipline scheme-free diversity measurement for journals. Jour. of the American Society for Information Science and Technology, 72(5):523–539, 2021. doi: 10.1002/asi.24433.
X. Cheng, X. Yan, Y. Lan, and J. Guo. Btm: Topic modeling over short texts. IEEE Trans.on Knowledge and Data Engineering, 26(12):2928–2941, 2014. doi: 10.1109/TKDE.2014.2313872.
R. Das, M. Zaheer, and C. Dyer. Gaussian LDA for topic models with word embeddings. In Proc. Annual Meeting of the Association for Computational Linguistics and Int. Joint Conf. on Natural Language Processing, pages 795–804, 2015. Association for Computational Linguistics. doi: 10.3115/v1/P15-1077.
S. Deerwester, S. T. Dumais, G. W. Furnas, T. K. Landauer, and R. Harshman. Indexing by latent semantic analysis. Jour. of the American Society for Information Science and Technology, 41(6):391–407, 1990. doi: 10.1002/(SICI)1097-4571(199009)41:6⟨391::AID-ASI1⟩3.0.CO;2-9.
J. Devlin, M. Chang, K. Lee, and K. Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proc. Conf. of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4171–4186, 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1423.
K. Ethayarajh. How contextual are contextualized word representations? Comparing the geometry of BERT, ELMo, and GPT-2 embeddings. In Proc. Conf. on Empirical Methods in Natural Language Processing, pages 55–65, 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-1006.
J. Ganesh, G. Manish, and V. Vasudeva. Doc2sent2vec: A novel two-phase approach for learning document representation. In Proc. ACM-SIGIR Int. Conf. on Research and Development in Information Retrieval, page 809–812, 2016. Association for Computing Machinery. doi: 10.1145/2911451.2914717.
D. Ganguly, D. Roy, M. Mitra, and G. J. F. Jones. Word embedding based generalized language model for information retrieval. In Proc. ACM-SIGIR Int. Conf. on Research and Development in Information Retrieval, SIGIR ’15, page 795–798, 2015. Association for Computing Machinery. doi: 10.1145/2766462.2767780.
N. Gillis and S. A. Vavasis. Fast and robust recursive algorithmsfor separable nonnegative matrix factorization. IEEE Trans.on Pattern Analysis and Machine Intelligence, 36(4): 698–714, 2014. doi: 10.1109/TPAMI.2013.226.
M. Grootendorst. Bertopic: Neural topic modeling with a class-based TF-IDF procedure. ArXiv, abs/2203.05794, 2022. doi: 10.48550/arXiv.2203.05794.
J. Guo, Y. Cai, Y. Fan, F. Sun, R. Zhang, and X. Cheng. Semantic models for the first-stage retrieval: A comprehensive review. ACM Trans.on Information Systems, 40(4), 2022. doi: 10.1145/3486250.
H. Gupta and M. Patel. Method of text summarization using LSA and sentence based topic modelling with BERT. In Int. Conf. on Artificial Intelligence and Smart Systems, pages 511–517, 2021. doi: 10.1109/ICAIS50930.2021.9395976.
S. Han, M. Shin, S. Park, C. Jung, and M. Cha. Unified neural topic model via contrastive learning and term weighting. In Proc. Conf. European Chapter of the Association for Computational Linguistics, pages 1802–1817, 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.eacl-main.132.
H. A. M. Hassan, G. Sansonetti, F. Gasparetti, and A. Micarelli. Semantic-based tag recommendation in scientific bookmarking systems. In Proc. ACM Conf. on Recommender Systems, page 465–469, 2018. Association for Computing Machinery. doi: 10.1145/3240323.3240409.
F. Incitti, F. Urli, and L. Snidaro. Beyond word embeddings: A survey. Information Fusion, 89:418–436, 2023. ISSN 1566-2535. doi: 10.1016/j.inffus.2022.08.024.
A. Jha, V. Rakesh, J. Chandrashekar, A. Samavedhi, and C. K. Reddy. Supervised contrastive learning for interpretable long-form document matching. ACM Trans.on Knowledge Discovery from Data, 17(2), 2023. doi: 10.1145/3542822.
Y. Jin, H. Zhao, M. Liu, L. Du, and W. Buntine. Neural attention-aware hierarchical topic model. In Proc. Conf. on Empirical Methods in Natural Language Processing, pages 1042–1052, 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.emnlp-main.80.
W. B. Johnson and J. Lindenstrauss. Extensions of lipschitz mappings into a hilbert space. In Conf. in Modern Analysis and Probability, volume 26 of Contemporary Mathematics, pages 189–206. American Mathematical Society, 1984. doi: 10.1090/conm/026/737400.
P. Karvelis, D. Gavrilis, G. Georgoulas, and C. Stylios. Topic recommendation using Doc2Vec. In Int. Joint Conf. on Neural Networks, pages 1–6, 2018. IEEE. doi: 10.1109/IJCNN.2018.8489513.
H. K. Kim, H. Kim, and S. Cho. Bag-of-concepts: Comprehending document representation through clustering words in distributed representation. Neurocomputing, 266:336–352, 2017. doi: 10.1016/j.neucom.2017.05.046.
K. Lang. 20 newsgroups dataset, 1995. URL http://people.csail.mit.edu/jrennie/20Newsgroups/.
Q. Le and T. Mikolov. Distributed representations of sentences and documents. In Proc. Int. Conf. on Machine Learning, pages 1188–1196, 2014. JMLR.org.
X. Ma, L. Wang, N. Yang, F. Wei, and J. Lin. Fine-tuning llama for multi-stage text retrieval. In Proc. ACM-SIGIR Int. Conf. on Research and Development in Information Retrieval, SIGIR ’24, page 2421–2425, 2024. Association for Computing Machinery. doi: 10.1145/3626772.3657951.
A. Meddeb and L. B. Romdhane. Using topic modeling and word embedding for topic extraction in Twitter. Procedia Computer Science, 207(C):790–799, 2022. ISSN 1877-0509. doi: 10.1016/j.procs.2022.09.134.
Y. Meng, Y. Zhang, J. Huang, Y. Zhang, and J. Han. Topic discovery via latent space clustering of pretrained language model representations. In Proc. World-Wide Web Conference, page 3143–3152, 2022. Association for Computing Machinery. doi:
1145/3485447.3512034.
T. Mikolov, K. Chen, G. S. Corrado, and J. Dean. Efficient estimation of word representations in vector space. ArXiv, abs/1301.3781, 2013. doi: 10.48550/arXiv.1301.3781.
T. Miller. Explanation in artificial intelligence: Insights from the social sciences. Artificial Intelligence, 267:1–38, 2019. doi: 10.1016/j.artint.2018.07.007.
D. Mimno, H. M. Wallach, E. Talley, M. Leenders, and A. McCallum. Optimizing semantic coherence in topic models. In Proc. Conf. on Empirical Methods in Natural Language Processing, page 262–272, 2011. Association for Computational Linguistics. doi: 10.5555/2145432.2145462.
F. Morstatter and H. Liu. In search of coherence and consensus: Measuring the interpretability of statistical topics. Jour. of Machine Learning Research, 18(1):6177–6208, 2017. ISSN 1532-4435. doi: 10.5555/3122009.3242026.
W. J. Murdoch, C. Singh, K. Kumbier, R. Abbasi-Asl, and B. Yu. Definitions, methods, and applications in interpretable machine learning. Proc. National Academy of Sciences, 116(44):22071–22080, 2019. doi: 10.1073/pnas.1900654116.
B. A. H. Murshed, S. Mallappa, J. Abawajy, M. A. N. Saif, H. D. E. Al-ariki, and H. M. Abdulwahab. Short text topic modelling approaches in the context of big data: taxonomy, survey, and analysis. Artificial Intelligence Review, 56(6):5133–5260, 2022. ISSN 0269-2821. doi: 10.1007/s10462-022-10254-w.
M. Nauta, J. Trienes, S. Pathak, E. Nguyen, M. Peters, Y. Schmitt, Y. Schlötterer, M. van Keulen, and C. Seifert. From anecdotal evidence to quantitative evaluation methods: A systematic review on evaluating explainable AI. ACM Computing Surveys, 55(13s), 2023. ISSN 0360-0300. doi: 10.1145/3583558.
D. Newman, J. H. Lau, K. Grieser, and T. Baldwin. Automatic evaluation of topic coherence. In Annual Conf. North American chapter of the association for computational linguistics, pages 100–108, 2010. Association for Computational Linguistics. doi: 10.5555/1857999.1858011.
A. Panigrahi, H. V. Simhadri, and C. Bhattacharyya. Word2Sense: Sparse interpretable word embeddings. In Proc. Annual Meeting of the Association for Computational Linguistics, pages 5692–5705, 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1570.
J. Pennington, R. Socher, and C. Manning. GloVe: Global vectors for word representation. In Proc. Conf. on Empirical Methods in Natural Language Processing, pages 1532–1543, 2014. Association for Computational Linguistics. doi: 10.3115/v1/D14-1162.
M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer. Deep contextualized word representations. In Annual Conf. North American chapter of the association for computational linguistics, pages 2227–2237, 2018. Association for Computational Linguistics. doi: 10.18653/v1/N18-1202.
T. Prouteau, N. Dugué, N. Camelin, and S. Meignier. Are embedding spaces interpretable? results of an intrusion detection evaluation on a large French corpus. In Proc. Language Resources and Evaluation Conference, pages 4414–4419, 2022. European Language Resources Association.
N. Reimers and I. Gurevych. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proc. Conf. on Empirical Methods in Natural Language Processing and Int. Joint Conf. on Natural Language Processing, pages 3982–3992, 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-1410.
S. Seifollahi, M. Piccardi, and A. Jolfaei. An embedding-based topic model for document classification. ACM Trans.Asian Low-Resources Language Information Processing, 20(3), 2021. doi: 10.1145/3431728.
L. K. Şenel, F. Şahinuç, V. Yücesoy, H. Schütze, T. Çukur, and A. Koç. Learning interpretable word embeddings via bidirectional alignment of dimensions with semantic concepts. Information Processing & Management, 59(3):102925, 2022. doi:
1016/j.ipm.2022.102925.
S. Sia, A. Dalmia, and S. J. Mielke. Tired of topic models? clusters of pretrained word embeddings make for fast and good topics too! In Proc. Conf. on Empirical Methods in Natural Language Processing, pages 1728–1736, 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.emnlp-main.135.
C. Singh, A. Askari, R. Caruana, and J. Gao. Augmenting interpretable models with large language models during training. Nature Communications, 14(1):7913, 2023. doi: 10.1038/s41467-023-43713-1.
J. Steinberger and K. Ježek. Text summarization and singular value decomposition. In Tatyana Yakhno, editor, Advances in Information Systems, pages 245–254, 2005. Springer Berlin Heidelberg.
A. Subramanian, D. Pruthi, H. Jhamtani, T. Berg-Kirkpatrick, and E. H. Hovy. Spine: Sparse interpretable neural embeddings. ArXiv, abs/1711.08792, 2017. doi: 10.48550/arXiv.1711.08792.
S. Tanabe, M. Ohta, A. Takasu, and J. Adachi. An approach to estimating cited sentences in academic papers using Doc2Vec. In Proc. Int. Conf. on Management of Digital EcoSystems, page 118–125, New York, NY, USA, 2018. Association for Computing Machinery. ISBN 9781450356220. doi: 10.1145/3281375.3281391.
J. Tao, L. Zhou, and K. Hickey. Making sense of the black-boxes: Toward interpretable text classification using deep learning models. Jour. of the American Society for Information Science and Technology, 74(6):685–700, 2023. doi: 10.1002/asi.24642.
F. Viegas, S. Canuto, C. Gomes, W. Luiz, T. Rosa, S. Ribas, L. Rocha, and M. A. Gonçalves. Cluwords: exploiting semantic word clustering representation for enhanced topic modeling. In Proc. ACM Int. Conf. on Web Search and Data Mining, pages 753–761, 2019. Association for Computing Machinery. doi: 10.1145/3289600.3291032.
E. M. Voorhees and D. K. Harman, editors. TREC Experiment and Evaluation in Information Retrieval. The MIT Press, 2005. ISBN 9780262220736.
I. Vulić and M. Moens. Monolingual and cross-lingual information retrieval models based on (bilingual) word embeddings. In Proc. ACM-SIGIR Int. Conf. on Research and Development in Information Retrieval, page 363–372, 2015. Association for Computing Machinery. doi: 10.1145/2766462.2767752.
Y. Wang and Y. Zhang. Nonnegative matrix factorization: A comprehensive review. IEEE Trans.on Knowledge and Data Engineering, 25(6):1336–1353, 2012. doi: 10.1109/TKDE.2012.51.
C. Wu, E. Kanoulas, and M. Rijke. Learning entity-centric document representations using an entity facet topic model. Information Processing & Management, 57(3):102216, 2020. ISSN 0306-4573. doi: 10.1016/j.ipm.2020.102216.
X. Yang, D. Lo, X. Xia, L. Bao, and J. Sun. Combining word embedding with information retrieval to recommend similar bug reports. In IEEE Int. Symposium on Software Reliability Engineering, pages 127–137, New York, NY, United States, 2016. IEEE. doi: 10.1109/ISSRE.2016.33.
M. Yuan, P. Lin, L. Rashidi, and J. Zobel. Asessment of the quality of topic models for information retrieval applications. In Proc. ACM-SIGIR Int. Conf. on Theory of Information Retrieval, ICTIR ’23, 2023. Association for Computing Machinery. doi:
1145/3578337.3605118.
J. Zhan, J. Mao, Y. Liu, M. Zhang, and S. Ma. Repbert: Contextualized text embeddings for first-stage retrieval. ArXiv, abs/2006.15498, 2020. doi: 10.48550/arXiv.2006.15498.
T. Zheng and M. Wang. Using SVD for topic modeling. Jour.of the American Statistical Association, 0(0):1–16, 2022. doi: 10.1080/01621459.2022.2123813.
G. Zuccon, B. Koopman, P. Bruza, and L. Azzopardi. Integrating and evaluating neural word embeddings in information retrieval. In Proc. Australian Document Computing Conf., 2015. Association for Computing Machinery. doi: 10.1145/2838931.2838936.
Y. Zuo, C. Li, H. Lin, and J. Wu. Topic modeling of short texts: A pseudo-document view with word embedding enhancement. IEEE Trans.on Knowledge & Data Engineering, 35(01):972–985, 2023. ISSN 1558-2191. doi: 10.1109/TKDE.2021.3073195.
Downloads
Published
Issue
Section
License
Copyright (c) 2025 Meng Yuan, Lida Rashidi, Justin Zobel (Author)
This work is licensed under a Creative Commons Attribution 4.0 International License.
CC-BY 4.0
