When More Metrics Do Not Improve Faithfulness Assessment: A Document-Level Evaluation of AI-Generated News Summaries
DOI:
https://doi.org/10.62177/jaet.v3i4.1811Keywords:
News Summarization, Factual Consistency, Source Faithfulness, Human Annotation, Metric Fusion, Document-Level ValidationAbstract
Automatic scores are attractive for screening artificial intelligence (AI)-generated news summaries, but their agreement with human judgments must be evaluated without sharing source articles between model fitting and testing. We examined whether a fixed linear fusion of five published scores improves the ranking of human-rated source faithfulness over the published entailment score alone. We reanalysed the Creative Commons licensed XSum hallucination-annotation release, using its existing human judgments and precomputed ROUGE-1, ROUGE-2, ROUGE-L, BERTScore, and entailment scores. Before inspecting outcomes, we fixed mechanical eligibility rules, a deterministic document-level 80/20 split, ridge regression with penalty 1.0, Spearman correlation as the primary endpoint, and 2,000 document-cluster bootstrap draws. The eligible file contained 1,992 system-summary records from 498 BBC article IDs; 399 articles and 1,596 records were used for fitting, and 99 articles and 396 records for testing. On the held-out set, fusion correlated with human faithfulness at ρ = 0.425 (95% bootstrap interval 0.325 to 0.515), versus ρ = 0.445 (0.342 to 0.532) for entailment. The prespecified difference was 0.020 ( 0.100 to 0.066). For the secondary factuality-vote target, correlations were 0.285 and 0.285 after rounding, with no detectable fusion advantage. This reanalysis shows no evidence that this fixed fusion improves ranking over entailment in the released sample. The labels do not assess source attribution or editorial fidelity, and no new human annotation or summary generation was performed.
Downloads
References
Narayan, S., Cohen, S. B., & Lapata, M. (2018). Don’t Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 1797–1807. https://doi.org/10.18653/v1/D18-1206
Maynez, J., Narayan, S., Bohnet, B., et al. (2020). On Faithfulness and Factuality in Abstractive Summarization. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 1906–1919. https://doi.org/10.18653/v1/2020.acl-main.173
Lin, C.-Y. (2004). ROUGE: A Package for Automatic Evaluation of Summaries. Text Summarization Branches Out, 74–81. https://aclanthology.org/W04-1013/
Zhang, T., Kishore, V., Wu, F., et al. (2020). BERTScore: Evaluating Text Generation with BERT. International Conference on Learning Representations. https://doi.org/10.48550/arXiv.1904.09675
Krys´cin´ski, W., McCann, B., Xiong, C., et al. (2020). Evaluating the Factual Consistency of Abstractive Text Summarization. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, 9332–9346. https://doi.org/10.1 8653/v1/2020.emnlp-main.750
Wang, A., Cho, K., & Lewis, M. (2020). Asking and Answering Questions to Evaluate the Factual Consistency of Summaries. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 5008–5020. https://doi.org/10.18653/v1/2020.acl-main.450
Durmus, E., He, H., & Diab, M. (2020). FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 5055–5070. https://doi.org/10.18653/v1/2020.acl-main.454
Scialom, T., Dray, P.-A., Lamprier, S., et al. (2021). QuestEval: Summarization Asks for Fact-based Evaluation. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 6594–6604. https://doi.org/10.18653/v1/20 21.emnlp-main.529
Laban, P., Schnabel, T., Bennett, P. N., et al. (2022). SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization. Transactions of the Association for Computational Linguistics, 10, 163–177. https://doi.org/10.1162/tacl_a_00453
Pagnoni, A., Balachandran, V., & Tsvetkov, Y. (2021). Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metrics. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 4812–4829. https://doi.org/10.18653/v1/2021.naacl-main. 383
Goyal, T., & Durrett, G. (2021). Annotating and Modeling Fine-grained Factuality in Summarization. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 1449–1462. https://doi.org/10.18653/v1/2021.naacl-main.114
Fabbri, A. R., Krys´cin´ski, W., McCann, B., et al. (2021). SummEval: Re-evaluating Summarization Evaluation. Transactions of the Association for Computational Linguistics, 9, 391–409. https://doi.org/10.1162/tacl_a_00373
Yuan, W., Neubig, G., & Liu, P. (2021). BARTScore: Evaluating Generated Text as Text Generation. Advances in Neural Information Processing Systems, 34, 27263–27277. https://proceedings.neurips.cc/paper/2021/hash/e4d2b6e6fdeca3e60e0f1 a62fee3d9dd-Abstract.html
Honovich, O., Aharoni, R., Herzig, J., et al. (2022). TRUE: Re-evaluating Factual Consistency Evaluation. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 3905–3920. https://doi.org/10.18653/v1/2022.naacl-main.287
Google Research Datasets. (n.d.). XSum hallucination annotations (commit 4abdbb2c807cb93c1df1188a1992d20a55cdcc34) [Data set]. GitHub. Retrieved October 10, 2026, from https://github.com/google-research-datasets/xsum_hallucination_ann otations
Downloads
How to Cite
Issue
Section
License
Copyright (c) 2026 Cheng Chen, Alina Andreeva , Ramin Jalalpoor, Jinjiang Liu, Chaofan Yue, Zhiwei Cheng, Cuong Huynh, Zyad Abdullah Al-Shuja, Xiangyu Li

This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
DATE
Accepted: 2026-10-26
Published: 2026-11-12







