When More Metrics Do Not Improve Faithfulness Assessment: A Document-Level Evaluation of AI-Generated News Summaries

Authors

  • Cheng Chen Nanchang Institute of Technology
  • Alina Andreeva Hangzhou We Young Geron Technology Co., Ltd.
  • Ramin Jalalpoor ITMO University
  • Jinjiang Liu St. Petersburg State University
  • Chaofan Yue ITMO University
  • Zhiwei Cheng Hangzhou We Young Geron Technology Co., Ltd.
  • Cuong Huynh ITMO University
  • Zyad Abdullah Al-Shuja ITMO University
  • Xiangyu Li Shanghai Jiao Tong University

DOI:

https://doi.org/10.62177/jaet.v3i4.1811

Keywords:

News Summarization, Factual Consistency, Source Faithfulness, Human Annotation, Metric Fusion, Document-Level Validation

Abstract

Automatic scores are attractive for screening artificial intelligence (AI)-generated news summaries, but their agreement with human judgments must be evaluated without sharing source articles between model fitting and testing. We examined whether a fixed linear fusion of five published scores improves the ranking of human-rated source faithfulness over the published entailment score alone. We reanalysed the Creative Commons licensed XSum hallucination-annotation release, using its existing human judgments and precomputed ROUGE-1, ROUGE-2, ROUGE-L, BERTScore, and entailment scores. Before inspecting outcomes, we fixed mechanical eligibility rules, a deterministic document-level 80/20 split, ridge regression with penalty 1.0, Spearman correlation as the primary endpoint, and 2,000 document-cluster bootstrap draws. The eligible file contained 1,992 system-summary records from 498 BBC article IDs; 399 articles and 1,596 records were used for fitting, and 99 articles and 396 records for testing. On the held-out set, fusion correlated with human faithfulness at ρ = 0.425 (95% bootstrap interval 0.325 to 0.515), versus ρ = 0.445 (0.342 to 0.532) for entailment. The prespecified difference was 0.020 ( 0.100 to 0.066). For the secondary factuality-vote target, correlations were 0.285 and 0.285 after rounding, with no detectable fusion advantage. This reanalysis shows no evidence that this fixed fusion improves ranking over entailment in the released sample. The labels do not assess source attribution or editorial fidelity, and no new human annotation or summary generation was performed.

Downloads

Download data is not yet available.

References

Narayan, S., Cohen, S. B., & Lapata, M. (2018). Don’t Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 1797–1807. https://doi.org/10.18653/v1/D18-1206

Maynez, J., Narayan, S., Bohnet, B., et al. (2020). On Faithfulness and Factuality in Abstractive Summarization. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 1906–1919. https://doi.org/10.18653/v1/2020.acl-main.173

Lin, C.-Y. (2004). ROUGE: A Package for Automatic Evaluation of Summaries. Text Summarization Branches Out, 74–81. https://aclanthology.org/W04-1013/

Zhang, T., Kishore, V., Wu, F., et al. (2020). BERTScore: Evaluating Text Generation with BERT. International Conference on Learning Representations. https://doi.org/10.48550/arXiv.1904.09675

Krys´cin´ski, W., McCann, B., Xiong, C., et al. (2020). Evaluating the Factual Consistency of Abstractive Text Summarization. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, 9332–9346. https://doi.org/10.1 8653/v1/2020.emnlp-main.750

Wang, A., Cho, K., & Lewis, M. (2020). Asking and Answering Questions to Evaluate the Factual Consistency of Summaries. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 5008–5020. https://doi.org/10.18653/v1/2020.acl-main.450

Durmus, E., He, H., & Diab, M. (2020). FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 5055–5070. https://doi.org/10.18653/v1/2020.acl-main.454

Scialom, T., Dray, P.-A., Lamprier, S., et al. (2021). QuestEval: Summarization Asks for Fact-based Evaluation. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 6594–6604. https://doi.org/10.18653/v1/20 21.emnlp-main.529

Laban, P., Schnabel, T., Bennett, P. N., et al. (2022). SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization. Transactions of the Association for Computational Linguistics, 10, 163–177. https://doi.org/10.1162/tacl_a_00453

Pagnoni, A., Balachandran, V., & Tsvetkov, Y. (2021). Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metrics. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 4812–4829. https://doi.org/10.18653/v1/2021.naacl-main. 383

Goyal, T., & Durrett, G. (2021). Annotating and Modeling Fine-grained Factuality in Summarization. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 1449–1462. https://doi.org/10.18653/v1/2021.naacl-main.114

Fabbri, A. R., Krys´cin´ski, W., McCann, B., et al. (2021). SummEval: Re-evaluating Summarization Evaluation. Transactions of the Association for Computational Linguistics, 9, 391–409. https://doi.org/10.1162/tacl_a_00373

Yuan, W., Neubig, G., & Liu, P. (2021). BARTScore: Evaluating Generated Text as Text Generation. Advances in Neural Information Processing Systems, 34, 27263–27277. https://proceedings.neurips.cc/paper/2021/hash/e4d2b6e6fdeca3e60e0f1 a62fee3d9dd-Abstract.html

Honovich, O., Aharoni, R., Herzig, J., et al. (2022). TRUE: Re-evaluating Factual Consistency Evaluation. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 3905–3920. https://doi.org/10.18653/v1/2022.naacl-main.287

Google Research Datasets. (n.d.). XSum hallucination annotations (commit 4abdbb2c807cb93c1df1188a1992d20a55cdcc34) [Data set]. GitHub. Retrieved October 10, 2026, from https://github.com/google-research-datasets/xsum_hallucination_ann otations

Downloads

How to Cite

Chen, C., Andreeva , A., Jalalpoor, R., Liu, J., Yue, C., Cheng, Z., Huynh, C., Al-Shuja, Z. A., & Li, X. (2026). When More Metrics Do Not Improve Faithfulness Assessment: A Document-Level Evaluation of AI-Generated News Summaries. Journal of Advances in Engineering and Technology, 3(4). https://doi.org/10.62177/jaet.v3i4.1811

Issue

Section

Articles

DATE

Received: 2026-10-20
Accepted: 2026-10-26
Published: 2026-11-12