Mutation-RAG：使用突變衍生知識進行檢索增強測試生成

簡易檢索 / 詳目顯示

回結果列表

研究生：	李昕融 Lee, Sin-Rong
論文名稱：	Mutation-RAG：使用突變衍生知識進行檢索增強測試生成 Mutation-RAG: Test Generation Using Mutation-Derived Knowledge Artifacts
指導教授：	蔡子傑
口試委員:	周承復吳曉光陳伶志
學位類別：	碩士 Master
系所名稱：	資訊學院 - 資訊科學系 Department of Computer Science
論文出版年：	2026
畢業學年度：	114
語文別：	中文
論文頁數：	51
中文關鍵詞：	突變測試、大型語言模型、單元測試、檢索增強生成
外文關鍵詞：	Mutation Testing, LLM, Unit Testing, Retrieval-Augmented Generation
相關次數：	點閱：78 下載：0
分享至:	分享至facebook 分享至twitter

查詢本校圖書館目錄查詢臺灣博碩士論文知識加值系統勘誤回報

本論文以突變測試為核心評估觀點，研究大型語言模型在單元測試生成中的實際效果。相較於只檢查生成測試是否語法正確、是否可執行，或是否提升程式覆蓋率，本研究更關注生成測試是否能揭露原始函式與未被既有測試殺死的突變體之間的行為差異。
本研究實作一套名為 mutgen 的系統。針對真實 Python pytest 專案，mutgen 以 mutmut 蒐集突變測試結果，建立已殺死 mutant 的測試案例語料，並針對每一筆指定 prompt mutant，在原始程式與突變版本上重放生成測試，以判斷測試是否能在原始程式上通過、並在突變版本上失敗。實驗比較兩種生成策略：單階段 mutant-aware generation 直接根據目標函式與 survived mutant 產生測試；兩階段 retrieval-augmented generation 則在第一輪生成失敗後，引入檢索到的 killed mutant-test examples 作為額外參考，並產生第二個測試。透過此設定，本論文分析檢索案例是否能協助大型語言模型形成更強的測試 oracle，並檢驗檢索上下文能否真正轉化為突變測試下的錯誤揭露能力。

This thesis studies LLM-based unit test generation through mutationoriented evaluation. Instead of measuring generated tests only by syntactic validity or coverage, the study asks whether a generated test can expose the behavioral difference between an original function and a survived mutant.
The thesis implements a system named mutgen. Given real Python pytest projects, mutgen collects mutation records with mutmut, builds a killed mutant-test example corpus, and evaluates generated tests by replaying them on the original and mutant versions. The study compares a single-stage mutant-aware generator with a two-stage retrieval-augmented generation process that invokes retrieval only after the first attempt fails. Through this setting, the thesis examines whether retrieval examples can help LLMs produce stronger test oracles for mutation-based test generation.

致謝 ......... i
摘要 ............. ii
Abstract ........... iii
目次 ............. iv
圖次 ............. vii
表次 .............viii

第一章　緒論 ............... 1
　1.1　研究背景 ........... 1
　1.2　研究動機 ............. 2
　1.3　研究目的 ............... 4

第二章　文獻探討 ............... 6
　2.1　單元測試（Unit Testing） .............. 6
　2.2　測試評估標準（Test Adequacy Criteria） ...... 6
　2.3　突變測試（Mutation Testing） ...................... 7
　2.4　大型語言模型在測試生成中的應用 ............... 9
　2.5　檢索增強生成（Retrieval-Augmented Generation, RAG）..... 10

第三章　研究方法 .......... 13
　3.1　研究設計與研究問題 ............ 13
　　3.1.1　Prompt-mutant case 的定義 .......... 14
　　3.1.2　成對比較設計 ............ 14
　　3.1.3　研究問題 ........... 14
　　3.1.4　研究變數與判讀原則 ............ 15
　3.2　方法架構與比較組 ......... 15
　3.3　離線階段：mutation records 與檢索語料建置 ....... 16
　3.4　線上階段：兩階段生成流程 ............ 17
　　3.4.1　第一階段：mutant-aware generation ......... 17
　　3.4.2　第二階段：retrieval-augmented generation ......... 18
　3.5　評估流程 .......... 18

第四章　實驗資料與系統實作 ........... 21
　4.1　受測專案 ................. 21
　4.2　Case 建立方式 .............. 23
　4.3　檢索語料來源 .............. 24
　4.4　系統實作 ................... 25
　　4.4.1　系統整體架構 .............. 25
　　4.4.2　離線資料處理 .............. 26
　　4.4.3　G1 與 G2 生成流程 ...........28
　　4.4.4　指定 mutant 重放評估實作 ............ 30
　　4.4.5　輸出紀錄與分析支援 ............ 31

第五章　實驗結果與分析 ................ 32
　5.1　實驗資料 .................... 32
　5.2　比較組 .................. 33
　5.3　評估與統計方式 .................. 33
　5.4　主要結果 .................. 34
　5.5　Coverage 與 mutation-based 評估 ............. 35
　5.6　分專案觀察 ............ 37
　5.7　檢索品質分析 .............. 37
　5.8　研究問題回應與討論 ............. 39
　　5.8.1　RQ1：單階段 mutant-aware generation 的效果 ........ 39
　　5.8.2　RQ2：第二階段 retrieval 是否帶來補強效果 ............. 39
　　5.8.3　RQ3：G2 成功與失敗的原因 .............. 40
　　5.8.4　對方法設計的啟示 .............. 41

第六章　結論與未來工作 .............. 43
　6.1　研究總結 ................ 43
　6.2　研究限制與未來工作 ............ 44

參考文獻 .......... 47

[1] Barry Boehm and Victor R. Basili. Software defect reduction top 10 list. Computer, 34(1):135–137, 2001. https://doi.org/10.1109/2.962984.
[2] Laura Inozemtseva and Reid Holmes. Coverage is not strongly correlated with test suite effectiveness. In Proceedings of the 36th International Conference on Software Engineering, ICSE 2014, pages 435–445. ACM, 2014. https://doi.org/10.1145/2568225.2568271.
[3] Akbar Siami Namin and James H. Andrews. The influence of size and coverage on test suite effectiveness. In Proceedings of the Eighteenth International Symposium on Software Testing and Analysis, ISSTA 2009, pages 57–68. ACM, 2009. https://doi.org/10.1145/1572272.1572280.
[4] Hadi Hemmati. How effective are code coverage criteria? In 2015 IEEE International Conference on Software Quality, Reliability and Security, QRS 2015, pages 151–156. IEEE, 2015. https://doi.org/10.1109/QRS.2015.30.
[5] Earl T. Barr, Mark Harman, Phil McMinn, Muzammil Shahbaz, and Shin Yoo. The oracle problem in software testing: A survey. IEEE Transactions on Software Engineering, 41(5):507–525, 2015. https://doi.org/10.1109/TSE.2014.2372785.
[6] Yinghao Chen, Zehao Hu, Chen Zhi, Junxiao Han, Shuiguang Deng, and Jianwei Yin. ChatUniTest: A framework for LLM-based test generation. In Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering, FSE Companion 2024, pages 572–576. ACM, 2024. https://doi.org/10.1145/3663529.3663801.
[7] Max Schäfer, Sarah Nadi, Aryaz Eghbali, and Frank Tip. An empirical evaluation of using large language models for automated unit test generation. IEEE Transactions on Software Engineering, 50(1):85–105, 2024. https://doi.org/10.1109/TSE.2023.3334955.
[8] Nadia Alshahwan, Jubin Chheda, Anastasia Finogenova, Beliz Gokkaya, Mark Harman, Inna Harper, Alexandru Marginean, Shubho Sengupta, and Eddy Wang. Automated unit test improvement using large language models at Meta. In Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering, FSE Companion 2024, pages 185–196. ACM, 2024. https://doi.org/10.1145/3663529.3663839.
[9] Lin Yang, Chen Yang, Shutao Gao, Weijing Wang, Bo Wang, Qihao Zhu, Xiao Chu, Jianyi Zhou, Guangtai Liang, Qianxiang Wang, and Junjie Chen. On the evaluation of large language models in unit test generation. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, ASE 2024, pages 1607–1619. ACM, 2024. https://doi.org/10.1145/3691620.3695529.
[10] René Just, Darioush Jalali, Laura Inozemtseva, Michael D. Ernst, Reid Holmes, and Gordon Fraser. Are mutants a valid substitute for real faults in software testing? In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, FSE 2014, pages 654–665. ACM, 2014. https://doi.org/10.1145/2635868.2635929.
[11] Mike Papadakis, Donghwan Shin, Shin Yoo, and Doo-Hwan Bae. Are mutation scores correlated with real fault detection? A large-scale empirical study on the relationship between mutants and real faults. In Proceedings of the 40th International Conference on Software Engineering, ICSE 2018, pages 537–548. ACM, 2018. https://doi.org/10.1145/3180155.3180183.
[12] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Advances in Neural Information Processing Systems, volume 33, NeurIPS 2020, pages 9459–9474, 2020. https://proceedings.neurips.cc/paper/2020/hash/6b493230205f780e1bc26945df7481e5-Abstract.html.
[13] Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. Retrieval-augmented generation for large language models: A survey, 2023. https://arxiv.org/abs/2312.10997.
[14] Richard A. DeMillo, Richard J. Lipton, and Frederick G. Sayward. Hints on test data selection: Help for the practicing programmer. Computer, 11(4):34–41, 1978. https://doi.org/10.1109/C-M.1978.218136.
[15] Yue Jia and Mark Harman. An analysis and survey of the development of mutation testing. IEEE Transactions on Software Engineering, 37(5):649–678, 2011. https://doi.org/10.1109/TSE.2010.62.
[16] Parishad BehnamGhader, Santiago Miret, and Siva Reddy. Can retriever-augmented language models reason? The blame game between the retriever and the language model. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 15492–15509. Association for Computational Linguistics, 2023. https://aclanthology.org/2023.findings-emnlp.1036/.
[17] Barry W. Boehm. Software Engineering Economics. Prentice-Hall, Englewood Cliffs, NJ, USA, 1981. ISBN 978-0138221225. https://dl.acm.org/doi/10.5555/539425.
[18] A. Jefferson Offutt. Investigations of the software testing coupling effect. ACM Transactions on Software Engineering and Methodology, 1(1):5–20, 1992. https://doi.org/10.1145/125489.125473.
[19] Richard A. DeMillo. Test adequacy and program mutation. In Proceedings of the 11th International Conference on Software Engineering, ICSE 1989, pages 355–356. IEEE, 1989. https://doi.org/10.1109/ICSE.1989.714449.
[20] Lech Madeyski, Wojciech Orzeszyna, Richard Torkar, and Mariusz Józala. Overcoming the equivalent mutant problem: A systematic literature review and a comparative experiment of second-order mutation. IEEE Transactions on Software Engineering, 40(1):23–42, 2014. https://doi.org/10.1109/TSE.2013.44.
[21] Michele Tufano, Dawn Drain, Alexey Svyatkovskiy, Shao Kun Deng, and Neel Sundaresan. Unit test case generation with transformers and focal context, 2020. https://arxiv.org/abs/2009.05617.
[22] Caroline Lemieux, Jeevana Priya Inala, Shuvendu K. Lahiri, and Siddhartha Sen. CodaMosa: Escaping coverage plateaus in test generation with pre-trained large language models. In Proceedings of the 45th International Conference on Software Engineering, ICSE 2023, pages 919–931. IEEE/ACM, 2023. https://doi.org/10.1109/ICSE48619.2023.00085.
[23] Cody Watson, Michele Tufano, Kevin Moran, Gabriele Bavota, and Denys Poshyvanyk. On learning meaningful assert statements for unit test cases. In Proceedings of the 42nd International Conference on Software Engineering, ICSE 2020, pages 1398–1409. ACM, 2020. https://doi.org/10.1145/3377811.3380429.
[24] Elizabeth Dinella, Gabriel Ryan, Todd Mytkowicz, and Shuvendu K. Lahiri. TOGA: A neural method for test oracle generation. In Proceedings of the 44th International Conference on Software Engineering, ICSE 2022, pages 2130–2141. ACM, 2022. https://doi.org/10.1145/3510003.3510141.
[25] Soneya Binta Hossain, Antonio Filieri, Matthew B. Dwyer, Sebastian Elbaum, and Willem Visser. Neural-based test oracle generation: A large-scale evaluation and lessons learned. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2023, pages 120–132. ACM, 2023. https://doi.org/10.1145/3611643.3616265.
[26] Stephen Robertson and Hugo Zaragoza. The probabilistic relevance framework: BM25 and beyond. Foundations and Trends in Information Retrieval, 3(4):333–389, 2009. https://doi.org/10.1561/1500000019.
[27] Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, pages 6769–6781. Association for Computational Linguistics, 2020. https://doi.org/10.18653/v1/2020.emnlp-main.550.
[28] Jiho Shin, Nima Shiri Harzevili, Reem Aleithan, Hadi Hemmati, and Song Wang. Retrieval-augmented test generation: How far are we?, 2024. https://arxiv.org/abs/2409.12682.

全文公開日期 2031/07/15

簡易檢索 / 詳目顯示

相關論文