資訊檢索（IR）是一項從大規模文本集合中找到與用戶查詢相關資訊的任務。 隨著大型語言模型（PLM）的發展達到了新的高度。密集檢索技術便是透過將查詢句和文本輸入大型語言模型，編碼成密集向量進行關聯度計算此項技術能處理語言多樣性和複雜性。大型預訓練語言模型的訓練資源需求高，因此參數高效率微調（PEFT）如適配器、LoRA（低秩適應）等技術相繼提出，旨在減少微調參數量並保持性能。然而研究指出，此類方法在資訊檢索任務中效果有限，訓練參數過少會影響梯度下降方向，導致模型性能下降。本研究想利用LoRA的靈活性，在不增加額外訓練參數的情況下，以LoRA矩陣再加權文句的向量， 增進訓練效果， 設計一個更加通用的模型架構，並與其他較先進的LoRA技術結合， 以應對PEFT方法在資訊檢索任務中的挑戰。

Information retrieval (IR) is the task of finding information related to user queries from large text collections. With the development of large pre-trained language models (PLMs) reaching new heights, dense retrieval tech-niques have emerged. These techniques involve encoding query sentences and texts into dense vectors using large language models to calculate rel-evance scores. This approach effectively handles linguistic diversity and complexity. However, training large pre-trained language models requires substantial resources. Consequently, parameter-efficient fine-tuning (PEFT) techniques, such as adapters and LoRA (Low-Rank Adaptation), have been proposed to reduce the number of fine-tuning parameters while maintaining performance. Nonetheless, studies indicate that these methods have limited effectiveness in IR tasks, as too few training parameters can affect the direc-tion of gradient descent, leading to degraded model performance. This study aims to leverage the flexibility of LoRA to enhance training effectiveness without increasing additional training parameters. By re-weighting sentence vectors with LoRA matrices, we design a more versatile model architecture. This architecture will be combined with other advanced LoRA techniques to address the challenges of PEFT methods in IR tasks.

[1] A. Aghajanyan, L. Zettlemoyer, and S. Gupta. Intrinsic dimensionality explains the effectiveness of language model fine-tuning, 2020.
[2] V. Boteva, D. Gholipour, A. Sokolov, and S. Riezler. A full-text learning to rank dataset for medical information retrieval. In N. Ferro, F. Crestani, M.-F. Moens, J. Mothe, F. Silvestri, G. M. Di Nunzio, C. Hauff, and G. Silvello, editors, Advances in Information Retrieval, pages 716–722, Cham, 2016. Springer International Pub-lishing.
[3] J. Chen, S. Xiao, P. Zhang, K. Luo, D. Lian, and Z. Liu. Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. arXiv preprint arXiv:2402.03216, 2024.
[4] T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer. Qlora: Efficient fine-tuning of quantized llms. Advances in Neural Information Processing Systems, 36, 2024.
[5] J. Frankle and M. Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks, 2019.
[6] S. Hayou, N. Ghosh, and B. Yu. Lora+: Efficient low rank adaptation of large models. arXiv preprint arXiv:2402.12354, 2024.
[7] N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Ges-mundo, M. Attariyan, and S. Gelly. Parameter-efficient transfer learning for nlp. In International conference on machine learning, pages 2790–2799. PMLR, 2019.
[8] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021.
[9] N. Hyeon-Woo, M. Ye-Bin, and T.-H. Oh. Fedpara: Low-rank hadamard product for communication-efficient federated learning, 2023.
[10] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[11] D. J. Kopiczko, T. Blankevoort, and Y. M. Asano. Vera: Vector-based random matrix adaptation, 2024.
[12] T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Ep-stein, I. Polosukhin, M. Kelcey, J. Devlin, K. Lee, K. N. Toutanova, L. Jones, M.-W. Chang, A. Dai, J. Uszkoreit, Q. Le, and S. Petrov. Natural questions: a benchmark for question answering research. Transactions of the Association of Computational Linguistics, 2019.
[13] P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. K¨uttler, M. Lewis, W.-t. Yih, T. Rockt¨aschel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Sys-tems, 33:9459–9474, 2020.
[14] X. L. Li and P. Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190, 2021.
[15] Y. Li, Y. Yu, C. Liang, P. He, N. Karampatziakis, W. Chen, and T. Zhao. Loftq: Lora-fine-tuning-aware quantization for large language models. arXiv preprint arXiv:2310.08659, 2023.
[16] S.-Y. Liu, C.-Y. Wang, H. Yin, P. Molchanov, Y.-C. F. Wang, K.-T. Cheng, and M.-H. Chen. Dora: Weight-decomposed low-rank adaptation. arXiv preprint arXiv:2402.09353, 2024.
[17] X. Liu, Y. Zheng, Z. Du, M. Ding, Y. Qian, Z. Yang, and J. Tang. Gpt understands, too. AI Open, 2023.
[18] K. Lu, A. Grover, P. Abbeel, and I. Mordatch. Pretrained transformers as universal computation engines, 2021.
[19] X. Ma, J. Guo, R. Zhang, Y. Fan, and X. Cheng. Scattered or connected? an opti-mized parameter-efficient tuning approach for information retrieval. In Proceedings of the 31st ACM International Conference on Information & Knowledge Manage-ment, pages 1471–1480, 2022.
[20] M. Maia, S. Handschuh, A. Freitas, B. Davis, R. McDermott, M. Zarrouk, and A. Balahur. Www’18 open challenge: Financial opinion mining and question an-swering. In Companion Proceedings of the The Web Conference 2018, WWW ’18, page 1941–1942, Republic and Canton of Geneva, CHE, 2018. International World Wide Web Conferences Steering Committee.
[21] N. Muennighoff, H. Su, L. Wang, N. Yang, F. Wei, T. Yu, A. Singh, and D. Kiela. Generative representational instruction tuning. arXiv preprint arXiv:2402.09906, 2024.
[22] J. Pfeiffer, A. Kamath, A. R¨uckl´e, K. Cho, and I. Gurevych. Adapterfusion: Non-destructive task composition for transfer learning. arXiv preprint arXiv:2005.00247, 2020.
[23] S.-A. Rebuffi, H. Bilen, and A. Vedaldi. Learning multiple visual domains with residual adapters. Advances in neural information processing systems, 30, 2017.
[24] N. Reimers and I. Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084, 2019.
[25] S. E. Robertson, S. Walker, S. Jones, M. M. Hancock-Beaulieu, M. Gatford, et al. Okapi at trec-3. Nist Special Publication Sp, 109:109, 1995.
[26] T. Salimans and D. P. Kingma. Weight normalization: A simple reparameterization to accelerate training of deep neural networks, 2016.
[27] V. Sanh, L. Debut, J. Chaumond, and T. Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter, 2020.
[28] N. Thakur, N. Reimers, J. Daxenberger, and I. Gurevych. Augmened sbert: Data augmentation method for improving bi-encoders for pairwise sentence scoring tasks. arXiv preprint arXiv:2010.08240, 2020.
[29] N. Thakur, N. Reimers, A. R¨uckl´e, A. Srivastava, and I. Gurevych. Beir: A heteroge-nous benchmark for zero-shot evaluation of information retrieval models. arXiv preprint arXiv:2104.08663, 2021.
[30] D. Wadden, S. Lin, K. Lo, L. L. Wang, M. van Zuylen, A. Cohan, and H. Hajishirzi. Fact or fiction: Verifying scientific claims. arXiv preprint arXiv:2004.14974, 2020.
[31] L. Wang, N. Yang, X. Huang, B. Jiao, L. Yang, D. Jiang, R. Majumder, and F. Wei. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint arXiv:2212.03533, 2022.
[32] A. Waswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In NIPS, 2017.
[33] L. Xu, H. Xie, S.-Z. J. Qin, X. Tao, and F. L. Wang. Parameter-efficient fine-tuning methods for pretrained language models: A critical review and assessment. arXiv preprint arXiv:2312.12148, 2023.
[34] S.-Y. Yeh, Y.-G. Hsieh, Z. Gao, B. B. W. Yang, G. Oh, and Y. Gong. Navigating text-to-image customization: From lycoris fine-tuning to model evaluation, 2024.
[35] Q. Zhang, M. Chen, A. Bukharin, P. He, Y. Cheng, W. Chen, and T. Zhao. Adaptive budget allocation for parameter-efficient fine-tuning. In The Eleventh International Conference on Learning Representations, 2023.