4.3其他低秩分解技术

4.3 其他低秩分解技术

4.3.1 delta-tuning

4.3.1.1 delta-tuning 介绍

4.3.1.2 delta-tuning 的数学解释

4.3.1.3 delta-tuning 代码实战

4.3.1.4 模型评估

4.3.2 DoRA: Weight-Decomposed Low-Rank Adaptation

4.3.2.1 DoRA 介绍

4.3.2.2 DoRA 的数学解释

4.3.2.3 DoRA 代码实战

4.3.2.4 模型评估

4.3.3 Matrix-Transformation Based Low-Rank Adaptation (MTLoRA)

4.3.3.1 MTLoRA 介绍

4.3.3.2 MTLoRA 的数学解释

4.3.3.3 MTLoRA 代码实战

4.3.3.4 模型评估

4.3.4 AFLoRA: Adaptive Freezing of Low Rank Adaptation

4.3.4.1 AFLoRA 介绍

4.3.4.2 AFLoRA 的数学解释

4.3.4.3 AFLoRA 代码实战

4.3.4.4 模型评估

4.3.5 the Mixture of LoRA Experts (MoLE) approach

4.3.5.1 MoLE 介绍

4.3.5.2 MoLE 的数学解释

4.3.5.3 MoLE 代码实战

4.3.5.4 模型评估

参考文献

Lialin, Vladislav, Vijeta Deshpande, and Anna Rumshisky. "Scaling down to scale up: A guide to parameter-efficient fine-tuning." arXiv preprint arXiv:2303.15647 (2023).

Ding, Ning, et al. "Parameter-efficient fine-tuning of large-scale pre-trained language models." Nature Machine Intelligence 5.3 (2023): 220-235.

Liu, Shih-Yang, et al. "Dora: Weight-decomposed low-rank adaptation." arXiv preprint arXiv:2402.09353 (2024).

Liang, Yao, Yuwei Wang, and Yi Zeng. "Matrix-transformation based low-rank adaptation (mtlora): A brain-inspired method for parameter-efficient fine-tuning." arXiv preprint arXiv:2403.07440 (2024).

Liu, Zeyu, et al. "AFLoRA: Adaptive Freezing of Low Rank Adaptation in Parameter Efficient Fine-Tuning of Large Models." arXiv preprint arXiv:2403.13269 (2024).

Wu, Xun, Shaohan Huang, and Furu Wei. "Mixture of lora experts." arXiv preprint arXiv:2404.13628 (2024).