<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="benedek-wolf-2024-prilora">
    <titleInfo>
        <title>PRILoRA: Pruned and Rank-Increasing Low-Rank Adaptation</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Nadav</namePart>
        <namePart type="family">Benedek</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Lior</namePart>
        <namePart type="family">Wolf</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2024-03</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Findings of the Association for Computational Linguistics: EACL 2024</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Yvette</namePart>
            <namePart type="family">Graham</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Matthew</namePart>
            <namePart type="family">Purver</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">St. Julian’s, Malta</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>With the proliferation of large pre-trained language models (PLMs), fine-tuning all model parameters becomes increasingly inefficient, particularly when dealing with numerous downstream tasks that entail substantial training and storage costs. Several approaches aimed at achieving parameter-efficient fine-tuning (PEFT) have been proposed. Among them, Low-Rank Adaptation (LoRA) stands out as an archetypal method, incorporating trainable rank decomposition matrices into each target module. Nevertheless, LoRA does not consider the varying importance of each layer. To address these challenges, we introduce PRILoRA, which linearly allocates a different rank for each layer, in an increasing manner, and performs pruning throughout the training process, considering both the temporary magnitude of weights and the accumulated statistics of the input to any given layer. We validate the effectiveness of PRILoRA through extensive experiments on eight GLUE benchmarks, setting a new state of the art.</abstract>
    <identifier type="citekey">benedek-wolf-2024-prilora</identifier>
    <location>
        <url>https://aclanthology.org/2024.findings-eacl.18</url>
    </location>
    <part>
        <date>2024-03</date>
        <extent unit="page">
            <start>252</start>
            <end>263</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T PRILoRA: Pruned and Rank-Increasing Low-Rank Adaptation
%A Benedek, Nadav
%A Wolf, Lior
%Y Graham, Yvette
%Y Purver, Matthew
%S Findings of the Association for Computational Linguistics: EACL 2024
%D 2024
%8 March
%I Association for Computational Linguistics
%C St. Julian’s, Malta
%F benedek-wolf-2024-prilora
%X With the proliferation of large pre-trained language models (PLMs), fine-tuning all model parameters becomes increasingly inefficient, particularly when dealing with numerous downstream tasks that entail substantial training and storage costs. Several approaches aimed at achieving parameter-efficient fine-tuning (PEFT) have been proposed. Among them, Low-Rank Adaptation (LoRA) stands out as an archetypal method, incorporating trainable rank decomposition matrices into each target module. Nevertheless, LoRA does not consider the varying importance of each layer. To address these challenges, we introduce PRILoRA, which linearly allocates a different rank for each layer, in an increasing manner, and performs pruning throughout the training process, considering both the temporary magnitude of weights and the accumulated statistics of the input to any given layer. We validate the effectiveness of PRILoRA through extensive experiments on eight GLUE benchmarks, setting a new state of the art.
%U https://aclanthology.org/2024.findings-eacl.18
%P 252-263

Markdown (Informal)

[PRILoRA: Pruned and Rank-Increasing Low-Rank Adaptation](https://aclanthology.org/2024.findings-eacl.18) (Benedek & Wolf, Findings 2024)

• PRILoRA: Pruned and Rank-Increasing Low-Rank Adaptation (Benedek & Wolf, Findings 2024)

ACL

• Nadav Benedek and Lior Wolf. 2024. PRILoRA: Pruned and Rank-Increasing Low-Rank Adaptation. In Findings of the Association for Computational Linguistics: EACL 2024, pages 252–263, St. Julian’s, Malta. Association for Computational Linguistics.