
LLM2Vec abstract
piano afro-rock
April 11th, 2024suno
Lyrics
[Bridge]
[Verse 1]
Large decoder-only language models
LLMs
are the state-of-the-art models
on most of today’s NLP tasks
[Verse 2]
Yet, the community is only slowly
adopting these models
for text embedding tasks
which require rich
contextualized representations.
[Verse 3]
In this work, we introduce LLM2Vec,
a simple unsupervised approach that
can transform any decoder-only LLM
into a strong text encoder.
[Chorus]
LLM2Vec consists of three simple steps:
One. Enabling bidirectional attention
Two. Masked next token prediction, and
Three. unsupervised contrastive learning.
[Verse 4]
We demonstrate the effectiveness of
LLM2Vec
by applying it to three popular LLMs
[Verse 5]
We outperform encoder-only models
by a large margin on word-level tasks
and reach a new unsupervised
state-of-the-art performance
on the Massive Text Embeddings Benchmark
MTEB.
[Verse 6]
Moreover, when combining
LLM2Vec
with supervised contrastive learning,
we achieve state-of-the-art performance
on MTEB
[Chorus]
LLM2Vec consists of three simple steps:
One. Enabling bidirectional attention
Two. Masked next token prediction, and
Three. unsupervised contrastive learning.
Three. unsupervised contrastive learning.
[]Verse 7]
Our strong empirical results
and extensive analysis
demonstrate that
LLMs can be effectively transformed
[Verse 8]
into universal text encoders
in a parameter-efficient manner
without the need
for expensive adaptation or
synthetic GPT-4 generated data.
[Chorus]
LLM2Vec consists of three simple steps:
One. Enabling bidirectional attention
Two. Masked next token prediction, and
Three. unsupervised contrastive learning.
Recommended

איך לייצר שיר?
מלודי אקוסטי פופ

Taubat Lah Sahabat
Arab Folk, Roots Worship, Fantasy, Anthem Worship, Indonesian Classical, Orchestra,

주기율표의 공간
dynamic powerful hard rock

Filhos do Céu
inspirado sonhador pop

Trippin' in Kuwait
arabian trap, arabic male vocals, oud melodies

1
Japan's animation opening

Meet Me Down In Hades
E-Punk Hyper-Pop Hip-Hop

Astral Projection
Pop punk, aggressive, piano riff

Chronik Beats
electronic edm

คืนนี้เราเมา
rhythmic playful pop

🌸美菩薩念經🌸
Dreamy k-pop

Paradoxical Mind
dubstep kawaii

vida minha
piano

Connect Me, Connect You
flamenco, metal, heavy metal

Gregarious Gally
witch house, accordion, 60BPM, intense, cinematic