LLM2Vec abstract

piano afro-rock

April 11th, 2024suno

Lyrics

[Bridge] [Verse 1] Large decoder-only language models LLMs are the state-of-the-art models on most of today’s NLP tasks [Verse 2] Yet, the community is only slowly adopting these models for text embedding tasks which require rich contextualized representations. [Verse 3] In this work, we introduce LLM2Vec, a simple unsupervised approach that can transform any decoder-only LLM into a strong text encoder. [Chorus] LLM2Vec consists of three simple steps: One. Enabling bidirectional attention Two. Masked next token prediction, and Three. unsupervised contrastive learning. [Verse 4] We demonstrate the effectiveness of LLM2Vec by applying it to three popular LLMs [Verse 5] We outperform encoder-only models by a large margin on word-level tasks and reach a new unsupervised state-of-the-art performance on the Massive Text Embeddings Benchmark MTEB. [Verse 6] Moreover, when combining LLM2Vec with supervised contrastive learning, we achieve state-of-the-art performance on MTEB [Chorus] LLM2Vec consists of three simple steps: One. Enabling bidirectional attention Two. Masked next token prediction, and Three. unsupervised contrastive learning. Three. unsupervised contrastive learning. []Verse 7] Our strong empirical results and extensive analysis demonstrate that LLMs can be effectively transformed [Verse 8] into universal text encoders in a parameter-efficient manner without the need for expensive adaptation or synthetic GPT-4 generated data. [Chorus] LLM2Vec consists of three simple steps: One. Enabling bidirectional attention Two. Masked next token prediction, and Three. unsupervised contrastive learning.

Recommended

Rhythm of the Jungle
Rhythm of the Jungle

drum 'n bass jungle

Summer Hit Raven
Summer Hit Raven

Summer, beat, rap, ambient

Tsunami
Tsunami

trip hop, heavy drum solo

techno
techno

electronics, mechanisms, retro style, edm, extreme.

Ups and Downs
Ups and Downs

Acoustic intro Post-grunge rockalternative male

Riches and Reign
Riches and Reign

gothic, piano, electric swing, witch house, catchy, atmospheric, mischievous, sultry female vocal

Forwarding
Forwarding

eclectic surreal cryptic hypnotic melodic deep synth electronica

Freezing
Freezing

space rock,new prog, progressive,electro rock, pop, hard rock

Özledim Seni
Özledim Seni

catchy pop acoustic

I See Dead People
I See Dead People

hard bass phonk eerie

Melow Electric Dream
Melow Electric Dream

electro-pop upbeat shimmering synths

Disfraz
Disfraz

blues rock only piano and slow pace. male voice.

Autumn's Dance
Autumn's Dance

post-rock, slow, melancholic, cello, atmospheric

Let’s Bounce
Let’s Bounce

Hip hop, 90s

City Lights
City Lights

upbeat hip hop rhythmic

Twirling in the Rain
Twirling in the Rain

Egyptian Electro swing