newsBAIR (Berkeley)Trust 88 · LabPublished 11mo agoLive · 1mo ago

What exactly does word2vec learn?

What exactly does word2vec learn, and how? Answering this question amounts to understanding representation learning in a minimal yet interesting language modeling task. Despite the fact that word2vec is a well-known precursor to modern language models, for many years, researchers lacked a quantitative and predictive theory describing its learning pro

Research BAIR (Berkeley)

Why these links exist

Every edge carries a method, confidence, and the source snippet that justified it — so bad links are debuggable.

LinkedLinked via unknownTransformer →
LinkedLinked via unknownHow Surprising Is Historical Italian to Language Models? Tokenization Tax, Comprehension Tax, and a Simple Mitigation →
LinkedLinked via unknownRepresentational Depth of Evaluation Awareness Shifts With Scale in Open-Weight Language Models →
PossiblePossibly related (embedding) · 47%TypeProbe: Recovering Type Representations from Hidden States of Pre-trained Code Models →
PossiblePossibly related (embedding) · 47%Complexity-Guided Component-wise Initialization for Language Model Pretraining →
PossiblePossibly related (embedding) · 50%Scalable Visual Pretraining for Language Intelligence →
PossiblePossibly related (embedding) · 51%Language Identification with Succinct Machine-Independent Traces →
PossiblePossibly related (embedding) · 50%Controlling Implicit Shortcut Reliance in L2 Spoken English Auto-markers →

Covers

glossary_termTransformer

Covers (incoming)

paperHow Surprising Is Historical Italian to Language Models? Tokenization Tax, Comprehension Tax, and a Simple Mitigation paperRepresentational Depth of Evaluation Awareness Shifts With Scale in Open-Weight Language Models paperTypeProbe: Recovering Type Representations from Hidden States of Pre-trained Code Models paperComplexity-Guided Component-wise Initialization for Language Model Pretraining paperScalable Visual Pretraining for Language Intelligence paperLanguage Identification with Succinct Machine-Independent Traces paperControlling Implicit Shortcut Reliance in L2 Spoken English Auto-markers paperExposure is Optional: Learning Unlike Coordination in Language Models paperWhat, Where, and How: Disentangling the Roles of Task, Language, and Model in Code Model Representations

Related across the graph

paperScalable Visual Pretraining for Language Intelligence paperLanguage Identification with Succinct Machine-Independent Traces paperExposure is Optional: Learning Unlike Coordination in Language Models glossary_termTransformer paperRepresentational Depth of Evaluation Awareness Shifts With Scale in Open-Weight Language Models paperControlling Implicit Shortcut Reliance in L2 Spoken English Auto-markers paperTypeProbe: Recovering Type Representations from Hidden States of Pre-trained Code Models paperHow Surprising Is Historical Italian to Language Models? Tokenization Tax, Comprehension Tax, and a Simple Mitigation paperComplexity-Guided Component-wise Initialization for Language Model Pretraining paperWhat, Where, and How: Disentangling the Roles of Task, Language, and Model in Code Model Representations