🧠 I Trained a Massive Word2Vec Model on 13 Billion Russian Fiction Words — Here’s What Happened

TL;DR: I built a lemma‑based Word2Vec model (CBOW, 300d) on a huge corpus of Russian fiction (13B tokens → 7.3B after cleaning). You can load it with Gensim and explore semantic neighborhoods of words like слово, язык, речь. The model captures literary semantics without stop words or grammar tags. Check it out on Hugging Face. Why another word2vec for Russian? Most pre‑trained Russian word2vec models are trained on web crawls, news, or mixed corpora. That’s fine for general NLP, but fiction has its own semantic rules. Poetic metaphors, archaic vocabulary, and author‑specific styles shift vector spaces. I wanted a model that: Is trained exclusively on fiction (from classic to modern authors). Uses lemmas (no inflection noise). Is small enough to ship (197 MB for the main model, plus two 5.8 GB arrays). Lets researchers and developers play with literary semantics. So I built one. And I’m sharing it under the MIT license. The Corpus by Numbers Metric Value Raw words before preprocessing 13.98 B After lemmatization & stop‑word removal 7.36 B Sentences (after cleaning short ones) 1.36 B Paragraphs processed ~539 M Stop words: removed using this list. Lemmatizer: Yandex Mystem. Sentence splitter: razdel.sentenize. All lemmas are lowercase, dictionary form, no part‑of‑speech tags (saves time & space). Training Details import gensim data = gensim.models.word2vec.LineSentence('splitted_lemmed_lines.txt') model = gensim.models.Word2Vec( data, vector_size=300, window=10, min_count=2, sg=0 # CBOW (faster, good for large corpora) ) model.save('cbow_300_10.model') CBOW, not skip‑gram – slightly better syntactic capture for fiction. Window size 10 – balances local and long‑range dependencies. Min count 2 – very rare words become noise; keep it clean. The training script is plain Gensim – no weird dependencies. You can retrain or fine‑tune if you have more data. What’s Inside the Download? When you clone from Hugging Face, you get: cbow_300_10.model (197 MB) – the main Gensim model object cbow_300_10.model.syn1neg.npy (5.8 GB) – negative sampling weights cbow_300_10.model.wv.vectors.npy (5.8 GB) – the actual word vectors README.md, tst.py – docs and a test script Yes, the two .npy files are large. But you can load the model without loading both if you only need similarities (Gensim does lazy loading). Or use model.wv directly. Quick Start – Get Similar Words import gensim model = gensim.models.Word2Vec.load("cbow_300_10.model") # Look at closest neighbours of "слово" (word) for word, score in model.wv.most_similar("слово", topn=10): print(f"{word}: {score:.4f}") Output: фраза: 0.7941 словечко: 0.6602 слог: 0.6322 реплика: 0.6015 словосочетание: 0.5928 изречение: 0.5818 высказывание: 0.5800 глагол: 0.5735 эпитет: 0.5615 сентенция: 0.5556 Notice how epithet and verb pop up – the model clearly learned linguistic meta‑concepts from fiction. Now try язык (language / tongue): наречие (adverb) 0.6684 диалект 0.6095 латынь 0.5892 язычок (little tongue) 0.5698 алфавит 0.5039 грамматика 0.5027 суахили 0.4977 идиома 0.4952 иврит 0.4950 произношение 0.4927 And речь (speech): монолог 0.6400 спич 0.5914 тирада 0.5900 фраза 0.5652 диалог 0.5477 проповедь 0.5334 слово 0.5217 разглагольствование 0.5193 филиппика 0.5184 декламация 0.5147 Cool Things You Can Do Author style analysis – Compare vector spaces of Tolstoy vs. modern authors. Semantic change over time – Not directly, but you can train separate models for 19th and 20th century subsets. Genre classification – Fiction vs. non‑fiction (though this model is pure fiction, so it will be biased). Creative writing tools – Find thematic associations, generate metaphor candidates. Educational – Show how word geometry reflects literary concepts. Known Limitations (Read Before You Complain) No POS tags – All lemmas are ambiguous (e.g., стекло could be noun “glass” or past tense of “to flow”). That’s a conscious trade‑off for performance. Only lemmas – No inflected forms. Want слова, слову, словом? You won’t find them; use the lemma слово. Fiction bias – This model will perform poorly on legal documents, tweets, or technical manuals. But that’s the point. Large memory footprint – The full model takes ~12 GB on disk. However, you can load just the vectors with KeyedVectors.load() to save RAM. Comparison with Other Russian Word2Vec Models Model Corpus Size POS Availability This one Fiction, 13B words 300d No MIT, HF w2v-russian-tolstoy Only Tolstoy 300d No MIT, HF w2v-russian-19c-fiction-lemmas 19th century prose 300d No HF RusVectores (web+news) Mixed, ~20B 300d Yes (tags) CC BY‑SA If you need a general‑purpose model with grammatical info, go for RusVectores. If you work with literary analysis, this one is your friend. Why You Should Star / Share / Fork 📦 Ready to use – pip install gensim and model = gensim.models.Word2Vec.load(...) 🧪 Reproducible – Full preprocessing script included in the HF repo. 📚 Academically referenced – Several papers already used it (see README). 🚀 Lightweight core – The 197 MB model loads in seconds; the big .npy files are optional. Get the Model Hugging Face: nevmenandr/w2v-russian-fiction License: MIT Clone with git lfs or download via huggingface_hub: from huggingface_hub import snapshot_download snapshot_download(repo_id="nevmenandr/w2v-russian-fiction", local_dir="./w2v_model") What I’d Love to See in the Comments 🧩 Use cases – Are you doing computational literary studies? Building a Russian story generator? Tell me! 🐛 Issues / improvements – Did you find a bug? A word that should be a lemma but isn’t? 📊 Benchmarks – How does it perform on your task compared to other models? 🧠 Funny analogies – Run model.most_similar(positive=['царь', 'женщина'], negative=['мужчина']) and share the result. (Hint: it’s not “царица” – fiction is weird.) Final Code Snippet – Try It Yourself import gensim model = gensim.models.Word2Vec.load("cbow_300_10.model") # Find words similar to "поэт" (poet) print("Neighbors of поэт:") for w, s in model.wv.most_similar("поэт", topn=5): print(f" {w}: {s:.4f}") # Analogy: Москва : Россия = Париж : ? result = model.wv.most_similar(positive=["Франция", "Москва"], negative=["Россия"], topn=1) print(f"\nМосква / Россия ≈ Париж / {result[0][0]} (score {result[0][1]:.4f})") Run it. Play with it. Break it. Then tell me in the comments what you found. Happy vector hunting! 🧙‍♂️ P.S. The model is called cbow_300_10.model – old‑school name, but it works like a charm.