How to train skip-gram models
The skip-gram models were trained using scripts based on pungen's implementation. These script require that the input data follows a specific preprocessing. This document explains how to download and prepare the data for using training the skip-gram models.
All commands are expected to be run from the root directory of the repository.
Where to download the data from
Chinese Wikipedia (维基百科)
- Download the dataset using HuggingFace.
from datasets import load_dataset
ds = load_dataset('wikimedia/wikipedia', '20231101.zh')
df = ds['train'].to_pandas()
df.to_csv('wiki-zh.txt', sep='\0', header=False, index=False, columns=['text'])
- Convert all the text to simplified Chinese using OpenCC.
opencc -i wiki-zh.txt -o wiki-zh-simplified.txt -c t2s.json
tokenization.py script using Python 3.11. Remember to edit the script to use the correct input and output files.
...
input = 'wiki-zh-simplified.txt'
output = 'wiki-zh-simplified-seg.txt'
...
Bookcorpus
- Download the dataset from here.
- Concatenate the two parts with
cat books_large_p*.txt > bookcorpus.txt
CroissantLLM Dataset
- Download using HuggingFace's datasets library.
from datasets import load_dataset
ds = load_dataset('croissantllm/croissant_dataset', data_files='french_303b_1/*/*.arrow')
df = ds['train'].to_pandas()
with open('croissant.txt', 'w', encoding='utf-8') as f:
for text in df['text']:
f.write(f'{text}\n')
BRWaC
- Request access through the corpus' official website.
- Download the corpus and extract the files.
- We use the CoNLL-U format, so we can take advantage of the text being already tokenized.
- Use our
scripts/preprocessing/brwac_conll.shscript to convert the CoNLL-U format to a simple text file. Make sure to edit the script to use the correct input and output files.
corpus_file="brwac.conll"
out_file="brwac.txt"
SBWC
- Download the clean corpus from here. Make sure that you have the clean version, with one sentence per line.
How to prepare the data
With the previous steps, you should have one file for each dataset, with one sentence per line. All files should be in the data directory. The next steps will convert the text to the format required by pungen.
Tokenize and lemmatize the texts
For each dataset, we need to tokenize and lemmatize the texts using spacy with the scripts/preprocessing/large_corpora.py script. This script will save the tokenized and lemmatized texts in the data/skipgram directory using the Parquet format for efficiency.
python scripts/preprocessing/large_corpora.py --input [CORPUS_NAME].txt \
--model [SPACY_MODEL_NAME]
Make sure to use the corresponding spacy model for each language.:
- Chinese:
zh_core_web_sm - English:
en_core_web_sm - French:
fr_core_news_sm - Portuguese:
pt_core_news_sm - Spanish:
es_core_news_sm
These models are not installed with the library. Install them with the training dependency group:
uv sync --group training
If the dataset is already tokenized and lemmatized (such as BRWAC or Bookcorpus), you can skip this step with the --no-spacy flag.
Build vocabulary
With the lemmatized texts, we first need to build a vocabulary of the most frequent lemmas in the corpus. We use the scripts/preprocessing/skipgram/build_vocab.py script for this.
python scripts/preprocessing/skipgram/build_vocab.py \
--corpus data/skipgram/[CORPUS_NAME].parquet \
--output data/preprocessed_skipgram/[CORPUS_NAME]/vocab.csv \
--max-vocab 50000 \
--min-freq 5
Convert tokens to IDs
With the vocabulary generated, we can now convert the corpus text into token IDs using the scripts/preprocessing/skipgram/convert_corpus_to_ids.py script.
python scripts/preprocessing/skipgram/convert_corpus_to_ids.py \
--corpus data/skipgram/[CORPUS_NAME].parquet \
--vocab data/ngrams/[LANGUAGE]/1gram.csv \
--output data/preprocessed_skipgram/[CORPUS_NAME]/corpus_ids.parquet
Create training data
Finally, we can create the training data for the skip-gram model using the scripts/preprocessing/skipgram/create_context_windows.py script. This script will create the context windows for each word in the corpus, saving them in a Parquet file.
python scripts/preprocessing/skipgram/create_context_windows.py \
--corpus data/preprocessed_skipgram/[CORPUS_NAME]/corpus_ids.parquet \
--vocab data/ngrams/[LANGUAGE]/1gram.csv \
--output data/preprocessed_skipgram/[CORPUS_NAME]/context_windows.parquet \
--min-dist 5 \
--max-dist 10
How to train the models
We provide a script to train the distant skip-gram models. The script is scripts/training/train_skipgram.py. To train a model, run:
python scripts/training/train_skipgram.py \
--context-windows data/preprocessed_skipgram/[CORPUS_NAME]/context_windows.parquet \
--vocab data/ngrams/[LANGUAGE]/1gram.csv \
--output results/skipgram/[LANGUAGE].pt \
--epochs 10
To check the parameters we used for training our models, check the scripts/experiments/run_skipgram_training.sh script.
Our trained models were converted with scripts/hub/export_skipgram.py and published to Hugging Face Hub.