Morpheme boundaries and language-model loss for tokenizers that read tags and lemmas
Contributors: Anish Chaudhuri, Maya Vendhan
A language model reads text as tokens, and a tokenizer decides where each token begins and ends. The commonly used methods, byte-pair encoding and the unigram model, build their vocabularies from the frequency of character sequences and use no grammatical information. Turkish öğrencilerin, "of the students", is öğrenci (student) followed by the plural suffix ler and the genitive suffix in. Byte-pair encoding trained on Turkish Wikipedia cuts it as öğrenc, iler and in, moving the last letter of the stem onto the plural suffix
Part-of-speech tags and lemmas are the cheapest grammatical information available, since taggers exist for more than sixty languages and tagging a training corpus is a one-time cost. A tokenizer can use them in several ways. Merge scores in byte-pair encoding can be weighted by the probability of the part-of-speech transition at each word boundary, merges inside nouns, verbs and adjectives can be weighted so that content words stay whole, and words can be split where the lemma ends before any subword training takes place, the method called Lemma2Word. Preliminary experiments with these methods had suggested large gains, measured without held-out words, gold boundaries or repeated runs, and a controlled comparison was needed before any of them could be recommended.
Setup. Every tokenizer is trained on the same Wikipedia paragraphs in English, Spanish, Danish, Tamil and Turkish with the same budget of 8,000 types, three times with different training samples. Tags and lemmas come from Stanza. Boundary precision and recall are measured against the lemma-derived morpheme boundaries of MorphScore, compression on held-out paragraphs, and modeling quality by the bits per character of a small language model trained on each tokenizer's output. Byte-pair encoding and the unigram model, trained on the same text, are the baselines.
Part-of-speech weighting. Weighting merges by the transition probability changes boundary F1 by less than the seed-to-seed variation in four of the five languages and leaves compression unchanged. The weighting only touches merges that attach a word to the space after it, and the tokens it produces span the same frequent pairs as unweighted BPE, of the in English and de la in Spanish. The mean transition probability at a spanned boundary is no higher than at any other boundary. A transition probability describes word order, and word order says nothing about where to cut inside a word.
Content-word weighting. Doubling the weight of merges inside nouns, verbs and adjectives moves vocabulary budget toward whole content words. In Tamil, where BPE almost never cuts at the stem boundary, F1 rises from 4.4% to 22.2%. In Spanish and Turkish it falls, because the whole words it buys replace the suffix tokens BPE would otherwise learn.
Lemma2Word. Splitting words at the lemma boundary raises boundary recall from 54.4% to 88.2% in English and from 7.4% to 84.5% in Tamil, and by 23 to 34 points in the other three languages, for four to eight percent more tokens. On words absent from the training text, where only the learned affix inventory can act, the gain survives in English, Tamil and Turkish, whose suffixes are long and distinctive, and mostly disappears in Spanish and Danish. A variant that calls the lemmatizer at tokenization time recovers almost every gold boundary, at the cost of running a neural lemmatizer on every word.
Baselines. The unigram model is the strongest tokenizer that uses no linguistic resource at all and the only one competitive in Tamil without a lemmatizer. Letting the space marker merge freely, as the original implementation does, matters as much as any of the linguistic signals. It raises boundary F1 over the standard word-initial marker in four languages, because whole words with a prefixed marker are no longer available as single tokens.
A language model as a segmenter. Claude Sonnet, asked to mark the morpheme boundaries of 400 words per language without changing a character, finds more of the gold boundaries than any tokenizer in four languages, 91% recall in English and 96% in Turkish, and trails Lemma2Word in Danish. Its precision is low because it also marks derivational affixes and compound parts that the gold standard leaves unmarked. In Tamil it sometimes wrote the underlying suffix instead of the written form, which the character constraint forbids. The answers have no fixed vocabulary and cost far more per word than any tokenizer, so the study measures what the model knows about morphology, not what a tokenizer built from it would do.
Language-model probe. A 7M-parameter language model trained for six passes over the same text with each tokenizer's output ranks the tokenizers in a different order from the boundary scores. The unigram model gives the lowest bits per character in every language, 2.53 against 2.81 for byte-pair encoding in English, and Lemma2Word, the best tokenizer by boundary F1, gives the highest. The part-of-speech weighting changes the figure by less than 0.03 bits. Finding more morpheme boundaries does not by itself make a tokenizer better for a language model, at least at this scale.