Lowercases and splits on any run of non-letter/non-digit characters,
so "Nieuw-Amsterdam" and "Nieuw Amsterdam" tokenize identically
regardless of which separator the data used. Unicode-aware (\p{L}),
so accented letters count as letters rather than splitting a word in
two. Case-only -- no diacritic folding; see tokenizeFolded for that.
Kept separate so nameMatchQuality can tell whether a match needed
folding to reach its tier (the "exact spelling" bonus).
Lowercases and splits on any run of non-letter/non-digit characters, so "Nieuw-Amsterdam" and "Nieuw Amsterdam" tokenize identically regardless of which separator the data used. Unicode-aware (
\p{L}), so accented letters count as letters rather than splitting a word in two. Case-only -- no diacritic folding; seetokenizeFoldedfor that. Kept separate sonameMatchQualitycan tell whether a match needed folding to reach its tier (the "exact spelling" bonus).