Four Strokes or Three

In 1993 Japan’s electronics industry association published a pamphlet objecting to Unicode’s Han unification.

The unification maps Chinese hanzi, Japanese kanji, Korean hanja and historical Vietnamese chữ Hán onto shared code points, treating regional glyph shapes as renderings of one underlying grapheme.

The complaint was not that software had failed to store a character. It was that software had decided which differences did not count.

The differences were real enough to draw. The grass radical has four strokes in Traditional Chinese and three in Simplified Chinese and Japanese. Other mappings bring Japanese shinjitai and kyūjitai forms under one identifier. The code point survives in every case, and the regional form is no longer available from the identifier alone.

That is the quiet version of a very common data failure. A pipeline strips accents, picks a Latin reading, folds case, or substitutes one script for another. Search improves, because near forms now find each other.

The join gets worse, because the same conversion deleted the only discriminator.

The code point renders text #

Unicode never claimed that Chinese, Japanese, Korean and Vietnamese records were the same thing. It created a common encoding treatment for character sets whose relationships are close enough for that particular purpose.

The field’s disagreement is about the boundary. Whether a given shape is an allograph, a variant of one grapheme, or a different character altogether depends on the use at hand, and the encoding was designed around one use.

None of which is cosmetic. The registry that Unicode maintains gives each script a short alphabetic code — Latin, Cyrillic, Han — and those codes are administrative coordinates for renderable text. They are not judgments about what language a text contains, and not verdicts on whether a script’s scholarly classification is settled.

That separation is the apparatus worth taking. A representation can carry a language property, a script property and a regional property at the same time, in three fields.

Treat any one of them as a substitute for another and the lost component becomes invisible rather than absent.

Language and script part company at the first real example #

BCP 47 keeps the layers apart deliberately. Its tags combine a language subcode with optional script and region subcodes, so zh-Hant-TW identifies Traditional-script Chinese as used in Taiwan and zh-Hans-CN identifies Simplified-script Chinese as used in China.

Both tags have Chinese in them.

They do not say the same thing, and a system that keeps only the language subcode has thrown the difference away while appearing to keep the information.

The opposite mistake is treating a script as a language. Devanagari writes well over a hundred languages, Hindi and Sanskrit and Marathi and Nepali and Konkani among them. Hangul was designed for Korean in the fifteenth century and has largely stayed with that one language.

The two names look comparable. Their relationship between script and language is nothing alike.

The historical record keeps making the point from the other direction. Chinese characters reached Japan as kanji, Korea as hanja and Vietnam as chữ Hán. Japan still uses them. Korea moved toward Hangul in the twentieth century. Vietnam replaced them with the Latin-based Quốc Ngữ.

One writing system crossed three language settings and then had three entirely different careers.

A normalised vendor name gets handled as though it has the simpler history: one label, progressively cleaned. A raw form and a reading are different records of the same object. A language tag and a script tag are different records of its setting.

A transformed string carrying neither is not a cleaner version of the record. It is a shorter one.

A reading cannot stand in for a source form #

The error becomes most tempting exactly where the original cannot be read.

Linear A is conventionally called logosyllabic on the strength of its resemblance to its deciphered relative Linear B. No Linear A text has been translated and its underlying language remains unidentified. Structural classification and a recovered reading are separate achievements, and only one of them has happened.

The Indus script is harder again. Farmer, Sproat and Witzel argued in 2004 that its short inscriptions and sign patterns mean it may not encode language at all. Parpola and Rao countered with modelling they say finds language-like patterning.

The question is open.

A transcription that pretended to be a pronunciation would not settle it. It would hide the unanswered question inside a familiar alphabet, where nobody would look for it again.

So the script record has learned to keep the raw object, because the reading may change. Barthel’s 1958 catalogue assigns Rongorongo six hundred glyph codes even though its function and date remain disputed. Mahadevan counted 419 distinct Indus signs; Wells counted nearly seven hundred four decades later.

That disagreement is not an inconvenience to be normalised away. It records different decisions about which shapes are variants, which are ligatures, and which are separate signs, and the raw corpus is what makes the disagreement resolvable at all.

The operational case has one advantage those objects do not. An undeciphered script may have no recoverable sound behind its source form. A data feed almost always has the raw bytes, right up until somebody overwrites them.

The collision has a location #

The useful question is not whether two normalised strings match. It is where in the pipeline their histories became one string.

Compare the collision groups at ingestion, then after Unicode handling, after script conversion, after transliteration, after accent stripping. Then look at what came off the characters at the step where a group suddenly expands.

If the originals were already identical, the supplier issued duplicates. If distinct source forms meet only after conversion, the conversion made the collision, and the conversion is the thing to change.

This is why a single canonical name is a poor primary key. It answers a retrieval question — what should be found together — and entity matching asks a different one: what may be treated as the same record.

Han unification decided which differences did not count, and for rendering text on a screen it decided correctly.

Nobody consulted it about primary keys.

The step where two records stopped being different is the step that took the identifying field off them, and it is the only place worth looking.