Encoding and normalisation

Arabic script is not one code point per visible letter, and two faithful copies of the same ayah from the same publisher can differ in bytes while rendering identically. Nothing about this throws. This page shows the bytes from two of our own datasets.

Here is what that means in a real word. The first word of the ayah Ḥafṣ numbers 1:4, from Quran Text:

Three letters a reader sees. text.length is 7: U+0645 U+064E U+0670 U+0644 U+0650 U+0643 U+0650 — each letter followed by its vowel, and the first also by a small superscript alif. Slice that string at index 2 and you have cut a letter away from the mark that belongs to it.

The same seven ayat, from two of our datasets

Quran Text and the tajwīd edition both carry al-Fātiḥa in Ḥafṣ, both from the King Fahd Complex's text. Compared byte by byte:

Not one ayah is byte-equal, and every one is equal after Unicode NFC. The bytes at the first difference in 1:1:

1:1, first difference at index 10
quran-text       U+0644 U+064E U+0651
tajwīd edition   U+0644 U+0651 U+064E

U+064E is a fatḥah and U+0651 is a shadda. Quran Text stores the vowel then the shadda; the tajwīd edition stores the shadda then the vowel. Both render as one letter with two marks on it. Both are legitimate Unicode. They are not equal strings.

In 1:7 the difference changes the length: one file writes آ as a single precomposed code point, U+0622, and the other as U+0627 alif followed by U+0653 maddah. Everything after it is shifted by one.

Why NFC does not save you

NFC does not resolve every divergence. Al-Fātiḥa is the easy case. The same comparison over the Ḥafṣ text of surahs 36, 112, 113 and 114, 98 ayat, against the tajwīd edition:

ayahs compared            98
byte-equal                 5
equal after NFC           40
different length          41

What NFC leaves alone: the tanwīn on a word written as U+08F0 (open fatḥatan) in Quran Text and U+0657 (inverted ḍammah) in the edition; and waqf marks, which the edition carries inline in 2,868 of its 6,236 ayat while Quran Text holds all of them in a separate marks layer and none in words. Those are different characters, not different orderings, and no normalisation form makes them equal.

NFC changes offsets. Even where it makes two strings equal, it does so by rewriting one of them, and the spans were computed against the un-rewritten text. Normalise 1:7 and its length drops from 90 to 89; every span after the آ now points one character early. NFC is a tool for comparing, never for indexing.

1:7  span [84, 88]  madd-tabee-kalimi.3
against the tajwīd edition   "لِّي"   U+0644 U+0651 U+0650 U+064A
against quran-text, joined   "ِّين"   U+0650 U+0651 U+064A U+0646

Encodings differ across readings too

The Ḥafṣ text writes the sukūn as U+06E1, a small high dotless head of khāʾ — the sign the Madinah muṣḥaf prints. The Warsh text writes U+0652, the ordinary Arabic sukūn:

Same word, same letters, different code point in the fourth position. So a search index built over one edition does not find the same word in another, and a string equality test between readings is meaningless even where the spelling is the same. The join is the word number, not the text.

The rules

a.normalize("NFC") === b.normalize("NFC") is a fine test for "probably the same ayah", and it is not sufficient across all of our texts. Quran Text's fold() and plain, and Elements' search field, exist because stripping marks off the printed spelling yourself deletes the long vowels a user types.

The whole comparison, runnable against the two payloads this site serves:

// node, from the repository root
const f = require("./public/demo/fatiha.json").hafs;
const t = require("./public/demo/tajweed-fatiha.json");
const starts = [...f.ayah_starts, f.words.length];
const cp = (s) => [...s].map((c) => "U+" + c.codePointAt(0).toString(16).toUpperCase().padStart(4, "0")).join(" ");

for (let a = 1; a <= 7; a++) {
  const joined = f.words.slice(starts[a - 1], starts[a]).join(" ");
  const edition = t.text[a];
  let i = 0;
  while (i < joined.length && joined[i] === edition[i]) i++;
  console.log(`1:${a}`, joined === edition, joined.normalize("NFC") === edition.normalize("NFC"), i);
  if (i < joined.length) console.log("  ", cp(joined.slice(i, i + 2)), "|", cp(edition.slice(i, i + 2)));
}

How this data is made

Quran Text's words are extracted unedited from the publisher's digital packages, and its marks layer is peeled off by a build that lists every sign it moved. The tajwīd edition is the text the annotation engine was run over, published beside the spans with a content digest; its own metadata describes it as a legacy application export rather than naming a publisher package.