|
| 1 | +# LCM Grammar JSON (format `lcm-grammar`, version 1) |
| 2 | + |
| 3 | +A deterministic, GUID-keyed JSON projection of the **parser-relevant subset** of a FieldWorks/LCM |
| 4 | +project: phonology, morphology, and lexicon — everything a morphological parser needs, and (aside |
| 5 | +from glosses, definitions, and names) nothing else. Produced by |
| 6 | +`SIL.LCModel.DomainServices.GrammarJsonServices.ExportGrammar`; the machine-checkable structure is |
| 7 | +[`lcm-grammar.schema.json`](lcm-grammar.schema.json) (validated against the exporter by |
| 8 | +`GrammarJsonServicesTests`). |
| 9 | + |
| 10 | +**What it is for:** grammar verification tooling ("does this change parse better?"), conformance |
| 11 | +fixtures for morphological-parser test suites, and field deployment (a small artifact that |
| 12 | +web/wasm parsers can load without LibLCM — as one measured data point, a real ~56 MB `.fwdata` |
| 13 | +project exported to roughly 2.4 MB pretty-printed, ~250 KB gzipped). |
| 14 | + |
| 15 | +**What it is not:** an editing format, a synchronization/merge format, or a replacement for |
| 16 | +`.fwdata`. It is a read-only projection; LCM remains the authority. Consuming an LCM Grammar JSON |
| 17 | +document never requires LibLCM or FieldWorks — that independence is the point. |
| 18 | + |
| 19 | +## 1. Conventions |
| 20 | + |
| 21 | +- **Envelope.** Every document is `{ "format": "lcm-grammar", "version": 1, ... }`. Consumers |
| 22 | + should reject any other format tag or major version. |
| 23 | +- **Naming.** camelCase field names owned by this spec — not a mirror of LCM class/property names. |
| 24 | + The exporter source documents which LCM property each field originates from. |
| 25 | +- **Cross-references.** Every reference to another object in the document is a FieldWorks GUID, |
| 26 | + rendered lowercase-hyphenated (`Guid.ToString()`'s default format) — never an `Hvo` |
| 27 | + (FieldWorks' in-session integer id, which is not durable). |
| 28 | +- **Determinism.** Two exports of the same data are byte-identical, and an independent |
| 29 | + implementation reading the same project must be able to reproduce the same bytes: |
| 30 | + - Output is pretty-printed with two-space indentation. |
| 31 | + - LCM owning/reference *sequences* (`OS`/`RS` properties) keep model order — that order is |
| 32 | + semantically meaningful (rule order, slot order, allomorph disjunctive order, ...). |
| 33 | + - Unordered LCM *collections* (`OC`/`RC` properties, repository enumerations — notably lexical |
| 34 | + entries) are sorted by the ordinal comparison of their lowercase GUID string. |
| 35 | + - Multistring values are arrays of `{"ws": tag, "form": text}` sorted ordinally by |
| 36 | + writing-system tag; empty alternatives are skipped. (Phoneme/boundary-marker |
| 37 | + `representations` concatenate codes in model order, sorting per-code.) |
| 38 | +- **Optional vs. absent.** Absent optional values and empty arrays are omitted entirely, never |
| 39 | + written as `null` or `[]` — with one exception: a lexical entry's `allomorphs` array is always |
| 40 | + present, even when empty. An omitted field means "no data"; consumers must not distinguish |
| 41 | + omission from emptiness. |
| 42 | +- **Tagged unions.** Polymorphic objects (natural classes, phonological rules, pattern contexts, |
| 43 | + compound rules, ad hoc rules, rule mappings, MSAs, entry refs) carry a `kind` discriminator. |
| 44 | +- **Never-silent skips.** Data the format cannot represent (an unknown morph type, a dangling |
| 45 | + reference, malformed parser-parameter XML) is skipped by the exporter with a message in its |
| 46 | + optional warnings collection — the document itself contains only well-formed data. |
| 47 | +- **Unicode normalization.** All exported text is NFC. (LCM holds strings NFD in memory; the raw |
| 48 | + `.fwdata` XML at rest is NFC — exporting NFC lets an independent reader of the raw XML |
| 49 | + reproduce the same bytes.) |
| 50 | +- **Name fields.** Name-like multistrings export the best analysis alternative, falling back to |
| 51 | + the best vernacular alternative (phoneme names, for example, are usually authored only in a |
| 52 | + vernacular writing system). A field with no usable value is exported as `""` (FieldWorks' |
| 53 | + internal `"***"` missing-value marker is normalized to `""`). |
| 54 | + |
| 55 | +## 2. Document structure |
| 56 | + |
| 57 | +Top level: `format`, `version`, `project`, `featureSystems`, `phonology`, `morphology`, |
| 58 | +`lexicon` — all always present. See the JSON Schema for every field's exact shape; the highlights |
| 59 | +per section: |
| 60 | + |
| 61 | +### `project` |
| 62 | +`name`; `vernacularWritingSystems` / `analysisWritingSystems` (ICU tags, default writing system |
| 63 | +first, then the remaining current writing systems in project order). |
| 64 | + |
| 65 | +### `featureSystems` |
| 66 | +`phonological` and `morphosyntactic` — FieldWorks' two independent feature systems, each with |
| 67 | +`closedFeatures` (values enumerated as guid/name/abbreviation symbols) and `complexFeatures`. |
| 68 | +Feature *structures* appear throughout the document as |
| 69 | +`{"values":[{"feature": guid, "value": {"kind": "closed"|"complex", "value": ...}}]}`, recursive |
| 70 | +through complex values. A structure's features resolve against whichever feature system its host |
| 71 | +belongs to; the two are never mixed in one structure. |
| 72 | + |
| 73 | +### `phonology` |
| 74 | +`phonemes` (per-writing-system `representations` with FieldWorks' dotted-circle placeholder |
| 75 | +U+25CC stripped; optional `features`, `basicIpaSymbol`), `boundaryMarkers` (excluding the built-in |
| 76 | +word-boundary marker — see appendix), `naturalClasses` (`kind:"segments"` extensional / |
| 77 | +`kind:"features"` intensional; `name` is the FieldWorks *abbreviation*, which is what environment |
| 78 | +strings reference), `environments` (the raw FieldWorks environment string, e.g. `/_[C]`, |
| 79 | +untokenized), `rules` (rewrite and metathesis, in `OrderNumber` order, disabled rules excluded), |
| 80 | +and `featureConstraints` (alpha-variable slots). |
| 81 | + |
| 82 | +Pattern positions use the recursive `PhonContext` union: `sequence`, `iteration` (min/max, -1 = |
| 83 | +unbounded), `segment`, `naturalClass` (with `plusVariables`/`minusVariables` alpha-variable |
| 84 | +agreement), `boundary`, `wordBoundary`, `variable`. Rewrite rules carry `direction` |
| 85 | +(`leftToRight` | `rightToLeft` | `simultaneous`), a structural description, and one or more |
| 86 | +right-hand sides (structural change, left/right context, required parts of speech, |
| 87 | +required/excluded rule features). A rewrite rule's `featureConstraintVariables` lists its |
| 88 | +alpha-variable feature constraints **in assignment order** — consumers assign variable names |
| 89 | +(α, β, γ, ...) in exactly this order, so the order is semantic, not cosmetic. Metathesis rules carry 0-based `leftSwitchIndex` / |
| 90 | +`rightSwitchIndex` into their structural description. |
| 91 | + |
| 92 | +### `morphology` |
| 93 | +`partsOfSpeech` (the full possibility tree, with per-POS inflection classes (recursive), |
| 94 | +`defaultInflectionClass`, `inflectableFeatures`, stem names (feature-structure `regions`), affix |
| 95 | +slots, and affix templates (`prefixSlots` innermost-to-outermost, `suffixSlots` in order, |
| 96 | +`isFinal`, disabled templates included with their flag)); `compoundRules` (`endocentric` / |
| 97 | +`exocentric`, disabled included); `adhocProhibitions` (allomorph- and morpheme-level, with |
| 98 | +`adjacency`: `anywhere` | `somewhereToLeft` | `somewhereToRight` | `adjacentToLeft` | |
| 99 | +`adjacentToRight`); `exceptionFeatures` (the merged registry of productivity restrictions and |
| 100 | +possibility-typed phonological rule features); `lexEntryInflTypes` (irregularly-inflected-form |
| 101 | +variant types, with `glossPrepend`/`glossAppend` and template `slots`); and `parserParameters` |
| 102 | +(parsed from FieldWorks' stored XML block: `notOnClitics` **defaults to true when absent**, |
| 103 | +`acceptUnspecifiedGraphemes` and `noDefaultCompounding` default to false, optional raw `strata` |
| 104 | +string, optional per-compound-rule `compoundRuleMaxApplications`). |
| 105 | + |
| 106 | +### `lexicon` |
| 107 | +`entries`, GUID-sorted. Each entry: `citationForm`, `lexemeMorphType` (the lexeme form's morph |
| 108 | +type — see appendix for the closed enum), `allomorphs` (**alternate forms first, lexeme form |
| 109 | +last** — this order carries allomorph-selection semantics), `msas`, `senses`, `entryRefs`. |
| 110 | + |
| 111 | +- **Allomorphs** cover stem allomorphs (`environments`, `stemName`), affix allomorphs |
| 112 | + (`environments`, `positions`, `inflectionClasses`, `msEnvFeatures`, `msEnvPartOfSpeech`), and |
| 113 | + affix processes (`process` with `input` pattern parts and `output` mappings: |
| 114 | + `insertNaturalClass`, `copyFromInput`, `insertSegments`, `modifyFromInput`). `copyFromInput` / |
| 115 | + `modifyFromInput` reference input parts **positionally** (1-based index into `input`); an affix |
| 116 | + process with an unrepresentable input part is therefore skipped whole rather than emitted with |
| 117 | + misaligned indices. Forms may contain FieldWorks' lexical-pattern bracket notation (e.g. |
| 118 | + `[C][V]d`) verbatim; tokenizing it is the consumer's concern. |
| 119 | +- **MSAs** are a tagged union: `stem`, `inflectional` (empty `slots` means the affix applies |
| 120 | + outside any template), `derivational` (from/to pairs), `unclassified`. |
| 121 | +- **Senses** are the entry's sense tree flattened pre-order (parent before its subsenses), each |
| 122 | + with `gloss`, `definition`, and an `msa` reference into the same entry's `msas`. |
| 123 | +- **Entry refs** are `variant` or `complexForm`: a ref is a `complexForm` only when it has |
| 124 | + complex-form types and no variant types; anything else (variant types only, both kinds, or |
| 125 | + neither) is a `variant`. `componentLexemes` may reference entries **or** senses. |
| 126 | + |
| 127 | +## 3. Versioning and evolution |
| 128 | + |
| 129 | +- The schema in this directory validates exactly what the current exporter emits; exporter and |
| 130 | + schema change together, in the same commit. |
| 131 | +- Within major version 1, changes are **additive only**: new optional fields may appear; existing |
| 132 | + fields never change meaning, type, or optionality. Consumers should ignore fields they do not |
| 133 | + recognize. |
| 134 | +- Anything that would break a faithful consumer requires a major-version bump of the `version` |
| 135 | + field. |
| 136 | + |
| 137 | +Planned additive extensions (not yet present): writing-system definitions (so a document can be |
| 138 | +imported with no source project behind it) and optional export filters with an `omits` envelope |
| 139 | +marker. A cross-implementation byte-equality gate against an independent `.fwdata` reader is |
| 140 | +planned to keep this spec honest. |
| 141 | + |
| 142 | +## 4. Appendix: built-in FieldWorks GUIDs |
| 143 | + |
| 144 | +These well-known objects are referenced by meaning rather than enumerated in the document. They |
| 145 | +are constant across all FieldWorks projects. |
| 146 | + |
| 147 | +**Word boundary** (`PhBdryMarker`, excluded from `boundaryMarkers`; pattern contexts referencing |
| 148 | +it are exported as `kind:"wordBoundary"`): `7db635e0-9ef3-4167-a594-12551ed89aaa`. |
| 149 | + |
| 150 | +**Morph types** (`MoMorphType` possibilities → the `morphType`/`lexemeMorphType` enum): |
| 151 | + |
| 152 | +| Enum value | GUID | |
| 153 | +|---|---| |
| 154 | +| `stem` | `d7f713e8-e8cf-11d3-9764-00c04f186933` | |
| 155 | +| `boundStem` | `d7f713e7-e8cf-11d3-9764-00c04f186933` | |
| 156 | +| `root` | `d7f713e5-e8cf-11d3-9764-00c04f186933` | |
| 157 | +| `boundRoot` | `d7f713e4-e8cf-11d3-9764-00c04f186933` | |
| 158 | +| `prefix` | `d7f713db-e8cf-11d3-9764-00c04f186933` | |
| 159 | +| `suffix` | `d7f713dd-e8cf-11d3-9764-00c04f186933` | |
| 160 | +| `infix` | `d7f713da-e8cf-11d3-9764-00c04f186933` | |
| 161 | +| `circumfix` | `d7f713df-e8cf-11d3-9764-00c04f186933` | |
| 162 | +| `proclitic` | `d7f713e2-e8cf-11d3-9764-00c04f186933` | |
| 163 | +| `enclitic` | `d7f713e1-e8cf-11d3-9764-00c04f186933` | |
| 164 | +| `clitic` | `c2d140e5-7ca9-41f4-a69a-22fc7049dd2c` | |
| 165 | +| `particle` | `56db04bf-3d58-44cc-b292-4c8aa68538f4` | |
| 166 | +| `phrase` | `a23b6faa-1052-4f4d-984b-4b338bdaf95f` | |
| 167 | +| `discontigPhrase` | `0cc8c35a-cee9-434d-be58-5d29130fba5b` | |
| 168 | +| `prefixingInterfix` | `af6537b0-7175-4387-ba6a-36547d37fb13` | |
| 169 | +| `infixingInterfix` | `18d9b1c3-b5b6-4c07-b92c-2fe1d2281bd4` | |
| 170 | +| `suffixingInterfix` | `3433683d-08a9-4bae-ae53-2a7798f64068` | |
| 171 | + |
| 172 | +FieldWorks' *simulfix* and *suprafix* morph types, `MoDerivStepMsa` analyses, and coordinate |
| 173 | +compound rules (`MoCoordinateCompound`) have no representation in this format; each is skipped |
| 174 | +with a warning. |
0 commit comments