## Problem statement <p>Enable high-quality narration of mathematical expressions</p> <p class="fragment"><br> 1. Done: <span class="concepty">"narrate-as-written"</span> cases are enabled via MathML Core<br><br> 2. Missing: <span class="concepty">"narrate-as-concept"</span> cases need additional information </p> <p class="fragment"><br>Method: New annotation language for MathML Core </p>

### Math notations are narrated * as written - $1,2$ read "one comma two" * as the concepts they stand for - $C-H$ read - "C minus H" for "minus", - "C to H" for "range", - "C bound to H" for "chemical bond" * phrased in context - "C bound to H" or "Carbon–hydrogen bond" - "C to H" or "the range from C to H" * as a composition of pieces - choice of connector words - $f:A\rightarrow B$ read "f is a map **from** A **to** B" * depends on modality (contd.)

### Modalities * Unimodal * single modality * needs to contain all relevant information * details: hidden outside of expression * e.g. textual context, UI toggles * Bimodal * two or more synchronized modalities * information often separated between media types. <br><br> ### Examples Unimodal | Bimodal :--- | :--- rendered on screen | rendered on screen/blackboard + speech rendered on braille display | braille display + speech spoken as audiobook | source syntax + speech plain-text source syntax | source syntax + rendered on screen

## Solution requirements <br> ## "Adoptable" <br> A blackbox that needs to be unpacked

## "Adoptable" 1. <span class="fragment highlight-current-concept">Lightweight:</span> remediation intervention only when needed (not on every node) 2. <span class="fragment highlight-current-concept">Small learning-curve:</span> "a convention so simple, you don't have to read the spec" 3. <span class="fragment highlight-current-concept">Low friction:</span><ul> <li> <span class="fragment highlight-current-green">Good</span> common questions: "what is this technical notation? <span class="fragment highlight-current-green">how do we narrate it?</span>"</li> <li> <span class="fragment highlight-current-red">Bad</span> common questions: "what is this intent syntax? <span class="fragment fade-in highlight-current-red">how should I use it?</span>"</li></ul> 4. <span class="fragment highlight-current-concept">Predictable:</span> minimize "surprise" in AT outcomes (<span class="fragment fade-in highlight-current-red">"why did it say that?"</span>) 5. <span class="fragment highlight-current-concept">Terse:</span> brief syntax is quicker to write; less overwhelming to review 6. <span class="fragment highlight-current-concept">Full:</span> solves the problem * in real-world math syntax * in real-world MathML Core syntax * for open-ended domains 7.<!-- .element: class="fragment" --> Golden middle: some requirements are in <span class="fragment highlight-red">trade-off tension</span>

## Partial annotation * Partial, but when? We need to define **when annotation is required**. So far: - whenever we use special language conventions to narrate - never if narrating exactly as written * Partial, but what? * concept names that participate in narration (encyclopedic) * argument/component structure (e.g. <span class="concepty">`binomial of $\$1$ and $\$2$`</span>)

## NNRFC Syntax * Short for "Names, numbered references and function calls" * Deyan's current preference (restriction on [Bruce's mini language](https://mathml-refresh.github.io/discussion-papers/semantics-mini)) * Grammar ```ebnf intent ::= name | expression name ::= concept | argref concept ::= [letter|digit|-]* arg ::= [digit]+ argref ::= '$' arg expression ::= name '(' intent [ ',' intent ]* ')' ``` * Example ```xml <mrow intent="$1($2)"> <mi arg="2">x</mi> <mo arg="1" intent="double-factorial">!!</mo> </mrow> ```

## When should we use ARIA? * When there is a clear need for **custom narration** * any of: author preference, AT corrections, special content * example: group operation spoken ["blob"](https://youtu.be/mJ8ZDdA10GY?t=70) in $(G,\cdot)$: ```xml <mrow intent="group($1,$2)"> <mo stretchy="false">(</mo> <mi arg="1">G</mi> <mo>,</mo> <mo arg="2" aria-label="blob">⋅</mo> <mo stretchy="false">)</mo> </mrow> ```

## The Narration Algorithm In a depth-first traversal over a `<math>` element, AT examines the node attributes, as follows: 1. if <span class="linky">`aria-hidden="true"`</span>: skip 2. if <span class="linky">`aria-label`</span>, `aria-labelledby`, `aria-braillelabel`: narrate per ARIA 3. if <span class="linky">`intent`</span>, transform NNRFC intent expression to narration string - keyed into <span class="concepty">"encyclopedic"</span> naming convention - bootstrap with a global list in the public domain with <span class="concepty">"alias"</span>es and <span class="concepty">"speech rule"</span>s - AT tool-specific enhancements 4. if <span class="linky">`n/a`</span>, check for narration rule match in <span class="concepty">"Western K12 Education"</span> rule set - standardize [~250](https://docs.google.com/spreadsheets/d/1EsWou1K5nxBdLPvQapdoA9h-s8lg_qjn8fJH64g9izQ/edit?usp=sharing) intent values, with selectors matching standard K12 notations. - AT tools should allow an "off" toggle. 5. if <span class="linky">`n/a`</span>, use <span class="concepty">"Read as Rendered"</span> rule set. - expected baseline for all AT tools, likely best narration choice for copy-editing. - standardize narrations for [the 31](https://www.w3.org/TR/2021/WD-mathml-core-20210816/#x2-1-elements-and-attributes) MathML Core elements. - standardize narrations for relevant attributes (e.g. `mathbackground`, `mathcolor`, `mathsize`, `mathvariant`). 6. Auxiliary: if <span class="linky">`aria-describedby`</span>: further properties via a UI action.

## Encyclopedic Names * Prototype: surveyed [>1,000 concepts](https://docs.google.com/spreadsheets/d/1EsWou1K5nxBdLPvQapdoA9h-s8lg_qjn8fJH64g9izQ/edit#gid=1818558895) with associated notations. * Avoids new nomenclature, follow names in scientific practice. * The author can use the **exact** concept name (word or phrase) they already know. * Nice auxiliary benefits * easy connection to encyclopedic resources, * associated [internationalization](https://www.wikidata.org/wiki/Q13520655), * each value has dual use - natural narration and machine-readable key

### More than AT Are other applications also enabled together with narration? * computation and CAS interchange? * i.e. path to Content MathML * richer e-Learning? * others? Let's do some due dilligence.

### MathML Difficulty 1 (D1) ### No Canonical Presentation ```xml <mrow intent="$1($2)"> <mi arg="2">x</mi> <mo arg="1" intent="double-factorial">!!</mo> </mrow> ``` ```xml <mrow intent="$1($2)"> <mi arg="2">x</mi> <mspace width="2px;"></mspace> <mo arg="1" intent="double-factorial">!!</mo> </mrow> ``` ```xml <mrow intent="double-factorial($1)"> <mi arg="1">x</mi> <mo>!</mo> <mo>!</mo> </mrow> ``` ```xml <mrow intent="$1($2($3,$4))"> <mo>(</mo> <mi arg="3">x</mi> <mo arg="2" intent="plus">+</mo> <mi arg="4">y</mi> <mo>)</mo> <mo arg="1" intent="double-factorial">!!</mo> </mrow> ```

### MathML Difficulty 2 (D2) ### No Canonical Content

### D2.1 No canonical concept names (aliases) * Which other names stand for a [double-factorial](https://en.wikipedia.org/wiki/Double_factorial)? * semifactorial, odd factorial (sub-case depending on input parity) * internationalization: * "подвійний-факторіал" (Ukranian), * "двоен-факториeл" (Bulgarian), * "двойной-факториал" (Russian) * "duplo fatorial" (Polish), * ... * By extension - no canonical narration.

### No canonical concept definitions (evolving methods) <br> Ok, we have a factorial. But which one was actually [computed](https://hbfs.wordpress.com/2020/03/31/factorial-approximations/)? <br> ```xml <mo arg="1" intent="factorial" aria-describedby="factorial-method-id-n">!</mo> ``` Possible cases: ```html <span id="factorial-method-id-1">computed via direct multiplication</span> <span id="factorial-method-id-2">computed via Stirling's formula</span> <span id="factorial-method-id-3">computed via Gosper's approximation</span> <span id="factorial-method-id-4">computed via Burnside's approximation</span> <span id="factorial-method-id-5">computed via Mortici's approximation</span> <span id="factorial-method-id-6">computed via Mohanty and Rummens's approximation</span> ```

### D2.2 No canonical content trees ### $1 \leq t \leq 2$ * Choice 1: n-ary relation ```xml <apply> <leq></leq> <cn type="integer">1</cn> <ci id="t">t</ci> <cn type="integer">2</cn> </apply> ``` * Choice 2: $1 \leq t$ and $t \leq 2$ ```xml <apply> <and></and> <apply> <leq></leq> <cn type="integer">1</cn> <ci id="t">t</ci> </apply> <apply> <leq></leq> <share href="#t"></share> <cn type="integer">2</cn> </apply> </apply> ``` * Choice 3: $t \in [1,2]$ ```xml <apply> <in></in> <ci>t</ci> <interval closure="closed"> <cn type="integer">1</cn> <cn type="integer">2</cn> </interval> </apply> ```

### D2.3 Every unit of presentation is overloaded * makes it hard to choose defaults, * makes it hard to draw "subject area" lines, * automatic "conceptual" narration not reliable without proofreading, * examples: the well-known symbols of arithmetic, overloaded in advanced fields of study. * do not map to the usual `<cn>` and `<plus>` elements in Content MathML, * would need dedicated `<csymbol>` entries.

### D2.3 Every unit of presentation is overloaded (examples) 1. zero; in a lecture for the [de Rham complex](https://youtu.be/QLnzIOGIvfo?t=4927): ```xml <mi aria-describedby="not-a-zero">0</mi> ``` ```html <span id="not-a-zero">the vector space consisting of only one element</span> ``` 2. one; when using the [indicator function](https://en.wikipedia.org/wiki/Indicator_function) ```xml <mn mathvariant="bold" intent="indicator-function">1</mn> ``` 3. plus; when representing the [kleene plus](https://en.wikipedia.org/wiki/Kleene_star#Kleene_plus) ```xml <mo intent="kleene-plus">+</mo> ```

### Future 1: find a [Happy Path](https://en.wikipedia.org/wiki/Happy_path) * What are great software steps for reaching accessible narration? * What are great vehicles for annotation? * Authoring tools or post-publication remediation * Advanced ecosystem tools can be a huge boost * Document-level declarations can save a lot of typing * QA via automatic validation during authoring / on submission * Custom "rule sets" need more prototyping, could have promise * Is a successful vision for AT a thin client or a heavy expert system? * What is the least unhappy path for error-reporting, debugging and adding corrections? Idea: start with the simplest syntax and increment complexity **after** prototypes demand it.

### Future 2: Consolidate goals * Accessibility annotations are not trivial, need focused work. * Any "default" rule set will induce **canonical** choices over MathML Core. * A lot of the ambitious ideas may fail in practice * Prototypes, tests? Timeline & investment. * Content MathML needs its own **new** Happy Path, and needs an even bigger investment than AT.

### Proposal Summary *a convention so simple, you don't have to read the spec* 1. Encyclopedic Names 2. Restricted, NNRFC syntax 3. "Western K12 Education" rule set 4. "Read as Rendered" rule set 5. ARIA interop

### Thank you! <br> "remember when viewing source would actually teach us something about how the website was built…" by [@LeaVerou](https://twitter.com/LeaVerou/status/1461035927212638214) ```xml <mrow intent="$1($2)"> <mi arg="2">x</mi> <mo arg="1" intent="double-factorial">!!</mo> </mrow> ```