AITP16
Math NLP
Corpora
arXiv
LaTeXML
https://creativecommons.org/licenses/by/4.0/
Research overview on large-scale natural language processing on math-rich scientific documents, as carried out by the KWARC research group.
2020-12-05
class: center, middle ### Language and Mathematics Model Pretraining in 2021 <br> *Preparing the large open datasets* <br><br> .footnote[**Deyan Ginev**] .remark-venue[SIGMathLing Seminar<br> December 7, 2020] <div style="position:absolute; top: 12px; right: 20px;"> <div style="position:relative;"> <a rel="license" href="https://creativecommons.org/licenses/by/4.0/" title="This work is licensed under a Creative Commons Attribution 4.0 International License"> <img alt="Creative Commons License" style="border-width:0" width="80" src="https://mirrors.creativecommons.org/presskit/buttons/80x15/png/by.png" /> </a> </div> </div> --- ## Quick History of arXMLiv * KWARC+NIST since 2006, converting arXiv to HTML via LaTeXML. * LaTeXML: main workhorse of DLMF, [arxiv-vanity.com](arxiv-vanity.com), [ARQMath](https://www.cs.rit.edu/~dprl/ARQMath/) * arXiv has accelerating growth / moving target <br><br><br> <div style="margin-left:-35px;"><img src="/sigmathling20/arxiv_growth_2020.png" width="650px"></div> --- ## Quick History of arXMLiv, 2020 * SIGMathLing annual releases: [dataset, embeddings, statements](https://sigmathling.kwarc.info/resources/arxmliv/) * \>96% of sources can be converted, now \>70% error-free <br><br><br> <div style="margin-left:-55px;"><img src="/sigmathling20/arxmliv_history_2020.png" width="650px"></div> --- ### Increase in using arXiv for Neural NLP 1. KWARC/NIST: [Scientific Statement Classification](https://www.aclweb.org/anthology/2020.lrec-1.153/) 2. AllenAI: [SciREX](https://github.com/allenai/SciREX) (uses latexml), [s2orc](https://github.com/allenai/s2orc), [SciTLDR](https://arxiv.org/abs/2004.15011) 3. OpenAI: [ATP for Metamath](https://arxiv.org/abs/2009.03393) 4. Google: [Autoformalization](https://link.springer.com/chapter/10.1007/978-3-030-53518-6_1) effort 5. Montréal: [Abstractive Neural Document Summarization](https://arxiv.org/abs/1909.03186) 6. Uni Maryland: [Logical structure](https://arxiv.org/abs/1709.00770) 7. a surprising variety of other incremental results ### arXiv is Openly Available as TeX and PDF - [s3](https://arxiv.org/help/bulk_data_s3) or [kaggle](https://www.kaggle.com/Cornell-University/arxiv) downloads - but you save a month+ of work by reusing our [html data](https://sigmathling.kwarc.info/resources/arxmliv/) --- ## Agenda 1. <span style='color:blue;'>Pretraining</span> 2. Large Open Datasets 3. Tokenizing Mathematics --- ## Pretraining .center[ Learning a task-agnostic representation of natural language,<br> via **self-supervision** over large amounts of unlabeled data.] <br> .center[ **+ Mathematics**: add high-quality tokenization of math syntax] --- count: false ## Pretraining .center[ Learning a task-agnostic representation of natural language,<br> via **self-supervision** over large amounts of unlabeled data.] <br> .center[ **+ Mathematics**: add high-quality tokenization of math syntax] ### How? - Classic: per-word global vector embeddings; - 2018-2020: **autoregressive** (AR) and **autoencoding** (AE) neural models.<br><br> - Variety of specific learning tasks, e.g. - generalized AR (xlnet), - masked (MLM), - replaced token detection (RTD);<br><br> - Employ subword tokenization (BPE, WordPiece) --- class: center, middle ### Why? Symbol grounding + Syntax --- class: center ## Pretraining Data in NLP model | size ($10^9$ tokens) | year :-------|------------------------:|----: BERT | 3.3 | 2018 Electra | 3.3 | 2020 ELMO | 5.5 | 2018 GPT-2 | 10 | 2018 <span style='color:gray'>Human</span> | <span style='color:gray'>~14</span> | <span style='color:gray'>-</span> XLNet | 33 | 2019 T5 | 34 | 2019 GPT-3 | 499 | 2020 GloVe | 840 | 2014 .remark-venue[<span style='color:gray'>Human = 500 words/minute x 16 h/day x 80 years] --- count: false class: center ## Pretraining Data in NLP model | size ($10^9$ tokens) | year :-------|------------------------:|----: BERT | 3.3 | 2018 <span class="circled-td">Electra</span> | 3.3 | 2020 ELMO | 5.5 | 2018 GPT-2 | 10 | 2018 <span style='color:gray'>Human</span> | <span style='color:gray'>~14</span> | <span style='color:gray'>-</span> XLNet | 33 | 2019 T5 | 34 | 2019 GPT-3 | 499 | 2020 <span class="circled-td">GloVe</span> | 840 | 2014 ✅ We've validated pretraining with arXMLiv 08.2019 .remark-venue[<span style='color:gray'>Human = 500 words/minute x 16 h/day x 80 years] --- ## Agenda 1. Pretraining 2. <span style='color:blue;'>Large Open Datasets</span> 3. Tokenizing Mathematics --- class: center ## Primary Resources (WIP) corpus | size ($10^9$ tokens) | genre :-------|------------------------:|----: PubMed Central | ~30 | scientific articles arXMLiv 2020 | ~17.5 | scientific articles <span style='color:gray'>Human</span> | <span style='color:gray'>~14</span> | <span style='color:gray'>-</span> StackExchange | (?) | Q&A PubMed | 3.7 | abstracts Wikipedia | 3 | encyclopedic Elsevier OA | ~0.4 | scientific articles WikiHow | ~0.2 | tutorials Art of problem solving | ~0.1 | encyclopedic PlanetMath | ~0.1 | encyclopedic ProofWiki | ~0.1 | encyclopedic Wiktionary | ~0.1 | dictionary deepmind AQuA | ~0.1 | math syntax Q&A deepmind math | ∞ | problem generator .remark-venue[<span style='color:gray'>Human = 500 words/minute x 16 h/day x 80 years] --- class: center ## Primary Resources (Non-math) corpus | size ($10^9$ tokens) | genre :-------|------------------------:|----: CommonCrawl | ~1000 | entire web Books 3 | 55 | books, 200k S2ORC | (?) | scientific articles, 13M Project Gutenberg | (?) | books, 60k Rational wiki | (?) | encyclopedic vikidia | (?) | encyclopedic wikibooks | (?) | encyclopedic wikiversity | (?) | encyclopedic --- ## Preprocessing Task 1. detect content pieces from each source dataset 2. normalize down HTML dialect and other markup 3. map custom markup to canonical arXMLiv-near HTML 4. map custom math syntax to canonical lexemes (via LaTeXML) 5. serialize as plain-text ready for pretraining input. --- class:center,middle Let us do a brief dataset tour.<br><br><br> Running example: <br><br><strong>martingale</strong> --- ### Samples: arXMLiv (1), martingale .footnote[source: [1806.05901](https://corpora.mathweb.org/preview/arxmliv/tex_to_html/1806.05901?style=simple) ] >`A probability measure blackboard_Q is an equivalent local martingale measure if blackboard_Q is equivalent to blackboard_P and every italic_X ∈ caligraphic_X ( 1 ) is a local martingale under blackboard_Q. We denote the family of equivalent local martingale measures by caligraphic_M and assume that...` --- ### Samples: arXMLiv (2), martingale .footnote[Source [1806.10798](https://corpora.mathweb.org/preview/arxmliv/tex_to_html/1806.10798?style=simple) ] > `italic_M start_sub italic_n + 1 end_sub start_sup ( italic_i ) end_sup , italic_i = 1 , 2 ,` > > `are martingale difference sequences with respect to the increasing italic_σ -fields` > > `caligraphic_F start_sub italic_n end_sub := italic_σ ( italic_x start_sub italic_m end_sub , italic_y start_sub italic_m end_sub , italic_M start_sub italic_m end_sub start_sup ( italic_i ) end_sup ; italic_i = 1 , 2 ; italic_m ≤ italic_n ) , italic_n ≥ 0 .` --- ### Samples: PubMed Central, martingale .footnote[source: `manuscript_subset/PMC0048XXXXX/PMC4853299.xml`] >`...where <italic>M<sub>i</sub></italic>(<italic>u</italic>) is a zero mean martingale. The <italic>i</italic>th term is a stochastic integral of a predictable vector process with respect to a martingale. Thus, <italic>D</italic> is itself a mean 0 vector-valued martingale. This proves the consistency of the ζ -estimator.` >`The <italic>i</italic> th term is a stochastic integral of a predictable vector process with respect to a martingale. Thus, <italic>D</italic> is itself a mean 0 vector valued martingale. This proves (ii).` In PMC, Math syntax can appear as simple HTML, raw tex, or pres. MathML. --- ### Samples: WikiHow, martingale .footnote[source: `Preparing for Dog Training, How to Train a Dog`] > A 6-foot leash and flat collar or **martingale** collar may be all the you need to start, besides your treats. > Consult a trainer for advice on other equipment like a “Promise Leader” head halter, a “No Pull” harness, a metal training collar, or other device. Puppies or small dogs generally do not need harsh equipment. --- ### Samples: Elsevier OA, martingale .footnote[source: `S0001870819303494.json`] <div class="middle center"><img src="/sigmathling20/elsevierOA_martingale.png" width="500px"></div> --- ### Samples: Rationalwiki, martingale .footnote[source: `Gambling`] >Over the long term, a cursory understanding of probability tells us that any finitely long streak of losses is guaranteed to occur eventually, giving the **martingale** a probability of ruin equal to one. --- ### Samples: Project Gutenberg, martingale .footnote[source: `The Chemistry of Cookery.53458.html`] >We were in the Mediterranean, where bonetta were very abundant, and every morning and evening I amused myself by spearing them from the **martingale** of the schooner, and so successfully that all hands (or rather mouths) were abundantly supplied with this delicious dark-fleshed, full-blooded, and high-flavoured fish. Gutenberg has a rich set of classical `martingale` uses. Deyan(EFL) surprise: third meaning! [Dolphin striker](https://en.wikipedia.org/wiki/Dolphin_striker) --- ### Aside: multi-modal grounding, Wikipedia .footnote[[dolphin striker](https://en.wikipedia.org/wiki/Dolphin_striker) (an older term for a martingale boom or simply a **martingale**; sometimes called a striker)] <div class="center"><img src="/sigmathling20/martingale_multimodal.png" width="300px"></div> cf. WS3, [Experience Grounds Language (EMNLP 2020)](https://www.aclweb.org/anthology/2020.emnlp-main.703/) --- ### Samples: PlanetMath, martingale .footnote[source: `60J65-LevyMartingaleCharacterization.tex`] ``` \begin{mythm}[Levy's martingale characterisation] Let $W(t), t\geq 0$, be a stochastic process and let $\mathcal{F}_t = \sigma(W_s,s\leq t)$ be the filtration generated by it. Then $W(t)$ is a Wiener process if and only if the following conditions hold: \begin{enumerate} \item $W(0)=0$ almost surely; \item The sample paths $t \mapsto W(t)$ are continuous almost surely; \item $W(t)$ is a martingale with respect to the filtration $\mathcal{F}_t$; \item $|W(t)|^2-t$ is a martingale with respect to $\mathcal{F}_t$. \end{enumerate} \end{mythm} ``` PlanetMath has a rich set of mathematical `martingale` uses. --- ### Samples: StackExchange (1), martingale .footnote[source: `cs/question/14073.html`] ``` Is there a universal lower semicomputable $\lambda$-supermartingale $t$ and a binary sequence $\tilde\omega$ such that $t$ always "goes up" along $\tilde\omega$ but never succeeds? ``` ----- .footnote[source: `cs/question/12113.html`] ``` I doubt that the professor will use $\sigma$-algebras and the like (unless they will mention martingales and will prefer to discuss them using these terms; no one would be able to follow). ``` --- ### Samples: StackExchange (2), martingale .footnote[source: `datascience/question/38548.html`] ``` In terms of prediction into the future, the price behaves as a <a href="https://en.wikipedia.org/wiki/Martingale_(probability_theory)" rel="nofollow noreferrer">martingale</a>, i.e. the best prediction for tomorrow's price is today's price. ``` ----- .footnote[source: `ai/question/20272.html`] ``` There's a lot of substantial and difficult theory in this area requiring measure theory leading to martingales and stochastic approximations. ``` --- ### Samples: StackExchange (3), martingale .footnote[source: `codereview/question/61901.html`] <div class="center"><img src="/sigmathling20/martingale_hs.png" width="450px"></div> --- ### Samples: StackExchange (4), martingale .footnote[source: `math/question/932368.html`] ``` Let $Z_n=\prod_{k=1}^n(1+X_k)$, so $Z_n$ a martingale. ``` ----- <br><br> .footnote[source: `mathoverflow/question/350579.html`] ``` Inequality (0) is also a special case of <a href="https://projecteuclid.org/euclid.aop/1176988477" rel="nofollow noreferrer">Theorem 3.1</a> for martingales in <span class="math-container">$(2,D)$</span>-smooth separable Banach spaces; note that a Hilbert space is <span class="math-container">$(2,1)$</span>-smooth. ``` --- ### Samples: Wikipedia, martingale .footnote[source: `Doob_decomposition_theorem`] ``` Then there exists a martingale <span class="texhtml "><i>M</i> = (<i>M<sub>n</sub></i>)<sub><i>n</i>∈<i>I</i></sub></span> and ... ``` ### Defining Pages STEM subset (via Kiwix) | other :----------|:------- 'Martingale_(betting_system)' | Martingale (tack) Martingale_central_limit_theorem | Martingale (collar) Martingale_difference_sequence | Martingale (+fencing strap) Martingale_pricing | Martingale (+theatrical twofer) 'Martingale_(probability_theory)' | Martingale_representation_theorem | --- ### Samples: Wiktionary, martingale .footnote[source: `forrard`] ``` <div class="citation-whole"><span class="cited-source"><b>1858</b>, Various, <cite>The Atlantic Monthly, Volume 2, Issue 12, October, 1858</cite><span></span> <sup><a class="external autonumber external" href="http://www.gutenberg.org/etext/10435"></a></sup>:</span> <dl><dd><div class="h-quotation"><span class="Latn e-quotation"> Here and there an adventurous genius coins a phrase for the benefit of posterity,--as we once heard a mate order a couple of men to "go <b>forrard</b> and trim the ship's whiskers," to the utter bewilderment of his captain, who, in thirty years' following of the sea, had never heard the martingale chains and stays so designated. </span></div></dd></dl></div></li> ``` --- ### Samples: Random Web Search .footnote[source: `www.horsenation.com/2012/08/15/the-aiding-abetting-amateur-where-did-the-word-martingale-come-from/`] >For you entertainment, we have asked non-horsey friends to define this term. Their responses are full of creativity and hilarity: > - martingale: n. a small bird with a high-pitched, song-like chirp. > - Martingale: N. (pronounced: Martin Ale) A fine international (or rather, intergalactic) pale ale brewed in Martin breweries from the finest hops known to man or Martian. > - martingale: n. a strong gusting wind accompanied by summer storms, known to take sea-voyagers off route. --- ## Agenda 1. Pretraining 2. Large Open Datasets<br><br> - Tracking issue at [KWARC/llamapun#59](https://github.com/KWARC/llamapun/issues/59) 3. <span style='color:blue;'>Tokenizing Mathematics</span> --- class:padded-table ## Tokenizing Math Syntax Preprocessing choices| `\mysqrt{a\by b}` :--------------------|:-------- placeholder token |`MathFormula` keep as-is |`\mysqrt{a\by b}` expand to primitives |`\sqrt{a \times b}` scrape PDF | `√ a × y` latexml lexemes | `square-root start_ARG` | `italic_a × italic_b end_ARG` fully discard | > Mainstream NLP datasets make **lossy** choices that > > lead to **noisy** or **damaged** math syntax tokens. --- ### Example 1 (arXiv:1703.06815, page 2) <div class="center middle boxed"> <img src='/sigmathling20/rewrite_example.png' width='700' /> </div> <br><br> 1) original TeX ``` We use the shorthand $\ifor{\theta}{I}$ for the i-formula formed from the formula $\theta$ and the instant $I$ by replacing all literals $L$ occurring in $\theta$ by $\ifor{L}{I}$, e.g. $\ifor{\lit{F}{V} \rightarrow \lit{F'}{V'}}{3}$ is a shorthand for $\ifor{\lit{F}{V}}{3} \rightarrow \ifor{\lit{F'}{V'}}{3}$. ``` --- count: false ### Example 1 (arXiv:1703.06815, page 2) <div class="center middle boxed"> <img src='/sigmathling20/rewrite_example.png' width='700' /> </div> <br><br> 2) pandoc ``` We use the shorthand $\ensuremath{[\theta]@I}$ for the i-formula formed from the formula $\theta$ and the instant $I$ by replacing all literals $L$ occurring in $\theta$ by $\ensuremath{[L]@I}$, e.g. $\ensuremath{[\mbox{\ensuremath{ F \! = \! V}} \rightarrow \mbox{\ensuremath{ F' \! = \! V'}}]@3}$ is a shorthand for $\ensuremath{[\mbox{\ensuremath{ F \! = \! V}}]@3} \rightarrow \ensuremath{[\mbox{\ensuremath{ F' \! = \! V'}}]@3}$. ``` --- count: false ### Example 1 (arXiv:1703.06815, page 2) <div class="center middle boxed"> <img src='/sigmathling20/rewrite_example.png' width='700' /> </div> <br><br> 3) textract/pdftotext ``` We use the shorthand [θ]@I for the i-formula formed from the formula θ and the instant I by replacing all literals L occurring in θ by [L]@I, e.g. [F = V → F ′ = V ′ ]@3 is a shorthand for [F = V ]@3 → [F ′ = V ′ ]@3. ``` --- count: false ### Example 1 (arXiv:1703.06815, page 2) <div class="center middle boxed"> <img src='/sigmathling20/rewrite_example.png' width='700' /> </div> <br><br> 4) latexml+llamapun: ``` We use the shorthand [ italic_θ ] @ italic_I for the i-formula formed from the formula italic_θ and the instant italic_I by replacing all literals italic_L occurring in italic_θ by [ italic_L ] @ italic_I , e.g. [ italic_F = italic_V → italic_F start_sup ′ end_sup = italic_V start_sup ′ end_sup ] @ 3 is a shorthand for [ italic_F = italic_V ] @ 3 → [ italic_F start_sup ′ end_sup = italic_V start_sup ′ end_sup ] @ 3 . ``` --- ### Example 2 (arXiv:cond-mat/0004345) <div class="center middle boxed"> <img src='/sigmathling20/genfrac_example.png' width='500' /> </div> <br><br> 1) original TeX: ``` V_\cC(\rho ,\varphi ,z)=\sum _{\fracnul{K,M=-\infty} {(Mr=-K\,{\rm mod}\frac{q}{n}})} ^{+\infty}\, \alpha^M_K(\rho )e^{\ri nM\varphi}e^{\ri (2\pi /a)Kz} ``` --- count: false ### Example 2 (arXiv:cond-mat/0004345) <div class="center middle boxed"> <img src='/sigmathling20/genfrac_example.png' width='500' /> </div> <br><br> 2) pandoc: ``` V_{\cal C}(\rho ,\varphi ,z)=\sum _{\genfrac{}{}{0pt}{}{K,M=-\infty} {(Mr=-K\,{\rm mod}\frac{q}{n}})} ^{+\infty}\, \alpha^M_K(\rho )e^{\mathrm{i}nM\varphi}e^{\mathrm{i}(2\pi /a)Kz} ``` --- count: false ### Example 2 (arXiv:cond-mat/0004345) <div class="center middle boxed"> <img src='/sigmathling20/genfrac_example.png' width='500' /> </div> <br><br> 3) textract/pdftotext ``` VC (ρ, ϕ, z) = +∞ X K,M =−∞ q (M r=−K mod n inMϕ i(2π/a)Kz αM e (11) K (ρ)e ) 5 ``` --- count: false ### Example 2 (arXiv:cond-mat/0004345) <div class="center middle boxed"> <img src='/sigmathling20/genfrac_example.png' width='500' /> </div> <br><br> 4) latexml+llamapun: ``` italic_V start_sub caligraphic_C end_sub ( italic_ρ , italic_φ , italic_z ) = ∑ start_sub FRACOP start_ARG italic_K , italic_M = - ∞ end_ARG start_ARG ( italic_M italic_r = - italic_K roman_mod divide start_ARG italic_q end_ARG start_ARG italic_n end_ARG end_ARG ) end_sub start_sup + ∞ end_sup italic_α start_sup italic_M end_sup start_sub italic_K end_sub ( italic_ρ ) italic_e start_sup roman_i italic_n italic_M italic_φ end_sup italic_e start_sup roman_i ( 2 italic_π / italic_a ) italic_K italic_z end_sup ``` --- ### Example 3 (arXiv:1703.06815, page 2) <div class="center middle boxed"> <img src='/sigmathling20/piecewise_example.png' width='500' /> </div> <br><br> 1) original TeX: ``` \begin{equation}\label{eq:evalpproposition} \epsilon (p,W) = \begin{cases} P^+ & \text{if } W \mmodels \ifor{A}{I} \\ 1 - P^+ & \text{otherwise} \end{cases} \end{equation} ``` --- count:false ### Example 3 (arXiv:1703.06815, page 2) <div class="center middle boxed"> <img src='/sigmathling20/piecewise_example.png' width='500' /> </div> <br><br> 2) pandoc: ``` $$\label{eq:evalpproposition} \epsilon (p,W) = \begin{cases} P^+ & \text{if } W \mathrel{||}\joinrel\Relbar\ensuremath{[A]@I} \\ 1 - P^+ & \text{otherwise} \end{cases}$$ ``` --- count:false ### Example 3 (arXiv:1703.06815, page 2) <div class="center middle boxed"> <img src='/sigmathling20/piecewise_example.png' width='500' /> </div> <br><br> 3) textract/pdftotext ``` ( P+ if W ||= [A]@I (8) ǫ(p, W ) = + 1−P otherwise 9 ``` --- count:false ### Example 3 (arXiv:1703.06815, page 2) <div class="center middle boxed"> <img src='/sigmathling20/piecewise_example.png' width='500' /> </div> <br><br> 4) latexml+llamapun: ``` italic_ϵ ( italic_p , italic_W ) = { start_row start_cell italic_P start_sup + end_sup end_cell start_cell if italic_W ||= [ italic_A ] @ italic_I end_cell end_row start_row start_cell 1 - italic_P start_sup + end_sup end_cell start_cell otherwise end_cell end_row ``` --- class: center, middle ## Thank you! **Questions?** .remark-venue[SIGMathLing Seminar<br> December 7, 2020] --- count:false ### Bonus: arXiv TeX Brutalism (1) .footnote[source: [hep-th/9810208](https://arxiv.org/abs/hep-th/9810208)] ``` $\bullet$ The Casimir algebra of ${\cal {B}}$ $_{3} $ ,obtained i.e. by Hamiltonian reduction for the Lie algebras ${\cal {B}}$ $_{_3}$ $^{11}$. ``` ----- Gotchas: - `\bullet` as indent - $\cal{B}_3$ and $\cal{B}_3^{11}$ split between math modes. 6 MathML elements for two actual math expressions. ----- Lexematized: ``` ∙ The Casimir algebra of caligraphic_B start_sub 3 end_sub , obtained i.e. by Hamiltonian reduction for the Lie algebras caligraphic_B start_sub start_sub 3 end_sub end_sub start_sup 11 end_sup . ``` --- count:false ### Bonus: arXiv TeX Brutalism (2) .footnote[Source [1508.05350](https://arxiv.org/abs/1508.05350)] ``` % line 91: \font\mybb=msbm10 at 12pt \def\bb#1{\hbox{\mybb#1}} \def\bR {\bb{R}} \def\bC {\bb{C}} \def\bH {\bb{H}} \def\bO {\bb{O}} % line 158: are related to the division algebras $\bR,\bC,\bH,\bO$. ``` ----- Lexematized: ``` are related to the division algebras blackboard_R , blackboard_C , blackboard_H , blackboard_O . ``` --- count:false ### Bonus: arXiv TeX Brutalism (3) .footnote[source: [2010.16273](https://arxiv.org/abs/2010.16273)] ``` ($\lesssim$~1 min) $\sim$60\% ``` ----- Lexematized: ``` ( ≲ 1 min ) ∼ 60 % ```