Skip to content
Snippets Groups Projects
Select Git revision
  • master default protected
  • ar5iv-04-2024-dataset
  • grounding-dataset-v1
  • fix-sidebar-gitlab-link
  • 13-collect-a-sigmathling-bibliography-2
  • add_bibliography
  • arxmliv-2018
7 results

arxmliv-embeddings-082018.md

Blame
  • layout: page
    title: arXMLiv 08.2018 - Word Embeddings; Token Model

    Part of the arXMLiv project at the KWARC research group

    Author

    • Deyan Ginev

    Current release

    • 08.2018

    Accessibility and License

    The content of this Dataset is licensed to SIGMathLing members for research and tool development purposes.

    Access is restricted to SIGMathLing members under the SIGMathLing Non-Disclosure-Agreement as for most arXiv articles, the right of distribution was only given (or assumed) to arXiv itself.

    Contents

    • An 11.5 billion token model for the arXMLiv 08.2018 dataset, including subformula lexemes
      • token_model.zip
    • 300 dimensional GloVe word embeddings for the arXMLiv 08.2018 dataset
      • glove.arxmliv.5B.300d.zip and vocab.arxmliv.zip
    • 300d GloVe word embeddings for individual subsets
      • glove.subsets.zip
    • Embeddings and vocabulary with math lexemes omitted: glove.arxmliv.nomath.11B.300d.zip and vocab.arxmliv.nomath.zip
      • added on July 20, 2019
      • used as a control when evaluating the contribution of formula lexemes
    • the main arXMLiv dataset is available separately here

    Token Model Statistics

    subset documents paragraphs sentences
    no_problem 137,864 4,646,203 21,533,963
    warning 705,095 45,797,794 183,246,777
    error 389,225 26,759,524 99,641,978
    complete 1,232,184 77,203,521 304,422,718
    subset words formulas inline cite numeric literals
    no_problem 430,217,995 20,910,732 3,709,520 11,177,753
    warning 3,175,663,430 281,832,412 25,337,574 83,606,897
    error 1,731,971,035 153,186,264 13,145,561 43,399,720
    complete 5,337,852,460 455,929,408 42,192,655 138,184,370

    GloVe Model Statistics

    subset tokens unique words unique words (freq 5+ )
    no_problem 622,968,267 715,433 219,304
    warning 7,203,536,205 3,478,235 666,317
    error 3,691,805,321 2,444,532 574,467
    complete 11,518,309,793 5,285,379 1,000,295

    Citing this Resource

    Please cite the main dataset when using the word embeddings, as they are generated and distributed jointly. Instructions here

    Download

    Download link (SIGMathLing members only)

    Generated via

    Generation Parameters

    • token model distributed as 3 subsets - no_problem, warning and error. complete model is derived via:

        cat token_model_no_problem.txt \
            token_model_warning.txt \
            token_model_error.txt > token_model_complete.txt
    • llamapun v0.2, corpus_token_model example used for token model extraction

      • processed logical paragraphs only (excluded non-textual modalities, e.g. tables, figures and their captions, bibliographies, as well as abstracts)
      • excluded paragraphs containing latexml errors (marked via a ltx_ERROR HTML class)
      • used llamapun math-aware sentence and word tokenization, with subformula math lexemes
      • marked up inline citations replaced with citationelement token
      • numeric literals replaced with NUM token (both in text and formulas)
      • ignored sentences with unnaturally long words (>25 characters) - almost always due to latexml conversion errors
    • GloVe repository at sha 07d59d

      • build/vocab_count -min-count 5
      • build/cooccur -memory 32.0 -window-size 15
      • build/shuffle -memory 32.0
      • build/glove -threads 16 -x-max 100 -iter 25 -vector-size 300 -binary 2

    Examples and baselines

    GloVe in-built evaluation (non-expert tasks e.g. language, relationships, geography)

    1. no_problem
    • Total accuracy: 27.47% (4028/14663)
    • Highest score: "gram3-comparative.txt", 75.30% (1003/1332)
    1. warning
    • Total accuracy: 31.97% (5351/16736)
    • Highest score: "gram3-comparative.txt", 77.40% (1031/1332)
    1. error
    • Total accuracy: 29.67% (4910/16549)
    • Highest score: "gram3-comparative.txt", 72.75% (969/1332)
    1. complete
    • Total accuracy: 35.48% (6298/17750)
    • Highest score: "gram3-comparative.txt", 76.65% (1021/1332)
    1. demo baseline: text8 demo (first 100M characters of Wikipedia)
    • Total accuracy: 23.62% (4211/17827)
    • Highest score: "gram6-nationality-adjective.txt", 58.65% (892/1521)

    Evaluation note: These in-built evlauation runs are provided as a sanity check that the generated GloVe models pass a basic baseline against the non-expert tasks in the default GloVe suite. One would need a scienctific discourse tailored set of test cases to evaluate the arXiv-based models competitively.

    Measuring word analogy

    In a cloned GloVe repository, start via:

    python2 eval/python/word_analogy.py --vocab_file vocab.arxmliv.txt --vectors_file glove.arxmliv.11B.300d.txt
    1. abelian is to group as disjoint is to ?
    • Top hit: union, cosine distance 0.653377
    1. convex is to concave as positive is to ?
    • Top hit: negative, cosine distance 0.786877
    1. finite is to infinte as abelian is to ?
    • Top hit: nonabelian, cosine distance 0.676042
    1. quantum is to classical as bottom is to ?
    • Top hit: top, cosine distance 0.734896
    1. eq is to proves as figure is to ?
    • Top hit: showing, cosine distance 0.668502
    1. italic_x is to italic_y as italic_a is to ?
    • Top hit: italic_b, cosine distance 0.902608

    Nearest word vectors

    In a cloned GloVe repository, start via:

    python2 eval/python/distance.py --vocab_file vocab.arxmliv.txt --vectors_file glove.arxmliv.11B.300d.txt
    1. lattice
    Word: lattice  Position in vocabulary: 488
    
                                   Word       Cosine distance
    
    ---------------------------------------------------------
    
                               lattices		0.853103
    
                             triangular		0.637767
    
                              honeycomb		0.626426
    
                                crystal		0.624397
    
                                 finite		0.614720
    
                                spacing		0.603067
    1. entanglement
    Word: entanglement  Position in vocabulary: 1568
    
                                   Word       Cosine distance
    
    ---------------------------------------------------------
    
                              entangled		0.780425
    
                           multipartite		0.730968
    
                            concurrence		0.691708
    
                             negativity		0.649595
    
                             tripartite		0.647623
    
                                quantum		0.640395
    
                               fidelity		0.640285
    
                          teleportation		0.616797
    
                                discord		0.613752
    
                                entropy		0.612341
    
                              bipartite		0.608034
    
                              coherence		0.606859
    
                            nonlocality		0.601337
    1. forgetful
    Word: forgetful  Position in vocabulary: 11740
    
                                   Word       Cosine distance
    
    ---------------------------------------------------------
    
                                functor		0.723472
    
                               functors		0.656184
    
                               morphism		0.598965
    1. eigenvalue
    Word: eigenvalue  Position in vocabulary: 1448
    
                                   Word       Cosine distance
    
    ---------------------------------------------------------
    
                            eigenvalues		0.893073
    
                            eigenvector		0.768380
    
                           eigenvectors		0.765241
    
                          eigenfunction		0.754222
    
                         eigenfunctions		0.686141
    
                             eigenspace		0.666098
    
                                  eigen		0.641422
    
                                 matrix		0.616723
    
                              eigenmode		0.613117
    
                             eigenstate		0.612188
    
                              laplacian		0.611396
    
                                largest		0.606122
    
                               smallest		0.605342
    
                             eigenmodes		0.604839
    
    1. riemannian
    Word: riemannian  Position in vocabulary: 2285
    
                                   Word       Cosine distance
    
    ---------------------------------------------------------
    
                              manifolds		0.765827
    
                               manifold		0.760806
    
                                 metric		0.719817
    
                                finsler		0.687826
    
                              curvature		0.676100
    
                                  ricci		0.664770
    
                                metrics		0.660804
    
                             riemmanian		0.651666
    
                              euclidean		0.644686
    
                             noncompact		0.643878
    
                            conformally		0.638984
    
                              riemanian		0.633814
    
                                 kahler		0.632680
    
                                endowed		0.622035
    
                            submanifold		0.613868
    
                           submanifolds		0.612716
    
                               geodesic		0.604488