33 karma · joined April 7, 2016
https://github.com/approach0/search-engine
Appeoach0 is a math-aware search engine. It is about to publish its first source-code release very recently.
As for my "works pretty well", please refer to my answer in another thread below. To be concise, I use subjective words on search engine effectiveness because NTCIR makes it difficult to compare my TeX search engine with "MathML search engine". But I have already shown better efficiency of my engine compared to Tangent, and an important factor is Tangent have to use LaTeXML to parse every TeX back to MathML. Without considering NTCIR, I am willing to make a comparison (probably after done my new version search engine) with some open-source established math engine (e.g. Tangent) on effectiveness and efficiency based on some corpus with both MathML (used by Tangent) and TeX (used by my engine) annotation.
At very first I tried to compare my results (MAP, recall, precision) with participants in NTCIR, but I take a lot efforts to get dataset, after which I find I cannot convert MathML back into TeX very confidently, most importantly, my parser-generated tree structure is fine-tuned and very dependent on TeX input, I cannot just take MathML tree structure directly, I need much more efforts than just importing an existing XML parser. Because of these, I can not compare my results with mainstream NTCIR researchers. But I definitely tried very hard, sadly I give up. If NTCIR someday can provide (even if request is needed) TeX data for competition, I will consider to (and able to, willing to) compare my results with NTCIR participants (in order to "prove" it).
Writing a TeX parser only for math search is not that difficult, I have written it, it parses most user-created document on math.stackexchange.com. Although I cannot convince you I get better results, I can argue parsing search-interested TeX subset is effortless (if you only care math-related TeX), I even opensourced my search engine TeX parser. Again, problem is not that easy to grab a XML parser and reuse it in my project, I believe a good math-aware search engine needs to get a tree structure very different from that a MathML structure represents, you get a tree by reusing MWS praser, so WHAT? That tree is not the tree I want, I need a lot effort to convert it, the easy way for me is to convert MathML back into TeX (Since I have already done that from TeX), sadly it turns out to be too complicated to worth giving a shot.
2. I wish NTCIR corpus is not that difficult to download (I once wrote a request for NTCIR corpus, but no one replies), please make it public accessible just like what MIaS does: https://mir.fi.muni.cz/mias/
3. My search engine (http://tkhost.github.io/opmes) is actually using structural method, but I still give up MathML and go parsing TeX directly instead. Why? In TeX I can just omit irrelevant command like "\color" and "\mbox", and only focus on a handful math-related TeX subset, and the result is great. Although my search engine can just handle "toy formula syntax", but maybe it is better than MathWebSearch (https://zbmath.org/formulae/) and even beat Tangent (http://saskatoon.cs.rit.edu/tangent/random) in long query. But in MathML, I have no idea why I need to read its lengthy spec, and I see no reason to write a MathML parser.
NTCIR-math conference (and its none-friendly website) makes me unwilling to submit a single paper.
BTW, if we want a <math> tag that no one will write by hand and only machine will try to understand, then think about why not HTML being designed as some open binary format in the first place?
Converting MathML back into LaTeX is possible but error-prone for most moderate-complex expressions (I tried it using haskell pandoc). This makes math-aware search engines have to include a MathML parser. And the most popular digital math document are still mostly written in LaTeX, math search engine thus needs another tool (e.g. LaTeXML) to convert LaTeX to much more lengthy MathML stuffs. As a researcher in this field, all I see is MathML brings a lot overhead to our life.
I think LaTeX is still the ideal way to "input" math expression, it is human-friendly and most commonly used math input. While WEB standard should focus on "rendering" LaTeX. I have to point it out that I am pretty comfortable about what MathJax provides, but if there needs to be a WEB standard on math, I wish some day the standard way to write math expression in HTML is something like this: <math> x = \frac{-b \pm \sqrt{b^2 - 4ac}} {2a} </math>
Instead of: <math display="block"> <mrow> <mi>x</mi> <mo>=</mo> <mfrac> <mrow> <mo>−</mo> <mi>b</mi> <mo>±</mo> <msqrt> <mrow> <msup> <mi>b</mi> <mn>2</mn> </msup> <mo>−</mo> <mn>4</mn> <mi>a</mi> <mi>c</mi> </mrow> </msqrt> </mrow> <mrow> <mn>2</mn> <mi>a</mi> </mrow> </mfrac> </mrow> </math>