Extension:Math/RESTBase
Configure Restbase to work with the Math extension
To install mathoid and restbase to work with the math extension the following steps are required:
Mathoid
npm i mathoid
cd node_modules/mathoid
edit config.yaml as follows
# Number of worker processes to spawn.
# Set to 0 to run everything in a single process without clustering.
# Use 'ncpu' to run as many workers as there are CPU units
num_workers: ncpu
# Log error messages and gracefully restart a worker if v8 reports that it
# uses more heap (note: not RSS) than this many mb.
worker_heap_limit_mb: 500
# Logger info
logging:
level: warn
# Statsd metrics reporter
metrics:
type: log
#host: localhost
#port: 8125
services:
- name: mathoid
# a relative path or the name of an npm package, if different from name
module: ./app.js
# optionally, a version constraint of the npm package
# version: ^0.4.0
# per-service config
conf:
port: 10042
# interface: localhost # uncomment to only listen on localhost
# more per-service config settings
svg: true
img: true
png: true
texvcinfo: true
speech: true
speech_on: false
chem: true
no_check: true
run mathoid as a screen session
screen
node server.js -c config.yaml
testing mathoid
Now the following test
curl -d 'q=E=mc^2' localhost:10042/mml
should return
<math xmlns="http://www.w3.org/1998/Math/MathML" display="block">
<semantics>
<mrow>
<mi>E</mi>
<mo>=</mo>
<mi>m</mi>
<msup>
<mi>c</mi>
<mn>2</mn>
</msup>
</mrow>
<annotation encoding="application/x-tex">E=mc^2</annotation>
</semantics>
</math>
RESTBase
npm i restbase
cd node_modules/restbase
edit config.yaml as follows
paths:
/{api:v1}:
x-modules:
# swagger options, overriding the shared ones from the merged specs (?)
- spec:
info:
version: 1.0.0-beta
title: Formulasearchengine REST API
description: >
This API aims to provide a preview of new features reagarding
Math rendering and search, supposed to be integrated to
the [wikimedia restbase api](https://wikimedia.org/api/rest_v1/?doc).
### High-volume access
- Don't perform more than 500 requests/s to this API.
- Set a unique `User-Agent` header that allows us to contact you
quickly. Email addresses or URLs of contact pages work well.
termsOfService: https://wikimediafoundation.org/wiki/Terms_of_Use
contact:
name: the Wikimedia Services team
url: http://mediawiki.org/wiki/RESTBase
license:
name: Apache2
url: http://www.apache.org/licenses/LICENSE-2.0
securityDefinitions: &wp/content-security/1.0.0
mediawiki_auth:
description: Checks permissions using MW api
type: apiKey
in: header
name: cookie
x-internal-request-whitelist:
- /http:\/\/[a-zA-Z0-9\.]+\/w\/api\.php/
# Override the base path for host-based (proxied) requests. In our case,
# we proxy https://{domain}/api/rest_v1/ to the API.
x-host-basePath: /api/rest_v1
x-route-filters:
- path: ./lib/normalize_title_filter.js
options:
redirect_cache_control: '{{options.purged_cache_control}}'
paths:
/media:
x-modules:
- path: v1/mathoid.yaml
options: '{{options.mathoid}}'
/testing:
x-modules:
- path: ../mathpipe.yaml
# options: '{{options.mathoid}}'
options: '{{options}}'
/{api:sys}:
x-modules:
- spec:
paths:
/mathoid:
x-modules:
- path: sys/mathoid.js
options: '{{options.mathoid}}'
/table:
x-modules:
- path: sys/table.js
options:
conf: '{{options.table}}'
/key_value:
x-modules:
- path: sys/key_value.js
/key_rev_value:
x-modules:
- path: sys/key_rev_value.js
/key_rev_latest_value:
x-modules:
- path: sys/key_rev_latest_value.js
/post_data: &sys_post_data
x-modules:
- path: sys/post_data.js
/events:
x-modules:
- path: sys/events.js
options: '{{options.events}}'
options: '{{options}}'
run restbase as a screen session
screen
node server.js -c config.yaml
testing restbase
Navigate to http://localhost:8081/wikimedia.org/v1/?doc you should be able to enter a texvc formula to the check command POST endpoing i.e. /media/math/check/{type} (See screenshot 1) In the response headers you should find something like
"x-resource-location": "4c0004393a88f350a93bcef62106d556c7fc827b"
If you copy the value 4c0004393a88f350a93bcef62106d556c7fc827b and paste it to the GET endpoint /media/math/render/{format}/{hash} as value you should be able to see the PNG image if you select that as an output format. (See screenshot 2)
Configuration Example for arbitrary domain in RESTBase setup
If you want to use domain other than wikimedia.org, you can follow below example;
- mathoid/config.yaml
# Number of worker processes to spawn.
# Set to 0 to run everything in a single process without clustering.
# Use 'ncpu' to run as many workers as there are CPU units
num_workers: 0
# Log error messages and gracefully restart a worker if v8 reports that it
# uses more heap (note: not RSS) than this many mb.
worker_heap_limit_mb: 250
# Logger info
logging:
level: trace
# streams:
# # Use gelf-stream -> logstash
# - type: gelf
# host: logstash1003.eqiad.wmnet
# port: 12201
# Statsd metrics reporter
metrics:
#type: log
#host: localhost
#port: 8125
services:
- name: mathoid
# a relative path or the name of an npm package, if different from name
module: ./app.js
# optionally, a version constraint of the npm package
# version: ^0.4.0
# per-service config
conf:
port: 10044
# interface: localhost # uncomment to only listen on localhost
# more per-service config settings
# the location of the spec, defaults to spec.yaml if not specified
# spec: ./spec.template.yaml
# allow cross-domain requests to the API (default '*')
#cors: '*'
# to disable use:
# cors: false
# to restrict to a particular domain, use:
# cors: restricted.domain.org
# content for the CSP headers
# csp: false # uncomment this line to disable sending them
# URL of the outbound proxy to use (complete with protocol)
# proxy: http://my.proxy.org:8080
# the list of domains for which not to use the proxy defined above
# no_proxy_list:
# - domain1.com
# - domain2.org
# the list of incoming request headers that can be logged; if left empty,
# the following headers are allowed: cache-control, content-length,
# content-type, if-match, user-agent, x-request-id
# log_header_whitelist:
# - cache-control
# - content-length
# - content-type
# - if-match
# - user-agent
# - x-request-id
# list of enabled renders
svg: true
img: true
png: true #new feature
speech: true #new feature
texvcinfo: true
speech_on: true
no_check: false
dpi: 180
svgo: false
# the user agent to use when issuing requests
# user_agent: service-template-node
# the template used for contacting the MW API
#mwapi_req:
# method: post
# uri: https://{{domain}}/w/api.php
# headers:
# user-agent: '{{user-agent}}'
# body: '{{ default(request.query, {}) }}'
# the template used for contacting RESTBase
#restbase_req:
# method: '{{request.method}}'
# uri: https://{{domain}}/api/rest_v1/{+path}
# query: '{{ default(request.query, {}) }}'
# headers: '{{request.headers}}'
# body: '{{request.body}}'
- restbase/config.yaml
services:
- name: restbase
module: hyperswitch
conf:
port: 7231
salt: secret
default_page_size: 125
user_agent: RESTBase
ui_name: RESTBase
ui_url: https://www.mediawiki.org/wiki/RESTBase
ui_title: RESTBase docs
spec:
x-request-filters:
- path: lib/security_response_header_filter.js
x-sub-request-filters:
- type: default
name: http
options:
allow:
- pattern: http://localhost/api.php
forward_headers: true
- pattern: http://localhost:8000
forward_headers: true
- pattern: http://localhost:10044
forward_headers: true
- pattern: /^https?:\/\//
paths:
/{domain:localhost}: &arbitary_domain
x-modules:
- path: projects/MR.yaml
options:
action:
# XXX Check API URL!
apiUriTemplate: http://localhost/api.php
# XXX Check the base RESTBase URI
baseUriTemplate: "{{'http://localhost:7231/{domain}/v1'}}"
parsoid:
# XXX Check Parsoid URL!
host: http://localhost:8000
mathoid:
# XXX Check Mathoid URL!
host: http://localhost:10044
table:
backend: sqlite
dbname: db.sqlite3
pool_idle_timeout: 20000
retry_delay: 250
retry_limit: 10
show_sql: false
/{domain:your.some.domain}: *arbitary_domain
# Finally, a standard service-runner config.
info:
name: restbase
logging:
name: restbase
level: info
- restbase/project/MR.yaml
# will work with Mathoid on any {domain}
paths:
/{api:v1}:
x-modules:
# swagger options, overriding the shared ones from the merged specs (?)
- spec:
info:
version: 1.0.0
title: Wikimedia REST API
description: Welcome to your RESTBase API.
#securityDefinitions: &wp/content-security/1.0.0
# mediawiki_auth:
# description: Checks permissions using MW api
# type: apiKey
# in: header
# name: cookie
# x-internal-request-whitelist:
# - /http:\/\/[a-zA-Z0-9\.]+\/w\/api\.php/
# Override the base path for host-based (proxied) requests. In our case,
# we proxy https://{domain}/api/rest_v1/ to the API.
#x-host-basePath: /api/rest_v1
x-route-filters:
- path: ./lib/normalize_title_filter.js
options:
redirect_cache_control: '{{options.purged_cache_control}}'
paths:
/media:
x-modules:
- path: v1/mathoid.yaml
options: '{{options.mathoid}}'
#/testing:
# x-modules:
# - path: ../mathpipe.yaml
# options: '{{options.mathoid}}'
/page:
x-modules:
- path: v1/content.yaml
options:
purged_cache_control: '{{options.purged_cache_control}}'
- path: v1/common_schemas.yaml # Doesn't really matter where to mount it.
/transform:
x-modules:
- path: v1/transform.yaml
options: '{{options}}'
/{api:sys}:
x-modules:
- spec:
paths:
/mathoid:
x-modules:
- path: sys/mathoid.js
options: '{{options.mathoid}}'
/table:
x-modules:
- path: sys/table.js
options:
conf: '{{options.table}}'
/key_value:
x-modules:
- path: sys/key_value.js
/key_rev_value:
x-modules:
- path: sys/key_rev_value.js
/key_rev_latest_value:
x-modules:
- path: sys/key_rev_latest_value.js
/post_data: &sys_post_data
x-modules:
- path: sys/post_data.js
/events:
x-modules:
- path: sys/events.js
options: '{{options.events}}'
/page_revisions:
x-modules:
- path: sys/page_revisions.js
/action:
x-modules:
- path: sys/action.js
options: '{{options.action}}'
/page_save:
x-modules:
- path: sys/page_save.js
/parsoid:
x-modules:
- path: sys/parsoid.js
options:
parsoidHost: '{{options.parsoid.host}}'
response_cache_control: '{{options.purged_cache_control}}'
options: '{{options}}'
- restbase/v1/mathoid.yaml
# Mathoid - math formula rendering service :simply replaced /wikimedia.org/sys to /{domain}/sys
tags:
- name: Math
description: formula rendering
paths:
/math/check/{type}:
post:
tags: ['Math']
summary: Check and normalize a TeX formula.
description: |
Checks the supplied TeX formula for correctness and returns the
normalised formula representation as well as information about
identifiers. Available types are tex and inline-tex. The response
contains the `x-resource-location` header which can be used to retrieve
the render of the checked formula in one of the supported rendering
formats. Just append the value of the header to `/media/math/{format}/`
and perform a GET request against that URL.
Stability: [stable](https://www.mediawiki.org/wiki/API_versioning#Stable).
produces:
- application/json
parameters:
- name: type
in: path
description: The input type of the given formula; can be tex or inline-tex
type: string
required: true
enum:
- tex
- inline-tex
- chem
- name: q
in: formData
description: The formula to check
type: string
required: true
responses:
'200':
description: Information about the checked formula
'400':
description: Invalid type
schema:
$ref: '#/definitions/problem'
default:
description: Error
schema:
$ref: '#/definitions/problem'
x-monitor: true
x-amples:
- title: Mathoid - check test formula
request:
params:
domain: wikimedia.org
type: tex
body:
q: E=mc^{2}
response:
status: 200
headers:
content-type: /^application\/json/
x-resource-location: /.+/
cache-control: 'no-cache'
body:
success: true
checked: /.+/
x-request-handler:
- get_from_sys:
request:
method: post
uri: /{domain}/sys/mathoid/check/{type}
headers: '{{ request.headers }}'
body: '{{ request.body }}'
/math/formula/{hash}:
get:
tags: ['Math']
summary: Get a previously-stored formula
description: |
Returns the previously-stored formula via `/media/math/check/{type}` for
the given hash.
Stability: [stable](https://www.mediawiki.org/wiki/API_versioning#Stable).
produces:
- application/json
parameters:
- name: hash
in: path
description: The hash string of the previous POST data
type: string
required: true
minLength: 1
responses:
'200':
description: Information about the checked formula
'404':
description: Data for the given hash cannot be found
schema:
$ref: '#/definitions/problem'
default:
description: Error
schema:
$ref: '#/definitions/problem'
x-monitor: false
x-request-handler:
- get_from_sys:
request:
method: get
uri: /{domain}/sys/mathoid/formula/{hash}
headers: '{{ request.headers }}'
/math/render/{format}/{hash}:
get:
tags: ['Math']
summary: Get rendered formula in the given format.
description: |
Given a request hash, renders a TeX formula into its mathematic
representation in the given format. When a request is issued to the
`/media/math/check/{format}` POST endpoint, the response contains the
`x-resource-location` header denoting the hash ID of the POST data. Once
obtained, this endpoint has to be used to obtain the actual render.
Stability: [stable](https://www.mediawiki.org/wiki/API_versioning#Stable).
produces:
- image/svg+xml
- application/mathml+xml
- image/png
parameters:
- name: format
in: path
description: The output format; can be svg or mml
type: string
required: true
enum:
- svg
- mml
- png
- name: hash
in: path
description: The hash string of the previous POST data
type: string
required: true
minLength: 1
responses:
'200':
description: The rendered formula
'404':
description: Unknown format or hash ID
schema:
$ref: '#/definitions/problem'
default:
description: Error
schema:
$ref: '#/definitions/problem'
x-monitor: false
x-setup-handler:
- init_svg:
uri: /{domain}/sys/key_value/mathoid.svg
body:
keyType: string
valueType: string
- init_mml:
uri: /{domain}/sys/key_value/mathoid.mml
body:
keyType: string
valueType: string
- init_png:
uri: /{domain}/sys/key_value/mathoid.png
body:
keyType: string
valueType: blob
x-request-handler:
- check_storage:
request:
method: get
uri: /{domain}/sys/key_value/mathoid.{$.request.params.format}/{$.request.params.hash}
headers:
cache-control: '{{ cache-control }}'
catch:
status: 404
return_if:
status: '2xx'
return:
status: 200
headers: "{{ merge({ 'cache-control': options.cache-control }, check_storage.headers) }}"
body: '{{ check_storage.body }}'
- postdata:
request:
uri: /{domain}/sys/mathoid/formula/{request.params.hash}
- mathoid:
request:
method: post
uri: /{domain}/sys/mathoid/render/{request.params.format}
headers:
content-type: application/json
x-resource-location: '{{ request.params.hash }}'
body: '{{postdata.body}}'
- Apache proxy config : /etc/httpd/conf.d/restbase.conf
#
# Restbase
#
<VirtualHost *:80>
ProxyPreserveHost On
ProxyPass /your.some.domain/ http://localhost:7231/your.some.domain/
ProxyPassReverse /your.some.domain/ http://localhost:7231/your.some.domain/
ProxyPass /localhost/ http://localhost:7231/localhost/
ProxyPassReverse /localhost/ http://localhost:7231/localhost/
# MediaWiki is located here:
DocumentRoot /var/www/html
</VirtualHost>
you can test the APIs at http://your.some.domain/your.some.domain/v1/
setup apache proxy
To load the images from the same location where the main wiki site is hosted a simple apache proxy can be used. Enable apache mod proxy ... and then add a config file along the lines of this file
cat /etc/apache2/sites-available/restbase.conf
<VirtualHost *:80>
ServerName restbase.formulasearchengine.com
ServerAlias api.formulasearchengine.com png.formulasearchengine.com
ServerAdmin wiki@physikerwelt.de
ProxyPreserveHost On
ProxyPass /v1/ http://localhost:8081/wikimedia.org/v1/
ProxyPass /wikimedia.org/v1/ http://localhost:8081/wikimedia.org/v1/
</VirtualHost>
a2ensite restbase.conf
You can use letsencrypt to also generate a HTTPS version for that site.
Helpful commands to check the state of the database
Clean cache
run cqlsh
use "local_group_default_T_mathoid_svg";
truncate data;
truncate meta;
use "local_group_default_T_mathoid_mml";
truncate data;
truncate meta;
use "local_group_default_T_mathoid_input";
truncate data;
truncate meta;
use "local_group_default_T_mathoid_png";
truncate data;
truncate meta;
Browse the data
Task retrieve the input given a key ... in this example we'll use 0f7fa7bb7d214c60f8d2f0b8d356c41bfe5c5d1d as key
run cqlsh
use "local_group_test_T_mathoid_input";
SELECT * from data WHERE key = '0f7fa7bb7d214c60f8d2f0b8d356c41bfe5c5d1d' and "_domain" = 'wikimedia.org';
_domain | key | tid | _del | content-location | content-sha256 | content-type | headers |latestTid | tags | value
---------------+------------------------------------------+--------------------------------------+------+------------------+----------------+------------------+-------------------------------------+-----------+------+----------------------------------------------------------------------------------------------------------------------------------------------------------------
wikimedia.org | 0f7fa7bb7d214c60f8d2f0b8d356c41bfe5c5d1d | e736aba4-9edc-11e5-927a-301f88a60cb9 | null | null | null | application/json | {"content-type":"application/json"} | null | null | {"q":"=nf(x)\\left({n-1 \\choose k-1}F(x)^{k-1}(1-F(x))^{(n-1)-(k-1)}-\\underbrace {{n-1 \\choose n}F(x)^{n}(1-F(x))^{(n-1)-n}} \\right)","type":"inline-tex"}