Extension:External Data/Retrieving and displaying data

External Data offers four basic ways to retrieve outside data and display it on the page: #external_value, #for_external_table, #display_external_table and #format_external_table. Although they are considered deprecated, you can also use the legacy #get_*_data functions to do the initial retrieval step; examples of both methods are shown below.

If Scribunto is installed, a Lua module can access external data by calling mw.ext.externalData.getExternalData().

The examples on this page all involve querying web data (the most popular data source), but External Data also supports retrieving data from many other data sources, including databases, LDAP servers and local files. For more information on these, and on web data retrieval itself, see the "Data sources" section in the navigation bar above. As of version 3.0, these other data sources can be used directly within #for_external_table and similar parser functions described here, specifying other necessary parameters such as |source= at the end of the parameter list; see #"Legacy" versus "Standalone" mode for details.

"Legacy" versus "Standalone" mode

Glossary entries: Standalone mode • Legacy mode

As of version 3.2 of this extension, external data sources are both retrieved and displayed within a single call #for_external_table and #display_external_table and similar parser functions. Parameters necessary for the data source itself (such as |source=, |db=, or |url=) are specified at the end of the parameter list. This obviates the need to use #get_external_data, #get_db_data or similar, and then call the display parser functions separately, which is how things were done in the 2.x series.

The new all-in-one method is called the standalone mode and the other method legacy mode. "Standalone" mode is preferred going forward, for best compatibility with Parsoid. New releases of External Data require an additional LocalSettings.php setting to enable the legacy mode.

Below is a comparison of the two methods, using the #external_value display function in a simple example. This one-line CSV file (with a header) is used as the external data source. The notable difference is that "legacy" mode requires a separate #get_web_data to actually fetch the data from the remote server. In this contrived example, "standalone" entails a bit more repetition (of the URL), but ends up being more concise in real-world usage with tabular data and display functions like #for_external_table.

Standalone mode Legacy mode Expected result
* Germany has population {{#external_value:Population|source=https://discoursedb.org/GermanyCSV.txt}}.
* Germany has area {{#external_value:Area|source=https://discoursedb.org/GermanyCSV.txt}}.
* Its capital is {{#external_value:Capital|source=https://discoursedb.org/GermanyCSV.txt}}.
 
{{#get_web_data:url=https://discoursedb.org/GermanyCSV.txt|data=population=Population,area=Area,capital=Capital}}
* Germany has population {{#external_value:population}}.
* Germany has area {{#external_value:area}}.
* Its capital is {{#external_value:capital}}.
  • Germany has population 82,411,001.
  • Germany has area 357,050 km².
  • Its capital is Berlin

Hint: The #arraymap parser function from Extension:Page Forms may come in handy for further parsing of individual values.

"Limited" versus "Intuitive" parsing

Glossary entries: Limited parsing • Intuitive parsing

Most of External Data's display parser functions accept an "expression" parameter: wikitext which is combined values from with the external data source and rendered to the page. In reference to this expression parameter, you will see "limited parsing" or "the old way" and "intutive parsing" mentioned in examples below.

Here are the differences:

  • The limited parsing or "old" method passes the wikitext expression as the first parameter.
    • example: {{#for_external_table:expression}} (note the lack of a leading pipe here)
    • This method does not correctly expand templates and parser functions within expression. It is maintained for backward compatibility, but is not preferred if you're using External Data ≥ 3.0 where "intuitive" parsing is supported.
  • The intuitive parsing or "new" method passes the wikitext expression as the second parameter, while the first is empty. This is the preferred method when using External Data ≥ 3.0.
    • example: {{{#for_external_table:|expression}}
    • With this method, templates and parser functions in expression, even wrapping the {{{variable}}} macros, will work just as they normally would in body text or within templates.

Displaying individual values

See also: Tag emulation mode

To display just a single value from an outside source, use the #external_value parser function:

{{#external_value:local variable name|optional fallback value}}

The optional fallback value can contain wiki markup, including additional {{#external_value:}} calls.

By default, #external_value displays an error message if it is called for a variable that has not been set, or if the specified data source is inaccessible, or the data source does not contain any data. There is no provision for a fallback/default value if the request fails, but you can disable the error message by adding $wgExternalDataVerbose = false; to LocalSettings.php:

#external_value example

Given this comma-delimited external data file[1]

Area,Borders,Capital,Population
"357,050 km²","Austria,Belgium,Czech Republic,Denmark,France,Luxembourg,Netherlands,Poland,Switzerland",Berlin,"82,411,001"

…the table below demonstrates how to retrieve individual values with #external_value.

Standalone mode Legacy mode Expected output
* Germany borders the following countries: {{#arraymap:{{#external_value:Borders|source=https://discoursedb.org/GermanyCSV.txt|format=csv with header}}|,|x|[[w:x|]]}}.
* Germany has population {{#external_value:Population|source=https://discoursedb.org/GermanyCSV.txt}}.
* Germany has area {{#external_value:Area|source=https://discoursedb.org/GermanyCSV.txt}}.
* Its capital is [[w:{{#external_value:Capital|source=https://discoursedb.org/GermanyCSV.txt}}|]].
 
{{#get_web_data:url=https://discoursedb.org/GermanyCSV.txt|format=csv with header|data=bordered countries=Borders,population=Population,area=Area,capital=Capital}}
* Germany borders the following countries: {{#arraymap:{{#external_value:bordered countries}}|,|x|[[w:x|]]}}.
* Germany has population {{#external_value:population}}.
* Germany has area {{#external_value:area}}.
* Its capital is [[w:{{#external_value:capital}}|]].

The |format= in the first #external_value of the Standalone example is for demonstration purposes only. In actuality, this file format (CSV with headers) is properly auto-detected.

For the bordered countries value, Page Forms' #arraymap function as well as the pipe trick are employed to split the comma-delimited value apart and make each country's name into a wikilink.

Displaying a table of values

If you want to retrieve and display tabular data, you can use any of the display functions #for_external_table, #display_external_table, or #format_external_table.

#for_external_table

The #for_external_table display function renders arbitrary wikitext for each row of tabular data source. As mentioned above, it can be called with the limited parsing format, but the recommended method supplies the expression as the second parameter, while the first is empty. This way, templates and parser functions in expression, even surrounding the {{{variable}}} macros, will work as expected.

Call this display function like this in "legacy" mode, when preceded by a {{#get_*_data}} function elsewhere on the page

{{#for_external_table:|expression}}

…or like this in "standalone" mode, where the parameters for the data source like |source= or |url= come at the end:

{{#for_external_table:|expression|optional data source parameters}}

The expression parameter is a string that contains one or more macros. You can think of these like parameters in the context of MediaWiki templates.

As with template parameters, macros are denoted by a name surrounded in curly brackets, with an optional value specified after a pipe, e.g. {{{var}}} and {{{var|default}}}. The default value used if the variable's value would be empty in the current loop. The expression string is then evaluated for each row of the external data source, macros are replaced by their corresponding values (or defaults), and the resulting wikitext is inserted into the page.

#for_external_table example

Let's try a similar example to #external_value above, but with multiple countries in a two-dimensional input file, rather than a single list of values. Given this external data file

Country,Area,Borders,Capital,Population
Algeria,"2,381,743 km²","Libya,Mali,Mauritania,Niger,Tunisia",Algiers,"32,854,002"
Angola,"1,246,700 km²","Democratic Republic of the  Congo,Namibia,Zambia",Luanda,"24,383,301"
Benin,"112,620 km²","Burkina Faso,Niger,Nigeria,Togo",Porto-Novo,"8,439,000"

the table below demonstrates how to retrieve rows of data from a delimited data file on a remote server using #for_external_data.[2]

Standalone mode Legacy mode
{| class="wikitable"
! Name
! Borders
! Population
! Area {{#for_external_table:|
{{!}}-
{{!}} {{{name}}}
{{!}} {{{borders}}}
{{!}} {{{population}}}
{{!}} {{{area}}}
| url = https://discoursedb.org/AfricaCSV.txt
| format = csv with header
| data = name=Country,borders=Borders,population=Population,area=Area
}}
|}

{{#get_web_data:url = https://discoursedb.org/AfricaCSV.txt
| format = csv with header
| data = name=Country,borders=Borders,population=Population,area=Area}}
{| class="wikitable"
! Name
! Borders
! Population
! Area {{#for_external_table:<nowiki/>
{{!}}-
{{!}} {{{name}}}
{{!}} {{{borders}}}
{{!}} {{{population}}}
{{!}} {{{area}}}}}
|}

As mentioned in the previous section, the |format=csv with header isn't strictly necessary here, as this particular data file's format would be auto-detected just fine. It's nonetheless shown for demonstration purposes.

The first or second argument to #for_external_table holds the wikitext expression that will be used to form the rows of the table from each incoming data record; it's surrounded by ordinary wikitext to create the top and bottom of the table. The {{!}} template is a standard MediaWiki trick to be able to use literal pipe characters in arguments to templates and parser functions.

Here's the expected output for either of the above methods:

Name Borders Population Area
Algeria Libya,Mali,Mauritania,Niger,Tunisia 32,854,002 2,381,743 km²
Angola Democratic Republic of the Congo,Namibia,Zambia 24,383,301 1,246,700 km²
Benin Burkina Faso,Niger,Nigeria,Togo 8,439,000 112,620 km²

Using #for_external_table for lists

Using #for_external_table to build an HTML table is the standard approach, but it may also be used to create lists:

{{#for_external_table:|
* {{{name}}} has the following borders: {{{borders}}}.
}}

See #Caveat: whitespace trimming if you're using the "old" method of passing the expression as the first parameter to the parser function.

Modifying values in place

There are two extra helpers for building URLs and query strings: {{{fieldname.urlencode}}} and {{{fieldname.htmlencode}}}[3]. See the table below for examples.

These are especially useful when passing the expression the "old" way, as the first parameter to the parser function, because built-in parser functions like {{urlencode:…}} are not recognized in that case. As noted above, the "new" way of passing the expression as the second parameter is recommended; this is supported from version 3.0 onward.

When used in an expression passed as the… Examples Notes
1st parameter
(old style) 		2nd parameter
(recommended style)
URL-encoding {{{fieldname.urlencode}}} {{urlencode:{{{fieldname}}}}} 
<!-- "old" style -->
{{#for_external_table: http://www.google.com/search?q={{{term.urlencode}}}}}

<!-- recommended style -->
{{#for_external_table:|http://www.google.com/search?q={{urlencode:{{{term}}}}}}}
As mentioned above, parser functions aren't understood in the first parameter (old style), so this wouldn't work: {{#for_external_table: http://www.google.com/search?q={{urlencode:{{{term}}}}}}}
HTML-encoding {{{fieldname.htmlencode}}} same; MediaWiki has no built-in direct replacement 
{{{field-name.htmlencode}}}
Analogous to PHP's htmlentities function

Caveat: whitespace trimming

If the wiki markup is passed as the first parameter to the display function ("limited" parsing), its leading whitespace is trimmed.

If a leading line break is required, e.g., to start a new table row or list item, prepend the parameter with <nowiki /> to preserve this line break like so:

{| class="wikitable"
! Name
! Borders
! Population
! Area {{#for_external_table:<nowiki />
{{!}}-
{{!}} {{{name}}}
{{!}} {{{borders}}}
{{!}} {{{population}}}
{{!}} {{{area}}}
}}
|}

#display_external_table

The #display_external_table display function is similar in concept to #for_external_table, but it passes the each row of values to a template which handles the display. This function is called as:

{{#display_external_table:
  template = template name
| data = set of parameters, separated by commas
| delimiter = delimiter
| intro template = template name (optional)
| outro template = template name (optional)
}}

An explanation of the parameters:

  • |template= - the name of the template into which each "row" of data will be passed
  • |data= - the data mappings between external variable and local template parameter; much like the data parameters for the other functions
  • |delimiter= - the separator used between one template call and the next; default is a newline. (To include newlines in the delimiter value, use "\n".)
  • |intro template= - a template displayed before the results set, only if there are any results
  • |outro template= - a template displayed after the results set, only if there are any results

For example, to display the data from the previous example in a table as before, you could create a template called "Country info row", that had the parameters "Country name", "Countries bordered", "Population" and "Area", and then call the #display_external_table parser function as follows:

Standalone mode Legacy mode
{| class="wikitable"
! Name
! Borders
! Population
! Area
{{#display_external_table:
    source = https://discoursedb.org/AfricaCSV.txt
  | format = CSV with header
  | data = Country name=Country,Countries bordered=borders,Population=population,Area=area
  | template = Country info row}}
|}

{{#get_web_data:url = https://discoursedb.org/AfricaCSV.txt
  | format = CSV with header
  | data = Country name=Country,Countries bordered=borders,Population=population,Area=area
}}
{| class="wikitable"
! Name
! Borders
! Population
! Area
{{#display_external_table:template = Country info row}}
|}

The template "Country info row" referenced by the |template= parameter should contain wikitext similar to following:

|-
| {{{Country name}}}
| {{{Countries bordered}}}
| {{{Population}}}
| {{{Area}}}

#format_external_table

The #format_external_table parser function is available if External Data ≥ 3.0 and Cargo ≥ 3.0 are installed. It passes the retrieved external data as a row-based two-dimensional array to the Cargo formatting engine, the same as used by the #cargo_query parser function. The function accepts the same parameters controlling the display as #cargo_query, as well as an optional |data= parameter which can be used to limit the number of displayed external variables, control their order and field aliases.

Example:

Standalone mode Legacy mode
{{#format_external_table:source=https://expert.ru/avtory/konstantin_pakhunov/
| format = HTML
| data = Title=article h2>a,URL=article h2>a.attr(href)
| display format = table}}
 
{{#get_external_data:
  url = https://expert.ru/avtory/konstantin_pakhunov/
| format = HTML
| data = title=article h2>a,url=article h2>a.attr(href)
}}
{{#format_external_table:display format = table | data = Title=title,URL=url}}

In Lua, the Cargo Lua function mw.ext.cargo.formatTable(table, parameters) can be applied to the results returned by mw.ext.externalData.getExternalData() function family to achieve similar results.

References