👨‍💻 about me home CV/Resume News 🖊️ Contact Github LinkedIn I’m a Haskeller 🏆 Best of LuaX hey bang ypp panda lsvg Fizzbuzz makex Calculadoira TPG todo pwd rrpi

LuaX and Pandoc based tools

LuaX and Pandoc based tools

makex.mk is a Makefile. It is intended to be included in any Makefile to easily install some tools based on LuaX and Pandoc to pre-process files and generate documents, using Lua as a common, simple and powerful scripting language.


Lots of software projects involve various tools, free as well as commercial, to build the software, run the tests, produce the documentation, … These tools use different data formats and scripting languages, which makes the projects less scalable and harder to maintain.

Sharing data between configuration files, documentations, tests results can then be painful and counter productive (the necessary glue is often more complex than the tools themselves).

Usually people script their build systems and processes with languages like Bash, Python, Javascript and make them communicate with plain text, YAML, JSON, XML, CSV, INI, TOML. Every script shall rely on specific (existing or not) libraries to read and write these data formats.

This document presents a commun and powerful data format and some tools to script the build process of a project and generate documentation.

To sum up the suggested solution is:


Only makex.mk is required. Other files from the makex repository are documentation and examples.

Just download makex.mk to a directory where other Makefiles can include it:

wget https://raw.githubusercontent.com/CDSoft/makex/master/makex.mk


curl -O https://raw.githubusercontent.com/CDSoft/makex/master/makex.mk

The full repository is available on Github: https://github.com/CDSoft/makex.


makex.mk usage is pretty simple:

  1. include makex.mk in another Makefile
  2. use $(LUAX), $(YPP), $(PANDOC_HTML), $(PANDOC_PDF), … to call luax, ypp, pandoc (to generate HTML or PDF files)

For a complete documentation, please refer to makex:

# makex defines some make variable that can be used to execute makex tools:
#     path to the LuaX interpreter (see https://github.com/CDSoft/luax)
#     path to the ypp executables (see https://github.com/CDSoft/ypp)
#     path to the upp executable (see https://github.com/CDSoft/upp)
#     path to the bang executable (see https://github.com/CDSoft/bang)
#     path to the panda script (see https://github.com/CDSoft/panda)
#     path to the pandoc executable (see https://pandoc.org)
#     path to a LaTeX template
#     (see https://github.com/Wandmalfarbe/pandoc-latex-template.git)
#     path to a LaTeX template
#     (see https://github.com/aaronwolen/pandoc-letter.git)
#     path to a CSS file (see https://benjam.info/panam)
#     shortcut to pandoc/panda with some default parameters
#     to generate Markdown documents
#     shortcut to pandoc/panda with some default parameters
#     to generate Github Markdown documents
#     shortcut to pandoc/panda with some default parameters
#     to generate HTML documents
#     shortcut to pandoc/panda with some default parameters
#     to generate PDF documents
#     shortcut to pandoc/panda with some default parameters
#     to generate beamer slideshows
#     shortcut to pandoc/panda with some default parameters
#     to generate a letter
#     path to the lsvg executable (see https://github.com/CDSoft/lsvg)
#     path to plantuml.jar
#     path to ditaa.jar
#     path to mmdc (Mermaid)
#     path to roger (Penrose)
#     path to the ghcup, ghc, cabal, stack executables
#     (see https://www.haskell.org/ghcup/)
#     ghc, cabal and stack commands executed through ghcup
# It also adds some targets:
# makex-clean
#     remove all makex tools
# makex-install
#     install all makex tools
# makex-install-luax
#     install luax
# makex-install-ypp
#     install ypp
# makex-install-upp
#     install upp
# makex-install-bang
#     install bang
# makex-install-pandoc
#     install pandoc
# makex-install-panda
#     install panda
# makex-install-plantuml
#     install PlantUML
# makex-install-ditaa
#     install ditaa
# makex-install-mermaid
#     install mermaid
# makex-install-penrose
#     install penrose
# makex-install-lsvg
#     install lsvg
# makex-install-ghcup
#     install ghcup
# help
#     runs the `welcome` target (user defined)
#     and lists the targets with their documentation

# The project configuration variables can be defined before including
# makex.mk.
# Makex update:
# wget http://cdelord.fr/makex/makex.mk

# MAKEX_INSTALL_PATH defines the path where tools are installed
MAKEX_INSTALL_PATH ?= /var/tmp/makex

# MAKEX_CACHE is the path where makex tools sources are stored and built

# MAKEX_HELP_TARGET_MAX_LEN is the maximal size of target names
# used to format the help message

# LUAX_VERSION is a tag or branch name in the LuaX repository
LUAX_VERSION ?= master

# YPP_VERSION is a tag or branch name in the ypp repository
YPP_VERSION ?= master

# UPP_VERSION is a tag or branch name in the upp repository
UPP_VERSION ?= master

# BANG_VERSION is a tag or branch name in the bang repository
BANG_VERSION ?= master

# PANDOC_VERSION is the version number of pandoc

# PANDOC_CLI_VERSION is the version number of pandoc-cli

# PANDOC_DYNAMIC_LINK is "no" to download a statically linked executable
# or "yes" to compile a dynamically linked executable with cabal

# PANDOC_LATEX_TEMPLATE_VERSION is a tag or branch name in the
# pandoc-latex-template repository

# PANDOC_LETTER_VERSION is a tag or branch name in the
# pandoc-letter repository

# PANAM_VERSION is a tag or branch name in the
# pan-am repository

# PANDA_VERSION is a tag or branch name in the Panda repository

# LSVG_VERSION is a tag or branch name in the lsvg repository
LSVG_VERSION ?= master

# GHCUP_INSTALL_BASE_PREFIX is the base of ghcup

# HASKELL_GHC_VERSION is the ghc version to install
HASKELL_GHC_VERSION ?= recommended

# HASKELL_CABAL_VERSION is the cabal version to install

# RUSTUP_HOME is the rustup installation path

# CARGO_HOME is the cargo installation path

# TYPST_COMPILATION is "no" to download a precompiled executable
# or "yes" to compile typst with cargo

# TYPST_VERSION is a tag or branch name in the
# typst repository

# PLANTUML_VERSION is the PlantUML version to install

# DITAA_VERSION is the ditaa version to install


The Makefile used to generate this document is a pretty self-explanatory example:

BUILD = .build

HTML_OPTS += --table-of-content
HTML_OPTS += --mathml

PDF_OPTS += --table-of-content
PDF_OPTS += --highlight-style tango

## Generates README.md and examples
all: ../README.md
all: $(BUILD)/makex.html
all: $(BUILD)/makex.pdf
all: $(BUILD)/letter.pdf

# first include makex.mk to add makex targets ($(LUAX), $(YPP), ...)
include ../makex.mk

# Note that comments starting with `##` are used by the `help` target
# to document targets (try `make help`)

## Clean the build directory
    rm -rf $(BUILD)

## Clean the build directory and all makex tools
mrproper: clean makex-clean


# This is the welcome message used by the `help` target.
# Note the single `#`, this is not a documentation for `help`.
    @echo '${TEXT_COLOR}makex${NORMAL} usage example'

export PANDA_IMG := img

# Preprocess a Markdown file with $(YPP).
# The preprocessed file is also a Markdown file
# that can be used by $(PANDA).
$(BUILD)/%.md: %.md makex.lua | $(YPP) $(DEPENDENCIES)
    @echo '${YPP_COLOR}[YPP]${NORMAL} ${TARGET_COLOR}$< -> $@${NORMAL}'
    @mkdir -p $(dir $@)
    @$(YPP) --MD --MT $@ --MF $(DEPENDENCIES)/$(notdir $@).ypp.d \
        -p . -l makex.lua $< -o $@

# Render an HTML file using $(PANDA) (i.e. pandoc and some Lua filters)
$(BUILD)/%.html: $(BUILD)/%.md | $(PANDA) $(PANAM_CSS) img $(DEPENDENCIES)
    @PANDA_TARGET=$@ \
    PANDA_DEP_FILE=$(DEPENDENCIES)/$(notdir $@).panda.d \
    $(PANDA_HTML) $(HTML_OPTS) $< -o $@

# Render a PDF file using $(PANDA) (i.e. pandoc and some Lua filters)
    @PANDA_TARGET=$@ \
    PANDA_DEP_FILE=$(DEPENDENCIES)/$(notdir $@).panda.d \
    $(PANDA_PDF) $(PDF_OPTS) $< -o $@

# Render an english letter using $(PANDA) (i.e. pandoc and some Lua filters)
$(BUILD)/letter.pdf: $(BUILD)/letter.md | $(PANDA) $(LETTER) img $(DEPENDENCIES)
    @PANDA_TARGET=$@ \
    PANDA_DEP_FILE=$(DEPENDENCIES)/$(notdir $@).panda.d \
    LANG=en \
    $(PANDA_LETTER) $< -o $@

$(BUILD)/letter.md: $(MAKEX_CACHE)/pandoc-letter/example/letter.md | $(PANDA)
    @echo '${PANDA_COLOR}[CP]${NORMAL} ${TARGET_COLOR}$< -> $@${NORMAL}'
    @cp $< $@
    @sed -i 's#example/#$(dir $<)#' $@

$(MAKEX_CACHE)/pandoc-letter/example/letter.md: $(LETTER)

# Render a Github Markdown file using $(PANDA)
../README.md: $(BUILD)/makex.md fix_links.lua | $(PANDA) img $(DEPENDENCIES)
    @PANDA_TARGET=$@ \
    PANDA_DEP_FILE=$(DEPENDENCIES)/$(notdir $@).panda.d \
    $(PANDA_GFM) -L fix_links.lua $< -o $@

    @mkdir -p $@

-include $(wildcard $(DEPENDENCIES)/*.d)


Lua is the perfect candidate for both a common data format and a script language.

What is Lua?

Lua is a powerful, efficient, lightweight, embeddable scripting language. It supports procedural programming, object-oriented programming, functional programming, data-driven programming, and data description.

Lua combines simple procedural syntax with powerful data description constructs based on associative arrays and extensible semantics. Lua is dynamically typed, runs by interpreting bytecode with a register-based virtual machine, and has automatic memory management with incremental garbage collection, making it ideal for configuration, scripting, and rapid prototyping.

Why choose Lua?

Lua is a proven, robust language

Lua has been used in many industrial applications (e.g., Adobe’s Photoshop Lightroom), with an emphasis on embedded systems (e.g., the Ginga middleware for digital TV in Brazil) and games (e.g., World of Warcraft and Angry Birds). Lua is currently the leading scripting language in games. Lua has a solid reference manual and there are several books about it. Several versions of Lua have been released and used in real applications since its creation in 1993. Lua featured in HOPL III, the Third ACM SIGPLAN History of Programming Languages Conference, in 2007. Lua won the Front Line Award 2011 from the Game Developers Magazine.

Lua is fast

Lua has a deserved reputation for performance. To claim to be “as fast as Lua” is an aspiration of other scripting languages. Several benchmarks show Lua as the fastest language in the realm of interpreted scripting languages. Lua is fast not only in fine-tuned benchmark programs, but in real life too. Substantial fractions of large applications have been written in Lua.

Lua is portable

Lua is distributed in a small package and builds out-of-the-box in all platforms that have a standard C compiler. Lua runs on all flavors of Unix and Windows, on mobile devices (running Android, iOS, BREW, Symbian, Windows Phone), on embedded microprocessors (such as ARM and Rabbit, for applications like Lego MindStorms), on IBM mainframes, etc.

Lua is powerful (but simple)

A fundamental concept in the design of Lua is to provide meta-mechanisms for implementing features, instead of providing a host of features directly in the language. For example, although Lua is not a pure object-oriented language, it does provide meta-mechanisms for implementing classes and inheritance. Lua’s meta-mechanisms bring an economy of concepts and keep the language small, while allowing the semantics to be extended in unconventional ways.

Lua is small

Adding Lua to an application does not bloat it. The tarball for Lua 5.4.4, which contains source code and documentation, takes 353K compressed and 1.3M uncompressed. The source contains around 30000 lines of C. Under 64-bit Linux, the Lua interpreter built with all standard Lua libraries takes 281K and the Lua library takes 468K.

Lua is free

Lua is free open-source software, distributed under a very liberal license (the well-known MIT license). It may be used for any purpose, including commercial purposes, at absolutely no cost. Just download it and use it.


LuaX is a Lua interpretor and REPL based on Lua 5.4.4, augmented with some useful packages. LuaX can also produce standalone executables from Lua scripts.

LuaX runs on several platforms with no dependency:

LuaX can cross-compile scripts from and to any of these platforms.

LuaX comes with a standard Lua interpretor and provides some libraries (embedded in a single executable, no external dependency required):

More information here: http://cdelord.fr/luax

Scripting with LuaX

LuaX can be used as a general programming language. There are plenty of good documentations for Lua and LuaX.

A big advantage of Lua is the usage of Lua tables as a common data format usable by various tools. It is Human-readable and structured. It can be generated by Lua scripts but also by any software producing text files.

Typical usages are:

The next chapters present some tools written in Lua/LuaX or using Lua as a scripting engine.


Ypp is a minimalist and generic text preprocessor using Lua macros.

Ypp is compiled by LuaX, i.e. Lua and LuaX functions and modules are available in macros.

More information here: http://cdelord.fr/ypp

Ypp is pretty simple. It searches for Lua expressions and replaces macros with their results.

Macro Result
@(...) Evaluates the Lua expression ... and replaces the macro by its result
@@(...) Executes the Lua chunk ... and replaces the macro by its result (if not nil)

Some expression do not require parentheses (function calls).


\sum_{i=1}^{100} i^2 = @F.range(100):map(function(x) return x*x end):sum()

is rendered as

\[ \sum_{i=1}^{100} i^2 = 338350 \]

Macros can also define variables reusable later by other macros.

    local foo = 42
    N = foo * 23 + 34
    local function sq(x) return x*x end
    function sumsq(n) return F.range(N):map(sq):sum() end

defines N (\(N = 1000\)) which can be read in a Lua expression or with @N and sumsq which computes the sum of squares.


\sum_{i=1}^{@N} i^2 = @sumsq(N)


\[ \sum_{i=1}^{1000} i^2 = 333833500 \]


Pandoc is a swiss-army knife to convert from and to a bunch of document formats.

A big advantage of Pandoc is the ability to use Lua scripts to define custom readers and writers for unsupported formats and also Lua filters to manipulate the pandoc abstract syntax tree (AST). This is the main pandoc feature exercised in this document.

Pandoc has an excellent documentation:

This document uses pandoc Lua filters with Panda (see next chapter) which bundles some useful filters in a single script.


Panda is a Pandoc Lua filter that works on internal Pandoc’s AST.

It provides several interesting features:

The documentation of Panda is here: http://cdelord.fr/panda


There are lots of examples in the documentation of panda. We will see here two of them.

Documentation extraction from source code

The source code can be documented by adding special marks in comments. The documentation shall be written in Markdown. The default mark is @@@ and can be customized.

For instance, the following C source contains documentation that can be extracted and included to a Pandoc document.

**`answer`** takes any question and returns the most relevant answer.


``` c
    const char *meaning = answer("What's the meaning of life?");

#include <unistd.h>

const char *answer(const char *question)
    /* Test all the possibilities */
    for (int y = 0; y < 7500000; y++)
        sleep(365 * 86400);
    /* Returns the most relevant answer */
    return "42";

To extract the documentation, panda provides a macro to replace a div element by the documentation chunks from a file. E.g.:


will be replaced by:

answer takes any question and returns the most relevant answer.


    const char *meaning = answer("What's the meaning of life?");


Diagrams can be embedded in Pandoc documents. Diagrams are specified as code blocks and are replaced by an image by panda.

```{.dot render="{{dot}}" width=67%}
digraph {
    input -> pandoc -> output
    pandoc -> panda -> {pandoc, diagrams}
    { rank=same; pandoc, panda }
    { rank=same; diagrams, output }

```{render="{{gnuplot}}" width=67%}
set xrange [-pi:pi]
set yrange [-1.5:1.5]
plot sin(x) lw 4, cos(x) lw 4


Haskell is a general-purpose, statically-typed, purely functional programming language with type inference and lazy evaluation.

Haskell related tools are installed with GHCup.


GHCup is the main installer for the general purpose language Haskell.


GHC is a state-of-the-art, open source, compiler and interactive environment for the functional language Haskell.


Cabal is a system for building and packaging Haskell libraries and programs. It defines a common interface for package authors and distributors to easily build their applications in a portable way. Cabal is part of a larger infrastructure for distributing, organizing, and cataloging Haskell libraries and programs.


Stack is a program for developing Haskell projects. It is aimed at Haskellers both new and experienced. It is cross-platform and aims to support fully users on Linux, macOS and Windows.

Stack features:

The documentation of Stack is here: https://docs.haskellstack.org/en/stable/


makex repository: https://github.com/CDSoft/makex

Lua: https://www.lua.org

Lua is a powerful, efficient, lightweight, embeddable scripting language. It supports procedural programming, object-oriented programming, functional programming, data-driven programming, and data description.

Lua documentation: https://www.lua.org/manual/5.4/

The reference manual is the official definition of the Lua language.

LuaX: https://github.com/CDSoft/luax

LuaX is a Lua interpretor and REPL based on Lua 5.4.6, augmented with some useful packages. LuaX can also produce standalone executables from Lua scripts.

UPP: https://github.com/CDSoft/upp

UPP is a minimalist and generic text preprocessor using Lua macros.

Pandoc: https://pandoc.org

Pandoc is a universal document converter. If you need to convert files from one markup format into another, pandoc is your swiss-army knife.

Pandoc manual: https://pandoc.org/MANUAL.html

Pandoc User’s Guide

Pandoc’s Markdown: https://pandoc.org/MANUAL.html#pandocs-markdown

Pandoc understands an extended and slightly revised version of John Gruber’s Markdown syntax. This document explains the syntax, noting differences from original Markdown.

Pandoc Lua filters: https://pandoc.org/lua-filters.html

Pandoc has long supported filters, which allow the pandoc abstract syntax tree (AST) to be manipulated between the parsing and the writing phase. Traditional pandoc filters accept a JSON representation of the pandoc AST and produce an altered JSON representation of the AST. They may be written in any programming language, and invoked from pandoc using the --filter option.

Although traditional filters are very flexible, they have a couple of disadvantages. First, there is some overhead in writing JSON to stdout and reading it from stdin (twice, once on each side of the filter). Second, whether a filter will work will depend on details of the user’s environment. A filter may require an interpreter for a certain programming language to be available, as well as a library for manipulating the pandoc AST in JSON form. One cannot simply provide a filter that can be used by anyone who has a certain version of the pandoc executable.

Starting with version 2.0, pandoc makes it possible to write filters in Lua without any external dependencies at all. A Lua interpreter (version 5.3) and a Lua library for creating pandoc filters is built into the pandoc executable. Pandoc data types are marshaled to Lua directly, avoiding the overhead of writing JSON to stdout and reading it from stdin.

Panda: https://github.com/CDSoft/panda

Panda is a Pandoc Lua filter that works on internal Pandoc’s AST.