Render LilyPond in Markdown

May 15, 2019 ยท 5 minute read

I want to write LilyPond code, a music typesetting markup language, directly in a markdown file and have it rendered directly to embeded SVG in an HTML page. This post explores the process I went through to get this working.

My typical workflow when writing about music on the web is to make a music image and then reference it in the markdown as an image. When doing lots of examples this gets tedious and my source files for the images are not contained within my content.

What is LilyPond?

LilyPond is a markup language similar to LaTeX but for making music scores. LilyPond takes an input text file like the following

\version "2.18.2"

upper = \relative c'' {
  \clef treble
  \key c \major
  \time 4/4

  a4 b c d
}

lower = \relative c {
  \clef bass
  \key c \major
  \time 4/4

  a2 c
}

\score {
  \new PianoStaff <<
    \new Staff = "upper" \upper
    \new Staff = "lower" \lower
  >>
  \layout { }
}

and it would render to this

Example LilyPond Output

Proposed API

If I were using Hugo, my markdown file would contain something like this

## Some LilyPond Markdown

{{< lilypond >}}
\score{
    \relative c'' {
        % some notes
        g8 e \tuplet 3/2 { f[ a c] } e d a b c4
    }
    \layout{}
}
{{< \lilypond >}}

Hugo's Shortcomings

The Hugo developers have chosen not to allow an exec shortcode. This would allow theme developers to execute arbitrary code on users machines and therefore seen as a security risk.

Custom Nunjucks Tag

Since I can't do it in Hugo I looked for another SSG that had a more flexible (and dangerous) template system. I love the jinja2 api so I tried the JS port Nunjucks. Nunjucks allows you to define custom tags.

I had a working prototype of Nunjucks just rendering some lilypond from a custom tag. Now I needed an SSG that would allow access to the Nunjucks API. So I went to StaticGen.com and filtered by template type.

I tried out 11ty and it worked out great.

The resulting call in my markdown looks like this

{% lilypond 'inline', 'preview', '0.7 in' %}
\score{
  \relative c'' {
    % some notes
    c d e f g
  }
  \layout{}
}
{% endlilypond %}

In my .eleventy.config I define a new Nunjucks tag

// The Lilypond extension
eleventyConfig.addNunjucksTag("lilypond", function(nunjucksEngine) {
  return new function() {
    // this block is from the Nunjucks docs
    // see https://mozilla.github.io/nunjucks/api.html#custom-tags
    this.tags = ['lilypond'];
    this.parse = function(parser, nodes, lexer) {
        var tok = parser.nextToken();
        var args = parser.parseSignature(null, true);
        parser.advanceAfterBlockEnd(tok.value);
        var body = parser.parseUntilBlocks('endlilypond');
        parser.advanceAfterBlockEnd();
        return new nodes.CallExtensionAsync(this, 'run', args, [body]);
    };
    // this is where the real processing occurs
    this.run = function(context, param, body, callback) {
      // hash the input so we can cache it for later
      let hash = MD5(body())
      // check if hash already exists in the cache
      // if cache miss then process the lilypond code
      fs.writeFile(${outputDir}/${hash}.ly, body(), function(err) {
        exec(`lilypond -dbackend=svg --output=${outputDir} ${outputDir}/${hash}.ly`
        , function(err, stdout, stderr) {
          // read contents of output LilyPond SVG
          // cache the result
          let ret = new nunjucksEngine.runtime.SafeString(lilypondSVG)
          // End the processing
          callback(null, ret);
          )
        })
      }
    }

I skipped a couple of things to make the example more readable.

See the code for the full example.

Highlight LilyPond Code

There are no good LilyPond syntax highlighters for the web. With Highlight.js, Prism.js, or Hugo you could use tex or latex highlighting but the results aren't great. I made an attempt at defining my own Prism.js syntax but ended up with a huge nasty regex.

A much better solution is to use an actual parser/lexer. Fortunately python-ly exposes an API to highlight lilypond code through the command line.

Python-ly is a command line tool used to process LilyPond files. You can transpose, reformat, and output syntax highlighted html from an input file. This is what the Frescobaldi LilyPond editor uses to highlight and manipulate LilyPond files.

After poking around and discovering some undocumented ways of using the python-ly CLI, I had something that provided some great syntax highlighting.

Compare the difference between the outputs of Hugo, PrismJS, and python-ly (screen shots taken from lilypond in markdown and python-ly highlight test):

Hugo using Chroma

Here is some lilypond using the latex highlight in Hugo (Chroma):

\score{
	\relative c'' {
      c8 | c8([ bes)] a a([ g)] f | f'4. b, | c4.~ c4
	}
	\layout{}
}

The only thing that gets highlighted are things preceded by a slash \.

PrismJS

And here is PrismJS with my own LilyPond definition.

Since Prism is using regex, it is hard to separate different contexts between strings or note names like in hills a -- dorn.

Prism Highlight
Prism Highlight with Regex

Python-Ly

And finally python-ly. I am using my own custom CSS to style the output. Each parsed token is wrapped in a span with a class name.

python-ly Highlight
python-ly Highlight with custom CSS

Custom Nunjucks Highlight Tag

I added another Nunjucks custom tag using the same method as before. This time I pass body() to python-ly.

// LilyPond Syntax highlight extension
eleventyConfig.addNunjucksTag("lilycode", function(nunjucksEngine) {
  return new function() {
    // define and parse tags
  }
  this.run = function(context, body, callback) {
    let execString = `
      echo "${body()}" |
      ly highlight -d full_html=false -d wrapper_tag=code -d document_id=language-lilypond
      `;
    exec(execString, function(err, stdout, stderr) {
      // wrap the code block with a pre as per convention
      let formatedHtml = `<pre class="language-lilypond">${stdout}</pre>`;
      let ret = new nunjucksEngine.runtime.SafeString(formatedHtml);
      callback(null, ret);
  }
}

See the actual code.

You can then style the output however you like. Here I am using a monokai inspired theme based on the Prism Okaidia theme. See my css.

To use the custom tag it looks like

{% lilycode %}
\score{
  \relative c'' {
    % some notes
    c d e f g
  }
  \layout{}
}
{% endlilycode %}

Conlusion

What I have learned from this process is that Hugo is great for blogging for most use cases. If you need a more powerful template system to process your markdown files try 11ty. I found 11ty easy to use with good documentation.