parsing in .woff2

In my last post, one piece of feedback I got was that the title was inaccurate because the sentiment parsing wasn't occurring on the font level but rather the typesetting level. This wasn't intentional but in retrospect was definitely true and drove me down a fun rabbit hole!

Every box holds plain text, and the formatting is applied by substitution rules inside the font rather than by a script. All of these fonts are downloadable as .woff2 files.

How it works

You may have noticed that in some fonts the f and i combine so that the cross of the f connects with the dash of the i. An example font of this in Google Docs is Lobster. This is called a ligature, and fonts support it through context-sensitive substitution rules. Those same rules are what we use to make a font do formatting.

Every OpenType font (.ttf, .otf, .woff2) carries a table of these substitution rules that the browser runs on each line before drawing it. For example the rule below says that an f followed by an i is drawn as one joined glyph in OpenType Feature File syntax. Fira Code does the same for => and !=, and rules can also look at the neighbours around a glyph, which makes them a pattern matcher running inside the font:

feature liga {
  sub f i by f_i;
} liga;

To make this process easier I put together a simple tool, which takes a grammar and generates the feature syntax for fontTools to build.

Formatting a git diff with a font is the easy case: a line that starts with + is green, with - is red, and with @@ is purple, and keep leaves the marker visible.

style added   = color "#1a7f37";
style removed = color "#cf222e";
style hunk    = from "LiberationSans-Bold.ttf", color "#8250df";

added   = line_start , "+"  , { any } -> added   keep;
removed = line_start , "-"  , { any } -> removed keep;
header  = line_start , "@@" , { any } -> hunk    keep;

The other 2 fonts shown above are a bit harder because they have state. For example for semfont we try to match the word with the sentiment, so that needs a rule to remember that a negation (in this case a not) came earlier than the word we aim to turn red. Similarly for formatting markdown we need to remember which of bold, italic and code tags are open. A font has no memory, so my solution is to carry the state in the glyphs themselves. This means each letter exists in a copy per state, and a rule swaps a letter for the copy that matches what came before it.

For example here is the grammar for bolding. It's basically the same as the markdown font grammar but for simplicity let's restrict the alphabet to a, b, c, asterisk, and a space.

alphabet = "a-c* ";
face      = "LiberationSans-Regular.ttf";
face bold = "LiberationSans-Bold.ttf";
toggle bold = "**" -> bold;

The last line says that ** flips bold on and off because in markdown bolding occurs when a word is in between those two symbols.

Now here's what it generates:

lookup PRIME   { sub @Plain by [a.s0 b.s0 c.s0 asterisk.s0 space.s0]; } PRIME;
lookup NULL_s0 { sub @Plain by null.s0; sub [a.s0 b.s0 c.s0 asterisk.s0 space.s0] by null.s0; } NULL_s0;
lookup TO_s1   { sub [a.s0 b.s0 c.s0 asterisk.s0 space.s0] by [a.s1 b.s1 c.s1 asterisk.s1 space.s1]; } TO_s1;
lookup NULL_s1 { sub @Plain by null.s1; sub [a.s0 b.s0 c.s0 asterisk.s0 space.s0] by null.s1; } NULL_s1;

lookup TOGGLE {
  sub @S0 asterisk.s0' lookup NULL_s1 asterisk.s0' lookup NULL_s1 [a.s0 b.s0 c.s0];
  sub [a.s1 b.s1 c.s1 asterisk.s1 null.s1] asterisk.s0' lookup NULL_s0 asterisk.s0' lookup NULL_s0;
  sub @S1 [a.s0 b.s0 c.s0 asterisk.s0 space.s0]' lookup TO_s1;
  ignore sub @Styled [asterisk.s0]';
  sub asterisk.s0' lookup NULL_s1 asterisk.s0' lookup NULL_s1 [a.s0 b.s0 c.s0];
} TOGGLE;

So let's break down the lookups:

1. PRIME renames every letter in the alphabet from a to a.s0, marking all of them as not bold.
2. NULL_s0 replaces a glyph with an invisible, zero-width glyph that carries the bold-off flag.
3. NULL_s1 does the same trick to carry the bold-on flag.
4. TO_s1 turns a bold-off letter into its bold-on copy, so a.s0 becomes a.s1.
5. TOGGLE is the main one. It invokes the other 3 helper lookups to generate the bolding.

A bit confusing for sure, but the nice part is the tool will handle all this generation for you so you don't have to spend time writing out the lookups yourself!

I had a great time exploring the world of fonts, feel free to check out semfont for the code and thanks for reading!

PS: One fun thing I noticed after publishing. On iOS the semfont box does not flip the color after a not, so "was not clean" shows green, while the Markdown and diff fonts work fine. Every browser on iOS, Chrome included, uses Apple's text engine instead of the one Chrome uses elsewhere, and it seems to handle chained font rules differently. I don't know exactly why yet, so if you do, let me know!