Fix memory-amplification DoS via uint16 underflow in ReadGlyph

[XavLimSG]

Hey, found a memory-amplification DoS in ReadGlyph() while auditing the codebase. Figured I'd just submit the fix directly since it's a one-liner.

The Bug

ReadGlyph() in glyph.cc computes per-contour point counts by subtracting consecutive endPtsOfContours values:

uint16_t num_points = point_index - last_point_index + (i == 0 ? 1 : 0);
glyph->contours[i].resize(num_points);

The TrueType spec requires endPtsOfContours to be monotonically increasing, but this isn't validated. If the values decrease, the uint16_t subtraction wraps — 100 - 200 becomes 65436 — and each affected contour allocates ~65K Point structs on the heap.

An attacker can alternate high/low values across many contours to force massive allocations from a tiny input.

Validated Impact

Tested against the real woff2_compress binary (not a test harness), measuring RSS via /proc/<pid>/statm from an external monitor process:

Crafted Font	Size	Contours	Peak RSS	Amplification
50 underflow contours	1,456 B	50	21.5 MB	15,484x
100 underflow contours	1,556 B	100	40.5 MB	27,264x
500 underflow contours	2,356 B	500	191.1 MB	85,063x

A 2.3 KB crafted .ttf forces the real binary to consume 191 MB before it crashes out. Scales linearly — with 32K contours in a ~65KB font you're looking at ~12 GB.

The attack surface is the encoder path: ConvertTTFToWOFF2 → NormalizeFontCollection → NormalizeGlyphs → ReadGlyph. Anything that accepts untrusted .ttf/.ttc files and runs them through the woff2 encoder is affected. The decoder (ConvertWOFF2ToTTF) uses a separate transformed glyph format and is not affected by this specific issue.

The Fix

One check before the subtraction — reject the font if endPtsOfContours values aren't monotonically increasing:

if (i > 0 && point_index < last_point_index) {
    return FONT_COMPRESSION_FAILURE();
}

This is consistent with the TrueType spec requirement and how other font parsers handle it.

Testing

After the fix, same malicious inputs:

Crafted Font	Before Fix	After Fix
500 underflow contours	191.1 MB	3.2 MB (baseline RSS)

Legitimate fonts unaffected — tested all 6 DejaVu system fonts:

• DejaVuSans.ttf (757 KB) → compresses and round-trips fine
• DejaVuSans-Bold.ttf, DejaVuSerif-Bold.ttf, DejaVuSerif.ttf, DejaVuSansMono.ttf, DejaVuSansMono-Bold.ttf — all compress successfully

No regressions on any well-formed font.