Complete scrub_surrogates' lone-surrogate coverage (CR follow-up to #146)

[cboos]

Summary

CodeRabbit's review on PR #146 flagged that scrub_surrogates raises UnicodeEncodeError on lone high surrogates (U+D800–U+DBFF) — verified empirically. Python's surrogateescape error handler only back-maps the low range (U+DC80–U+DCFF) produced when decoding raw bytes; lone high surrogates have no mapping and the encode step raises on them. The function name's promise ("scrub lone surrogates") implied broader coverage than the implementation delivered.

This PR pre-substitutes high surrogates with U+FFFD via re.sub before the surrogateescape encode step. Low-range handling unchanged. Both ranges now produce U+FFFD as the on-disk sentinel.

Verification

>>> "\ud800".encode("utf-8", errors="surrogateescape")
UnicodeEncodeError: 'utf-8' codec can't encode character '\ud800' in position 0: surrogates not allowed
>>> "\udcb2".encode("utf-8", errors="surrogateescape")
b'\xb2'   # OK — back-mapped to raw byte

Pre-fix, scrub_surrogates(chr(0xD800)) raised. Post-fix:

>>> scrub_surrogates(chr(0xD800))  # U+D800
'�'
>>> scrub_surrogates(chr(0xDCB2))  # U+DCB2 (unchanged behaviour)
'�'

Test plan

• test/test_surrogate_encoding.py::test_various_lone_surrogates_handled — parametrize extended over the full lone-surrogate range: U+DC80, U+DCB2, U+DCFF (low), U+D800, U+DBFF (high). Renamed from test_various_low_surrogates_handled for accuracy.
• test/test_tui_surrogate.py::test_session_export_scrubs_lone_surrogate — parametrized over both ranges (low U+DCB2, high U+D800). This also closes CR's parametrize-the-TUI-test nitpick from the same review (one commit addresses both findings).
• just ci clean — 1721+ collected, ruff + pyright + ty all green.

Out-of-scope but related

CR's review mentioned the proposed regex r"[\ud800-\udbff]" covers exactly the high-surrogate range. The full Unicode surrogate block is U+D800-U+DFFF; the low half (U+DC00-U+DCFF) is what surrogateescape already handles via the encode round-trip, so we only need to pre-substitute the high half. Function docstring updated to spell out the two-mechanism strategy.

Summary by CodeRabbit

• Bug Fixes

  • Enhanced handling of invalid Unicode surrogate code points in data processing and export operations. Both high and low surrogates are now properly converted to standard replacement characters for strict UTF-8 compatibility.

• Tests

  • Expanded test coverage for surrogate character encoding validation across the full lone-surrogate Unicode range.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 7c993a13-b888-4687-bb20-8b996317a262

📥 Commits

Reviewing files that changed from the base of the PR and between df088a7 and c9736a9.

📒 Files selected for processing (3)

• claude_code_log/cache.py
• test/test_surrogate_encoding.py
• test/test_tui_surrogate.py

---

📝 Walkthrough

Walkthrough

The PR extends surrogate handling in the UTF-8 scrubbing helper to cover both low (U+DC80–U+DCFF) and high (U+D800–U+DBFF) lone surrogate code points. A regex pattern pre-substitutes high surrogates before the existing surrogateescape round-trip, and test parametrization expands coverage across both ranges.

Changes

Extend Surrogate Handling to High Surrogates

Layer / File(s)	Summary
High Surrogate Regex and Scrub Logic claude_code_log/cache.py	Adds _HIGH_SURROGATE_RE regex pattern to match U+D800–U+DBFF and updates scrub_surrogates to pre-substitute matched high surrogates with U+FFFD before surrogateescape encode/decode normalization.
Encoding Test Parametrization test/test_surrogate_encoding.py	Renames test_various_low_surrogates_handled to test_various_lone_surrogates_handled and expands parametrization from 3 low surrogates (\udc80, \udcb2, \udcff) to 5 total (\udc80, \udcb2, \udcff, \ud800, \udbff).
TUI Test Refactoring test/test_tui_surrogate.py	Removes module-level LONE_SURROGATE constant, refactors test_session_export_scrubs_lone_surrogate into a parametrized test covering both low and high surrogates, and updates post-export assertions to verify parametrized surrogate absence.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related PRs

• daaain/claude-code-log#144: Introduced the base scrub_surrogates helper with surrogateescape→replace normalization for low surrogates; this PR extends it to cover high surrogates.
• daaain/claude-code-log#146: Also modifies the same scrub_surrogates helper in claude_code_log/cache.py.

Poem

🐰 A rabbit's ode to surrogate squashing:
High and low surrogates dance in the glow,
Pre-subbed with U+FFFD, now they must go!
Regex and UTF-8 in perfect accord,
Tests parametrized—coverage restored!
Clean bytes at last, the scrubbing is true. 🌟

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately summarizes the main change: completing lone-surrogate coverage in scrub_surrogates by addressing high surrogates, which is the primary focus of the PR across all modified files.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch dev/scrub-high-surrogates

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[cboos]

No idea what happened with #146 opened by @KeloYuan (... who also disappeared!).

Well, in any case, thanks KeloYuan for the contribution and the hint to fix the remaining places in the TUI where we had this issue.