LLVM/clang/rust asan could use some attention

[iximeow]

paging out some notes for the next person who's wanting ASAN on a Helios dev system (or illumos generally). all of this could/should be issues on the various projects themselves, but I'd like to keep track of sum total work somewhere, and here's as good as anywhere.

first, LLVM does not (yet! this could be you!) have an illumos target, so the digging here uses "solaris" and "sunos" names; those are currently the relevant ones for illumos uses of LLVM. and so previously:

LLVM

back in 2017, ro got sanitizer support into LLVM for Solaris. D40898 mentions having used sanitizers "inside the gcc tree", which I think I should read as "linking in the asan runtime", but honestly I'm not totally sure. this dates back to Clang 7. there's a mention of additional compiler-rt work needed for x86-64, but I'm not sure how that translates to now, 8 years on.

the additional diffs linked in that commit eventually get you to the implementation bits of asan today: sanitizer_common and sanitizer_solaris.cpp in there. sanitizer_solaris.cpp has some FIXMEs that look more worrying than I think they are? a few of these seem fine (like getdents)?

some notes for a future "run asan tests" check here, as the AddressSanitizer docs seem a bit stale. in a fresh clone of LLVM:

# you want LLVM_OPTIMIZED_TABLEGEN because running tablegen
# is a big chunk of time in the middle of the build
#
# you don't need `clang` etc unless you want to run those at the end of the build (I do)
#
# on illumos, I needed to include -DCOMPILER_RT_BUILD_SANITIZERS=ON as well for
# compiler-rt to have check-* targets
cmake -S llvm -B build -G Ninja \
    -DCMAKE_BUILD_TYPE=Debug -DLLVM_ENABLE_RUNTIMES=compiler-rt \
    -DLLVM_ENABLE_PROJECTS="'clang;clang-tools-extra;lld' \
    -DLLVM_OPTIMIZED_TABLEGEN=ON

# does the build,
cmake --build build --target compiler-rt

# runs all the compiler-rt tests,
cmake --build build --target check-compiler-rt

# ... but on illumos this ends out at `check-compiler-rt does nothing, no tools built.` .. uh oh.
# might be a symptom of some partially-wired-up .. something?
# in fact on illumos the following was empty before setting COMPILER_RT_BUILD_SANITIZERS
cmake --build build --target help | grep check

clang

clang supports asan (and a few others) on sparc and x86 (not -64) since some patches around the same time. the exclusion of 64-bit code is why this fails:

# clang-17 -fsanitize=address lzcnt.c
clang-17: error: unsupported option '-fsanitize=address' for target 'x86_64-pc-solaris2.11'

if we see ASAN working on x86-64 now, it might be reasonable to add llvm::Triple::x86_64 to the tolerated triples in getSupportedSanitizers(). because Solaris and illumos have the same driver, we'd want to ask Solaris folks if they could give it a try, or if there's some divergence that means we land back at llvm#53919.

rustc

a while ago Rust got support sanitizer=address on illumos and Solaris: rust @ 8539a3c001. for some reason there's no referenced PR on the GitHub side, so I'm not sure what the history on this is.. but since llvm places the Solaris compiler-rt under sunos I worry this did not actually work and that perhaps no one had run the Rust asan tests at first? additionally rust-lang/rust#123615 deserves an update if someone looks at this issue and has gotten asan working on illumos/Solaris x86_64.

at least on illumos actually using -Z sanitizer=address fails trying to link against librust<-channel>_rt.asan.a. there is at least one Rust change needed for this to work right: iximeow/rust@d22c0f9. this points Rust at the LLVM-produced sunos target libclang_rt, which is retained as librust<-channel>_rt.asan.a. if LLMV had an illumos target, this probably should be able to "Just Work" like any other common_libs().

as I note in that commit, I think we're also missing a --enable-sanitizers in the Rust Dockerfile so in dist builds of rustc etc we're not actually getting the runtime library for the allegedly-supported feature.

once we see ASAN working in LLVM it would be prudent to use that LLVM in Rust and see where ./x.py test --force-rerun tests/ui/sanitizer/address.rs --verbose -- --no-capture gets us. currently this hangs in ASAN trying to report a memory check failure in a place I think it isn't intended to fail, but in a test that is expected to trip ASAN. the rest of tests/ui/sanitizer/ in Rust are probably informative, but this seems like a good place to start.