Implement EL1 MMU linearmap by YinhuaChen-vivo · Pull Request #407 · vivoblueos/kernel

YinhuaChen-vivo · 2026-04-30T01:41:34Z

Summary

This PR implements an EL1 MMU linear mapping for the AArch64 kernel, using TTBR1_EL1 to map physical memory into the kernel’s high virtual address space. It also updates the AArch64 boot path so the kernel switches from the early identity-mapped execution environment into the high virtual address space after eret.

The default kernel virtual base for qemu_virt64_aarch64 is now configured as:

0xFFFFFF8000000000

Main Changes

Add CONFIG_KERNEL_VIRT_START for AArch64.
Add AArch64 EL1 linear map support in mmu.rs.
Introduce helpers:
- kernel_phys_to_virt()
- kernel_virt_to_phys()
Split MMU initialization into:
- init_el1_enable_mmu()
- init_el1_boot_linearmap()
Configure TTBR1_EL1 for the kernel linear map.
Add a 39-bit kernel virtual address layout using T1SZ/T0SZ.
Update the AArch64 boot flow to jump to jump_to_high_va() after eret.
Adjust stack and boot addresses when transitioning between physical and virtual addresses.
Update the qemu_virt64_aarch64 linker script to link the kernel at the high virtual address while loading it at the physical address.
Convert board MMIO/DRAM constants to kernel virtual addresses.
Update VirtIO address translation so MMIO and shared buffers correctly translate between physical and virtual addresses.
Use physical entry addresses when booting secondary CPUs through PSCI.

Memory Layout

The kernel now uses a high virtual linear map for AArch64 EL1:

VA = PA + KERNEL_VIRT_START
PA = VA - KERNEL_VIRT_START

Files Changed

arch/arm/arm64/Kconfig
- Adds CONFIG_KERNEL_VIRT_START.
kernel/src/arch/aarch64/mmu.rs
- Adds linear map page table setup.
- Adds physical/virtual conversion helpers.
- Enables TTBR1_EL1 walks for the kernel linear map.
- Documents the AArch64 linear mapping layout.
kernel/src/arch/aarch64/mod.rs
- Updates the EL2-to-EL1 boot path.
- Adds jump_to_high_va().
- Handles board-specific TTBR0_EL1 clearing for qemu_virt64_aarch64.
- Converts secondary CPU boot entry to a physical address.
kernel/src/boards/qemu_virt64_aarch64/config.rs
- Converts MMIO constants to kernel virtual addresses.
kernel/src/boards/qemu_virt64_aarch64/init.rs
- Converts DRAM_BASE to a kernel virtual address.
kernel/src/boards/qemu_virt64_aarch64/link.x
- Links the kernel at the high virtual address.
- Uses AT(...) to preserve the physical load address.
kernel/src/devices/virtio.rs
- Adds AArch64-aware VirtIO physical/virtual address translation.

Notes

Some AArch64 boot behavior is currently specific to qemu_virt64_aarch64, especially the TTBR0_EL1 clearing logic. FIXME comments were added to mark this as board-specific code that should be generalized when additional AArch64 platforms are adapted.

YinhuaChen-vivo · 2026-04-30T01:41:47Z

build_prs

github-actions · 2026-04-30T01:43:16Z

Job is started, see https://github.com/vivoblueos/kernel/actions/runs/25142977751.

github-actions · 2026-04-30T01:55:45Z

✅ All jobs completed successfully, see https://github.com/vivoblueos/kernel/actions/runs/25142977751.

YinhuaChen-vivo · 2026-04-30T02:22:24Z

build_prs

github-actions · 2026-04-30T02:24:26Z

Job is started, see https://github.com/vivoblueos/kernel/actions/runs/25144067446.

YinhuaChen-vivo · 2026-04-30T02:27:35Z

build_prs

github-actions · 2026-04-30T02:32:46Z

Job is started, see https://github.com/vivoblueos/kernel/actions/runs/25144203899.

github-actions · 2026-04-30T02:40:21Z

❌ Job failed. Failed jobs: check_format (failure), build_and_check_boards (failure), see https://github.com/vivoblueos/kernel/actions/runs/25144067446.

github-actions · 2026-04-30T02:45:17Z

✅ All jobs completed successfully, see https://github.com/vivoblueos/kernel/actions/runs/25144203899.

xuchang-vivo · 2026-04-30T04:07:41Z

Please draw the current kernel's memory layout.

xuchang-vivo · 2026-04-30T04:18:40Z

[AI-generated review summary]

I found one boot/SMP concern that should be addressed before merging.

enable_el1_mmu() and el1_add_linearmap() still initialize TABLE_MANAGER / LINEARMAP_MANAGER only on CPU0, but this PR removes the old PAGETABLE_INIT_DONE release/acquire wait and sev/wfe wakeup. The comment still says other cores wait, while non-CPU0 now goes straight to programming TTBR0/TTBR1. If any AArch64 secondary reaches this path before CPU0 has finished both table initializations, it can install an empty page table; on qemu this becomes especially fragile after TTBR0 is cleared and the kernel relies on TTBR1 for high-VA accesses.

Please either keep a synchronization point for both page tables before secondary cores program TTBR0/TTBR1, or document the boot-order guarantee that makes it impossible for secondary cores to enter this code until CPU0 has completed both initializations.

One related question: the boot path now comments out virt_init() and writes only HCR_EL2.RW inline. That changes the EL2 setup for all AArch64 boards, not just the qemu high-VA path. If this is intentional, please call out why dropping the existing hyp_init() setup is safe.

YinhuaChen-vivo · 2026-04-30T08:25:30Z

Please draw the current kernel's memory layout.

YinhuaChen-vivo · 2026-04-30T08:35:25Z

enable_el1_mmu() and el1_add_linearmap() still initialize TABLE_MANAGER / LINEARMAP_MANAGER only on CPU0, but this PR removes the old PAGETABLE_INIT_DONE release/acquire wait and sev/wfe wakeup. The comment still says other cores wait, while non-CPU0 now goes straight to programming TTBR0/TTBR1. If any AArch64 secondary reaches this path before CPU0 has finished both table initializations, it can install an empty page table; on qemu this becomes especially fragile after TTBR0 is cleared and the kernel relies on TTBR1 for high-VA accesses.

Please either keep a synchronization point for both page tables before secondary cores program TTBR0/TTBR1, or document the boot-order guarantee that makes it impossible for secondary cores to enter this code until CPU0 has completed both initializations.

You’re right. I removed the synchronization code because I found that qemu_virt64_aarch64 only boots CPU0 initially, and the other cores won’t come up until CPU0 executes secondary_cpu_setup. That’s why I took out those sync routines — I thought they were unnecessary.
But you’re correct. AArch64 platforms other than QEMU may behave differently, so those synchronization codes should be kept.

One related question: the boot path now comments out virt_init() and writes only HCR_EL2.RW inline. That changes the EL2 setup for all AArch64 boards, not just the qemu high-VA path. If this is intentional, please call out why dropping the existing hyp_init() setup is safe.

You're right. I commented them out during the initial development because I assumed the EL2 MMU was disabled, and that EL2 would encounter errors when accessing symbol addresses with a linear offset.

However, I found that nearly all the code in virt_init() is compiled into PC-relative addressing instructions by rustc, meaning this code won’t cause errors during the initialization phase. But if an exception ever occurs that causes the CPU to enter EL2—with the EL2 MMU still disabled—these offset exception entry addresses will trap the CPU in a nested EL2 exception infinite loop.

That said, during testing, I’ve confirmed that no normal operations trigger EL2 exceptions under the current setup. So I’ve decided to uncomment the code.

YinhuaChen-vivo · 2026-04-30T08:46:59Z

build_prs

github-actions · 2026-04-30T08:48:26Z

Job is started, see https://github.com/vivoblueos/kernel/actions/runs/25156221089.

github-actions · 2026-04-30T08:54:27Z

❌ Job failed. Failed jobs: build_and_check_boards (failure), see https://github.com/vivoblueos/kernel/actions/runs/25156221089.

YinhuaChen-vivo · 2026-04-30T08:56:38Z

build_prs

github-actions · 2026-04-30T08:58:08Z

Job is started, see https://github.com/vivoblueos/kernel/actions/runs/25156635635.

github-actions · 2026-04-30T09:04:45Z

❌ Job failed. Failed jobs: build_and_check_boards (failure), see https://github.com/vivoblueos/kernel/actions/runs/25156635635.

YinhuaChen-vivo · 2026-04-30T09:05:57Z

build_prs

github-actions · 2026-04-30T09:07:26Z

Job is started, see https://github.com/vivoblueos/kernel/actions/runs/25157043440.

github-actions · 2026-04-30T09:19:55Z

✅ All jobs completed successfully, see https://github.com/vivoblueos/kernel/actions/runs/25157043440.

xuchang-vivo · 2026-05-06T01:48:27Z

 static mut TABLE_MANAGER: PageTableManager = PageTableManager::new();

+#[used]
+#[link_section = ".data"]


Why is it necessary to explicitly specify placement in the .data section? Won't it be automatically placed there?

Yes, LINEARMAP_MANAGER needs to be explicitly placed in the .data section.

Since the initial contents of PageTableManager::new() are all zero-initialized, Rust/LLVM tends to place it in the .bss section instead of .data.

I verified this with:

nm out/shell_test/bin/shell | grep LINEARMAP_MANAGER

By default, it is placed into the .bss section.

Once LINEARMAP_MANAGER is in .bss, after the linear mapping is set up, when execution reaches kernel/kernel/src/boot.rs in the call chain init -> init_runtime -> init_bss(), the function will zero out the entire .bss section.

This resets the EL1 page tables and eventually causes kernel boot failure.

Therefore, LINEARMAP_MANAGER must be explicitly placed in the .data section.

xuchang-vivo · 2026-05-06T01:50:55Z

+#[cfg(target_board = "qemu_virt64_aarch64")]
+pub(crate) const KERNEL_VIRT_START: u64 = u64::MAX << KERNEL_VA_BITS;
+#[cfg(not(target_board = "qemu_virt64_aarch64"))]
+pub(crate) const KERNEL_VIRT_START: u64 = 0;


Here, using kconfig for configuration might be a better choice. It's best not to hardcode assumptions about the target_board in the kernel code. If it's needed temporarily, please add a FIXME note indicating that it will be modified later.

han-jiang277 · 2026-05-07T02:19:36Z

Auto code review

★ Insight ─────────────────────────────────────

The most interesting question in this PR is whether core::ptr::addr_of!(STATIC) as u64 yields a physical or virtual address when running pre-MMU on a kernel linked at high VAs. The answer hinges on Rust's code model: aarch64-unknown-none* uses RelocModel::Static with default Small code model, which emits adrp + add :lo12: (PC-relative). PC-relative loads naturally yield the runtime physical address when executing at PA — the same trick the explicit adrp in enable_el1_mmu relies on.
The new ldr x1, ={stack_end} in enter_el1! looks like it might be unnecessary, but it actually compensates for init's sub sp, x1, x8 (which subtracts cpu_offset again). Without reloading x1 with the unmodified stack_end, sp would have cpu_offset subtracted twice. Subtle but correct.
A "small code model" PC-relative kernel that runs at both PA (pre-MMU) and high VA (post-MMU) only works because the relative offsets between any two symbols are preserved — what changes is the absolute base. This is why the kernel only needs a trampoline (jump_to_high_va) at the boundary: one absolute jump to switch which "base" the same relative addresses resolve against.

No issues found. Checked for bugs and CLAUDE.md compliance.

🤖 Generated with Claude Code

han-jiang277 · 2026-05-12T01:53:28Z

    asm::isb_sy();
 }
+
+pub fn el1_add_linearmap() {


this function looks like a init function, may rename as init_el1_boot_linearmap or el1_init_kernel_linearmap

done.
I rename:
enable_el1_mmu -> init_el1_enable_mmu
el1_add_linearmap -> init_el1_boot_linearmap

YinhuaChen-vivo · 2026-05-12T06:53:49Z

Please draw the current kernel's memory layout.

I add memory layout diagram in mmu.rs as comments:

// ============================================================================
// Linear Mapping Layout (AArch64 EL1)
// ============================================================================
//
// KERNEL_VA_BITS = 39   →  512 GB virtual address space
// LINEAR_OFFSET  = 0xFFFF_FF80_0000_0000  (u64::MAX << 39)
//
//   Virtual Address Space (TTBR1, 39-bit)
//   ┌─────────────────────────────────────────────────────────┐ 0xFFFF_FFFF_FFFF_FFFF
//   │                                                         │
//   │  ┌─────────────────────────────────────────────────────┐│ 0xFFFF_FFFF_0000_0000 + 4 GB
//   │  │  Linear-mapped DRAM (4 × 1 GB L1 blocks)           ││
//   │  │  PA 0x0000_0000_0000 → VA 0xFFFF_FF80_0000_0000    ││
//   │  │  PA 0x0000_4000_0000 → VA 0xFFFF_FF80_4000_0000    ││
//   │  │  PA 0x0000_8000_0000 → VA 0xFFFF_FF80_8000_0000    ││
//   │  │  PA 0x0000_C000_0000 → VA 0xFFFF_FF80_C000_0000    ││
//   │  └─────────────────────────────────────────────────────┘│ 0xFFFF_FF80_0000_0000 ← KERNEL_VIRT_START
//   │                                                         │
//   │  (unmapped)                                             │
//   │                                                         │
//   └─────────────────────────────────────────────────────────┘ 0xFFFF_FF80_0000_0000
//
//   Translation:
//     VA = PA + LINEAR_OFFSET   (kernel_phys_to_virt)
//     PA = VA - LINEAR_OFFSET   (kernel_virt_to_phys)
//
// ============================================================================

YinhuaChen-vivo · 2026-05-12T08:03:26Z

build_prs

github-actions · 2026-05-12T08:04:44Z

Job is started, see https://github.com/vivoblueos/kernel/actions/runs/25721648189.

1.disable virtualization 2.rename enable_mmu -> enable_el1_mmu 3.mmu.rs: add el1_add_linearmap() 4.invoke el1_add_linearmap when booting blueos on aarch64 platform 5.after eret, jump to 'jump_to_high_va' 6.add 'ldr x1, ={stack_end}' 7.change jump_to_high_va: br x16 jump to aarch64::init 8.add KERNEL_OFFSET in qemu_virt64_aarch64/link.x 9.add kernel_virt_to_phys / kernel_phys_to_virt 10.change virtio: virt_to_phys/phys_to_virt

…aarch64

enable_el1_mmu -> init_el1_enable_mmu el1_add_linearmap -> init_el1_boot_linearmap

github-actions · 2026-05-12T08:10:58Z

❌ Job failed. Failed jobs: build_and_check_boards (failure), see https://github.com/vivoblueos/kernel/actions/runs/25721648189.

YinhuaChen-vivo · 2026-05-12T08:11:35Z

build_prs

github-actions · 2026-05-12T08:13:07Z

Job is started, see https://github.com/vivoblueos/kernel/actions/runs/25722028627.

github-actions · 2026-05-12T08:26:37Z

✅ All jobs completed successfully, see https://github.com/vivoblueos/kernel/actions/runs/25722028627.

xuchang-vivo reviewed May 6, 2026

View reviewed changes

YinhuaChen-vivo mentioned this pull request May 7, 2026

AArch64 EL1 4KB paging #408

Draft

han-jiang277 reviewed May 12, 2026

View reviewed changes

han-jiang277 approved these changes May 12, 2026

View reviewed changes

YinhuaChen-vivo force-pushed the feature-linearmap branch from bd01482 to be9323d Compare May 12, 2026 06:50

YinhuaChen-vivo added 9 commits May 12, 2026 16:09

Cancel zeroing TTBR0_EL1 on all AArch64 platforms except qemu_virt64_…

f054bc2

…aarch64

format

20b5f47

uncomment 'bl {virt_init}'

1b872e3

restore sync code in enable_el1_mmu & el1_add_linearmap

f74b0c2

Add an ASCII memory layout diagram in mmu.rs

8e15cd2

remove KERNEL_VIRT_START setting from mmu.rs -> Kconfig

b61a7d7

add FIXME at the hardcoded board-specific configuration

bc0ec18

Rename some functions

ef75e98

enable_el1_mmu -> init_el1_enable_mmu el1_add_linearmap -> init_el1_boot_linearmap

YinhuaChen-vivo force-pushed the feature-linearmap branch from b697912 to ef75e98 Compare May 12, 2026 08:11

Conversation

YinhuaChen-vivo commented Apr 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Main Changes

Memory Layout

Files Changed

Notes

Uh oh!

YinhuaChen-vivo commented Apr 30, 2026

Uh oh!

github-actions Bot commented Apr 30, 2026

Uh oh!

github-actions Bot commented Apr 30, 2026

Uh oh!

YinhuaChen-vivo commented Apr 30, 2026

Uh oh!

github-actions Bot commented Apr 30, 2026

Uh oh!

YinhuaChen-vivo commented Apr 30, 2026

Uh oh!

github-actions Bot commented Apr 30, 2026

Uh oh!

github-actions Bot commented Apr 30, 2026

Uh oh!

github-actions Bot commented Apr 30, 2026

Uh oh!

xuchang-vivo commented Apr 30, 2026

Uh oh!

xuchang-vivo commented Apr 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

YinhuaChen-vivo commented Apr 30, 2026

Uh oh!

YinhuaChen-vivo commented Apr 30, 2026

Uh oh!

YinhuaChen-vivo commented Apr 30, 2026

Uh oh!

github-actions Bot commented Apr 30, 2026

Uh oh!

github-actions Bot commented Apr 30, 2026

Uh oh!

YinhuaChen-vivo commented Apr 30, 2026

Uh oh!

github-actions Bot commented Apr 30, 2026

Uh oh!

github-actions Bot commented Apr 30, 2026

Uh oh!

YinhuaChen-vivo commented Apr 30, 2026

Uh oh!

github-actions Bot commented Apr 30, 2026

Uh oh!

github-actions Bot commented Apr 30, 2026

Uh oh!

xuchang-vivo May 6, 2026

Choose a reason for hiding this comment

Uh oh!

YinhuaChen-vivo May 12, 2026

Choose a reason for hiding this comment

Uh oh!

xuchang-vivo May 6, 2026

Choose a reason for hiding this comment

Uh oh!

YinhuaChen-vivo May 12, 2026

Choose a reason for hiding this comment

Uh oh!

han-jiang277 commented May 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Auto code review

Uh oh!

han-jiang277 May 12, 2026

Choose a reason for hiding this comment

Uh oh!

YinhuaChen-vivo May 12, 2026

Choose a reason for hiding this comment

Uh oh!

YinhuaChen-vivo commented May 12, 2026

Uh oh!

YinhuaChen-vivo commented May 12, 2026

Uh oh!

YinhuaChen-vivo commented Apr 30, 2026 •

edited

Loading

xuchang-vivo commented Apr 30, 2026 •

edited

Loading

han-jiang277 commented May 7, 2026 •

edited

Loading