feat(android): 交付 APK v1 移动端运行时与悬浮窗录音 by HKLHaoBin · Pull Request #641 · Open-Less/openless

HKLHaoBin · 2026-06-10T15:34:47Z

User description

摘要

为 OpenLess 增加 Android APK v1 移动端运行时：在保持桌面端零破坏的前提下，完成应用内录音、麦克风权限、悬浮窗前台服务、手势驱动的听写/问答管线，以及 debug APK CI 构建链路；修复真机 native 加载与 JNI 初始化等多轮闪退。

修复 / 新增 / 改进

新增 Android mobile/desktop 分层运行时：lib.rs、mobile_runtime.rs、mobile_stubs/*、PlatformCapabilities 门控，前端 platform.ts / ipc.ts 在移动端隐藏桌面专属能力
新增移动端响应式 UI：主界面、设置页 Android 权限区块、QaPanel 移动端录音入口、「打开 OpenLess」跳回主应用
新增 Android debug APK CI：.github/workflows/android-apk.yml（workflow_dispatch + v*-tauri tag）、四 ABI 分包产物、merge-android-v1-manifest.mjs / copy-android-scaffolding.mjs
新增悬浮窗 v3：OpenLessOverlayService、AccessibilityService、JNI bridge（android_jni.rs / android_native_bridge.rs）、听写与 QA 管线接线
新增悬浮窗手势与偏好：androidOverlayActivationMode、左滑动作、取消滑向、androidOverlaySizeDp；搁置「键盘触发显示」并迁移历史值为 background
新增 Android 麦克风权限：UI 触发申请、grant 状态持久化、原生权限查询、绕过桌面麦克风 gate
修复真机 __cxa_pure_virtual：build.rs 在 CARGO_CFG_TARGET_OS=android 时链接 c++_static + c++abi
修复 ndk-context 未初始化、FGS 启动超时、clipboard / accessibility JNI 签名与生命周期、overlay 重复窗口等多轮真机崩溃
修复 CI 回归：macos-private-api 恢复至 base tauri features；windows_ime_profile stub 暴露；clipboard delay 去重
改进 set_settings_common 抽取共享偏好写入；.gitignore 忽略 Android 调试日志与本地 CI 产物；补充 docs/android-mobile-apk-overlay-plan.md

兼容

不包含：
- Android 本地 ASR（Foundry / Sherpa / Qwen）
- IME v2 完整跨 App 文本插入主路径
- 「弹出键盘时显示悬浮窗」（已搁置，历史值静默迁移为 background）
- Release 签名 APK（当前 CI 仅产出 debug APK）
对现有用户 / 本地环境 / 构建流程的影响：
- 桌面 macOS / Windows / Linux 功能语义与热键 / 托盘 / 浮窗逻辑不变；变更均经 #[cfg(mobile)] / #[cfg(not(mobile))] / capability 门控隔离
- 不执行 tauri:android:* 的开发者无额外依赖；现有 cargo check 与桌面 release 流程不受影响
- 新增可选 Android CI job，不修改 release-tauri.yml 桌面发布链路
- Android 真机构建需 ANDROID_HOME / NDK_HOME 及 aarch64-linux-android 等 Rust target

测试计划

命令：cd openless-all/app && npm run build && cargo check --manifest-path src-tauri/Cargo.toml
结果：桌面侧编译通过，无 mobile 分层污染
证据路径：本地 cargo check 输出
命令：gh workflow run "Android APK (debug)" --repo appergb/openless --ref openless-android → gh run watch <run-id> --exit-status
结果：四 ABI debug APK artifact openless-android-debug 上传成功
证据路径：GitHub Actions run URL / 下载的 OpenLess-android-debug-run-*.apk
命令：adb install -r <apk> → adb shell am start -n com.openless.app/.MainActivity → 授权麦克风与悬浮窗 → 应用内录音 → 悬浮窗手势（点按听写 / 左滑 QA / 录音中上滑取消）→ adb logcat -b crash -d
结果：无 __cxa_pure_virtual / ndk-context / FGS timeout 闪退；录音转写与 QA 流程可完成
证据路径：test-captures/openless_after_install.png、test-captures/openless_app_recording_after_tap.png、test-captures/openless_app_recording_after_stop.png、test-captures/run73-start.png、test-captures/qa-repeat-current.png

PR Type

Enhancement, Bug fix

Description

Add Android mobile runtime with platform capabilities gating
Implement floating overlay v3 with dictation and QA pipelines
Add microphone and accessibility permission handling
Enable debug APK CI and fix JNI initialization crashes

Diagram Walkthrough

flowchart LR
  Entry["Main Entry"] --> Check{"Platform"}
  Check --> Desktop["Desktop Runtime"]
  Check --> Mobile["Mobile Runtime"]
  Mobile --> Overlay["Floating Overlay"]
  Mobile --> Permissions["Permissions"]
  Permissions --> Mic["Microphone"]
  Permissions --> OverlayPerm["Overlay"]
  Permissions --> Accessibility["Accessibility"]
  Overlay --> Gestures["Gesture Dictation/QA"]

File Walkthrough

Relevant files

Enhancement

19 files

commands.rs `Refactor commands for mobile conditional compilation and add Android` `commands`	+284/-37
lib.rs `Split runtime into desktop and mobile with macro handler selection`	+357/-150
coordinator.rs `Add Android overlay and QA session management logic`	+649/-11
jni.rs `Add JNI bridge for Android system APIs`	+510/-0
mobile_runtime.rs `Implement minimal mobile Tauri runtime`	+81/-0
native_bridge.rs `Add native bridge for overlay communication`	+396/-0
types.rs `Add Android-specific types and enums`	+323/-0
overlay.rs `Implement floating overlay lifecycle`	+145/-0
accessibility.rs `Implement accessibility service integration`	+123/-0
external_url.rs `Add external URL opening support`	+70/-0
insert.rs `Add text insertion strategy for Android`	+51/-0
mod.rs `Module declaration for Android routines`	+25/-0
AndroidPermissionsPanel.tsx `Add Android permissions UI settings panel`	+386/-0
androidMicrophonePermission.ts `Implement microphone permission bridge`	+104/-0
platform.ts `Add PlatformCapabilities detection for mobile`	+122/-0
androidMicrophonePermission.ts `Add Android microphone permission hook`	+1/-0
selection.rs `Provide mobile stub for selection module`	+54/-0
combo_hotkey.rs `Provide mobile stub for combo hotkey`	+46/-0
hotkey.rs `Provide mobile stub for hotkey module`	+49/-0

Configuration changes

1 files

android-apk.yml `Add CI workflow for debug APK building`	+241/-0

Additional files

80 files

ci.yml	+1/-0
README.md	+6/-0
android-mobile-apk-overlay-plan.md	+315/-0
README.md	+79/-0
androidIpc.ts	+43/-0
androidTypes.ts	+38/-0
MicrophonePermissionActivity.kt	+49/-0
OpenLessAccessibilityCommandReceiver.kt	+36/-0
OpenLessAccessibilityService.kt	+373/-0
OpenLessAndroidPreferences.kt	+112/-0
OpenLessAppContext.kt	+13/-0
OpenLessApplication.kt	+84/-0
OpenLessNative.kt	+42/-0
OpenLessOverlayBridge.kt	+33/-0
OpenLessOverlayService.kt	+831/-0
OpenLessPermissionBridge.kt	+46/-0
OverlayPermissionActivity.kt	+27/-0
README.md	+27/-0
AndroidManifest.v1.snippet.xml	+7/-0
AndroidManifest.v3.snippet.xml	+20/-0
openless_accessibility_config.xml	+9/-0
openless_ime_method.xml	+9/-0
package.json	+6/-0
copy-android-scaffolding.mjs	+187/-0
merge-android-overlay-manifest.mjs	+174/-0
merge-android-v1-manifest.mjs	+133/-0
Cargo.toml	+20/-9
build.rs	+11/-0
default.json	+1/-0
mobile.json	+18/-0
ic_overlay_logo.xml	+4/-0
download.rs	+8/-6
dictation.rs	+52/-35
qa.rs	+2/-2
insertion.rs	+62/-29
qa_hotkey.rs	+33/-0
shortcut_binding.rs	+38/-0
unicode_keystroke.rs	+40/-0
permissions.rs	+47/-2
persistence.rs	+158/-57
recorder.rs	+46/-7
types.rs	+150/-1
tauri.android.conf.json	+19/-0
tauri.conf.json	+3/-0
App.tsx	+116/-23
AudioCue.tsx	+11/-10
AutoUpdate.tsx	+7/-16
AutoUpdateGate.tsx	+11/-2
FloatingShell.tsx	+183/-20
Onboarding.tsx	+443/-76
SettingsModal.tsx	+52/-45
WindowChrome.tsx	+4/-3
Row.tsx	+5/-3
SelectLite.tsx	+10/-0
en.ts	+103/-0
ja.ts	+69/-0
ko.ts	+69/-0
zh-CN.ts	+103/-0
zh-TW.ts	+69/-0
ipc.ts	+168/-27
stylePrefs.test.ts	+6/-0
types.ts	+48/-1
useMobileLayout.ts	+25/-0
History.tsx	+20/-10
Overview.tsx	+224/-14
QaPanel.tsx	+244/-17
_atoms.tsx	+22/-6
AboutSection.tsx	+25/-3
BetaChannelSection.tsx	+9/-1
LocalModelSection.tsx	+10/-2
MicrophoneSelect.tsx	+3/-1
PermissionsSection.tsx	+66/-21
ProvidersSection.tsx	+32/-11
RecordingInputSection.tsx	+21/-4
ShortcutsSection.tsx	+12/-0
shared.tsx	+9/-4
tabs.tsx	+25/-6
tokens.css	+6/-0
tsconfig.json	+6/-2
vite.config.ts	+18/-4

引入独立的 Tauri 移动端运行时，具备专有的调用处理器（invoke handler）、移动端存根（mobile stubs）以及针对特定目标的 Cargo 依赖。由此，Android 构建版本将不再引入仅限桌面的 crate（如 updater、autostart、single-instance、global-hotkey、enigo）。桌面端行为在 run_desktop() 逻辑下保持不变。新增 PlatformCapabilities（包含 supportsAutoUpdate、supportsInAppDictation 等字段）并据此对前端进行门控处理：在概览页启用应用内听写；Android 入门引导不再因麦克风权限处于“未确定”状态而阻塞；同时在移动端隐藏桌面热键、本地 ASR 以及自动更新/Beta 渠道的相关 UI。包含 Android 清单文件代码片段（v1 版本的 RECORD_AUDIO；标记为未来版本的 v2/v3 IME/叠加层权限）、Kotlin 基础架构、JNI 权限/IME/叠加层存根、移动端 tauri.android.conf.json 以及 docs/android-mobile-apk-overlay-plan.md 文档。

引入了 GitHub Actions 工作流，以实现 Android 应用调试 APK 构建的自动化。该工作流在匹配 'v*-tauri' 模式的标签推送时触发，并支持手动调度。它包括设置 Java 环境、Android SDK 和 NDK、安装依赖项、构建前端以及合并 APK 清单等步骤。生成的 APK 将作为 artifact 上传，并可附加到 GitHub Releases 中。此外，还更新了 package.json，新增了一个用于合并 Android 清单的脚本。

在文档中添加了 Android capability 的平台隔离详情。更新了 default.json 以指定桌面端支持的平台，并更新了 mobile.json 将 Android 定义为目标平台。这确保了应用程序能够正确处理权限并实现特定于平台的行为。

更新了文档和 default.json，将“macos”统一改为“macOS”，以保持各平台间的一致性。此更改增强了清晰度，并维护了平台命名规范的统一。

Co-authored-by: Cursor <cursoragent@cursor.com>

• 分别删除了 android_ime.rs 和 android_overlay.rs 中未使用的 AndroidImeState 和 AndroidOverlayPermissionState 导入。 • 更新了 insertion.rs 中的条件编译标志，在非 macOS 和非 Android 的检查中排除了 iOS。 • 在 lib.rs 的 app_invoke_handler_mobile 中添加了 #[macro_export] 以提高可访问性。 • 调整了 types.rs 中的类型检查，将移动平台纳入功能条件中。

在 app_invoke_handler_mobile 宏中，将直接命令引用替换为 crate 限定路径，以提高整个代码库的清晰度和一致性。

• 引入了 set_settings_common 来处理持久化用户偏好设置和发布更改的核心逻辑。 • 更新了各平台特定的 set_settings 函数以使用新的通用函数，从而提高了代码复用性和清晰度。 • 确保非移动平台的托盘麦克风菜单刷新的主线程处理保持不变。

• 添加了 Gradle 设置操作以优化构建流程。 • 引入了带有重试逻辑的 Gradle Wrapper 预热步骤，以提高构建过程中的可靠性。

Fixes startup UnsatisfiedLinkError for __cxa_pure_virtual when loading libopenless_lib.so. Co-authored-by: Cursor <cursoragent@cursor.com>

Plain c++_static left __cxa_pure_virtual undefined at dlopen; whole-archive pulls required symbols from libc++_static.a. Co-authored-by: Cursor <cursoragent@cursor.com>

build-script cfg(target_os=android) never ran on Linux CI hosts; use runtime target check so whole-archive c++_static applies. Co-authored-by: Cursor <cursoragent@cursor.com>

static=c++_static failed without -L; CARGO_CFG_TARGET_OS ensures the link runs on Linux CI hosts. Co-authored-by: Cursor <cursoragent@cursor.com>

Co-authored-by: Cursor <cursoragent@cursor.com>

Compile windows_ime_profile on all desktop targets so get_windows_ime_status can call its non-Windows stub. Remove duplicate CLIPBOARD_RESTORE_DELAY cfg on Windows. Co-authored-by: Cursor <cursoragent@cursor.com>

HKLHaoBin · 2026-06-10T15:37:46Z

目前还有两个功能没有写完：一个是 QA 功能，另一个是取消录音功能。不过在这两个功能完成之前，我决定先请求合并这个 PR。因为如果研发周期再拖长，产生的冲突可能会达到无法处理的程度。为了尽可能跟上项目的最新变更，建议先合并这个 PR，随后在最新的 Beta 版本中完成剩余的 QA 功能和取消录音功能。

github-actions · 2026-06-10T15:37:48Z

PR Reviewer Guide 🔍

(Review updated until commit `8eebffb`)

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 4 🔵🔵🔵🔵⚪
🧪 No relevant tests
🔒 No security concerns identified
⚡ No major issues detected

Consolidate Rust android_* modules, Kotlin scaffolding, and frontend Android UI under openless-all/app/android/ with crate::android and @android imports to improve maintainability without changing runtime behavior. Co-authored-by: Cursor <cursoragent@cursor.com>

Load android/types.rs from types.rs directly so backend-tests can compile types without the full android module tree. Co-authored-by: Cursor <cursoragent@cursor.com>

github-actions · 2026-06-10T16:35:03Z

Persistent review updated to latest commit 237a60a

HKLHaoBin · 2026-06-10T17:12:53Z

Stage deletions left over from the android/ directory migration so the working tree matches the relocated kotlin and manifest sources. Co-authored-by: Cursor <cursoragent@cursor.com>

github-actions · 2026-06-11T04:22:58Z

Persistent review updated to latest commit 58c7c91

H-Chris233 · 2026-06-11T09:00:39Z

有冲突，请解决一下

github-actions · 2026-06-11T10:45:14Z

Persistent review updated to latest commit 841215e

github-actions · 2026-06-11T11:35:22Z

Persistent review updated to latest commit 582af09

HKLHaoBin · 2026-06-11T11:38:44Z

合并安卓的可以先放缓，因为我又新增了一些功能，并尝试修复一些已经存在的问题。
如果需要合并，请合并 58c7c91 这个提交的代码。

github-actions · 2026-06-11T11:58:41Z

Persistent review updated to latest commit cef63e1

扩展 Android overlay/accessibility 类型与 JNI 桥接，优化悬浮窗交互与权限引导 UI，并同步 coordinator 移动端录音协调逻辑。 Co-authored-by: Cursor <cursoragent@cursor.com>

github-actions · 2026-06-11T13:24:27Z

Persistent review updated to latest commit 8eebffb

HKLHaoBin · 2026-06-11T13:50:29Z

可以合并全部了。

HKLHaoBin and others added 30 commits June 8, 2026 00:00

fix(reqwest): 从 reqwest 依赖中移除 native-tls 特性，并改为在非移动平台的客户端构建器中条件式使用。

a4e5f65

fix(docs): 规范化能力文档和 JSON 文件中的 macOS 大写

266fff0

更新了文档和 default.json，将“macos”统一改为“macOS”，以保持各平台间的一致性。此更改增强了清晰度，并维护了平台命名规范的统一。

Fix Android APK build: TLS, capabilities, arboard

c768da2

Co-authored-by: Cursor <cursoragent@cursor.com>

重构：更新 app_invoke_handler_mobile 宏中的命令引用以使用 crate 路径

4892b00

在 app_invoke_handler_mobile 宏中，将直接命令引用替换为 crate 限定路径，以提高整个代码库的清晰度和一致性。

feat(ci): 通过 Gradle 设置和 Wrapper 预热增强 Android APK 构建工作流

c4eb1ac

• 添加了 Gradle 设置操作以优化构建流程。 • 引入了带有重试逻辑的 Gradle Wrapper 预热步骤，以提高构建过程中的可靠性。

Link Android NDK c++_static for cpal/oboe C++ runtime.

d5c2e49

Fixes startup UnsatisfiedLinkError for __cxa_pure_virtual when loading libopenless_lib.so. Co-authored-by: Cursor <cursoragent@cursor.com>

Use whole-archive static=c++_static for Android libc++ link.

f759dc8

Plain c++_static left __cxa_pure_virtual undefined at dlopen; whole-archive pulls required symbols from libc++_static.a. Co-authored-by: Cursor <cursoragent@cursor.com>

Fix Android libc++ link via CARGO_CFG_TARGET_OS in build.rs.

d8de25d

build-script cfg(target_os=android) never ran on Linux CI hosts; use runtime target check so whole-archive c++_static applies. Co-authored-by: Cursor <cursoragent@cursor.com>

Use -lc++_static for Android NDK link in build.rs.

79ef162

static=c++_static failed without -L; CARGO_CFG_TARGET_OS ensures the link runs on Linux CI hosts. Co-authored-by: Cursor <cursoragent@cursor.com>

Link Android libc++ via NDK -lc++_static with whole-archive.

6362ddf

Co-authored-by: Cursor <cursoragent@cursor.com>

Link Android c++_static and c++abi for missing C++ ABI symbols.

bd85353

Co-authored-by: Cursor <cursoragent@cursor.com>

Avoid Android startup microphone probe panic

6e1365d

Avoid Android permission JNI startup panic

9d5c455

Add responsive mobile app shell

19e3933

Skip provider prompt on Android startup

b6a6f2b

Fix mobile settings layout

e1e7c74

Request Android microphone permission from UI

387cc22

Persist Android microphone grant state

74ebe1e

Treat Android microphone stream errors as granted

c67c2dc

Prefer stored Android microphone grant

4cd4dcb

Ignore Android WebView microphone preflight denial

0523710

Fix Android external links

51b6cda

Add Android audio settings permission

b624af2

Bypass Android desktop microphone gate

f35b4db

Initialize Android context for audio backend

a215bcd

fix(ci): expose windows_ime_profile stub and dedupe clipboard delay

3e02c00

Compile windows_ime_profile on all desktop targets so get_windows_ime_status can call its non-Windows stub. Remove duplicate CLIPBOARD_RESTORE_DELAY cfg on Windows. Co-authored-by: Cursor <cursoragent@cursor.com>

HKLHaoBin and others added 2 commits June 11, 2026 00:18

fix(ci): include android types via path for backend-tests harness

237a60a

Load android/types.rs from types.rs directly so backend-tests can compile types without the full android module tree. Co-authored-by: Cursor <cursoragent@cursor.com>

github-actions Bot added the Review effort 4/5 label Jun 10, 2026

chore(android): remove migrated android-scaffolding from git index

58c7c91

Stage deletions left over from the android/ directory migration so the working tree matches the relocated kotlin and manifest sources. Co-authored-by: Cursor <cursoragent@cursor.com>

github-actions Bot added Review effort 5/5 and removed Review effort 4/5 labels Jun 11, 2026

Fix Android overlay gestures and permissions

841215e

Complete Android setup and QA permission flow

582af09

Add Android icon safe zone

cef63e1

github-actions Bot added Review effort 4/5 and removed Review effort 5/5 labels Jun 11, 2026

feat(android): 完善悬浮窗手势、权限面板与无障碍粘贴链路

8eebffb

扩展 Android overlay/accessibility 类型与 JNI 桥接，优化悬浮窗交互与权限引导 UI，并同步 coordinator 移动端录音协调逻辑。 Co-authored-by: Cursor <cursoragent@cursor.com>

H-Chris233 added enhancement New feature or request platform:android Android-specific large-change Large or risky change experimental Experimental feature or change needs-review Needs review labels Jun 11, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(android): 交付 APK v1 移动端运行时与悬浮窗录音#641

feat(android): 交付 APK v1 移动端运行时与悬浮窗录音#641
HKLHaoBin wants to merge 83 commits into
Open-Less:betafrom
HKLHaoBin:openless-android

HKLHaoBin commented Jun 10, 2026 •

edited by github-actions Bot

Loading

Uh oh!

HKLHaoBin commented Jun 10, 2026

Uh oh!

github-actions Bot commented Jun 10, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jun 10, 2026

Uh oh!

HKLHaoBin commented Jun 10, 2026

Uh oh!

github-actions Bot commented Jun 11, 2026

Uh oh!

H-Chris233 commented Jun 11, 2026

Uh oh!

github-actions Bot commented Jun 11, 2026

Uh oh!

github-actions Bot commented Jun 11, 2026

Uh oh!

HKLHaoBin commented Jun 11, 2026

Uh oh!

github-actions Bot commented Jun 11, 2026

Uh oh!

github-actions Bot commented Jun 11, 2026

Uh oh!

HKLHaoBin commented Jun 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

HKLHaoBin commented Jun 10, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

User description

摘要

修复 / 新增 / 改进

兼容

测试计划

PR Type

Description

Diagram Walkthrough

File Walkthrough

Uh oh!

HKLHaoBin commented Jun 10, 2026

Uh oh!

github-actions Bot commented Jun 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Reviewer Guide 🔍

(Review updated until commit 8eebffb)

Uh oh!

github-actions Bot commented Jun 10, 2026

Uh oh!

HKLHaoBin commented Jun 10, 2026

摘要

修复 / 新增 / 改进

兼容

测试计划

Uh oh!

github-actions Bot commented Jun 11, 2026

Uh oh!

H-Chris233 commented Jun 11, 2026

Uh oh!

github-actions Bot commented Jun 11, 2026

Uh oh!

github-actions Bot commented Jun 11, 2026

Uh oh!

HKLHaoBin commented Jun 11, 2026

Uh oh!

github-actions Bot commented Jun 11, 2026

Uh oh!

github-actions Bot commented Jun 11, 2026

Uh oh!

HKLHaoBin commented Jun 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

HKLHaoBin commented Jun 10, 2026 •

edited by github-actions Bot

Loading

github-actions Bot commented Jun 10, 2026 •

edited

Loading

(Review updated until commit `8eebffb`)