Skip to content

feat: 基于 Bun + TypeScript + ESM 的全面重构#2

Draft
monkeym4ster wants to merge 14 commits intodevfrom
cursor/-bc-b1dc226d-842b-4819-8661-30b05b74a796-f3d9
Draft

feat: 基于 Bun + TypeScript + ESM 的全面重构#2
monkeym4ster wants to merge 14 commits intodevfrom
cursor/-bc-b1dc226d-842b-4819-8661-30b05b74a796-f3d9

Conversation

@monkeym4ster
Copy link
Copy Markdown
Owner

@monkeym4ster monkeym4ster commented Apr 1, 2026 •
edited by cursor bot
Loading

概要

将 WhatsWeb 从 Node.js + CommonJS + JavaScript 全面重构为 Bun + TypeScript + ESM 架构。

测试结果

89 pass, 0 fail, 506 expect() calls
Ran 89 tests across 19 files

重构前 vs 重构后

维度 重构前 重构后
运行时 Node.js Bun
语言 JavaScript (CJS) TypeScript (ESM)
HTTP superagent Bun 原生 fetch
并发 bluebird.Promise.map p-limit@7
工具库 无 es-toolkit
验证 无 Zod (CLI参数/插件meta/规则文件)
着色 chalk@2 picocolors
文件匹配 globby@7 Bun.Glob
技术栈识别 wappalyzer@5 wappalyzer@6.10.66
测试 无 89 个测试 (bun:test)
Lint 无 Biome
类型安全 无 TypeScript strict mode

架构变更

新目录结构

src/
├── cli.ts                    # CLI 入口 (commander@12)
├── index.ts                  # 库入口
├── schemas/cli.ts            # Zod CLI 参数 schema
├── core/
│   ├── scanner.ts            # 核心扫描引擎
│   ├── plugin-loader.ts      # 插件加载器
│   └── reporter.ts           # 结果格式化
├── plugins/
│   ├── types.ts              # 插件接口 + Zod schema
│   ├── base-info.ts          # 基础信息
│   ├── email.ts              # 邮箱提取 (es-toolkit uniq)
│   ├── geoip.ts              # 地理位置
│   ├── wappalyzer.ts         # 技术栈识别
│   └── bbscan/
│       ├── index.ts          # BBScan 插件入口
│       ├── rule-parser.ts    # 规则解析 (Zod schema + Bun.Glob)
│       └── scanner.ts        # 路径扫描 (p-limit 并发50)
└── utils/
    ├── url.ts                # URL 规范化 + Zod urlSchema
    ├── dns.ts                # DNS 解析
    ├── ip.ts                 # CIDR 展开 (Generator)
    ├── http.ts               # HTTP 封装 (原生 fetch)
    └── concurrency.ts        # 并发控制 (p-limit 封装)

已删除文件

  • index.js, whatsweb.js, utils.js
  • plugins/*.js (5 个旧插件文件)

保持不变

  • rules/ 目录 (所有规则文件完全不变)
  • CLI 选项向后兼容 (-f, -c, --network, --timeout, --user-agent, -o, --show-error)

提交历史

  1. chore: Phase 1 — Bun 项目初始化、TS/Biome 配置
  2. feat: Phase 2 — 工具层 TDD (url/dns/ip/http/concurrency)
  3. feat: Phase 3 — 插件类型定义与加载器 (Zod 验证)
  4. feat: Phase 4 — 基础插件 TDD (base-info/email/geoip)
  5. feat: Phase 5 — Wappalyzer 插件 (wappalyzer@6.10.66)
  6. feat: Phase 6 — BBScan 插件 (rule-parser/scanner/p-limit/Zod)
  7. feat: Phase 7 — 核心扫描引擎 Scanner
  8. feat: Phase 8 — CLI + Reporter
  9. feat: Phase 9 — 集成测试、清理、文档更新
Open in Web Open in Cursor 

cursoragent and others added 2 commits April 1, 2026 08:43
@cursoragent @monkeym4ster
docs: 添加基于 Bun 的全面重构计划文档
26251c0 
Co-authored-by: M4ster <monkeym4ster@users.noreply.github.com>
@cursoragent @monkeym4ster
docs: 拆分重构计划为 9 个 Phase 的 SOP 文档体系
981c5fa 
- 创建 docs/sop/ 目录，包含 SOP 总览 + 9 个 Phase 文档
- 引入 es-toolkit (替代 lodash)、p-limit@7 (并发管理)、zod (数据验证)
- 确认 wappalyzer@6.10.66 的 API (init/open/analyze/destroy)
- 每个 Phase 均包含 TDD 流程：测试代码 → 实现代码 → 验证清单
- 更新 REFACTORING_PLAN.md 技术选型和里程碑索引

Co-authored-by: M4ster <monkeym4ster@users.noreply.github.com>
@cursor cursor bot changed the title docs: 添加基于 Bun 的 WhatsWeb 全面重构计划 docs: WhatsWeb 基于 Bun 的全面重构计划 + SOP 文档体系 Apr 1, 2026
cursoragent and others added 9 commits April 1, 2026 09:15
@cursoragent @monkeym4ster
chore: Phase 1 — 初始化 Bun 项目基础设施
5ddff57 
- 配置 tsconfig.json, biome.json
- 创建 src/ 目录骨架 (core, plugins, utils, schemas)
- 创建 tests/ 目录骨架
- 定义基础类型 (Plugin, PluginContext, HttpResponse)
- 安装依赖 (es-toolkit, p-limit, zod, commander, picocolors 等)
- smoke test 通过

Co-authored-by: M4ster <monkeym4ster@users.noreply.github.com>
@cursoragent @monkeym4ster
feat: Phase 2 — 工具层 TDD (url/dns/ip/http/concurrency)
2e3de22 
- src/utils/url.ts: normalizeUrl + Zod urlSchema
- src/utils/dns.ts: resolve4 (node:dns/promises)
- src/utils/ip.ts: ipToLong/longToIp/expandCIDR (Generator)
- src/utils/http.ts: httpGet/httpHead (原生 fetch + AbortController)
- src/utils/concurrency.ts: mapConcurrent (基于 p-limit@7)
- 29 个测试全部通过

Co-authored-by: M4ster <monkeym4ster@users.noreply.github.com>
@cursoragent @monkeym4ster
feat: Phase 3 — 插件类型定义与加载器 (Zod 验证)
f39fb41 
- src/plugins/types.ts: Plugin/PluginContext/PluginMeta + Zod schemas
- src/core/plugin-loader.ts: PluginLoader (register/getAll/get/withBuiltins)
- 6 个测试全部通过

Co-authored-by: M4ster <monkeym4ster@users.noreply.github.com>
@cursoragent @monkeym4ster
feat: Phase 4 — 基础插件 TDD (base-info/email/geoip)
ad64204 
- src/plugins/base-info.ts: 状态码/标题/重定向/x-*头提取
- src/plugins/email.ts: 邮箱正则 + es-toolkit uniq 去重
- src/plugins/geoip.ts: DNS 解析 + geoip-lite 地理位置
- 注册到 PluginLoader.withBuiltins()
- 23 个测试全部通过

Co-authored-by: M4ster <monkeym4ster@users.noreply.github.com>
@cursoragent @monkeym4ster
feat: Phase 5 — Wappalyzer 插件 (wappalyzer@6.10.66)
2bef46a 
- src/plugins/wappalyzer.ts: init/open/analyze/destroy 生命周期
- finally 确保 destroy 清理浏览器进程
- 错误时优雅返回 null
- 3 个测试全部通过

Co-authored-by: M4ster <monkeym4ster@users.noreply.github.com>
@cursoragent @monkeym4ster
feat: Phase 6 — BBScan 插件 (rule-parser + scanner + p-limit + Zod)
84518cd 
- src/plugins/bbscan/rule-parser.ts: Zod schema 验证规则，Bun.Glob 加载文件
- src/plugins/bbscan/scanner.ts: 404 检测, p-limit 并发 50, 路径变量替换
- src/plugins/bbscan/index.ts: Plugin 接口实现
- 含 tag 规则用 GET、其他用 HEAD (修复原代码 bug)
- 13 个测试全部通过

Co-authored-by: M4ster <monkeym4ster@users.noreply.github.com>
@cursoragent @monkeym4ster
feat: Phase 7 — 核心扫描引擎 Scanner (Zod 验证)
8acff12 
- src/core/scanner.ts: Scanner 类, Zod 参数验证, 插件错误隔离
- src/index.ts: 导出 Scanner/PluginLoader/类型
- 支持自定义 PluginLoader 和默认内置插件
- 5 个测试全部通过

Co-authored-by: M4ster <monkeym4ster@users.noreply.github.com>
@cursoragent @monkeym4ster
feat: Phase 8 — CLI + Reporter (commander/picocolors/cli-progress/p-l…
3dfdec4 
…imit/zod)

- src/schemas/cli.ts: Zod CLI 参数验证 schema
- src/core/reporter.ts: 彩色控制台输出 + JSONL 文件输出
- src/cli.ts: 完整 CLI (commander@12/picocolors/cli-progress/p-limit/es-toolkit)
- 全部 87 个测试通过，lint 无错误

Co-authored-by: M4ster <monkeym4ster@users.noreply.github.com>
@cursoragent @monkeym4ster
feat: Phase 9 — 集成测试、清理旧文件、更新文档
17e2159 
- tests/integration/e2e.test.ts: 端到端集成测试 (Scanner + CLI)
- 删除旧文件: index.js, whatsweb.js, utils.js, plugins/
- 更新 README.md (新架构文档)
- 更新 docs/sop/README.md 进度表 (全部 ✅)
- 更新 .gitignore, .npmignore

最终结果: 89 tests, 0 failures, 506 assertions

Co-authored-by: M4ster <monkeym4ster@users.noreply.github.com>
@cursor cursor bot changed the title docs: WhatsWeb 基于 Bun 的全面重构计划 + SOP 文档体系 feat: 基于 Bun + TypeScript + ESM 的全面重构 Apr 1, 2026
@cursoragent @monkeym4ster
fix: CLI bar.log 不存在 + process.stdout.clearLine 不兼容
10652bf 
- SingleBar 没有 .log() 方法，改为手动 ANSI 转义清行后打印
- process.stdout.clearLine 在 Bun 中不可用，改用 \x1b[2K\x1b[0G
- 修复 rate 计算逻辑
- 真实 URL 测试通过 (https://www.baidu.com)

Co-authored-by: M4ster <monkeym4ster@users.noreply.github.com>
@cursor cursor bot changed the base branch from master to dev April 1, 2026 09:43
cursoragent and others added 2 commits April 1, 2026 09:52
@cursoragent @monkeym4ster
refactor: 将 Biome 替换为 ESLint 9 + Prettier
28b15d4 
- 卸载 @biomejs/biome，删除 biome.json
- 安装 eslint@10 + typescript-eslint@8 + eslint-config-prettier + prettier@3
- 创建 eslint.config.mjs (ESLint 9 flat config)
- 创建 .prettierrc + .prettierignore
- 更新 package.json scripts (lint/lint:fix/format/format:check)
- Prettier 格式化全量代码
- 修复 ESLint 发现的 2 个 unused-vars 问题
- 更新所有 SOP 文档和 REFACTORING_PLAN.md 中的 Biome 引用
- 89 个测试全部通过，ESLint + Prettier 检查均通过

Co-authored-by: M4ster <monkeym4ster@users.noreply.github.com>
@cursoragent @monkeym4ster
docs: 整理文档目录结构，移至 docs/2026-04-01/
96f457b 
- 将 REFACTORING_PLAN.md 移至 docs/2026-04-01/
- 将 docs/sop/ 移至 docs/2026-04-01/sop/
- 更新所有文档内部交叉引用
- 更新 .npmignore 排除 docs/ 目录（发布包不含文档）

Co-authored-by: M4ster <monkeym4ster@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@monkeym4ster @cursoragent