support custom vocabularies via esm.sh with user trust prompt

[Adityakumar37]

For the prompt I used window/showMessageRequest server sends it, VSCode renders the Allow/Deny dialog, no client changes needed for that part. For persisting the response I used globalState via two custom notifications between the server and extension since the server can't access globalState directly. On startup the extension pushes already-trusted identifiers to the server so the user doesn't get asked again.

One annoying thing Node.js doesn't allow import() over https: and esm.sh returns a redirect file instead of actual code. So I fetch the redirect, pull out the real path, fetch that, and load it as a data: URL.
New files are language-server/src/services/vocabulary-loader.js and .vscode/launch.json (no debug config existed in the repo). vscode/src/extension.ts handles the globalState side.

Would love a review on this pretty sure the overall approach is right but there might be edge cases I've missed, especially around the esm.sh fetching logic and how the notifications are structured between server and extension.

[jdesrosiers]

I don't think your hack around not having an HTTPS loader is going to work. I setup an example vocab at https://github.com/jdesrosiers/json-schema-dialect-example.

When you fetch https://esm.sh/gh/jdesrosiers/json-schema-dialect-example@66eb31a you get,

/* esm.sh - gh/jdesrosiers/json-schema-dialect-example@66eb31a */
import "/@hyperjump/browser?target=es2022";
import "/@hyperjump/json-schema@^1.17.5/draft-2020-12?target=es2022";
import "/@hyperjump/json-schema@^1.17.5/experimental?target=es2022";
export * from "/gh/jdesrosiers/json-schema-dialect-example@66eb31a/es2022/json-schema-dialect-example.mjs";
export { default } from "/gh/jdesrosiers/json-schema-dialect-example@66eb31a/es2022/json-schema-dialect-example.mjs";

As you can see, the output can be much more complex than a single export that we can parse out and fake. It could be arbitrarily complex and will only work reliably if it can be executed as JavaScript.

[Adityakumar37]

HI @jdesrosiers

Here's what I did:

Created https-loader.js which hooks into Node's module resolution via register() from node:module. It intercepts https:// imports by fetching the source directly, and resolves relative imports like /@hyperjump/browser?target=es2022 against the parent URL so the full transitive dependency graph works correctly.

In extension.ts the loader is registered by passing --import /path/to/https-loader.js in execArgv when starting the language server process, so it's active before any module loading happens.

In vocabulary-loader.js, the old fetch/regex/data URL hack is gone. It's now just await import("https://esm.sh/${identifier}") and the loader handles everything. Also added a #loaded Set and a #loading Map to handle the race condition where onInitialized and onDidChangeConfiguration both fire at startup for the same identifier — the Map ensures concurrent calls share a single Promise so mod.default() only runs once.

Tested with gh/jdesrosiers/json-schema-dialect-example@66eb31a and it loads cleanly with no errors.

Would love your review and any feedback on the approach

[jdesrosiers]

Did you do a full end-to-end test? When I tried, I get:

Encountered unknown dialect 'https://example.com/dialect/example

The package was imported successfully, but the dialect isn't loaded. Something's not right.

[Adityakumar37]

Hey jdesrosiers, thanks for the end-to-end test and the example repoit really helped me track down what was wrong.

Fixed a CJS/ESM dual-instance problem when the remote package imported @hyperjump/*, Node was loading a fresh copy separate from what the server uses, so defineVocabulary calls were going nowhere. Added a bridge on globalThis so the remote package gets the same instances as the server.

Also wired --import https-loader.mjs into both run and debug execArgv (it was only in debug before), and added a ready gate in ValidateSchemaFeature and ValidateWorkspaceFeature to prevent the "Unknown dialect" race condition on startup.

Please let me know if I've missed anything or if the approach looks off to you happy to fix whatever needs changing.

[jdesrosiers]

I think I found a better approach than the bridge. I figured out that esm.sh supports an ?external query parameter. It tells esm.sh to no transform the imports for the listed packages, which makes it easy to handle them in the loader and have it use the local package.

Here's what I got working. See if you can use this approach.

https://gist.github.com/jdesrosiers/1f915bfd4f72e52f9b7766d225500fcf

[Adityakumar37]

Hey @jdesrosiers, switched to the ?external= approach works great!
One thing I changed from your gist: used createRequire(import.meta.url) instead of createRequire(pathToFileURL("."))

[jdesrosiers]

Something's not quite right. The language server is reporting that it loaded the dialect, but it's not looking like it's registered.

[Adityakumar37]

Hi @jdesrosiers I got it working. If this looks like the right direction I'll clean it up and push a final version

[jdesrosiers]

It seems to be working! This is so low priority that I shouldn't be paying attention to it right now, but I can't help it because I find this feature so fascinating.

After some clean up, there are two more things that are needed. One of those things is neovim support. I tried it and it does support the popup (which surprised me), but it crashes when it receives the response. If there's a way to skip the popup for neovim, that would be fine because it requires that users set the config themselves, it doesn't get loaded automatically like in vscode.

The other thing that's missing is tests. This is a tricky one. I think the only meaningful way to test this is to actually load a vocabulary like we've been doing in our manual tests. I think we're going to have to make that example a permanent fixture for testing.

[Adityakumar37]

Sorry for responding late.
Made the changes you suggested:

1. Removed the vocab reload from validate-schema.js, so it only loads once on init now (also had to fix a similar reload happening in validate-workspace.js's config-change handler, since that was double-loading too)

2.Expanded ESM_EXTERNALS to include all @hyperjump/json-schema exports

3.Removed the addUriSchemePlugin("http"/"https") calls
Moved the onInitialize/onInitialized logic into VocabularyLoader itself

4.Removed json.validate.enable and the server deps from vscode/package.json

5.Added a check so the trust popup gets skipped if the client doesn't support showMessage (should fix the neovim crash, though I haven't tested on actual neovim yet, just confirmed the capability check works in vscode)

One thing I noticed while testing: when validating against the example vocab's dialect, I get "Location https://example.com/dialect/example is untrusted" in the Problems panel. I tested with and without the addUriSchemePlugin calls and it happens either way, so it doesn't seem related to anything in this PR — looks like it's VSCode's workspace trust blocking the remote $schema fetch. Not sure what the right fix is here, would appreciate a pointer if you have one.