v5.4.0 — Parse Server 8/9 compatibility, hybrid-search RAG, and MCP streaming transport

[AdrianCurtin]

v5.4.0 — Parse Server 8/9 compatibility, hybrid-search RAG, and MCP streaming transport

This branch is the 5.4.0 release (dev2 → main). It spans: a layer of Parse Server 8.x/9.x compatibility fixes and capability detection, full webhook trigger coverage (including the non-object auth/LiveQuery triggers and a fix that makes beforeFind/afterFind actually route), hybrid (lexical + vector) retrieval with reranking, and a consolidated MCP Streamable HTTP transport with disconnection hardening. It also carries MFA, email-verification, Parse::Audience, and test-tooling fixes.

Breaking changes

• BREAKING: Parse::User#disable_mfa_master_key! now fails closed. Because it bypasses MFA verification via the master key, it refuses to run without an authorization signal: pass admin_role: (the library verifies the operator's role membership) or allow_unverified: true (assert the operator was authorized out-of-band). Callers that previously passed only authorized_by: now raise Parse::MFA::ForbiddenError. Migration: add admin_role: or allow_unverified: true.
• BREAKING: Parse.call_function / client.call_function results are now ORM-typed for registered classes. The decoder (see "Parse Server 8.x / 9.x compatibility") rebuilds a returned object of a registered class into a Parse::Object, so an in-process Ruby caller reading a field goes through the property getter and gets the ORM type — an enum / symbolize: property yields a Symbol (:active, not "active"), a date yields Parse::Date, a pointer yields Parse::Pointer — where a pre-5.4.0 caller read the raw JSON String/Hash. Scope is narrow: it affects only the client-side return of a cloud-function call read in Ruby; HTTP/JSON consumers are unaffected (JSON has no symbols), and webhook trigger returns (beforeSave/afterSave/etc.) are unaffected since those responses go to Parse Server and never pass through the result decoder. Migration: a cloud function whose JSON shape is a contract should return an explicit plain Hash and coerce typed fields (status: obj.status.to_s) rather than whole Parse::Objects — returning objects (and obj.as_json, which still emits the __type envelope) is what triggers the rebuild; Ruby callers reading typed fields should expect the ORM type or normalize at the read site.

Parse Server 8.x / 9.x compatibility

• FIXED: Query#read_pref now rides the REST query body (readPreference), not just the X-Parse-Read-Preference header — Parse Server maps no such header, so over REST the preference was silently ignored and every scoped read hit the primary.
• FIXED: LiveQuery field projection emits the keys subscription option (Parse Server 7.0 renamed it from fields), with fields kept as an alias for older servers — a projected subscription was silently receiving every column on 7.0+.
• BREAKING: cloud-function results decode __type-encoded Parse objects back into Parse::Object / Parse::Pointer (Parse Server 8.0 began encoding returned objects; 9.0 made it unconditional). Decoding is conservative — an unregistered class is left as a raw Hash, plain data passes through untouched. This makes call_function results ORM-typed for registered classes (enums read as Symbols, etc.) for in-process Ruby callers — see Breaking changes above for the consequence and migration.
• NEW: Parse.server_supports?(:capability) / Parse.server_features — a capability probe built on the memoized serverInfo fetch, so future server changes can be feature-gated rather than discovered by breakage. Prefers the advertised features block and falls back to version inference, failing open to the current server line.
• IMPROVED: Query#explain surfaces actionable guidance (proactive one-shot warning + reactive message, and the explain_query agent tool) when it hits Parse Server 9.0's allowPublicExplain: false default, instead of a bare 403.

Webhook trigger coverage

• NEW: first-class routing for the non-object trigger shapes — the auth triggers (beforeLogin, afterLogin, afterLogout, beforePasswordResetRequest) and the LiveQuery triggers (beforeConnect, beforeSubscribe, afterEvent). Parse::Webhooks::Payload gains matching predicates (before_login? … after_event?, plus auth_trigger? / live_query_trigger?), an event accessor and clients / subscriptions connection counters, and captures the top-level sessionToken connect/subscribe carry into #session_token (so #user_client / #user_agent work, while keeping the token out of as_json and the request log). Dispatch matches Parse Server's response contract: the body is ignored for all seven, so a before* handler returning false (which Parse Server would resolve as {success:false} and allow) is converted to a rejection. None of these run ActiveModel save/create/destroy callbacks even though the auth triggers carry a _User/_Session.
• FIXED: beforeFind / afterFind webhook triggers now route. Parse Server omits the class name from the find payload body entirely (the matched objects carry no className and there is no top-level one), so the SDK could not resolve parse_class and the dispatcher never invoked the handler. The class is now threaded from the webhook URL path (<endpoint>/<trigger>/<className>) into the Payload. This is also a correctness fix, not just feature completeness: an unrouted afterFind returned {"success": true} (not an objects array), which Parse Server rejects — so a registered afterFind previously broke every matching query with a connection error. The path segment is charset-validated before use as a routing key.
• FIXED: :vector columns are now stripped from afterFind webhook payload objects (the route-derived class is the only way to resolve the model, since the find payload carries no className; vector_visibility :public classes keep them).
• NEW: file (@File) and connection (@Connect) triggers now have a full register/fetch/delete lifecycle; Parse::API::PathSegment.trigger_class_name! accepts the @-prefixed pseudo-classes.
• CHANGED: beforeCreate / afterCreate are no longer presented as registerable webhook triggers (Parse Server has no such type); they remain ActiveModel callbacks that run inside the beforeSave / afterSave handler. Registering a create trigger raises a clear error pointing to the save trigger.

Webhook handler ergonomics

• IMPROVED: a registered webhook handler can now use an explicit return value. Handlers previously ran via instance_exec, so a bare return raised LocalJumpError when the handler was defined inside a method (initializer, class body, config block); they now run as a method on the payload, giving return ordinary semantics. The legacy idioms (last expression, next, break) still set the result, and self is still the payload.
• NEW: payload.after_response { … } (alias defer) runs work after the webhook response is sent, off the client's critical path (search indexing, cache warming, fan-out). Uses rack.after_reply (Puma/Unicorn) when available, else a detached thread; callbacks run in registration order, are isolated, and fire only on the success path. In-process only (does not survive a worker restart) — use a durable queue for work that must happen.

Webhook trigger coverage audit

• NEW: Parse::Webhooks.trigger_audit — a master-key operator audit that cross-references three sources of trigger truth across every registered class and reports where they drift: a model's ActiveModel callbacks, the locally registered webhook blocks (Parse::Webhooks.routes), and the triggers actually registered with Parse Server (hooks/triggers). It surfaces the non-obvious rule that a callback runs server-side for non-Ruby clients only when both a local webhook block and the matching server trigger are registered — a callback declared on its own is inert for JS/Swift/REST/Dashboard writes. Findings: callbacks_inert, route_not_registered, orphan_server_trigger, and local_only_callbacks (*_update/*_validation callbacks no server trigger can run). Framework-internal callbacks are filtered out by source location. Returns a Hash (or a human-readable summary with pretty: true); network: false audits callbacks against local routes without a master key.

Parse Server feature coverage

• NEW: context: propagation on create_object / update_object, call_function / call_function_with_session, and Parse.call_function — serialized to X-Parse-Cloud-Context and exposed to Cloud Code triggers; Webhooks::Payload#context reads it on the receive side.
• NEW: Parse::User#verify_password(password) / Users#verify_password(username, password) validate credentials via POST /verifyPassword (credentials in the body, mirroring login, so the plaintext password stays out of URLs/logs) without minting a session — a step-up / re-auth primitive.
• NEW: Parse::Error::EmailNotVerifiedError from Parse::User.login! distinguishes "verify your email" (preventLoginWithUnverifiedEmail, code 205) from bad credentials. It subclasses Parse::Error::AuthenticationError, so existing rescue AuthenticationError handlers keep catching it (non-breaking).
• NEW: Query#exclude_keys(*fields) (excludeKeys), LiveQuery subscribe(watch: [...]) (update events only when named fields change, 7.0+), Query#aggregate(pipeline, raw_values:, raw_field_names:) (9.9.0 rawValues/rawFieldNames), Query#hint(index_name) (REST + mongo-direct), and the :field.contained_by => [...] ($containedBy) constraint.

Retrieval (RAG): hybrid search + reranking

• NEW: Class.hybrid_search(text:, lexical:, vector:, k:, fusion:) fuses a lexical Atlas Search branch with a $vectorSearch branch via reciprocal-rank fusion (RRF). Two independent aggregations are required because $vectorSearch must be stage 0; each branch enforces ACL/CLP/protectedFields independently before fusion, so fused rows are already access-filtered. Results carry #hybrid_score, #hybrid_ranks, #vector_score/#search_score.
• NEW: Parse::VectorSearch::Hybrid.rrf (pure fusion math) and .rank_fusion_supported? (Atlas 8.0+ native $rankFusion detection via a cached behavioural probe, not version-string parsing).
• NEW: Parse::Retrieval::Reranker cross-encoder protocol with a deterministic Reranker::Fixture and a Reranker::Cohere adapter (/v2/rerank); Parse::Retrieval.retrieve now accepts hybrid: and rerank: (previously reserved, raising NotImplementedError), with tenant scope enforced authoritatively in both branches.
• NEW: Parse::Embeddings::SpendCap — opt-in per-tenant cumulative embedding-token cap with hard-refuse, charged at the semantic_search agent-tool boundary (admin agents exempt).
• CHANGED: PipelineSecurity admits $rankFusion (read-only, stage-0 Atlas operator) for the opt-in native path.

Retrieval (RAG): completeness

• NEW: Class.embed_pending! backfills null :vector fields via objectId-cursor pagination; Parse::Object#compute_embedding! forces a digest-tracked in-place recompute without a save.
• NEW: vector_visibility :owner_only | :public controls whether :vector properties appear in as_json by default (:owner_only is the safe default; an explicit include_vectors: always wins).
• IMPROVED: webhook trigger payloads strip declared :vector columns from object/original/update/objects by default (a :public class keeps them).

MCP: Streamable HTTP transport + disconnection hardening

• NEW: Parse::Agent::MCPRackApp.new(transport: :streamable_http) (and Parse::Agent.rack_app(transport:)) enables the full MCP 2025-06-18 Streamable HTTP transport in one switch — POST→SSE streaming plus the server→client GET / notification stream — equivalent to streaming: true, notifications: true. Streamable HTTP is now documented as the primary embedded-Rack transport. transport: is a closed enum (:streamable_http / :legacy / nil); passing it alongside an explicit streaming:/notifications:, or an unknown value, raises ArgumentError.
• NOTE: the default transport is unchanged — an existing rack_app { ... } keeps its non-streaming behavior until it opts in. The switch needs a streaming-capable Rack server (Puma/Falcon/Unicorn) and has no effect under the WEBrick MCPServer.
• CHANGED: max_concurrent_dispatchers: now defaults to a finite 100 (was nil/unlimited), so a streaming surface is bounded out of the box — the cap fires a 503 JSON-RPC -32000 instead of spawning unbounded orphan-prone threads. Pass an explicit integer to resize, or nil to knowingly run uncapped (logs a one-time warning); a non-positive/non-integer value raises ArgumentError.
• NEW: disconnect observability — MCPRackApp.abandoned_dispatcher_count (process-wide counter of genuine orphans) plus a parse.agent.mcp_dispatcher_abandoned ActiveSupport::Notifications event on every premature close. On disconnect the dispatcher's cancellation token is tripped and the orphan is bounded by the per-tool Timeout and the clean MongoDB/REST I/O deadlines; it is intentionally not force-killed (a Thread#kill would skip the driver's connection-invalidation and risk returning a half-used pooled connection).
• CHANGED: custom tools registered via Parse::Agent::Tools.register now have their declared timeout: (default 30s) actually enforced — Tools.invoke wraps the handler in Timeout.timeout, raising ToolTimeoutError (previously the custom-handler path ran unbounded). register rejects a non-positive timeout:. Migration: a custom tool that legitimately runs longer than 30s must now declare an explicit timeout:.

Auth and accounts

• FIXED: Parse::User MFA lifecycle — setup_mfa!, setup_sms_mfa!, confirm_sms_mfa!, disable_mfa!, disable_mfa_master_key! no longer raise an internal argument error before reaching the server; mfa_enabled?/mfa_status report correctly after an ordinary fetch (a leak-safe {status: "enabled"} projection is preserved while the TOTP secret and recovery codes are stripped). Self-service disable_mfa! proves possession of the current code then unlinks the provider, confirming the disable from the server's own view.
• NEW: interactive console MFA login — rake client:console prompts for a TOTP/recovery code (or reads PARSE_LOGIN_MFA) when logging into an enrolled account.
• NEW: Parse::User.request_email_verification(email) (and the instance form) re-sends the verification email for a registered, unverified user, mirroring request_password_reset.
• FIXED: Parse::Audience#query is stored as a JSON string on the wire to match Parse Server's _Audience.query column type, so saving a hash query no longer fails the server schema check. Public API unchanged (assign/read a Hash).

Performance and tooling

• CHANGED: Parse::AtlasSearch role_cache_ttl now defaults to 30s (was 120) so role grants/revokes reflect in $search ACL decisions sooner.
• CHANGED: test tasks run through Bundler (bundle exec ruby) to avoid a minitest activation/load error on individual files; README documents the requirement.
• IMPROVED: ACL documentation clarifies the default :owner_else_private policy, its private fallback, and how to override it via set_default_acl / acl_policy.

Notes for reviewers

• Hybrid fusion runs client-side by default. The native single-roundtrip $rankFusion path is opt-in (fusion: { method: :rrf_native }) and falls back to client-side fusion when the cluster does not support it. Detection and the native pipeline shape are unit-tested, but native execution is not exercised in CI (no Atlas 8.0 cluster).
• afterFind/beforeFind routing was verified end-to-end against a live Parse Server 9.9.0 (the find payload genuinely carries no className; the callback URL uses the snake_case trigger form). A Docker-gated integration test guards it.
• MCP forcible disconnect-reclaim is a deliberate non-goal. The orphaned dispatcher is bounded (cap + per-tool Timeout + clean I/O deadlines) but not force-killed; this was reviewed against the connection-pool corruption risk a Thread#kill would introduce.
• LiveQuery webhook triggers are delivered over HTTP only in a co-located single-process setup; beforeConnect is effectively in-process only.
• The Reranker::Cohere /v2/rerank response parsing is tested against a stubbed HTTP layer rather than a live key.

[Copilot]

Pull request overview

This PR updates the test/developer workflow to run reliably under Bundler while also shipping a sizable 5.4.0 feature set: hybrid lexical+vector search (RRF + optional $rankFusion), retrieval reranking, embedding spend caps, tighter vector exposure defaults (serialization + webhooks), and expanded MFA/email/push integration coverage plus supporting docker test-stack wiring.

Changes:

• Run test files via bundle exec and add test helper support for post-signup login to obtain a live session token under Parse Server 9.x behavior.
• Add major RAG/search and security-related features: hybrid search + reranking, spend-cap metering, vector visibility controls, and vector scrubbing in webhook payloads.
• Expand integration/unit coverage, examples, and docs; bump version to 5.4.0 and update Parse Server docker/test-stack configuration (MFA/auth, push, capturing email).

Reviewed changes

Copilot reviewed 52 out of 54 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
test/test_helper_integration.rb	Adds login_after_signup! helper to ensure live session tokens in integration tests.
test/lib/parse/vector_visibility_test.rb	Adds unit tests for new vector_visibility behavior and webhook redaction expectations.
test/lib/parse/vector_searchable_hybrid_test.rb	Tests Class.hybrid_search wrapper kwargs threading and object hydration with hybrid metadata.
test/lib/parse/vector_search_hybrid_test.rb	Tests RRF math, $rankFusion probe caching, native pipeline shape, and orchestration behavior.
test/lib/parse/user_save_signup_integration_test.rb	Adjusts integration tests to log in after signup to obtain session tokens.
test/lib/parse/user_authdata_strip_test.rb	Adds test ensuring MFA authData is reduced to leak-safe status only.
test/lib/parse/retrieval_retrieve_test.rb	Updates retrieval tests for hybrid routing and rerank behavior.
test/lib/parse/retrieval_reranker_test.rb	Adds unit tests for reranker protocol, fixture, and Cohere adapter parsing/redaction.
test/lib/parse/push_integration_test.rb	Fixes integration setup and adds server-backed push/install/audience lifecycle tests with cleanup.
test/lib/parse/mfa_totp_flow_integration_test.rb	Adds end-to-end TOTP MFA integration tests against MFA-enabled server.
test/lib/parse/mfa_test.rb	Fixes provisioning URI assertion to account for URL encoding.
test/lib/parse/live_query_integration_test.rb	Makes LiveQuery event tests deterministic by using public ACL objects + timeouts.
test/lib/parse/embeddings_spend_cap_test.rb	Adds unit tests for embedding token spend cap behavior.
test/lib/parse/embed_pending_test.rb	Adds unit tests for embedding backfill and compute_embedding!.
test/lib/parse/email_verification_disruptive_test.rb	Adds disruptive integration test that recreates server with email verification enabled.
test/lib/parse/client_rest_password_reset_integration_test.rb	Adds client-mode password reset integration coverage using capturing adapter.
test/cloud/dummy-push-adapter.js	Adds test-only push adapter enabling deterministic _PushStatus lifecycle tests.
test/cloud/capturing-email-adapter.js	Adds test-only email adapter capturing outgoing messages into Parse for assertions.
scripts/start-parse.sh	Wires push adapter, MFA auth config file, capturing email adapter, and public server URL for test stack.
scripts/docker/Dockerfile.parse	Pins Parse Server image tag to 9.9.0 for specific security fixes/features.
scripts/docker/docker-compose.verifyemail.yml	Adds compose override enabling verifyUserEmails for disruptive test.
README.md	Adds “What’s new in 5.4” summary, examples section, and bundle exec testing guidance; bumps version text.
Rakefile	Routes per-file test execution through bundle exec and adds optional MFA login flow for console.
lib/parse/webhooks/payload.rb	Adds vector column scrubbing for webhook payload object/original/update/objects.
lib/parse/vector_search/hybrid.rb	Introduces hybrid search module (RRF + optional native $rankFusion path + probe cache).
lib/parse/two_factor_auth/user_extension.rb	Fixes MFA API calls to pass session/master opts correctly; revises disable flow.
lib/parse/stack/version.rb	Bumps gem version to 5.4.0.
lib/parse/retrieval/retriever.rb	Enables hybrid: and rerank: in retrieval, adds rerank integration and hybrid wiring.
lib/parse/retrieval/reranker/cohere.rb	Adds Cohere /v2/rerank adapter with hardened HTTP handling.
lib/parse/retrieval/reranker.rb	Adds reranker protocol/base, fixture reranker, and Cohere autoload.
lib/parse/retrieval/agent_tool.rb	Charges embedding spend cap for semantic_search and maps breaches to rate-limit errors.
lib/parse/pipeline_security.rb	Allows $rankFusion as an Atlas stage-0 operator.
lib/parse/model/object.rb	Documents default ACL policy and adds hybrid score/ranks accessors; updates vector serialization default logic.
lib/parse/model/core/vector_searchable.rb	Adds vector_visibility DSL and hybrid_search API + hybrid hit builder.
lib/parse/model/core/embed_managed.rb	Adds compute_embedding! and embed_pending! bulk backfill API.
lib/parse/model/classes/user.rb	Preserves leak-safe MFA status projection while stripping sensitive authData; adds email verification request APIs.
lib/parse/model/classes/audience.rb	Fixes _Audience.query persistence by storing JSON string on wire and exposing Hash API.
lib/parse/model/acl.rb	Clarifies default ACL policy documentation (:owner_else_private).
lib/parse/embeddings/spend_cap.rb	Adds per-tenant embedding spend cap implementation (disabled by default).
lib/parse/embeddings.rb	Requires new spend cap module.
lib/parse/atlas_search.rb	Reduces role cache TTL default from 120s to 30s.
lib/parse/api/users.rb	Adds POST /verificationEmailRequest client method with rate-limit tracking.
Gemfile.lock	Bumps gem version and adds rotp/rqrcode (and deps).
Gemfile	Adds rotp and rqrcode to test/development group for MFA tests/QR.
examples/README.md	Adds index of runnable example scripts and common setup.
examples/rag_chatbot.rb	Adds end-to-end RAG example using managed embeddings + agent retrieval + LLM add-in.
examples/live_query_listener.rb	Adds interactive LiveQuery listener example scoped to a user session.
examples/basic_server.rb	Adds privileged (master-key) setup + schema/CRUD example.
examples/basic_client.rb	Adds unprivileged client example demonstrating ACL enforcement.
docs/mongodb_direct_guide.md	Documents enforcement behavior for Atlas index stages including $rankFusion and hybrid search.
docs/client_sdk_guide.md	Links to new runnable client/server examples.
docs/atlas_vector_search_guide.md	Documents hybrid search, reranking, and spend cap behavior; links to RAG example.
CHANGELOG.md	Adds detailed 5.4.0 changelog entry.
.gitignore	Ensures examples/README.md is not ignored.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 55 out of 57 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 101 out of 103 changed files in this pull request and generated 1 comment.