#6: Update telemetry payload metadata and spec library

README.md

@@ -29,7 +29,7 @@ The project uses `speq` skills and recorded specs to drive planning, implementat
 
 - Non-blocking telemetry recording through an in-memory queue and background sender
 - JSON-over-HTTP `POST` delivery
-- Protocol-compatible message format with `version`, `timestamp`, and `features`
+- Protocol-compatible message format with `category`, protocol `version`, `productVersion`, `timestamp`, and `features`
 - Exponential backoff with retry timeout
 - Clean shutdown via `AutoCloseable`
 - Environment-variable control for disabling telemetry and overriding the endpoint
@@ -41,7 +41,7 @@ The project uses `speq` skills and recorded specs to drive planning, implementat
 ```java
 import com.exasol.telemetry.TelemetryClient;
 import com.exasol.telemetry.TelemetryConfig;
-TelemetryConfig config = TelemetryConfig.builder("my-app").build();
+TelemetryConfig config = TelemetryConfig.builder("my-app", "1.2.3").build();
 
 try (TelemetryClient client = TelemetryClient.create(config)) {
     client.track("startup");

doc/app-user-guide.md

@@ -1,16 +1,41 @@
 # App User Guide
 
-Applications using `telemetry-java` are expected to inform users when usage tracking is active.
+This guide is for end users of applications that use `telemetry-java`.
+
+It explains:
+
+- which data is collected
+- how to see whether telemetry is enabled
+- how to disable telemetry
 
 ## What Is Collected?
 
-The library is designed to send feature-usage events only. It does not collect logs, stack traces, high-frequency data, numeric data, or personally identifiable information.
+The library collects only the data needed for feature-usage telemetry:
+
+- product name, sent as the telemetry category
+- product version
+- which application features are used
+- when those features are used
 
-Messages sent to the server contain the protocol version, the message timestamp, and a `features` map from feature name to a list of usage timestamps.
+It does not collect:
+
+- personally identifiable information
+- general-purpose diagnostic logs
+- stack traces
+- high-frequency event streams
+- numeric measurements
 
 The library sends telemetry to `https://metrics.exasol.com`.
 
-## Opt-Out
+For Exasol's general privacy information, see the [Exasol Privacy Policy](https://www.exasol.com/privacy-policy/).
+
+## How To See Whether Telemetry Is Enabled
+
+Applications can use lifecycle log messages from the library to show whether telemetry is enabled or disabled.
+
+When telemetry is disabled, the library does not enqueue or send usage events.
+
+## How To Disable Telemetry
 
 Host applications can disable telemetry globally by setting environment variable `EXASOL_TELEMETRY_DISABLE` to any non-empty value.
 
@@ -25,7 +50,3 @@ CREATE OR REPLACE JAVA SCALAR SCRIPT MY_UDF(...) RETURNS VARCHAR(100) AS
 In Exasol UDF script options, each environment variable declaration must end with a semicolon and the value must not be quoted.
 
 Telemetry is also disabled automatically when environment variable `CI` is set to any non-empty value, so CI and test environments do not emit usage data by default.
-
-When telemetry is disabled, the library does not enqueue or send usage events.
-
-The library also emits lifecycle log messages so users can see whether telemetry is enabled or disabled, when data is sent, and when telemetry stops.

doc/integration-guide.md

@@ -6,7 +6,7 @@
 
 ```java
 String projectShortTag = "MyApp";
-TelemetryConfig config = TelemetryConfig.builder(projectShortTag).build();
+TelemetryConfig config = TelemetryConfig.builder(projectShortTag, "1.2.3").build();
 
 try (TelemetryClient client = TelemetryClient.create(config)) {
     client.track("checkout-started");
@@ -15,7 +15,7 @@ try (TelemetryClient client = TelemetryClient.create(config)) {
 
 ## Required Configuration
 
-- A project short tag at startup. The library adds it to every accepted telemetry event.
+- A project short tag and a product/library version at startup. The library adds the project tag as the telemetry category and includes the configured productVersion in every accepted telemetry event.
 - An optional HTTP endpoint for JSON `POST` delivery. If omitted, the default endpoint is `https://metrics.exasol.com`.
 
 ## Required Documentation
@@ -60,9 +60,9 @@ For Exasol UDF integration tests, disable telemetry explicitly so test execution
 
 - Tracking calls are non-blocking and enqueue events into a bounded in-memory queue.
 - Delivery happens on a background sender thread.
-- The JSON payload format matches the Python protocol shape: `version`, `timestamp`, and `features`.
-- Multiple queued events may be batched into a single payload, with timestamps grouped by fully qualified feature name.
-- The configured project short tag prefixes feature names in the payload, for example `MyApp.checkout-started`.
+- The JSON payload format includes `category`, protocol `version`, `productVersion`, `timestamp`, and `features`.
+- Multiple queued events may be batched into a single payload, with timestamps grouped by caller-provided feature name.
+- The configured project short tag is emitted as top-level `category`; feature names are preserved as provided, for example `checkout-started`.
 - Failed delivery uses exponential backoff and stops when the configured retry timeout is reached.
 - Closing `TelemetryClient` flushes pending work before returning and stops background threads.
 - Calling `track(...)` after `TelemetryClient` is closed is a no-op.

specs/_recorded/2026-04-15-change-tracking-message-metadata/config/client-identity/spec.md

@@ -0,0 +1,16 @@
+# Feature: client-identity
+
+Defines the required product identity values that the host application configures once and the library attaches to every emitted telemetry message.
+
+## Background
+
+Telemetry messages carry three stable identity fields: `category`, which is the configured project tag; `version`, which is the telemetry protocol version; and `productVersion`, which is the integrating product or library version. Feature names remain arbitrary caller-provided strings and MUST NOT duplicate project identity.
+
+## Scenarios
+
+### Scenario: Requires project tag and productVersion when creating telemetry configuration
+
+* *GIVEN* the host application creates telemetry configuration
+* *WHEN* the host application provides a blank project tag or a blank `productVersion`
+* *THEN* the library SHALL reject configuration creation
+* *AND* the library MUST require both values before a telemetry client can be created

specs/_recorded/2026-04-15-change-tracking-message-metadata/config/tracking-controls/spec.md

@@ -0,0 +1,21 @@
+# Feature: tracking-controls
+
+Preserves environment-based tracking control behavior while keeping message identity metadata stable.
+
+## Background
+
+Environment variables may disable telemetry or override the delivery endpoint, but they do not redefine the configured project identity attached to emitted messages.
+
+## Scenarios
+
+<!-- DELTA:CHANGED -->
+### Scenario: Overrides the configured endpoint via environment variable
+
+* *GIVEN* the host application configures an endpoint, project tag, and `productVersion` in code
+* *AND* the host environment defines an endpoint override
+* *WHEN* the library initializes
+* *THEN* the library SHALL use the environment-provided endpoint for delivery
+* *AND* the library SHALL continue to emit the configured project tag as the `category` field
+* *AND* the library SHALL continue to emit the configured `productVersion` as the `productVersion` field
+* *AND* the library SHALL continue to emit protocol `version`=`0.2.0`
+<!-- /DELTA:CHANGED -->

specs/_recorded/2026-04-15-change-tracking-message-metadata/delivery/async-delivery/spec.md

@@ -0,0 +1,31 @@
+# Feature: async-delivery
+
+Delivers accepted usage events asynchronously while serializing explicit client identity metadata in each payload.
+
+## Background
+
+The telemetry protocol now carries project identity in top-level message fields instead of encoding it into feature names. The JSON `version` field remains the telemetry protocol version and is incremented to `0.2.0`, while `productVersion` carries the integrating product or library version. Async delivery remains responsible for emitting valid JSON over HTTP without blocking the caller thread, including correct escaping of caller-provided feature names in JSON object keys.
+
+## Scenarios
+
+<!-- DELTA:CHANGED -->
+### Scenario: Sends queued events asynchronously over HTTP
+
+* *GIVEN* the library is configured with an endpoint, project tag, and `productVersion`
+* *AND* an accepted usage event is queued for delivery
+* *WHEN* the background sender processes the queue
+* *THEN* the library SHALL submit a protocol message as JSON using HTTP `POST`
+* *AND* the library SHALL include `category`, `version`, `productVersion`, `timestamp`, and `features` fields in that JSON payload
+* *AND* the library SHALL emit protocol `version`=`0.2.0`
+* *AND* the library SHALL perform network delivery without blocking the calling thread
+<!-- /DELTA:CHANGED -->
+
+<!-- DELTA:CHANGED -->
+### Scenario: Batches multiple drained events into a single protocol message
+
+* *GIVEN* multiple accepted telemetry events are present when the background sender drains the queue
+* *WHEN* the background sender emits the next protocol message
+* *THEN* the library SHALL include the queued events in a single JSON payload
+* *AND* the library SHALL group timestamps by caller-provided feature name in the `features` map
+* *AND* the library SHALL correctly JSON-escape caller-provided feature names when serializing the `features` map
+<!-- /DELTA:CHANGED -->

specs/_recorded/2026-04-15-change-tracking-message-metadata/plan.md

@@ -0,0 +1,119 @@
+# Plan: change-tracking-message-metadata
+
+## Summary
+
+This plan replaces project-tag-prefixed feature names with explicit top-level message metadata. It keeps the JSON `version` field as the telemetry protocol version, increments that protocol version to `0.2.0`, and adds a separate `productVersion` field for the host product/library version. It also updates the tracking, delivery, and tracking-controls specs so they consistently require caller-provided feature names in the payload without feature-name validation beyond ignoring `null` values.
+
+## Design
+
+### Context
+
+The current contract mixes two identities into one field by encoding the configured project tag into every feature name, while the issue requires a top-level `category` field and a required host product/library version. The JSON payload already has a `version` field, so changing its meaning would break the protocol contract unless the host-side version is given a distinct name. The permanent spec library also lacks a feature that owns required client identity inputs, so the new runtime contract would remain underspecified unless the plan adds one.
+
+- **Goals** — Move project identity out of feature names, require host applications to configure both project tag and product/library version, preserve application-chosen feature names without validation apart from ignoring `null` values, keep the protocol `version` field, and make emitted payloads carry `category` and `productVersion` explicitly.
+- **Non-Goals** — Change retry behavior, add extra payload fields, impose feature-name rules, or redesign endpoint/environment override behavior.
+
+### Decision
+
+Add a new permanent `config/client-identity` feature that defines required project tag and `productVersion` inputs. Keep the protocol `version` field in the JSON message and increment it to `0.2.0`. Update the tracking API so accepted feature names are sent exactly as provided rather than being qualified with the project tag or validated by the library, except that `null` feature names are ignored and not queued. Update async delivery so every JSON payload includes `category`, `version`, `productVersion`, `timestamp`, and `features` with correct JSON escaping for feature keys.
+
+#### Architecture
+
+```
+┌──────────────────┐
+│ TelemetryConfig  │
+│ projectTag       │
+│ productVersion   │
+└────────┬─────────┘
+         │
+         ▼
+┌──────────────────┐
+│ TelemetryClient  │
+│ track(feature)   │
+│ no tag prefixing │
+└────────┬─────────┘
+         │
+         ▼
+┌──────────────────┐
+│ Message          │
+│ category         │
+│ version=0.2.0    │
+│ productVersion   │
+│ timestamp        │
+│ features         │
+└──────────────────┘
+```
+
+### Consequences
+
+| Decision | Alternatives Considered | Rationale |
+|----------|------------------------|-----------|
+| Add `config/client-identity` as a new permanent feature | Fold `productVersion` requirements into `tracking-api` only | Required configuration inputs are part of the public setup contract and need a clear permanent owner |
+| Send raw feature names and add top-level `category` | Keep prefixing features and also add `category` | Prefixing would duplicate project identity and violate the issue acceptance criteria |
+| Keep `version` as the protocol version and add `productVersion` for the host product/library version | Reinterpret `version` as the host version | Preserves the existing protocol field semantics and avoids overloading one field with two meanings |
+
+## Features
+
+| Feature | Status | Spec |
+|---------|--------|------|
+| client-identity | NEW | `config/client-identity/spec.md` |
+| tracking-controls | CHANGED | `config/tracking-controls/spec.md` |
+| async-delivery | CHANGED | `delivery/async-delivery/spec.md` |
+| tracking-api | CHANGED | `tracking/tracking-api/spec.md` |
+
+## Requirements
+
+| Requirement | Details |
+|-------------|---------|
+| Required identity inputs | Host applications SHALL provide a non-blank project tag and a non-blank `productVersion` when creating telemetry configuration |
+| Payload identity | Emitted telemetry messages SHALL include `category` equal to the configured project tag |
+| Protocol version | Emitted telemetry messages SHALL include `version` equal to the protocol version `0.2.0` |
+| Product version | Emitted telemetry messages SHALL include `productVersion` equal to the configured host product/library version |
+| Feature semantics | Emitted `features` keys SHALL use the caller-provided feature names without adding the project tag as a prefix or validating the chosen names, except that `null` feature names SHALL be ignored |
+| JSON encoding | The library SHALL correctly JSON-escape caller-provided feature names when serializing the `features` map |
+
+## Implementation Tasks
+
+1. Extend `TelemetryConfig` to require and expose both project tag and `productVersion` through the public builder entry point.
+2. Remove feature-name qualification and feature-name validation from `TelemetryClient` so it queues caller-provided feature names without project-tag prefixing or library-side filtering, while ignoring `null` feature names.
+3. Update `Message` serialization to emit `category`, protocol `version`=`0.2.0`, configured `productVersion`, `timestamp`, and caller-provided feature keys with correct JSON escaping.
+4. Adjust helper/test fixtures such as `RecordingHttpServer` and any hard-coded payload assertions to use the new builder contract and payload shape.
+5. Update docs and Javadocs where they still describe project-tag-prefixed feature names or outdated payload field semantics.
+
+## Dead Code Removal
+
+| Type | Location | Reason |
+|------|----------|--------|
+| Field | `src/main/java/com/exasol/telemetry/TelemetryClient.java` | The cached feature-prefix field becomes obsolete once feature qualification is removed |
+| Methods | `src/main/java/com/exasol/telemetry/TelemetryClient.java` | Feature qualification and most feature-name validation helpers become obsolete once raw feature names are emitted without validation apart from ignoring `null` |
+| Test assertions | `src/test/java/com/exasol/telemetry/*` | Assertions expecting prefixed feature keys or the old protocol version string become obsolete under the new payload contract |
+
+## Verification
+
+### Scenario Coverage
+
+| Scenario | Test Type | Test Location | Test Name |
+|----------|-----------|---------------|-----------|
+| Requires project tag and productVersion when creating telemetry configuration | Unit | `src/test/java/com/exasol/telemetry/TelemetryConfigTest.java` | `requiresProjectTagAndProductVersion` |
+| Overrides the configured endpoint via environment variable | Integration | `src/test/java/com/exasol/telemetry/TrackingControlsIT.java` | `overridesConfiguredEndpointWithoutChangingPayloadIdentity` |
+| Sends queued events asynchronously over HTTP with configured identity metadata | Integration | `src/test/java/com/exasol/telemetry/TrackingApiIT.java` | `recordsFeatureUsageEventWithCategoryProtocolVersionAndProductVersion` |
+| Batches multiple drained events into a single protocol message | Integration | `src/test/java/com/exasol/telemetry/AsyncDeliveryIT.java` | `batchesMultipleDrainedEventsIntoSingleProtocolMessage` |
+| Records a feature usage event without project-tag prefixing or feature-name validation | Integration | `src/test/java/com/exasol/telemetry/TrackingApiIT.java` | `recordsFeatureUsageEventWithoutPrefixingOrValidation` |
+| Ignores null feature names | Integration | `src/test/java/com/exasol/telemetry/TrackingApiIT.java` | `ignoresNullFeatureNames` |
+| Emits protocol version `0.2.0`, configured `productVersion`, and escaped feature names in JSON | Unit | `src/test/java/com/exasol/telemetry/MessageTest.java` | `serializesProtocolVersionProductVersionAndEscapedFeatureNames` |
+| Keeps caller-thread overhead low for accepted tracking | Integration | `src/test/java/com/exasol/telemetry/TrackingApiIT.java` | `keepsCallerThreadOverheadLowForAcceptedTracking` |
+
+### Manual Testing
+
+| Feature | Command | Expected Output |
+|---------|---------|-----------------|
+| client-identity, tracking-controls, async-delivery, tracking-api | `mvn verify` | Tests confirm required config identity, protocol `version=0.2.0`, configured `productVersion`, unvalidated caller-provided feature names except ignored `null` values, correct JSON escaping, and top-level `category` across payload-producing paths |
+
+### Checklist
+
+| Step | Command | Expected |
+|------|---------|----------|
+| Build | `mvn package` | Exit 0 |
+| Test | `mvn test` | 0 failures |
+| Lint | `mvn verify` | Exit 0 |
+| Format | `mvn verify` | Exit 0; no dedicated formatter command is defined in the mission |

specs/_recorded/2026-04-15-change-tracking-message-metadata/tasks.md

@@ -0,0 +1,13 @@
+# Tasks: change-tracking-message-metadata
+
+## Phase 2: Implementation
+- [x] 2.1 Update telemetry configuration and payload model to require project tag and `productVersion`, keep protocol `version`, and emit `category`, `version=0.2.0`, and `productVersion`
+- [x] 2.2 Remove feature-name prefixing, ignore `null` feature names, and preserve caller-provided feature names for delivery
+- [x] 2.3 Update unit and integration tests plus helper fixtures for the new builder contract and payload shape
+- [x] 2.4 Update docs and Javadocs that still describe prefixed feature names or outdated payload field semantics
+
+## Phase 3: Verification
+- [x] 3.1 Run targeted tests for configuration, payload serialization, transport, tracking API, delivery, and tracking controls
+- [x] 3.2 Run `mvn test`
+- [x] 3.3 Run `mvn verify`
+- [x] 3.4 Generate verification report