Add stream topic label to realtime ingestion delay metrics#18175
Open
rsrkpatwari1234 wants to merge 22 commits intoapache:masterfrom
Open
Add stream topic label to realtime ingestion delay metrics#18175rsrkpatwari1234 wants to merge 22 commits intoapache:masterfrom
rsrkpatwari1234 wants to merge 22 commits intoapache:masterfrom
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #18175 +/- ##
============================================
+ Coverage 63.18% 63.33% +0.15%
- Complexity 1616 1627 +11
============================================
Files 3214 3229 +15
Lines 195838 196724 +886
Branches 30251 30411 +160
============================================
+ Hits 123734 124590 +856
+ Misses 62236 62151 -85
- Partials 9868 9983 +115
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
noob-se7en
added
ingestion
noob-se7en
reviewed
Apr 14, 2026
| IngestionInfo ingestionInfo = _ingestionInfoMap.get(partitionId); | ||
| @Nullable String streamTopicName = ingestionInfo != null ? ingestionInfo._streamTopicName : null; | ||
| if (StringUtils.isNotBlank(streamTopicName)) { | ||
| _serverMetrics.setOrUpdatePartitionGaugeForStreamTopic(_metricName, streamTopicName, partitionId, |
Contributor
There was a problem hiding this comment.
We should look into adding the label in current existing metric itself instead of creating a new metric.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
With multi-topic ingestion, ingestion delay gauges only keyed by table and partition group id made it hard to tell which Kafka (or other stream) topic was behind, often requiring indirect mapping from partition to topic in config (apache/pinot#18099).
Approach
Reuse the existing JMX → Prometheus naming pattern already used for other stream metrics:
tableNameWithType-topic-partitionGroupId, which matches server.yml rules so topic and partition are exported as labels.
Changes
AbstractMetrics:setOrUpdatePartitionGaugeForStreamTopic / removePartitionGaugeForStreamTopicbuild the same composite key as stream-topic meters and register viasetOrUpdateTableGauge / removeTableGauge.IngestionDelayTracker:updateMetricstakes an optional streamTopicName; if non-blank, delay gauges use the topic-aware helpers; if blank, behavior stays on the legacy per-partition gauge names (backward compatible).IngestionInfo: stores the topic used for registration so removals/shutdown clear the correct MBeans.RealtimeTableDataManager / RealtimeSegmentDataManager: pass _streamConfig.getTopicName() into ingestion metric updates (including “delay zero” paths).Note : Series for REALTIME_INGESTION_DELAY_MS and END_TO_END_REALTIME_INGESTION_DELAY_MS will include a topic label when a non-blank topic is configured; dashboards or alerts that matched only the old metric shape may need updates.
Fixes #18099