Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
996f614
impl(showcase): port observability fixture for mock OTLP testing
quartzmo Apr 10, 2026
18edd84
impl(showcase): port tracing scenarios for integration tests
quartzmo Apr 10, 2026
5e07264
impl(showcase): add tracing tests and update go.mod
quartzmo Apr 10, 2026
c995e56
impl(showcase): add cloud trace test, update makefile and showcase.bash
quartzmo Apr 10, 2026
0c1d1af
impl(showcase): stabilize tracing tests and upgrade dependencies
quartzmo Apr 13, 2026
b8dbbce
docs(showcase): document how telemetry tests work
quartzmo Apr 13, 2026
565a811
update showcase/README.md
quartzmo Apr 13, 2026
f31c1bd
fix(showcase): remove replace directive from go.mod to fix CI apidiff
quartzmo Apr 13, 2026
48b9737
gofmt -w .
quartzmo Apr 13, 2026
02c9e73
fix(showcase): normalize status.message in tests to handle environmen…
quartzmo Apr 13, 2026
5c154c8
chore(showcase): add license headers to new test files
quartzmo Apr 13, 2026
62548d9
add assertion for rpc.system.name: http
quartzmo Apr 14, 2026
3e483c3
add REST assertion for status.message
quartzmo Apr 14, 2026
3858618
add assertions for last span in a retry sequence
quartzmo Apr 14, 2026
e65ea61
test: add REST coverage to Cloud Trace integration tests
quartzmo Apr 14, 2026
e2de8c5
remove unused metrics and logging types from observability_fixture_te…
quartzmo Apr 15, 2026
3c4ffab
test: refactor trace assertions to use lookup and assert helpers
quartzmo Apr 15, 2026
b6d0a91
test: eliminate DYNAMIC placeholder asserts constraints on individual…
quartzmo Apr 15, 2026
d41178f
Merge branch 'main' into showcase-tracing-tests
quartzmo Apr 15, 2026
8a4d384
remove unneeded comments
quartzmo Apr 17, 2026
d727d5b
test: implement strict attribute assertions and constraint callbacks
quartzmo Apr 17, 2026
3571b0b
test: make resource assertions conditional on GCP environment
quartzmo Apr 17, 2026
6beb98a
test: use checkAbsent constraint to simplify assertCapturedSpan
quartzmo Apr 17, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,9 @@ test:
go install ./cmd/protoc-gen-go_gapic
cd showcase && ./showcase.bash && cd .. && ./test.sh

test-telemetry:
cd showcase && ./showcase.bash -tags=telemetry -run TestObservability_Tracing_CloudTrace_Integration

install:
go install ./cmd/protoc-gen-go_gapic
go install ./cmd/protoc-gen-go_cli
Expand Down
35 changes: 35 additions & 0 deletions showcase/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,3 +54,38 @@ and ensures all the necessary dependencies are up to date.

One can use `make clean` to revert any changes and delete any of the test
artifacts created by `showcase.bash`.

## How the telemetry tests work

The telemetry tests in `cloud_trace_test.go` are integration tests that verify end-to-end tracing to the real Google Cloud Trace backend. They require specific setup and are separated from normal Showcase tests.

### Prerequisites and Environment Variables

These tests require a real Google Cloud project to send traces to.
* **`GCLOUD_TESTS_GOLANG_PROJECT_ID`**: Set this environment variable to your GCP project ID before running the tests.
* **Application Default Credentials (ADC)**: You must have valid credentials configured in your environment (e.g., via `gcloud auth application-default login`).

### Build Tag

The test file `cloud_trace_test.go` uses the `//go:build telemetry` constraint. This ensures that these tests are not run during normal `go test` invocations, as they require external network access and specific credentials.

### Running the Tests

You can run these tests using the specific target in the repository root:
```bash
make test-telemetry
```
This target internally runs `./showcase.bash -tags=telemetry -run TestObservability_Tracing_CloudTrace_Integration`.

### Viewing the Traces

To verify that the tests successfully sent spans to Cloud Trace:
1. Go to the **Trace Explorer** in the Google Cloud Console for your project.
2. Adjust the time range to **"Last 1 hour"** (or a narrow window around when you ran the test) to ensure recent spans are visible.
3. Look for spans with `service.name: test-app`.
4. You should see root spans named `APP-Success`, `APP-ServerFailure`, `APP-ClientFailure`, and `APP-Retry`. Spans generated by the Showcase client will have `rpc.method` set to `google.showcase.v1beta1.SequenceService/AttemptSequence` (or similar) and contain `gcp.client.*` telemetry attributes.
5. Note that you may also see failing spans for `TraceService/GetTrace` due to the test polling the backend before the traces were fully indexed.

### Cleaning up

The tests use the standard `showcase.bash` infrastructure, which automatically starts and stops the local Showcase server. Any artifacts created during generation can be cleaned up by running `make clean` in the repository root.
248 changes: 248 additions & 0 deletions showcase/cloud_trace_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,248 @@
// Copyright 2026 Google LLC
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// https://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

//go:build telemetry
// +build telemetry

package showcase

import (
"context"
"encoding/hex"
"os"
"testing"
"time"

trace "cloud.google.com/go/trace/apiv1"
"cloud.google.com/go/trace/apiv1/tracepb"
showcase "github.com/googleapis/gapic-showcase/client"
gax "github.com/googleapis/gax-go/v2"
"go.opentelemetry.io/contrib/detectors/gcp"
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
"go.opentelemetry.io/otel/sdk/resource"
sdktrace "go.opentelemetry.io/otel/sdk/trace"
semconv "go.opentelemetry.io/otel/semconv/v1.26.0"
"golang.org/x/oauth2"
"golang.org/x/oauth2/google"
"google.golang.org/api/option"
"google.golang.org/grpc"
"google.golang.org/grpc/credentials"
"google.golang.org/grpc/credentials/insecure"
"google.golang.org/grpc/credentials/oauth"
)

// setupCloudTrace configures and installs a new tracer provider with a GCP
// resource detector that points to telemetry.googleapis.com. It retrieves the
// project ID from the environment or default credentials, configures telemetry
// environment variables, and initializes the OTLP exporter to send traces to
// Cloud Trace.
func setupCloudTrace(t *testing.T) string {
Comment thread
quartzmo marked this conversation as resolved.
ctx := context.Background()
creds, err := google.FindDefaultCredentials(ctx, "https://www.googleapis.com/auth/cloud-platform")
if err != nil {
t.Skipf("Skipping Cloud Trace integration test: %v", err)
}
projectID := os.Getenv("GCLOUD_TESTS_GOLANG_PROJECT_ID")
if projectID == "" {
projectID = creds.ProjectID
}
if projectID == "" {
t.Skip("Skipping Cloud Trace integration test: no project ID found in GCLOUD_TESTS_GOLANG_PROJECT_ID or default credentials")
}

gax.TestOnlyResetIsFeatureEnabled()
t.Cleanup(gax.TestOnlyResetIsFeatureEnabled)
os.Setenv("GOOGLE_SDK_GO_EXPERIMENTAL_TRACING", "true")
t.Cleanup(func() { os.Unsetenv("GOOGLE_SDK_GO_EXPERIMENTAL_TRACING") })

// Set OTEL_RESOURCE_ATTRIBUTES with project_id for the telemetry endpoint
os.Setenv("OTEL_RESOURCE_ATTRIBUTES", "gcp.project_id="+projectID)
t.Cleanup(func() { os.Unsetenv("OTEL_RESOURCE_ATTRIBUTES") })

// The telemetry endpoint requires a quota project when using ADC user credentials
os.Setenv("GOOGLE_CLOUD_QUOTA_PROJECT", projectID)
t.Cleanup(func() { os.Unsetenv("GOOGLE_CLOUD_QUOTA_PROJECT") })

grpcCreds, err := oauth.NewApplicationDefault(ctx)
if err != nil {
t.Fatalf("failed to create gRPC credentials: %v", err)
}

// Initialize the OTLP exporter to point to telemetry.googleapis.com
exp, err := otlptracegrpc.New(ctx,
otlptracegrpc.WithEndpoint("telemetry.googleapis.com:443"),
otlptracegrpc.WithDialOption(grpc.WithPerRPCCredentials(grpcCreds)),
otlptracegrpc.WithTLSCredentials(credentials.NewClientTLSFromCert(nil, "")),
otlptracegrpc.WithHeaders(map[string]string{"x-goog-user-project": projectID}),
)
if err != nil {
t.Fatalf("failed to create OTLP exporter: %v", err)
}

res, err := resource.New(ctx,
resource.WithDetectors(gcp.NewDetector()),
resource.WithTelemetrySDK(),
resource.WithAttributes(
semconv.ServiceNameKey.String("test-app"),
),
)
if err != nil {
t.Fatalf("failed to create resource: %v", err)
}

tp := sdktrace.NewTracerProvider(
sdktrace.WithBatcher(exp),
sdktrace.WithResource(res),
)
oldTP := otel.GetTracerProvider()
t.Cleanup(func() { otel.SetTracerProvider(oldTP) })
otel.SetTracerProvider(tp)
t.Cleanup(func() {
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
tp.Shutdown(ctx)
})

return projectID
}

func verifyTraceExists(t *testing.T, ctx context.Context, traceClient *trace.Client, projectID string, traceID [16]byte) {
traceIDStr := hex.EncodeToString(traceID[:])
t.Logf("Looking for trace %s in project %s", traceIDStr, projectID)

var found bool
for i := 0; i < 15; i++ {
req := &tracepb.GetTraceRequest{
ProjectId: projectID,
TraceId: traceIDStr,
}
_, err := traceClient.GetTrace(ctx, req)
if err == nil {
found = true
break
}
t.Logf("Attempt %d: trace %s not found yet, retrying...", i+1, traceIDStr)
time.Sleep(2 * time.Second)
}

if !found {
t.Errorf("Trace %s was not found in Cloud Trace backend", traceIDStr)
}
}

func TestObservability_Tracing_CloudTrace_Integration(t *testing.T) {
projectID := setupCloudTrace(t)
ctx := context.Background()

transports := []string{"grpc", "rest"}
for _, transport := range transports {
t.Run(transport, func(t *testing.T) {
var clientOpts []option.ClientOption
if transport == "grpc" {
clientOpts = []option.ClientOption{
option.WithEndpoint("127.0.0.1:7469"),
option.WithTokenSource(oauth2.StaticTokenSource(&oauth2.Token{AccessToken: "dummy-token"})),
option.WithGRPCDialOption(grpc.WithTransportCredentials(insecure.NewCredentials())),
}
} else {
clientOpts = []option.ClientOption{
option.WithEndpoint("http://127.0.0.1:7469"),
option.WithTokenSource(oauth2.StaticTokenSource(&oauth2.Token{AccessToken: "dummy-token"})),
}
}

var seqClient *showcase.SequenceClient
var err error
if transport == "grpc" {
seqClient, err = showcase.NewSequenceClient(ctx, clientOpts...)
} else {
seqClient, err = showcase.NewSequenceRESTClient(ctx, clientOpts...)
}
if err != nil {
t.Fatalf("failed to create sequence client: %v", err)
}
t.Cleanup(func() { seqClient.Close() })

var echoClient *showcase.EchoClient
if transport == "grpc" {
echoClient, err = showcase.NewEchoClient(ctx, clientOpts...)
} else {
echoClient, err = showcase.NewEchoRESTClient(ctx, clientOpts...)
}
if err != nil {
t.Fatalf("failed to create echo client: %v", err)
}
t.Cleanup(func() { echoClient.Close() })

traceClient, err := trace.NewClient(ctx)
if err != nil {
t.Fatalf("failed to create trace client: %v", err)
}
t.Cleanup(func() { traceClient.Close() })

t.Run("Success", func(t *testing.T) {
ctxSpan, span := otel.Tracer("test-tracer").Start(ctx, "APP-Success-"+transport)
if transport == "grpc" {
_ = runTracingSuccessScenario(ctxSpan, t, seqClient)
} else {
_ = runTracingSuccessScenarioREST(ctxSpan, t, seqClient)
}
span.End()
traceID := span.SpanContext().TraceID()
otel.GetTracerProvider().(*sdktrace.TracerProvider).ForceFlush(ctx)
verifyTraceExists(t, ctx, traceClient, projectID, traceID)
})

t.Run("ServerFailure", func(t *testing.T) {
ctxSpan, span := otel.Tracer("test-tracer").Start(ctx, "APP-ServerFailure-"+transport)
if transport == "grpc" {
_ = runTracingServerFailureScenario(ctxSpan, t, seqClient)
} else {
_ = runTracingServerFailureScenarioREST(ctxSpan, t, seqClient)
}
span.End()
traceID := span.SpanContext().TraceID()
otel.GetTracerProvider().(*sdktrace.TracerProvider).ForceFlush(ctx)
verifyTraceExists(t, ctx, traceClient, projectID, traceID)
})

t.Run("ClientFailure", func(t *testing.T) {
ctxSpan, span := otel.Tracer("test-tracer").Start(ctx, "APP-ClientFailure-"+transport)
if transport == "grpc" {
_ = runTracingClientFailureScenario(ctxSpan, t, seqClient)
} else {
_ = runTracingClientFailureScenarioREST(ctxSpan, t, seqClient)
}
span.End()
traceID := span.SpanContext().TraceID()
otel.GetTracerProvider().(*sdktrace.TracerProvider).ForceFlush(ctx)
verifyTraceExists(t, ctx, traceClient, projectID, traceID)
})

t.Run("Retry", func(t *testing.T) {
ctxSpan, span := otel.Tracer("test-tracer").Start(ctx, "APP-Retry-"+transport)
if transport == "grpc" {
_ = runTracingRetryScenario(ctxSpan, t, seqClient)
} else {
_ = runTracingRetryScenarioREST(ctxSpan, t, seqClient)
}
span.End()
traceID := span.SpanContext().TraceID()
otel.GetTracerProvider().(*sdktrace.TracerProvider).ForceFlush(ctx)
verifyTraceExists(t, ctx, traceClient, projectID, traceID)
})
})
}
}
36 changes: 25 additions & 11 deletions showcase/go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -4,41 +4,55 @@ go 1.25.0

require (
cloud.google.com/go v0.123.0
cloud.google.com/go/auth v0.20.0
cloud.google.com/go/iam v1.5.3
cloud.google.com/go/trace v1.11.7
github.com/google/go-cmp v0.7.0
github.com/googleapis/gapic-showcase v0.38.0
github.com/googleapis/gax-go/v2 v2.19.0
github.com/googleapis/gax-go/v2 v2.21.0
go.opentelemetry.io/contrib/detectors/gcp v1.43.0
go.opentelemetry.io/otel v1.43.0
go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploggrpc v0.19.0
go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc v1.43.0
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.43.0
go.opentelemetry.io/otel/sdk v1.43.0
go.opentelemetry.io/otel/sdk/log v0.19.0
go.opentelemetry.io/otel/sdk/metric v1.43.0
go.opentelemetry.io/otel/trace v1.43.0
go.opentelemetry.io/proto/otlp v1.10.0
golang.org/x/oauth2 v0.36.0
google.golang.org/api v0.272.0
google.golang.org/api v0.274.0
google.golang.org/genproto v0.0.0-20260319201613-d00831a3d3e7
google.golang.org/genproto/googleapis/rpc v0.0.0-20260319201613-d00831a3d3e7
google.golang.org/grpc v1.79.3
google.golang.org/genproto/googleapis/rpc v0.0.0-20260401024825-9d38bb4040a9
google.golang.org/grpc v1.80.0
google.golang.org/protobuf v1.36.11
)

require (
cloud.google.com/go/auth v0.18.2 // indirect
cloud.google.com/go/auth/oauth2adapt v0.2.8 // indirect
cloud.google.com/go/compute/metadata v0.9.0 // indirect
cloud.google.com/go/longrunning v0.8.0 // indirect
github.com/GoogleCloudPlatform/opentelemetry-operations-go/detectors/gcp v1.32.0 // indirect
github.com/cenkalti/backoff/v5 v5.0.3 // indirect
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/felixge/httpsnoop v1.0.4 // indirect
github.com/go-logr/logr v1.4.3 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/google/s2a-go v0.1.9 // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/googleapis/enterprise-certificate-proxy v0.3.14 // indirect
github.com/grpc-ecosystem/grpc-gateway/v2 v2.28.0 // indirect
go.opentelemetry.io/auto/sdk v1.2.1 // indirect
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.64.0 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.64.0 // indirect
go.opentelemetry.io/otel v1.42.0 // indirect
go.opentelemetry.io/otel/metric v1.42.0 // indirect
go.opentelemetry.io/otel/trace v1.42.0 // indirect
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.67.0 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.67.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.43.0 // indirect
go.opentelemetry.io/otel/log v0.19.0 // indirect
go.opentelemetry.io/otel/metric v1.43.0 // indirect
golang.org/x/crypto v0.49.0 // indirect
golang.org/x/net v0.52.0 // indirect
golang.org/x/sync v0.20.0 // indirect
golang.org/x/sys v0.42.0 // indirect
golang.org/x/text v0.35.0 // indirect
golang.org/x/time v0.15.0 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20260316180232-0b37fe3546d5 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20260401024825-9d38bb4040a9 // indirect
)
Loading
Loading