Skip to content

Go: Data race triggered by Encrypt/Decrypt (trips Go race detector) #902

Open
Open
Go: Data race triggered by Encrypt/Decrypt (trips Go race detector)#902
@jalaziz

Description

@jalaziz
jalaziz
opened on May 3, 2026

While running tests using the AWS Encryption SDK for Go, we are seeing failed tests due to Go's race detector.

The relevant bits of the stack trace are:

WARNING: DATA RACE
Write at 0x00c000010d38 by goroutine 20:
  github.com/dafny-lang/DafnyRuntimeGo/v4/dafny.(*ArraySequence).IsString_set_()
      /home/runner/go/pkg/mod/github.com/dafny-lang/!dafny!runtime!go/v4@v4.11.2/dafny/dafnyFromDafny.go:937 +0x30
  github.com/dafny-lang/DafnyRuntimeGo/v4/dafny.(*CompanionStruct_Sequence_).SetString()
      /home/runner/go/pkg/mod/github.com/dafny-lang/!dafny!runtime!go/v4@v4.11.2/dafny/dafnyFromDafny.go:202 +0x18
  github.com/dafny-lang/DafnyRuntimeGo/v4/dafny.(*ArraySequence).SetString()
      /home/runner/go/pkg/mod/github.com/dafny-lang/!dafny!runtime!go/v4@v4.11.2/dafny/dafnyFromDafny.go:948 +0x2c
  github.com/aws/aws-cryptographic-material-providers-library/releases/go/smithy-dafny-standard-library/Base64.(*CompanionStruct_Default___).FromIndicesToChars()
      /home/runner/go/pkg/mod/github.com/aws/aws-cryptographic-material-providers-library/releases/go/smithy-dafny-standard-library@v0.3.0/Base64/Base64.go:272 +0x48
  github.com/aws/aws-cryptographic-material-providers-library/releases/go/smithy-dafny-standard-library/Base64.(*CompanionStruct_Default___).EncodeUnpadded()
      /home/runner/go/pkg/mod/github.com/aws/aws-cryptographic-material-providers-library/releases/go/smithy-dafny-standard-library@v0.3.0/Base64/Base64.go:290 +0x4c
  github.com/aws/aws-cryptographic-material-providers-library/releases/go/smithy-dafny-standard-library/Base64.(*CompanionStruct_Default___).Encode()
      /home/runner/go/pkg/mod/github.com/aws/aws-cryptographic-material-providers-library/releases/go/smithy-dafny-standard-library@v0.3.0/Base64/Base64.go:361 +0x94
  github.com/aws/aws-cryptographic-material-providers-library/releases/go/mpl/Materials.(*CompanionStruct_Default___).InitializeEncryptionMaterials.func2()
      /home/runner/go/pkg/mod/github.com/aws/aws-cryptographic-material-providers-library/releases/go/mpl@v0.3.0/Materials/Materials.go:224 +0x10a4
  github.com/aws/aws-cryptographic-material-providers-library/releases/go/mpl/Materials.(*CompanionStruct_Default___).InitializeEncryptionMaterials()
      /home/runner/go/pkg/mod/github.com/aws/aws-cryptographic-material-providers-library/releases/go/mpl@v0.3.0/Materials/Materials.go:246 +0xb58
  github.com/aws/aws-cryptographic-material-providers-library/releases/go/mpl/DefaultCMM.(*DefaultCMM).GetEncryptionMaterials_k()
      /home/runner/go/pkg/mod/github.com/aws/aws-cryptographic-material-providers-library/releases/go/mpl@v0.3.0/DefaultCMM/DefaultCMM.go:353 +0xc24
  github.com/aws/aws-cryptographic-material-providers-library/releases/go/mpl/AwsCryptographyMaterialProvidersTypes.(*CompanionStruct_ICryptographicMaterialsManager_).GetEncryptionMaterials()
      /home/runner/go/pkg/mod/github.com/aws/aws-cryptographic-material-providers-library/releases/go/mpl@v0.3.0/AwsCryptographyMaterialProvidersTypes/AwsCryptographyMaterialProvidersTypes.go:3643 +0x50
  github.com/aws/aws-cryptographic-material-providers-library/releases/go/mpl/DefaultCMM.(*DefaultCMM).GetEncryptionMaterials()
      /home/runner/go/pkg/mod/github.com/aws/aws-cryptographic-material-providers-library/releases/go/mpl@v0.3.0/DefaultCMM/DefaultCMM.go:281 +0x50
  github.com/aws/aws-encryption-sdk/releases/go/encryption-sdk/EncryptDecryptHelpers.(*CompanionStruct_Default___).GetEncryptionMaterials()
      /home/runner/go/pkg/mod/github.com/aws/aws-encryption-sdk/releases/go/encryption-sdk@v0.3.0/EncryptDecryptHelpers/EncryptDecryptHelpers.go:691 +0x144
  github.com/aws/aws-encryption-sdk/releases/go/encryption-sdk/AwsEncryptionSdkOperations.(*CompanionStruct_Default___).Encrypt()
      /home/runner/go/pkg/mod/github.com/aws/aws-encryption-sdk/releases/go/encryption-sdk@v0.3.0/AwsEncryptionSdkOperations/AwsEncryptionSdkOperations.go:380 +0x1184
  github.com/aws/aws-encryption-sdk/releases/go/encryption-sdk/ESDK.(*ESDKClient).Encrypt()
      /home/runner/go/pkg/mod/github.com/aws/aws-encryption-sdk/releases/go/encryption-sdk@v0.3.0/ESDK/ESDK.go:435 +0x110
  github.com/aws/aws-encryption-sdk/releases/go/encryption-sdk/awscryptographyencryptionsdksmithygenerated.(*Client).Encrypt()
      /home/runner/go/pkg/mod/github.com/aws/aws-encryption-sdk/releases/go/encryption-sdk@v0.3.0/awscryptographyencryptionsdksmithygenerated/api_client.go:38 +0x958
  <truncated>/encryption.(*awsEncryptionClient).Encrypt()
      ....
  <truncated>/encryption.TestHandler_Encrypt_And_Decrypt.func3()
      ....
  testing.tRunner()
      /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.8.linux-arm64/src/testing/testing.go:1934 +0x164
  testing.(*T).Run.gowrap1()
      /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.8.linux-arm64/src/testing/testing.go:1997 +0x3c

Previous write at 0x00c000010d38 by goroutine 19:
  github.com/dafny-lang/DafnyRuntimeGo/v4/dafny.(*ArraySequence).IsString_set_()
      /home/runner/go/pkg/mod/github.com/dafny-lang/!dafny!runtime!go/v4@v4.11.2/dafny/dafnyFromDafny.go:937 +0x30
  github.com/dafny-lang/DafnyRuntimeGo/v4/dafny.(*CompanionStruct_Sequence_).SetString()
      /home/runner/go/pkg/mod/github.com/dafny-lang/!dafny!runtime!go/v4@v4.11.2/dafny/dafnyFromDafny.go:202 +0x18
  github.com/dafny-lang/DafnyRuntimeGo/v4/dafny.(*ArraySequence).SetString()
      /home/runner/go/pkg/mod/github.com/dafny-lang/!dafny!runtime!go/v4@v4.11.2/dafny/dafnyFromDafny.go:948 +0x2c
  github.com/aws/aws-cryptographic-material-providers-library/releases/go/smithy-dafny-standard-library/Base64.(*CompanionStruct_Default___).FromIndicesToChars()
      /home/runner/go/pkg/mod/github.com/aws/aws-cryptographic-material-providers-library/releases/go/smithy-dafny-standard-library@v0.3.0/Base64/Base64.go:272 +0x48
  github.com/aws/aws-cryptographic-material-providers-library/releases/go/smithy-dafny-standard-library/Base64.(*CompanionStruct_Default___).EncodeUnpadded()
      /home/runner/go/pkg/mod/github.com/aws/aws-cryptographic-material-providers-library/releases/go/smithy-dafny-standard-library@v0.3.0/Base64/Base64.go:290 +0x4c
  github.com/aws/aws-cryptographic-material-providers-library/releases/go/smithy-dafny-standard-library/Base64.(*CompanionStruct_Default___).Encode()
      /home/runner/go/pkg/mod/github.com/aws/aws-cryptographic-material-providers-library/releases/go/smithy-dafny-standard-library@v0.3.0/Base64/Base64.go:361 +0x94
  github.com/aws/aws-cryptographic-material-providers-library/releases/go/mpl/Materials.(*CompanionStruct_Default___).InitializeEncryptionMaterials.func2()
      /home/runner/go/pkg/mod/github.com/aws/aws-cryptographic-material-providers-library/releases/go/mpl@v0.3.0/Materials/Materials.go:224 +0x10a4
  github.com/aws/aws-cryptographic-material-providers-library/releases/go/mpl/Materials.(*CompanionStruct_Default___).InitializeEncryptionMaterials()
      /home/runner/go/pkg/mod/github.com/aws/aws-cryptographic-material-providers-library/releases/go/mpl@v0.3.0/Materials/Materials.go:246 +0xb58
  github.com/aws/aws-cryptographic-material-providers-library/releases/go/mpl/DefaultCMM.(*DefaultCMM).GetEncryptionMaterials_k()
      /home/runner/go/pkg/mod/github.com/aws/aws-cryptographic-material-providers-library/releases/go/mpl@v0.3.0/DefaultCMM/DefaultCMM.go:353 +0xc24
  github.com/aws/aws-cryptographic-material-providers-library/releases/go/mpl/AwsCryptographyMaterialProvidersTypes.(*CompanionStruct_ICryptographicMaterialsManager_).GetEncryptionMaterials()
      /home/runner/go/pkg/mod/github.com/aws/aws-cryptographic-material-providers-library/releases/go/mpl@v0.3.0/AwsCryptographyMaterialProvidersTypes/AwsCryptographyMaterialProvidersTypes.go:3643 +0x50
  github.com/aws/aws-cryptographic-material-providers-library/releases/go/mpl/DefaultCMM.(*DefaultCMM).GetEncryptionMaterials()
      /home/runner/go/pkg/mod/github.com/aws/aws-cryptographic-material-providers-library/releases/go/mpl@v0.3.0/DefaultCMM/DefaultCMM.go:281 +0x50
  github.com/aws/aws-encryption-sdk/releases/go/encryption-sdk/EncryptDecryptHelpers.(*CompanionStruct_Default___).GetEncryptionMaterials()
      /home/runner/go/pkg/mod/github.com/aws/aws-encryption-sdk/releases/go/encryption-sdk@v0.3.0/EncryptDecryptHelpers/EncryptDecryptHelpers.go:691 +0x144
  github.com/aws/aws-encryption-sdk/releases/go/encryption-sdk/AwsEncryptionSdkOperations.(*CompanionStruct_Default___).Encrypt()
      /home/runner/go/pkg/mod/github.com/aws/aws-encryption-sdk/releases/go/encryption-sdk@v0.3.0/AwsEncryptionSdkOperations/AwsEncryptionSdkOperations.go:380 +0x1184
  github.com/aws/aws-encryption-sdk/releases/go/encryption-sdk/ESDK.(*ESDKClient).Encrypt()
      /home/runner/go/pkg/mod/github.com/aws/aws-encryption-sdk/releases/go/encryption-sdk@v0.3.0/ESDK/ESDK.go:435 +0x110
  github.com/aws/aws-encryption-sdk/releases/go/encryption-sdk/awscryptographyencryptionsdksmithygenerated.(*Client).Encrypt()
      /home/runner/go/pkg/mod/github.com/aws/aws-encryption-sdk/releases/go/encryption-sdk@v0.3.0/awscryptographyencryptionsdksmithygenerated/api_client.go:38 +0x958
  <truncated>/encryption.(*awsEncryptionClient).Encrypt()
      ....
  <truncated>/encryption.TestHandler_DecryptAndVerify.func1()
      ...
  testing.tRunner()
      /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.8.linux-arm64/src/testing/testing.go:1934 +0x164
  testing.(*T).Run.gowrap1()
      /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.8.linux-arm64/src/testing/testing.go:1997 +0x3c

Goroutine 20 (running) created at:
  testing.(*T).Run()
      /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.8.linux-arm64/src/testing/testing.go:1997 +0x6e0
  <truncated>/encryption.TestHandler_Encrypt_And_Decrypt()
      ....
  testing.tRunner()
      /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.8.linux-arm64/src/testing/testing.go:1934 +0x164
  testing.(*T).Run.gowrap1()
      /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.8.linux-arm64/src/testing/testing.go:1997 +0x3c

Goroutine 19 (running) created at:
  testing.(*T).Run()
      /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.8.linux-arm64/src/testing/testing.go:1997 +0x6e0
  <truncated>/encryption.TestHandler_DecryptAndVerify()
      ....
  testing.tRunner()
      /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.8.linux-arm64/src/testing/testing.go:1934 +0x164
  testing.(*T).Run.gowrap1()
      /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.8.linux-arm64/src/testing/testing.go:1997 +0x3c
==================
--- FAIL: TestHandler_Encrypt_And_Decrypt (0.00s)
    --- FAIL: TestHandler_Encrypt_And_Decrypt/Decrypts_ciphertext_produced_in_a_separate_Encrypt_call (0.05s)
        testing.go:1617: race detected during execution of test
    --- FAIL: TestHandler_Encrypt_And_Decrypt/Successfully_encrypts_and_decrypts (0.07s)
        testing.go:1617: race detected during execution of test
--- FAIL: TestHandler_DecryptAndVerify (0.00s)
    --- FAIL: TestHandler_DecryptAndVerify/Returns_plaintext_when_expected_context_matches (0.05s)
        testing.go:1617: race detected during execution of test
    --- FAIL: TestHandler_DecryptAndVerify/Empty_expected_context_behaves_like_Decrypt (0.05s)
        testing.go:1617: race detected during execution of test

It appears that the race is actually within Dafny, but filing the issue here as it's affecting the encryption SDK.

I used Claude to help investigate the root cause, the full detailed analysis provides much more detail.

AI Root Cause Analysis

Affected versions

  • github.com/aws/aws-encryption-sdk/releases/go/encryption-sdk v0.3.0
  • github.com/aws/aws-cryptographic-material-providers-library/releases/go/mpl v0.3.0
  • github.com/aws/aws-cryptographic-material-providers-library/releases/go/smithy-dafny-standard-library v0.3.0
  • github.com/dafny-lang/DafnyRuntimeGo/v4 v4.11.2
  • Go 1.25.x

Summary

The Go release of the AWS Encryption SDK is not goroutine-safe for concurrent use within a single process. Two goroutines that each call Client.Encrypt (or Decrypt) on independent ESDKClient instances — i.e. with no shared client state of their own — still race on a process-global ArraySequence inside the Dafny runtime. The Go -race detector flags this on every concurrent invocation.

Reproducer

package main

import (
    "context"
    "sync"
    "testing"

    mpl "github.com/aws/aws-cryptographic-material-providers-library/releases/go/mpl/awscryptographymaterialproviderssmithygenerated"
    mpltypes "github.com/aws/aws-cryptographic-material-providers-library/releases/go/mpl/awscryptographymaterialproviderssmithygeneratedtypes"
    esdkclient "github.com/aws/aws-encryption-sdk/releases/go/encryption-sdk/awscryptographyencryptionsdksmithygenerated"
    esdktypes "github.com/aws/aws-encryption-sdk/releases/go/encryption-sdk/awscryptographyencryptionsdksmithygeneratedtypes"
)

func TestConcurrentEncryptRaces(t *testing.T) {
    ctx := context.Background()
    var wg sync.WaitGroup
    for i := 0; i < 2; i++ {
        wg.Add(1)
        go func() {
            defer wg.Done()
            matProv, _ := mpl.NewClient(mpltypes.MaterialProvidersConfig{})
            keyring, _ := matProv.CreateRawAesKeyring(ctx, mpltypes.CreateRawAesKeyringInput{
                KeyName:      "k",
                KeyNamespace: "ns",
                WrappingKey:  []byte("superSecureKeySecureKey32bytes32"),
                WrappingAlg:  mpltypes.AesWrappingAlgAlgAes256GcmIv12Tag16,
            })
            c, _ := esdkclient.NewClient(esdktypes.AwsEncryptionSdkConfig{})
            _, _ = c.Encrypt(ctx, esdktypes.EncryptInput{
                Plaintext:         []byte("hello"),
                EncryptionContext: map[string]string{"k": "v"},
                Keyring:           keyring,
            })
        }()
    }
    wg.Wait()
}

Run with go test -race. The race detector reports the issue reliably.

Race trace (excerpt)

WARNING: DATA RACE
Write at 0x... by goroutine A:
  DafnyRuntimeGo/v4/dafny.(*ArraySequence).IsString_set_()
      dafny/dafnyFromDafny.go:937 +0x30
  DafnyRuntimeGo/v4/dafny.(*CompanionStruct_Sequence_).SetString()
      dafny/dafnyFromDafny.go:202 +0x18
  DafnyRuntimeGo/v4/dafny.(*ArraySequence).SetString()
      dafny/dafnyFromDafny.go:948 +0x2c
  smithy-dafny-standard-library/Base64.(*CompanionStruct_Default___).FromIndicesToChars()
      Base64/Base64.go:272 +0x48
  smithy-dafny-standard-library/Base64.(*CompanionStruct_Default___).EncodeUnpadded()
      Base64/Base64.go:290 +0x4c
  smithy-dafny-standard-library/Base64.(*CompanionStruct_Default___).Encode()
      Base64/Base64.go:361 +0x94
  mpl/Materials.(*CompanionStruct_Default___).InitializeEncryptionMaterials.func2()
      Materials/Materials.go:224 +0x10a4
  ...
  encryption-sdk/ESDK.(*ESDKClient).Encrypt()

Previous write at 0x... by goroutine B:
  [identical stack]

Root cause

Base64.FromIndicesToChars() returns a process-global ArraySequence (the base64 alphabet). Base64.Encode then calls SetString(true) on it, which dispatches to ArraySequence.IsString_set_ — an unsynchronized field write on the receiver (dafnyFromDafny.go:937). Because the alphabet sequence is shared, every concurrent Encrypt/Decrypt walks this code path and races on the same isString field.

The writes happen to be idempotent (always true), so on x86/ARM the field's final value is still correct in practice. But under the Go memory model, unsynchronized concurrent writes are undefined behavior, and -race flags them on every run, which makes the SDK unusable in any Go test suite that runs with -race (including the default for any project that values race-free CI).

Expected

Concurrent calls to Encrypt/Decrypt from independent ESDKClient instances complete without a race detector warning.

Suggested fix

Either:

  1. Make ArraySequence.IsString_set_ race-safe — e.g. back isString with sync/atomic or set it once via sync.Once since the value never transitions back.
  2. Avoid mutating the shared alphabet sequence in Base64.Encode; treat the alphabet as an immutable value and produce new sequences for the encoded output.

Option 2 is preferable because it eliminates a class of similar latent races in the broader Dafny stdlib and matches the immutable-sequence semantics Dafny otherwise advertises.

Workarounds

  • Run encryption tests serially (-parallel=1 or no t.Parallel()).
  • Add a process-wide mutex around every SDK call. This serializes all encryption in the process, which is unacceptable for servers that issue per-request KMS-backed encryption.

Neither is satisfying for a server runtime that does concurrent encryption per request.

Impact

Any Go service that calls Client.Encrypt/Decrypt from per-request goroutines (i.e. the typical pattern for encrypting PII on incoming HTTP requests) hits this race continuously. Production behavior is likely benign in practice because the racing write is idempotent, but the SDK cannot be cleanly validated under go test -race, and there is no upstream guidance on whether the SDK is intended to be goroutine-safe in this configuration.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions