discussion/Custom-Transfer-Agent-vs-Batch-API-Server

[bwalsh]

ADR: Git LFS Integration Strategy — Custom Transfer Agent vs Batch API Server

Status

Discussion

---

Background

We are integrating Git LFS with a DRS-backed storage system that supports:

• Client-managed buckets
• Authorization bindings (IAM role, workload identity, broker reference)
• Short-lived credential minting at access time

Git LFS supports two fundamentally different integration models. There are two primary integration surfaces in Git LFS:

1. Client-side: Custom Transfer Agent
2. Server-side: LFS Batch API

These are fundamentally different architectural patterns and support different use cases.

Both can interact with DRS-backed storage, but they differ significantly in:

• Interoperability
• Federation support
• Security guarantees
• Governance capabilities
• SaaS compatibility
• Scientific reproducibility

Decision Drivers

• Interoperability with stock Git clients
• Federation across organizations
• Credential isolation and security
• Compatibility with hosted Git platforms
• Multi-user collaboration
• Operational simplicity
• Alignment with GA4GH / DRS design principles

This ADR documents the architectural differences, supported use cases, trade-offs,
and the “Add URL” (external object registration) workflow.

---

Architecture Overview

Custom Transfer Agent

flowchart LR
    A[Git Client] --> B[Custom Transfer Agent]
    B --> C[Object Store]
    B --> D[DRS APIs]

Loading

Option 1 — Git LFS Custom Transfer Agent (Client-Side)

Description

A custom transfer agent is configured in the Git client via:

git config lfs.customtransfer.<name>.path <binary>

Instead of contacting an LFS server, the Git LFS client:

• Invokes the custom agent binary
• Streams object metadata (OID, size)
• Delegates upload/download directly to the agent

The agent is responsible for:

• Authentication
• Storage interaction
• Credential resolution
• Progress reporting

No LFS Batch API server is involved.

Architecture

Git Client
   │
   │ invokes
   ▼
Custom Transfer Agent
   │
   │ interacts with
   ▼
Object Store / DRS

---

Supported Use Cases

Use Case	Supported
Single-user research workflows	✅
Direct upload to S3/GCS via DRS	✅
Air-gapped or private deployments	✅
Power users with controlled environments	✅
Experimental storage backends	✅
Bypassing Git hosting provider	✅

---

Characteristics

• No server required
• Full control over upload semantics
• Easy to embed DRS-specific logic
• Zero dependency on Git hosting platform support
• Tight coupling between client and storage logic

---

Limitations

Not Supported or Problematic

Use Case	Limitation
Multi-user collaboration	❌ Every user must install identical agent
Public Git hosting integration	❌ GitHub/GitLab do not invoke custom agents
Transparent user experience	❌ Requires client config
Centralized authorization policy	❌ Logic is pushed to clients
Server-side audit of batch requests	❌ Harder to enforce uniformly
Fine-grained repo policy enforcement	❌ Distributed across clients
Web-based uploads	❌ No web compatibility

---

2. LFS Batch API Server (Server-Side)

Description

A standard Git LFS server implements:

POST /info/lfs/objects/batch

The client sends:

{
  "operation": "upload",
  "objects": [
    { "oid": "...", "size": 123 }
  ]
}

The server responds with per-object actions:

{
  "objects": [
    {
      "oid": "...",
      "actions": {
        "upload": {
          "href": "https://signed-url",
          "header": { ... }
        }
      }
    }
  ]
}

The client then uploads directly to object storage using returned URLs.

---

Architecture

Git Client
   │
   │ HTTP
   ▼
LFS Batch API Server
   │
   │ resolves auth binding
   ▼
DRS Control Plane
   │
   ▼
Object Store

Supported Use Cases

Use Case	Supported
Multi-user collaboration	✅
Hosted Git platforms	✅
Enterprise SSO / OAuth	✅
Centralized policy enforcement	✅
Auditable batch requests	✅
Transparent client UX	✅
Repo-scoped storage policies	✅
SaaS deployment model	✅

---

Characteristics

• Fully compatible with stock Git LFS clients
• No client customization required
• Centralized policy and credential resolution
• Aligns with DRS access-time credential minting
• Enables fine-grained repo-level isolation

---

Comparison

Dimension	Custom Transfer Agent	Batch API Server
Requires LFS server	❌	✅
Requires client modification	✅	❌
Works with GitHub/GitLab	❌	✅
Centralized authorization	❌	✅
Repo-scoped isolation	Weak	Strong
Multi-tenant SaaS	❌	✅
Air-gapped research	✅	Possible
Operational simplicity	Client-heavy	Server-heavy
Auditable batch control	Limited	Strong
GA4GH alignment	Medium	High

---

Add URL / External Object Registration Semantics

Overview

The “Add URL” workflow allows registration of a pre-existing external object
without re-uploading it.

Supports:

• Large datasets already stored in cloud buckets
• Cross-project reuse
• Federated research workflows

---

Definitions

No User Upload

The client does not upload object bytes.

No Transfer

No object bytes are transferred at all (server already knows sha256).

---

Mode A — URL + sha256 + size

sequenceDiagram
    participant U as User
    participant C as Git Client
    participant S as LFS/DRS Server
    participant O as Object Store

    U->>C: git lfs add-url <url> --sha256 --size
    C->>O: HEAD <url>
    C->>S: Verify sha256 exists
    S-->>C: Exists
    C->>C: Write pointer file

Loading

Transfer semantics:

• No user upload
• No transfer (if already indexed)

---

Mode B — URL Only

sequenceDiagram
    participant U as User
    participant C as Git Client
    participant S as LFS/DRS Server
    participant O as Object Store

    U->>C: git lfs add-url <url>
    C->>O: HEAD <url>
    C->>S: Resolve sha256
    alt Known
        S-->>C: sha256 + size
    else Unknown
        S->>O: GET object (ingest)
        S-->>C: sha256 computed
    end
    C->>C: Write pointer file

Loading

Transfer semantics:

• No user upload
• Server-side transfer may occur

---

Error Cases

Condition	Error
Size mismatch	SIZE_MISMATCH
Checksum mismatch	CHECKSUM_MISMATCH
Unstable URL	UNSTABLE_OBJECT_SOURCE
Object modified after registration	IMMUTABILITY_VIOLATION
Source not accessible	SOURCE_NOT_ACCESSIBLE

---

Architectural Comparison

Dimension	Custom Agent	Batch API Server
Hosted Git Compatible	No	Yes
Centralized Policy	No	Yes
Multi-tenant SaaS	No	Yes
Auditability	No	Yes
Safe Federated Add-URL	No	Yes
Immutability Enforcement	Weak	Strong

---

Critical Gap: What Is NOT Supported If Only Custom Transfer Agent Is Used

If we rely solely on a custom transfer agent:

1. Hosted Git integration is impossible

   • GitHub and GitLab do not execute custom agents.
   • Users cannot push from standard environments.

2. Multi-user standardization is fragile

   • Every collaborator must install and configure the agent.
   • Version drift causes inconsistencies.

3. No centralized policy enforcement

   • Bucket and authorization logic lives on client machines.
   • Hard to enforce repo-specific storage controls.

4. No transparent federation

   • External collaborators cannot push without installing software.
   • Violates "it just works with Git" principle.

5. Reduced security posture

   • More credential resolution logic distributed to clients.
   • Harder to audit access centrally.

6. No web or CI integration

   • CI/CD systems require custom agent install.
   • Web-based file operations cannot use it.

---

When Custom Transfer Agent Is Appropriate

• Research-only deployments
• Internal platform experiments
• Developer tooling
• Transitional architecture
• Air-gapped or highly controlled environments

---

When Batch API Server Is Required

• Production multi-tenant environments
• Public Git hosting compatibility
• Enterprise SSO integration
• GA4GH federated ecosystems
• DRS-backed storage federation at scale

---

Discussion

For a federated, multi-tenant, GA4GH-aligned system:

A Git LFS Batch API server is required.

The custom transfer agent may remain as a complementary tool but cannot be the sole integration surface.

[bwalsh]

add-url

---

First case: user supplies url + sha256 + size

• Case A (url+sha+size):
  “Client validates reachability and size via HEAD; server checks sha256 existence in authoritative index; no upload is performed.”

✅ This is workable without downloading the object to the user’s filesystem, and without uploading bytes again, if the server has (or can confirm) an authoritative “exists by sha256” index.

Flow

1. User runs: git drs add-url <url> --sha256 <...> --size <...>

2. Client does a HEAD (or equivalent) to sanity-check:

   • URL reachable
   • Content-Length matches size (when available)

3. Server checks authoritative index: sha256 -> storage location exists

4. Client writes LFS pointer locally using provided sha/size

5. On push, no upload is needed because the server/batch path returns “already exists”.

Caveat: HEAD can confirm size, but generally cannot confirm sha256 (unless the origin provides a digest header or you have an agreed metadata convention).

---

Second case: user supplies url only

• Case B (url-only):
  “Client validates reachability via HEAD; server attempts to resolve sha256 via existing metadata mapping; if not resolvable, server must read bytes to compute sha256 (no user upload, but server ingestion may occur).”

A server cannot “calculate sha256” without reading the bytes

There is no way to compute SHA-256 from HEAD alone. HEAD typically gives you metadata (like Content-Length, ETag, Last-Modified), but not a cryptographic hash of the content.

So one of these must be true:

Option 2A — Server already knows the sha256 (metadata lookup)

This is the “no data transfer” version:

• Server derives an identity from the URL metadata (e.g., ETag, Content-Length, maybe a version ID) and looks up a previously-computed mapping:

  • url + etag + size -> sha256

• If found, the server returns sha256 and confirms existence.

✅ No upload, and no server download needed.
❗ Requires a prior ingestion/registration pipeline that populated that mapping.

Option 2B — Server fetches bytes once to compute sha256 (“no user upload”)

If the sha256 is not already known, the server must fetch the object bytes (GET) to hash it.

• This is still no upload from the user
• But it is not “no transfer” — the server is doing ingestion work

So the correct statement is:

“No user upload happens; the server may ingest/read the object to compute sha256 if it is not already known.”

If your intent is “absolutely no byte movement,” then Option 2B violates that.

---

Practical implication for “custom transfer agent only”

Your intended flows are basically saying:

• Existence and sha256 resolution are server responsibilities
• Client can create the pointer without downloading bytes only if it can obtain sha256+size from the server (or user provides them)

That means: even if you’re using a custom transfer agent, you still need a server-side API (not necessarily full LFS Batch API, but at least an “OID resolution” service), e.g.:

• POST /resolve-external { url } -> { sha256, size, exists }
• or POST /exists { sha256 } -> { exists }

Otherwise, the client has no way to confidently create a correct pointer without downloading and hashing locally.

---

[bwalsh]

Below is a curated reading list for understanding Git LFS integration from both sides:

• Client-side: Custom Transfer Agent
• Server-side: LFS Batch API
• Plus supporting specs that matter when building a DRS-backed implementation.

This is structured in reading order: start with protocol basics, then client internals, then server API, then advanced topics.

---

📚 Core Git LFS Specifications (Start Here)

1️⃣ Git LFS Protocol & Batch API (Primary Spec)

Git LFS Batch API

• Defines /objects/batch
• Defines upload, download, verify actions
• Defines request/response schema
• Required reading for server implementation

👉 https://github.com/git-lfs/git-lfs/blob/main/docs/api/batch.md

Why it matters:

• This is the canonical interoperability surface.
• Everything server-side must comply with this.

---

2️⃣ Git LFS Pointer File Format

Defines the LFS pointer structure:

version https://git-lfs.github.com/spec/v1
oid sha256:<digest>
size <bytes>

👉 https://github.com/git-lfs/git-lfs/blob/main/docs/spec.md

Why it matters:

• Critical for “add URL” correctness.
• Defines object identity rules.

---

🖥 Client-Side: Custom Transfer Agent

3️⃣ Custom Transfer Agent Protocol

Defines:

• How Git LFS invokes custom agents
• JSON message exchange format
• Upload/download progress reporting
• Error handling

👉 https://github.com/git-lfs/git-lfs/blob/main/docs/custom-transfers.md

Why it matters:

• Required to implement a custom transfer binary.
• Defines the STDIN/STDOUT protocol.

---

4️⃣ git-lfs-config (Configuration Reference)

Covers:

• lfs.customtransfer.*
• lfs.standalonetransferagent
• lfs.url
• Override behavior

👉 https://github.com/git-lfs/git-lfs/blob/main/docs/man/git-lfs-config.adoc

Why it matters:

• Explains how clients discover and configure custom agents.
• Important for deployment strategy.

---

5️⃣ git-lfs-transfer (Internal behavior)

Helpful for understanding:

• Transfer queue
• Concurrency
• Retry behavior
• Multipart support

👉 https://github.com/git-lfs/git-lfs/blob/main/docs/man/git-lfs-transfer.adoc

Why it matters:

• Explains client-side behavior during push/pull.
• Important when designing compatible servers.

---

🌐 Server-Side: Batch API Implementation

6️⃣ Git LFS Server Discovery

How Git LFS discovers the LFS endpoint:

• .lfsconfig
• lfs.url
• Remote URL + /info/lfs

👉 https://github.com/git-lfs/git-lfs/blob/main/docs/api/server-discovery.md

Why it matters:

• Critical when LFS server is on a different host than Git server.

---

7️⃣ Basic Transfer Adapter (Default HTTP Model)

Describes the “basic” transfer adapter:

• Signed URL model
• Direct-to-object-store upload

👉 https://github.com/git-lfs/git-lfs/blob/main/docs/api/basic-transfers.md

Why it matters:

• Matches DRS-style short-lived credential minting.
• Core to Batch API implementation.

---

8️⃣ Git LFS Locks API (Optional but Relevant)

Defines:

• Lock/unlock endpoints
• Collaborative workflows

👉 https://github.com/git-lfs/git-lfs/blob/main/docs/api/locking.md

Why it matters:

• Not required for minimal implementation.
• Important for full GitHub/GitLab parity.

---

🔐 Integrity, Hashing, and Object Semantics

9️⃣ Git Object Model (Context)

Understanding:

• How Git interacts with LFS
• When OIDs are computed
• Clean/smudge filters

👉 https://git-scm.com/docs/gitattributes

Why it matters:

• Explains when LFS hashing occurs.
• Important for “add URL” design.

---

🔟 Git LFS Clean/Smudge Filters

Describes:

• How LFS replaces large files with pointers
• How downloads occur on checkout

👉 https://github.com/git-lfs/git-lfs/blob/main/docs/man/git-lfs-smudge.adoc
👉 https://github.com/git-lfs/git-lfs/blob/main/docs/man/git-lfs-clean.adoc

Why it matters:

• Helps reason about pointer creation and fetch behavior.

---

🧪 Advanced / Optional Topics

11️⃣ TUS (Resumable Uploads)

Git LFS can support TUS-based resumable uploads.

👉 https://github.com/git-lfs/git-lfs/blob/main/docs/api/tus-transfers.md

Why it matters:

• Relevant for very large genomic datasets.

---

12️⃣ Error Handling & Response Semantics

See:

• Batch API error object
• HTTP status codes

In:
👉 https://github.com/git-lfs/git-lfs/blob/main/docs/api/batch.md

Why it matters:

• Needed to implement correct failure modes.
• Critical for immutability enforcement.

---

🧭 Suggested Reading Order

If you are:

Implementing a Batch API server

1. Batch API spec
2. Basic transfer model
3. Server discovery
4. Pointer format
5. Clean/smudge filters

Implementing a Custom Transfer Agent

1. Custom transfer spec
2. git-lfs-config
3. Transfer internals
4. Pointer format

Designing a DRS-backed hybrid

1. Batch API
2. Custom transfer spec
3. Server discovery
4. Basic transfer model
5. Clean/smudge filters

---

🎯 If You Are Building GA4GH-Aligned Infrastructure

Pay particular attention to:

• Batch API
• Basic transfer model
• Pointer format
• Clean/smudge lifecycle

Those four define 90% of interoperability.

[bwalsh]

minimum viable “compliant Git LFS server” checklist

• i.e., the smallest feature set that works with stock Git LFS clients and interoperates correctly.

• basic transfer mode with signed URLs (the most common and DRS-aligned pattern). ([GitHub]1)

---

Minimum Viable Compliant LFS Server Checklist

A. Server discovery and endpoint shape

• Provide an LFS base URL that Git LFS can discover:

  • Default derived from Git remote by appending .git/info/lfs, or
  • Explicitly configured via lfs.url / .lfsconfig in the repo. ([GitHub]2)

• Expose the Batch endpoint at:

  • POST <lfs_base>/objects/batch ([GitHub]1)

B. Authentication and authorization

• Require authentication on objects/batch (and any other endpoints you expose).

• Enforce authorization per repository scope (repo/project/tenant), not just per object hash.

• Return correct HTTP codes for auth failures:

  • 401 (not authenticated) / 403 (authenticated but not allowed).
    (Git LFS clients handle these as hard failures; this is important for predictable behavior.) ([GitLab Docs]3)

C. Implement Batch API correctly

• Accept a Batch request containing:

  • operation: download or upload
  • objects: array of { oid, size }
  • transfers: include support for "basic" ([GitHub]1)

• Return a Batch response containing, for each object:

  • oid, size

  • either:

    • actions (success path), or
    • error (with a message + code) ([GitHub]1)

D. Basic transfer actions (data-plane URLs)

For each object returned in the Batch response:

Download operation

• Return actions.download with:

  • href (URL to download bytes)
  • header (optional headers)
  • expires_at or expires_in ([GitHub]4)

Upload operation

• If object does not exist: return actions.upload with href/header/expires_*. ([GitHub]1)
• If object already exists: return the object with no upload action (or otherwise indicate “present” per spec behavior). This is how clients skip uploads. ([GitHub]1)

E. Pointer / object identity correctness

• Treat oid as sha256 and ensure it corresponds to the Git LFS pointer spec:

  • oid sha256:<hex>
  • size <bytes> ([GitHub]5)

• Enforce size consistency:

  • If a client requests upload/download for {oid,size} that conflicts with known metadata, return an error (do not serve wrong bytes). ([GitHub]1)

F. Optional but strongly recommended for correctness

• Support the verify action for uploads:

  • Batch may return actions.verify
  • Client will POST to it after uploading (if provided) ([GitHub]4)

• On verify:

  • Validate size and (if feasible) checksum/ETag expectations
  • Mark object “committed/available” in your index

G. Operational basics

• Make action URLs short-lived and scoped:

  • per-object, per-operation
  • minimal permissions

• Log:

  • auth principal
  • repo scope
  • operation (upload/download)
  • oid + size
  • decision (served actions vs skipped vs error)

---

What you can skip in an MVP (still “works” with stock clients)

• Locks API (collaboration feature) — optional. ([GitHub]1)
• TUS or multipart extensions — optional (add later). ([GitHub]4)
• Repository creation endpoints — out of scope for LFS; repo context comes from the URL / host platform. ([GitHub]2)

---

MVP “definition of done” sanity test

• git lfs env shows your lfs.url / discovered endpoint is correct. ([GitHub]2)
• git lfs push calls POST …/objects/batch and receives upload actions for missing objects. ([GitHub]1)
• Re-push of same commit causes client to skip uploads (server indicates objects already present). ([GitHub]6)
• git lfs fetch calls POST …/objects/batch and receives download actions, and checkout succeeds. ([GitHub]1)

---

[bwalsh]

For Auth
See https://github.com/calypr/git-drs-auth-helper

[kellrott]

The goal of this project is to make a client that can work against a standard GA4GH DRS server. This seems like its not working toward that goal