[REVIEW] model-supply-chain: bind composed model artifact identity

[mlodygbelmondo]

Skill Being Reviewed

Skill name: model-supply-chain
Skill path: skills/ai-security/model-supply-chain/

False Positive Analysis

Benign deployment that can be over-reported if the reviewer treats the adapter as an untrusted standalone model:

deployment: support-classifier-prod
base_model:
  source: huggingface.co/org/base-model
  revision: 9f4c2b1d8e8f4b2a6f0d9c1e7a2c3b4d5e6f7890
  digest: sha256:base...
adapter:
  source: internal-registry.example.com/lora/support-router
  revision: 3b7a9c0e2d44f1e9a6a9dbef04e4a7a1d56a92ac
  digest: sha256:adapter...
tokenizer:
  source: same as base model
  revision: 9f4c2b1d8e8f4b2a6f0d9c1e7a2c3b4d5e6f7890
prompt_template_digest: sha256:template...
quantization:
  method: awq
  artifact_digest: sha256:quantized...
runtime:
  server: vllm
  image_digest: sha256:image...

Why this is a false positive risk:

The skill correctly flags unpinned model sources, unsafe serialization, weak provenance, and third-party adapters as supply chain risks. However, a LoRA/PEFT adapter, tokenizer, prompt template, quantized artifact, and runtime image are not independently sufficient evidence of the deployed model. The security decision depends on the exact composed artifact. If reviewers only see a third-party adapter name, they may over-report it even when it is pinned, internally ingested, digest-verified, compatibility-checked against the base model, and served from a controlled runtime.

Coverage Gaps

Missed variant 1: Adapter is pinned, but the base model floats

base = AutoModelForCausalLM.from_pretrained("vendor/base-model")
model = PeftModel.from_pretrained(base, "internal/security-reviewed-lora", revision="3b7a9c0")

Why it should be caught:

The adapter revision is controlled, but the effective model can still change whenever the base model, tokenizer, or config changes. The current skill asks reviewers to check unpinned model versions and adapter sources, but the output format does not require a single composed identity record that proves every component is pinned and digest-bound.

Missed variant 2: Tokenizer or chat template drift changes model behavior without changing weights

Weights digest: unchanged
Tokenizer revision: latest
chat_template: updated in registry
Observed behavior: system/developer boundaries are tokenized or framed differently

Why it should be caught:

For LLM deployments, the tokenizer, generation config, special tokens, and chat/prompt template materially affect behavior. A supply chain review that only records model weight hashes can miss registry or template drift that changes safety, routing, tool-use, or classification behavior while preserving the same weight artifact.

Missed variant 3: Quantization or format conversion creates a new artifact with no provenance

python convert.py --model org/base --format gguf --quant q4_k_m --out model.gguf

Why it should be caught:

Quantized GGUF/AWQ/GPTQ/ONNX artifacts are deployable model artifacts. Even if the source weights were pinned, the conversion command, converter version, calibration data, output digest, and signer/attestation need to be captured. Otherwise a production model can be replaced or misbuilt after the reviewed source model passed provenance checks.

Missed variant 4: Runtime loads additional adapters dynamically

Startup model: base@sha256:base + safety-adapter@sha256:safe
Runtime route: /v1/adapters/load accepts adapter_id from config service
Loaded after deployment: promo-adapter@latest
Review evidence: only startup manifest

Why it should be caught:

Some serving stacks can hot-load adapters, merge adapters, or select adapters by route/tenant. The current skill mentions adapter/plugin sources, but it should require runtime adapter registry evidence and an update policy so the reviewed artifact cannot drift after deployment.

Edge Cases

• Multiple adapters can be merged or stacked; the review should record ordering, merge method, and final artifact digest where applicable.
• A safetensors base model avoids pickle-style code execution but does not prove the tokenizer, adapter, conversion output, or runtime image is trusted.
• A model card can describe the base model while saying nothing about the fine-tuned adapter, prompt template, quantized artifact, or serving configuration.
• revision="main" or a branch name is not equivalent to a commit hash for reproducible review evidence.
• A cache hit can hide registry drift; reviewers should distinguish local cache evidence from the remote revision/digest that will be pulled by fresh deployments.
• Per-tenant adapters and retrieval/router configurations can make one deployment name correspond to multiple effective models.

Remediation Quality

• Fix resolves the vulnerability
• Fix doesn't introduce new security issues
• Fix doesn't break functionality
• Issues found: Add a composed model artifact identity step and output table. Require reviewers to record base model, adapter(s), tokenizer, config, prompt/chat template, quantization/conversion output, runtime image, registry revision, digest, source authority, update policy, and runtime adapter registry evidence. Add Not Evaluable outcomes when any deployed component cannot be pinned to a revision and digest.

Comparison to Other Tools

Tool / Framework	Catches this?	Notes
SLSA provenance	Partial	Strong model for artifact identity and build provenance, but the skill needs to apply it to composed ML artifacts, not only training pipelines.
Hugging Face Hub revisions	Partial	Supports downloading from a specific revision, but the reviewer must verify that base, adapter, tokenizer, and configs are all fixed.
PEFT / LoRA loader review	Partial	Shows adapter loading behavior, but does not by itself prove base-model compatibility, composition order, or runtime update policy.
Safetensors	Partial	Reduces unsafe deserialization risk, but does not bind tokenizer, adapter, prompt template, quantization, or runtime provenance.
Container/image signing	Partial	Helps prove serving image identity, but not the complete model composition unless the manifest includes model component digests.

Overall Assessment

Strengths:

• Strong coverage of model provenance, unsafe serialization, training data lineage, fine-tuning pipeline integrity, inference dependencies, model cards, and backdoor checks.
• Good explicit warning that safetensors is not a complete supply-chain solution.
• Useful SLSA framing for model training pipelines and artifacts.

Needs improvement:

• The model inventory output records model source, format, checksum, pinned version, and model card, but not the full deployed composition.
• Adapter/plugin source checks are mentioned, but runtime adapter loading, adapter order, and adapter/base compatibility are not first-class evidence.
• Tokenizer, generation config, prompt/chat template, quantized/conversion artifact, and serving image drift can change behavior without changing the original model weights.
• The output lacks confidence and Not Evaluable fields for missing component digests or runtime registry evidence.

Priority recommendations:

1. Add a Composed Model Artifact Identity table with deployment, base model source/revision/digest, adapter source/revision/digest/order, tokenizer revision/digest, config/generation config digest, prompt/chat template digest, quantization/conversion method, converted artifact digest, runtime image digest, runtime adapter registry, update policy, source authority, confidence, and Not Evaluable reason.
2. Require reviewers to flag branch/tag-only references such as main, latest, or mutable config-service adapter IDs as High risk for production deployments unless an internal registry pins immutable digests.
3. Add a runtime drift check for hot-loaded adapters, per-tenant adapter selection, cache-only evidence, and registry updates after the reviewed manifest.
4. Add remediation guidance to publish a signed model bill of materials or deployment manifest that binds all model components and runtime artifacts together.

Sources Checked

• OWASP Top 10 for LLM Applications 2025, LLM03 Supply Chain: https://genai.owasp.org/llmrisk/llm03-supply-chain-vulnerabilities/
• SLSA Provenance, artifact digest fields: https://slsa.dev/spec/v1.2/provenance
• Hugging Face Hub CLI revision downloads: https://huggingface.co/docs/huggingface_hub/en/guides/cli
• Hugging Face Transformers PEFT adapter loading: https://huggingface.co/docs/transformers/v4.42.0/en/peft
• Hugging Face Safetensors documentation: https://huggingface.co/docs/safetensors

Bounty Info

• I have read and agree to the CONTRIBUTING.md bounty terms
• Preferred payment method: to be provided privately after acceptance

[auruminternum]

Opened follow-up improvement PR implementing this review: https://github.com/UnitOneAI/SecuritySkills/pull/191\n\nPreferred payment method can be provided privately after acceptance.

[mlodygbelmondo]

Thanks for opening an implementation PR.

For clarity, this issue is my submitted Skill Review under the repository's review bounty terms. Payment details can be provided privately after maintainer acceptance, as noted in the issue body.

[JamesJi79]

Review: Issue #186 — model-supply-chain: bind composed model artifact identity

Reviewer: Hermes Agent
Date: 2026-06-03
Commit reviewed: e4bf0ec on branch pr/191
Author: auruminternum
Skill file: skills/ai-security/model-supply-chain/SKILL.md
Status: APPROVED with minor suggestions

---

Summary

The commit adds a new Step 1A — Composed Model Artifact Identity section to the model-supply-chain skill, plus supporting changes across the Context table, Output Format, Findings Classification, Supply Chain Maturity Summary, Common Pitfalls, and References. This is the artifact identity binding feature requested in Issue #186.

The patch touches the single file SKILL.md, comprising 71 insertions and 1 deletion across 7 logical change sites.

---

What Was Changed (Audit of Each Diff Hunk)

1. Context table — three new rows (lines 85-87 of current file)

Added rows for:

• Tokenizer and prompt/chat template
• Quantization/conversion output
• Runtime adapter registry

Assessment: Excellent additions. These three items were genuine blind spots in the original context table — a reviewer could gather evidence on the base model but miss the mutable supporting components that collectively define the deployed model's behavior.

2. Step 1A — Composed Model Artifact Identity (new section, ~50 lines)

This is the core of the change. It includes:

• Scenario description: Clearly explains that the effective deployed model is a composition of base weights, adapters, tokenizer, configs, quantized artifacts, runtime image, and route-specific adapter policies.
• What to look for: 6 concrete inspection points (pinned base + floating tokenizers, mutable adapter references, stacked adapters without merge order, quantized artifacts without provenance, runtime hot-loading, cache-only evidence).
• Detection commands: Grep and Glob patterns for composed model loading, quantization, and runtime adapter loading.
• Composed identity evidence table: A 12-column table mapping Deployment → Base model → Adapter(s) → Tokenizer → Config/template → Quantized artifact → Runtime image → Runtime adapter registry → Update policy → Source authority → Confidence → Not Evaluable reason.
• Findings table: 6 conditions with severity ratings.
• Remediation guidance: 5 actionable recommendations.

Assessment: Thorough and well-structured. The evidence table is the standout feature — it forces the reviewer to systematically enumerate every component of a composed deployment, which is exactly the kind of rigor needed for artifact identity binding. The detection commands are practical and specific to real ML frameworks (PEFT, vLLM, TGI, GGUF, AWQ, GPTQ, ONNX, TensorRT).

3. Output Format — Composed Artifact Identity table (post Model Inventory)

A slimmed-down version of the evidence table is added to the skill's output template.

Assessment: Good consistency with the internal table above. Slightly redundant — but for a report template this is appropriate since it documents what the reviewer should produce.

4. Findings Category — "Composed Artifact Identity" added to category list

The - **Category:** line in the finding template now includes Composed Artifact Identity as an option alongside Provenance, Training Data, etc.

Assessment: Correct and necessary — without this, findings from Step 1A would have no category to map to.

5. Supply Chain Maturity Summary — new row added

"Composed artifact identity" added as a domain row.

Assessment: Consistent with the new section. Maintains the matrix structure.

6. Common Pitfalls — new entry #6

"Hashing only the base weights. For adapter-based or converted deployments, the base model digest is not the deployed model identity."

Assessment: Directly reinforces the theme of the PR. Concise and actionable.

7. References — 3 new links added

• Hugging Face Hub download revision docs
• Hugging Face Transformers PEFT adapter loading docs
• SLSA Provenance artifact digest fields spec

Assessment: Pertinent and from authoritative sources. Good additions.

---

Deeper Quality Assessment

Correctness

The technical claims are accurate:

• Tokenizer and chat template changes can alter model behavior without weight changes (e.g., different tokenization can cause different attention patterns; chat template injection can alter safety behavior).
• Quantization outputs are distinct deployable artifacts that need their own provenance (converter version, calibration data, output digest).
• Runtime adapter hot-loading (vLLM LoRA, TGI adapter routing) is a real production pattern that undermines static review.
• The severity ratings (High for unpinned components, Medium for unrecorded merge order) are appropriate.

Completeness

The section covers adapters, tokenization, quantization, runtime images, and runtime adapter registries. One possible addition:

Suggestion: Consider adding "prompt/system prompt configs" explicitly alongside chat templates in the detection commands. Some deployments load system prompts or guardrail configs from mutable external files that aren't captured by chat_template or prompt_template greps.

This is a minor gap — the section does already mention "prompt templates" in the detection description, so it's more about grep completeness than conceptual coverage.

Consistency with Existing Structure

The new Step 1A slots naturally between Step 1 (Model Provenance Verification) and Step 2 (Training Data Lineage). This is a logical ordering — you first establish the identity of all artifacts, then verify their provenance (Step 1), then check the composed identity (Step 1A), then move on to data lineage (Step 2).

The severity language, table style, and detection command format exactly match the existing convention. The codebase consistency is high.

Integration with Rest of Skill

The changes propagate correctly through:

• Context gathering table → new rows
• Step flow → new section inserted
• Output format → new report table
• Findings classification → new category
• Maturity summary → new domain row
• Common pitfalls → new numbered item
• References → expanded

This is a thorough cross-cutting update rather than a grafted-on section.

Edge Cases / Potential Issues

1. 12-column evidence table: The Evidence Table in Step 1A has 12 columns. In the output template, it's abbreviated to 11 columns (removing "Source authority" and combining slightly). This inconsistency could confuse reviewers who try to map between the two. Suggestion: Align the columns between the internal guidance table and the output template table.

2. "Confidence" column: The table includes a "Confidence" column (High/Medium/Low) but the findings table doesn't reference how confidence feeds into severity. Suggestion: Add a brief note explaining that low confidence in the composed identity should escalate finding severity (e.g., "If confidence is Low for any row, treat as a finding even if individual components are pinned — the overall identity is not provable").

3. Tokenizer regex: The detection grep for tokenizer is very broad (Grep: "tokenizer|chat_template|generation_config|prompt_template|special_tokens"). This might produce noise from library code or test fixtures. This is acceptable for a grep-based search but worth noting.

4. No mention of model registry tools (MLflow, DVC, Weights & Biases): These tools are commonly used to track model artifacts and their compositions. Adding a note about checking model registry manifests could strengthen the section. Suggestion: Add a detection line for mlflow|dvc|wandb|model.registry in model tracking configs.

---

Conclusion

This is a high-quality, production-ready change that addresses a genuine blind spot in model supply chain security: the gap between reviewing a single "model" artifact and reviewing the composed identity of the actual deployed system. The author demonstrates clear understanding of modern ML deployment patterns (PEFT adapters, vLLM hot-loading, quantization pipelines) and translates them into actionable review steps.

The code is well-structured, consistent with the existing skill, and properly integrated across all relevant sections of the document.

Recommendation: APPROVED with the minor suggestions above (column alignment, confidence explanation, MLflow/DVC note). None are blocking.

---

Files Created

• /private/tmp/review_186.md — this review file