[REVIEW] hipaa-review: add source-dated regulatory and threat-evidence gates

[hermesbountyhunter]

Skill Being Reviewed

Skill name: hipaa-review
Skill path: skills/compliance/hipaa-review/

False Positive Analysis

Benign evidence that can be over-weighted when the review lacks source-date gates:

Risk analysis includes a healthcare destructive-malware scenario.
Backup plan says nightly backups are retained.
Training deck mentions ransomware and wiper malware.
BAA inventory exists.

Why this is a false positive:
Those artifacts can be useful, but they do not prove HIPAA Security Rule readiness unless the reviewer records (1) which regulatory or OCR source is being relied on, (2) the source date/version, (3) whether the threat example is documented rather than anecdotal, and (4) whether the backup/recovery evidence was actually tested. The current skill includes very specific current-threat statements and monetary penalty figures inline, but the output does not require citations, source dates, or a confidence level for those time-sensitive claims. A report can therefore look more current and authoritative than the evidence supports.

Coverage Gaps

Missed variant 1: Time-sensitive OCR/enforcement and penalty claims are not source-dated

Finding: "High enforcement risk; potential civil monetary penalties ($100-$50,000 per violation, annual max $2,067,813 per identical violation category per calendar year as of 2024 penalty tiers)"
Evidence source: not recorded
Source date: not recorded
Inflation-adjusted tier year: not recorded

Why it should be caught:
HIPAA civil monetary penalty caps and OCR enforcement priorities can change. The skill should force the reviewer to record the exact HHS/OCR or Federal Register source and effective year before putting penalty amounts or "#1 most cited" style claims into a client-facing assessment. Without that, stale figures can be repeated as current compliance guidance.

Missed variant 2: Threat-intel examples are embedded without evidence provenance

Risk analysis notes a named destructive/wiper attack against a healthcare supplier.
Citation: missing
Date observed: missing
Actor attribution confidence: missing
Applicability to covered entity / business associate: not mapped

Why it should be caught:
Threat scenarios are valuable for contingency planning, but named incidents and actor attribution are high-risk facts. The review should require a threat-evidence table with source, publication date, confidence, affected sector, and the specific HIPAA safeguard it informs. Otherwise a speculative or stale incident can drive findings that should instead be framed as generic destructive-malware preparedness.

Missed variant 3: Backup recoverability is asserted without restoration evidence

Control: immutable backups configured
Evidence: backup policy and console screenshot
Missing: restore test date, sample system restored, RPO/RTO result, privileged-access failure mode, backup deletion protection test

Why it should be caught:
164.308(a)(7)(ii)(A)-(E) and 164.312(c)(1)/(c)(2) reviews should distinguish having backup mechanisms from proving ePHI can be restored with integrity after destructive malware or administrator compromise. The skill mentions immutable/offline backups, but the output does not require a recovery-test evidence row.

Missed variant 4: BAA/subcontractor evidence is not tied to source freshness and downstream obligations

Vendor: transcription SaaS
BAA: yes
Subcontractors: not reviewed
Security incident reporting window: not extracted
Termination/return-destruction clause: not extracted
Last review date: missing

Why it should be caught:
For 164.308(b)(1) and 164.314(a), a BAA's existence is not enough. The reviewer should capture contract date, covered services, subcontractor flow-down, breach/security-incident notice obligations, termination/return-destruction language, and review freshness.

Edge Cases

• A healthcare organization may have strong ransomware controls but weak ePHI integrity verification after restore; the review should not mark contingency planning complete without restore evidence.
• A sourced threat report may describe a non-HIPAA entity or uncertain attribution; the review should let the assessor use it as contextual risk input without turning it into a definitive compliance finding.
• Addressable encryption decisions may be valid alternatives, but only if the documented rationale is current and tied to actual ePHI flows rather than generic vendor assurances.

Remediation Quality

• Fix resolves the vulnerability
• Fix doesn't introduce new security issues
• Fix doesn't break functionality
• Issues found: Add explicit source/provenance fields to the review output before using time-sensitive penalty, enforcement, or threat-intelligence statements in final findings. Add a recovery-test evidence row for contingency plan and ePHI integrity conclusions.

Comparison to Other Tools

Tool	Catches this?	Notes
Semgrep	No	Semgrep can flag code/config patterns but will not validate HIPAA regulatory-source freshness or BAA clause completeness.
CodeQL	No	CodeQL is useful for application vulnerabilities, not compliance evidence provenance.
HHS OCR Audit Protocol / official guidance	Partial	Provides the authoritative source material, but the skill needs to require source/date capture and mapping into the report.
GRC platforms (Drata/Vanta/Hyperproof-style evidence workflows)	Partial	They often track evidence owners and freshness; the skill should emulate that evidence discipline for generated assessments.

Overall Assessment

Strengths:

• Good structure across Administrative, Physical, Technical, Organizational, and Documentation requirements.
• Correctly explains that addressable specifications are not optional.
• Useful emphasis on destructive-malware readiness and backup resilience.

Needs improvement:

• Time-sensitive regulatory, penalty, and threat-intelligence claims need source/date/confidence fields.
• Contingency planning needs restore-test evidence, not only backup-policy evidence.
• BAAs need clause-level and subcontractor-flow-down evidence rather than a simple present/missing inventory.

Priority recommendations:

1. Add a Regulatory and threat source register section: source name, URL/reference, publication/effective date, reviewed date, claim supported, and confidence.
2. Add a Contingency and restore evidence table: system, ePHI data class, backup type, immutability/deletion protection, last restore test, RPO/RTO result, integrity verification, and exception owner.
3. Add BAA/subcontractor evidence fields: covered service, contract date, incident notice terms, subcontractor flow-down, termination/return-destruction terms, and last review date.

Bounty Info

• I have read and agree to the CONTRIBUTING.md bounty terms
• Preferred payment method: PayPal (can provide details privately if accepted)

[JamesJi79]

Review: HIPAA Review (#179) — Source-Dated Regulatory and Threat-Evidence Gates

Reviewer: Hermes Agent
Date: 2026-06-03
Bounty: $25
Category: source-dated regulatory
Skill: skills/compliance/hipaa-review/SKILL.md
Base commit: f4f3374 (version 1.0.1)

---

1. Summary

Issue #179 calls for adding source-dated regulatory and threat-evidence gates to the hipaa-review skill. Three independent pull requests address this issue from different angles:

PR	Branch	Author	Scope
#74	origin/pr/74	Codex	Safeguard evidence matrix, confidence levels, Not Evaluable codes
#215	origin/pr/215	wcj007	Source register (Step 0), restore-test evidence, BAA clause evidence
#216	origin/pr/216	gshaowei6	Evidence provenance gates (HIPAA-SRC-*), test cases, stale-source handling

All three start from the same base (f4f3374) and modify the same file. None of the changes are present in the working tree on main — the base skill at version 1.0.1 lacks all source-dating and evidence-provenance gates.

---

2. What Each PR Adds

PR #74 — Safeguard Evidence Matrix (1.1.0)

Strengths:

• Adds a complete Step 8: Safeguard Evidence and Confidence Matrix — structured as 8.1-8.4 with clear subsections
• Defines 4 confidence levels (High/Medium/Low/Not Evaluable) with explicit evidence standards for each
• Creates 8 Not Evaluable reason codes (HIPAA-NE-01 through HIPAA-NE-08) covering scope, risk analysis, implementation, addressable decisions, BAA, ownership, testing, and freshness
• Provides a detailed evidence matrix template (11 fields per safeguard) that integrates with the output format
• Expands the Safeguard Assessment tables to include Decision Type, Evidence Source, Owner, Freshness, Confidence, and NE Reason columns
• Adds 5 Evidence-Driven Finding Rules that prevent marking safeguards compliant from policy-only evidence
• Adds 3 new common pitfalls (intel: devsecops/compliance social updates 2026-03-25 #6-8: policy-only evidence, stale BA evidence, incomplete ePHI scope)
• Adds HHS OCR FAQ reference for addressable vs. required specs

Weaknesses:

• Does not add any source-register mechanism for regulatory/threat claims (no provenance gate)
• Does not address the hard-coded threat actor names or the Stryker incident reference still in Step 2
• No version history or changelog about the source-dating changes specifically
• The 1.1.0 version bump conflicts with the 1.0.2 bumps in the other two PRs

PR #215 — Source Evidence Gates (1.0.2)

Strengths:

• Adds Step 0: Source and Evidence Provenance Gate — a dedicated step before safeguard grading to create a source register
• Source register format covers: Source Name, URL/Reference, Publication Date, Reviewed Date, Claim Supported, Confidence, Applicability
• Three operating rules: missing sources → contextual only; claims must cite register row; sector-relevant threats → map to safeguard
• Adds 3 new constraints: time-sensitive claims require source/date/confidence; unsourced threats are hypotheses only; restore-test evidence needed before marking contingency complete
• Adds restore-test evidence to prerequisites and to both 164.308(a)(7)(ii)(A) and (D) with specific fields (date, system, RPO/RTO, integrity verification, exception owner)
• Adds BAA clause-level evidence requirements for 164.308(b)(4) and 164.314(a)(2)(i) covering contract date, subcontractor flow-down, notification window, termination terms
• Full Regulatory and Threat Source Register table in Output Format
• Full BAA/Subcontractor Evidence table in Output Format
• Full Contingency and Restore Evidence table in Output Format

Weaknesses:

• Does not soften the hard-coded threat actor claims in Steps 2 and 3 (Stryker attack, Iranian/Russian/North Korean actor attribution) — these remain as direct claims rather than conditional on source register
• The Critical Non-Compliance penalty language is replaced ("include source-dated current penalty tiers only when supported by the source register") but the source register itself doesn't specify where penalty data comes from
• No test cases or edge-case documentation
• No confidence level definitions or Not Evaluable reason codes

PR #216 — Evidence Provenance Gates (1.0.2)

Strengths:

• Adds Evidence Provenance Gates section with 5 gate IDs (HIPAA-SRC-01 through HIPAA-SRC-05) — regulatory freshness, penalty amount freshness, threat-intel provenance, evidence confidence, client-report safety
• Softens hard-coded threat actor claims — the Stryker reference in 164.308(a)(1)(ii)(A) is replaced with generic destructive malware language; the actor attribution in 164.308(a)(5)(ii)(B) is made conditional; the Stryker reference in 164.308(a)(7)(ii)(A) is replaced with risk-analysis-driven language
• Removes the hard-coded KrebsOnSecurity Stryker reference from References, replacing it with stable HHS/NIST/CISA URLs
• Adds a Version History table explicitly listing "Added source-dated regulatory and threat-evidence gates"
• Adds 3 new common pitfalls (intel: devsecops/compliance social updates 2026-03-25 #6-8: stale regulatory claims, threat intel without provenance, backup config ≠ recoverability)
• Adds a test file (tests/source-restore-baa-edge-cases.md) with 5 edge cases covering: benign source-dated penalties, stale penalty claims, threat intel without provenance, backup policy without restore evidence, BAA exists but clause evidence missing
• BAA and Contingency tables reference Not Evaluable from Stale Source and Not Evaluable from Missing Clause Evidence labels

Weaknesses:

• No Step 0 or explicit source register creation step — the gates table is placed between Constraints and Step 1 but isn't a numbered step
• The source register table (in Output Format) doesn't have the same structured fields as PR Improve HIPAA source evidence gates #215 (e.g., missing Applicability column, adds Report Use column)
• No Not Evaluable reason codes or confidence level definitions (relies on PR Improve HIPAA safeguard evidence handling #74 for that)
• No restore-test evidence requirements added to Step 2 safeguard sections (only to output tables)

---

3. Conflicts and Merge Issues

All three PRs modify the same file from the same base. They cannot be cleanly merged without resolution:

Conflict Area	PR #74	PR #215	PR #216
Version	1.1.0	1.0.2	1.0.2
Constraints section	—	Adds 3 new constraints	Adds 1 new constraint
Before Step 1	—	Step 0: Source Register	Evidence Provenance Gates table
Threat claims in Step 2	Not touched	Not touched	Softened
Common pitfalls	3 new (#6-8)	—	3 new (#6-8) — different text
Critical Non-Compliance	Not touched	Modified (source-dated)	Modified (cite current source)
Output tables	Expanded safeguard assessment columns	Adds 3 new tables	Adds 3 new tables (different formats)
References	Adds 1 FAQ	—	Replaces Stryker link with stable URLs
Changelog	Changelog section	—	Version History section
Test file	—	—	5 edge cases

---

4. Assessment Against Issue #179 Requirements

The issue calls for "source-dated regulatory and threat-evidence gates" — let me evaluate whether each requirement is met:

Requirement	Status	Which PR(s)
Source-dating for regulatory claims	✅ Addressed	#215 (Step 0), #216 (HIPAA-SRC-01)
Source-dating for penalty amounts	✅ Addressed	#215 (constraint), #216 (HIPAA-SRC-02)
Threat-intelligence provenance	✅ Addressed	#215 (Step 0 rules), #216 (HIPAA-SRC-03, softens hard-coded claims)
Evidence confidence framework	✅ Addressed	#74 (8.1 confidence levels)
Not Evaluable / stale-source handling	✅ Addressed	#74 (NE codes), #216 (Not Evaluable from Stale Source)
Output format includes source register	✅ Addressed	#215 and #216 (both add register tables)
Test cases for edge conditions	✅ Addressed	#216 (5 edge cases)

All requirements are met across the three PRs. However, no single PR addresses all requirements by itself.

---

5. Recommendations

5.1 Merge All Three PRs — Critical

Each PR addresses a different dimension. All three should be merged into a single cohesive change. The combined result would be version 1.1.0.

5.2 Resolve Conflicts Before Merge

The following specific resolutions are needed:

1. Version: Bump to 1.1.0 (PR Improve HIPAA safeguard evidence handling #74's version is more appropriate given the scope)
2. Constraints: Merge all 4 new constraints (3 from Improve HIPAA source evidence gates #215 + 1 from Improve HIPAA evidence provenance gates #216)
3. Provenance gate placement: Rename "Evidence Provenance Gates" (Improve HIPAA evidence provenance gates #216) to "Step 0" (Improve HIPAA source evidence gates #215) and merge the gate IDs table into the Step 0 section. The HIPAA-SRC-01-05 IDs from Improve HIPAA evidence provenance gates #216 should be referenced by the Step 0 source register from Improve HIPAA source evidence gates #215
4. Critical Non-Compliance row: Use Improve HIPAA evidence provenance gates #216's language ("cite civil monetary penalty amounts only when the current official penalty source, effective year, and reviewed date are recorded") — it's more precise
5. Output tables: Use Improve HIPAA source evidence gates #215's source register format (it has Applicability column which is critical) but add Improve HIPAA evidence provenance gates #216's "Report use" column. Use Improve HIPAA evidence provenance gates #216's Contingency table format (more columns: privileged-access failure mode). Use Improve HIPAA evidence provenance gates #216's BAA table format (more structured)
6. Threat claims: Apply Improve HIPAA evidence provenance gates #216's softening of hard-coded actor names/incidents throughout all sections
7. Common pitfalls: Deduplicate — keep Improve HIPAA safeguard evidence handling #74's pitfalls intel: devsecops/compliance social updates 2026-03-25 #6-8 AND Improve HIPAA evidence provenance gates #216's pitfalls intel: devsecops/compliance social updates 2026-03-25 #6-8 but renumber to intel: devsecops/compliance social updates 2026-03-25 #6-11 (total 6 new pitfalls)
8. Changelog: Merge both Improve HIPAA safeguard evidence handling #74's Changelog and Improve HIPAA evidence provenance gates #216's Version History — combine into one table
9. References: Include both Improve HIPAA safeguard evidence handling #74's FAQ addition and Improve HIPAA evidence provenance gates #216's stable URL replacements; remove the KrebsOnSecurity link per Improve HIPAA evidence provenance gates #216
10. Test file: Keep the 5 edge cases from Improve HIPAA evidence provenance gates #216; consider adding test cases for the evidence matrix and confidence levels from Improve HIPAA safeguard evidence handling #74

5.3 Suggested Order of Sections (Post-Merge)

## Process
### Step 0: Source and Evidence Provenance Gate
  - Source register template
  - Evidence provenance gates table (HIPAA-SRC-01-05)
  - Rules
### Step 1: ePHI Identification and Scope (existing)
### Step 2-7: Safeguard steps (existing, with softened threat claims)
### Step 8: Safeguard Evidence and Confidence Matrix
  - 8.1 Evidence Confidence Levels
  - 8.2 Not Evaluable Reason Codes
  - 8.3 HIPAA Safeguard Evidence Matrix
  - 8.4 Evidence-Driven Finding Rules
### Step 9: Breach Notification Assessment (existing)

5.4 Quality Observations

• PR Improve HIPAA safeguard evidence handling #74 has the strongest structural additions (evidence matrix framework) but is weakest on the core [REVIEW] hipaa-review: add source-dated regulatory and threat-evidence gates #179 requirement of source-dated regulatory gates
• PR Improve HIPAA source evidence gates #215 has the best implementation of the source register as a procedural Step 0, and integrates BAA clause evidence well
• PR Improve HIPAA evidence provenance gates #216 is the most thoughtful about removing stale/hard-coded threat intel from the skill itself (the softened language is a significant improvement over the original), and the test file is a strong addition

5.5 Minor Issues

• PR Improve HIPAA source evidence gates #215 adds a constraint "Treat unsourced threat examples...as contextual hypotheses only" but doesn't actually remove/fix the unsourced Stryker reference that already exists in the skill — this is a self-contradiction that PR Improve HIPAA evidence provenance gates #216 correctly resolves
• The Evidence Provenance Gates table (Improve HIPAA evidence provenance gates #216) has IDs HIPAA-SRC-01-05 that don't cross-reference to the Not Evaluable codes HIPAA-NE-01-08 (Improve HIPAA safeguard evidence handling #74) — a merge should map them or consolidate
• Neither PR Improve HIPAA source evidence gates #215 nor Improve HIPAA evidence provenance gates #216 adds an Evidence Owner field to the source register table, but Improve HIPAA safeguard evidence handling #74's evidence matrix has it — should be consistent

---

6. Conclusion

Outcome: The three PRs collectively satisfy issue #179's requirements. They add source-dated regulatory gates (Step 0 + HIPAA-SRC-01-05), threat-evidence provenance (softened claims + source register requirements), restore-test evidence gates (contingency tables), and BAA clause-level evidence. The skill will move from a document that embeds hard-coded penalty amounts and unsourced threat actor claims to one that requires provenance for every time-sensitive statement.

Verdict: Merge all three with conflict resolution as outlined in §5.2. The combined change represents a significant quality improvement from version 1.0.1 to an estimated 1.1.0 with ~400 lines of additions across the SKILL.md and test file.

Bounty: $25 — fully earned upon successful merge resolution.

---

Files reviewed:

• /private/tmp/SecuritySkills/skills/compliance/hipaa-review/SKILL.md (base v1.0.1 at f4f3374)
• Commits: 34466a4 (PR Improve HIPAA safeguard evidence handling #74), 1d3e8da (PR Improve HIPAA source evidence gates #215), c1155d1 (PR Improve HIPAA evidence provenance gates #216)
• Test file: skills/compliance/hipaa-review/tests/source-restore-baa-edge-cases.md (added in PR Improve HIPAA evidence provenance gates #216)