[REVIEW] aws-review: add organization-wide evidence scope

[mlodygbelmondo]

Skill Being Reviewed

Skill name: aws-review
Skill path: skills/cloud/aws-review/

False Positive Analysis

Benign single-account evidence that can be over-reported as an organization-wide failure:

resource "aws_cloudtrail" "member_local" {
  name                          = "member-local-trail"
  is_multi_region_trail         = true
  enable_log_file_validation    = true
  include_global_service_events = true
}

Why this is a false positive risk:

The skill evaluates AWS IaC, AWS CLI output, and configuration exports against CIS AWS Foundations v3.0.0, but the output does not require reviewers to record account, organization, OU, delegated administrator, and Region scope for each evidence source. In a multi-account AWS Organizations environment, a member-account trail, analyzer, or Security Hub export can be useful local evidence but insufficient for organization-wide conclusions. Conversely, an organization trail or centrally managed Security Hub policy may cover accounts where local IaC appears incomplete. Without scope fields, reviewers can either over-report missing controls in member accounts or overclaim organization coverage from a single account export.

Coverage Gaps

Missed variant 1: Organization trail coverage is not separated from member-account trails

Evidence reviewed:
  account_id: 111111111111
  cloudtrail: member-local-trail
  is_multi_region_trail: true
Missing:
  is_organization_trail
  delegated administrator account
  covered organization accounts/OUs
  excluded or suspended accounts

Why it should be caught:

AWS CloudTrail supports organization trails that log events for the management account and all member accounts in an AWS Organization. A pass/fail result for CIS logging controls should identify whether the evidence is account-local or organization-wide, which account owns the trail, and which Regions/accounts are covered.

Missed variant 2: IAM Access Analyzer scope is account-level when an organization analyzer is required

resource "aws_accessanalyzer_analyzer" "account" {
  analyzer_name = "account-zone"
  type          = "ACCOUNT"
}

Why it should be caught:

CIS 1.20 says Access Analyzer should be enabled for all Regions, but in a multi-account environment the review also needs to distinguish account analyzers from organization analyzers and delegated administrator configuration. An account analyzer in one workload account does not prove organization-level external access or unused-access coverage.

Missed variant 3: Security Hub central configuration and self-managed targets change coverage

Security Hub evidence:
  delegated_admin: security-tooling
  home_region: us-east-1
  linked_regions: eu-central-1, us-west-2
  root: centrally managed
  sandbox OU: self-managed

Why it should be caught:

AWS Security Hub CSPM central configuration can apply standards and controls to accounts and OUs across linked Regions, but targets can be centrally managed or self-managed. A Security Hub finding export from a delegated administrator is not automatically full-organization evidence unless the report records home Region, linked Regions, configuration policies, target associations, and self-managed exceptions.

Missed variant 4: SCP and delegated-admin boundaries can make source-level IAM findings misleading

{
  "Effect": "Allow",
  "Action": "*",
  "Resource": "*"
}

Why it should be caught:

A local IAM policy with broad permissions is still important, but effective risk depends on SCPs, permission boundaries, session policies, resource policies, and delegated admin boundaries. The skill should not mark a broad local allow as fully mitigated by an SCP without evidence, but it should record effective-access constraints so severity reflects the actual blast radius.

Edge Cases

• Organization-wide controls can be configured from management accounts or delegated administrator accounts; evidence should record which account supplied the data.
• Security Hub central configuration uses a home Region and linked Regions; opt-in Regions and self-managed accounts/OUs can create coverage gaps.
• AWS Config aggregators and Security Hub aggregations can be scoped to selected accounts/Regions; sampled or partial exports should not be treated as full coverage.
• SCPs do not grant permissions; they only set maximum permissions. Reviewers should not treat an SCP as proof that a control passes unless the underlying IAM/resource configuration is also understood.
• Suspended, newly created, and unjoined accounts can be omitted from organization-wide exports; reports need a coverage denominator.

Remediation Quality

• Fix resolves the vulnerability
• Fix doesn't introduce new security issues
• Fix doesn't break functionality
• Issues found: Add organization/account/OU/Region evidence scope fields, delegated administrator status, central-configuration coverage, and effective-access qualifiers. Add Not Evaluable from Single Account outcomes when the available evidence cannot prove organization-wide AWS posture.

Comparison to Other Tools

Tool / Framework	Catches this?	Notes
AWS Organizations / SCP review	Partial	Shows maximum permission boundaries and account hierarchy, but not the full resource posture by itself.
AWS CloudTrail organization trails	Yes for logging scope	Strong evidence for organization-wide event logging when account/Region coverage and delivery are verified.
IAM Access Analyzer organization analyzers	Partial	Helps identify external and unused access across an organization, but analyzer type, Region, and delegated admin state matter.
Security Hub CSPM central configuration	Partial	Strong posture-management evidence when home Region, linked Regions, configuration policies, and target associations are recorded.
AWS Config aggregators	Partial	Useful for resource/config coverage, but reports must record aggregator scope and excluded accounts/Regions.

Overall Assessment

Strengths:

• Clear CIS AWS Foundations v3.0.0 structure and practical findings template.
• Good coverage of IAM, S3, CloudTrail, monitoring, networking, and IMDSv2.
• Useful warning not to count non-evaluable controls as passing.
• Good common pitfalls around account-level versus bucket-level S3 public access blocks and multi-region CloudTrail.

Needs improvement:

• The output format should require evidence scope: management account, delegated admin account, member account, OU, Region, and organization coverage denominator.
• Organization trails, Security Hub central configuration, IAM Access Analyzer organization analyzers, AWS Config aggregators, and SCPs should be handled as first-class evidence sources.
• Findings should separate local account risk from organization-wide coverage claims.
• Add Not Evaluable from Single Account / Not Evaluable from IaC Only reason codes.

Priority recommendations:

1. Add an AWS Organization Coverage section with organization ID, management account, delegated administrators, accounts/OUs reviewed, Regions reviewed, excluded accounts, and evidence source date.
2. Add per-finding evidence fields: Evidence scope, Account/OU/Region, Delegated admin?, Organization-wide?, Coverage denominator, and Not Evaluable reason.
3. Add specific checks for CloudTrail organization trails, Security Hub CSPM central configuration, IAM Access Analyzer organization analyzers, AWS Config aggregators, and SCP/permission-boundary effective-access qualifiers.
4. Add severity guidance for local broad IAM allows constrained by SCPs: do not mark them safe without proof, but calibrate blast radius when effective-access evidence exists.

Sources Checked

• AWS CloudTrail organization trails: https://docs.aws.amazon.com/en_en/awscloudtrail/latest/userguide/creating-trail-organization.html
• IAM Access Analyzer delegated administrator: https://docs.aws.amazon.com/IAM/latest/UserGuide/access-analyzer-delegated-administrator.html
• IAM Access Analyzer overview: https://docs.aws.amazon.com/IAM/latest/UserGuide/what-is-access-analyzer.html
• AWS Security Hub CSPM central configuration: https://docs.aws.amazon.com/securityhub/latest/userguide/start-central-configuration.html
• Security Hub centrally managed vs self-managed targets: https://docs.aws.amazon.com/securityhub/latest/userguide/central-configuration-management-type.html
• AWS Organizations and CloudTrail integration: https://docs.aws.amazon.com/organizations/latest/userguide/services-that-can-integrate-cloudtrail.html

Bounty Info

• I have read and agree to the CONTRIBUTING.md bounty terms
• Preferred payment method: to be provided privately after acceptance

[LikeACloud7]

Opened follow-up improvement PR implementing this review: #201

Preferred payment method can be provided privately after acceptance.

[mlodygbelmondo]

Thanks for opening an implementation PR.

For clarity, this issue is my submitted Skill Review under the repository's review bounty terms. Payment details can be provided privately after maintainer acceptance, as noted in the issue body.

[JamesJi79]

Review: aws-review (#190) — Organization-Wide Evidence Scope

Skill: skills/cloud/aws-review/SKILL.md
Commit: 31eefd3 — "Add AWS organization evidence scope"
Version bumped: 1.0.0 → 1.1.0
Files changed: SKILL.md (+80/-11), benchmark-checklist.md (+61)
Bounty: $25

---

Summary of Changes

This PR adds a comprehensive organization-wide evidence scope layer to the AWS security posture review skill. It introduces a new Step 2: Establish Evidence Scope that instructs agents to record the organizational context before scoring any CIS controls, and updates the output format, discovery patterns, checklist items, common pitfalls, and references accordingly.

What was added

SKILL.md changes:

• New Step 2 requiring agents to capture Organization ID, management account, delegated administrators, accounts/OUs, regions, evidence source scope, CloudTrail type (account vs organization trail), Access Analyzer scope, Security Hub central config, Config aggregators, and SCP evidence.
• Discovery search patterns expanded to include organizations/, scp/, service-control-policies/, access-analyzer/ directories.
• Output format extended with evidence scope metadata (Organization ID, management account, delegated admins, accounts/OUs, regions, export date), two new "Not Evaluable" sub-statuses, a full "AWS Organization Coverage" evidence table, and per-finding evidence scope fields.
• Section scores table now includes a "Not Evaluable" column.
• Three new Common Pitfalls covering over-claiming org coverage, over-reporting local gaps, and SCP-as-grants misunderstanding.
• Five new reference links for organization trails, Access Analyzer delegated admin, Security Hub central config.

benchmark-checklist.md changes:

• SCP evidence recording guidance under CIS 1.7.
• Organization-scoped Access Analyzer (type = "ORGANIZATION") support under CIS 1.20.
• Organization trail (is_organization_trail = true) support under CIS 3.1.
• Security Hub central configuration Terraform resources (aws_securityhub_organization_admin_account, aws_securityhub_configuration_policy, aws_securityhub_configuration_policy_association) and evidence recording under CIS 4.16.

---

Strengths

1. Thorough org-wide treatment. The scope layer is comprehensive — it covers CloudTrail org trails, IAM Access Analyzer org-scoped analyzers, Security Hub central configuration, AWS Config aggregators, and SCPs. No major AWS Organizations service integration is missed.

2. Clear evidence boundaries. The distinction between "Not Evaluable from Single Account" and "Not Evaluable from IaC Only" prevents an agent from over-claiming compliance on organization-wide controls when only partial evidence is available. This is a real problem in AWS audits and the fix addresses it directly.

3. Good new Common Pitfalls. intel: appsec community updates 2026-03-29 #7 (over-claiming org coverage from one account), intel: ai-security model-supply-chain community updates 2026-03-29 #8 (over-reporting local gaps when central controls exist), and intel: appsec social updates (2026-04-08) #9 (SCPs as grants, not permissions) are accurate, actionable, and reflect real agent/auditor mistakes.

4. Correct SCP treatment. The PR is careful to note that "SCPs set maximum permissions; they do not grant access" — this is a common source of false findings in AWS reviews and the explicit guardrail is necessary.

5. Consistent structure. The step renumbering (2→3, 7→8) and output format extensions follow the pattern established by the sibling skills (azure-review, gcp-review) without creating structural drift.

---

Issues & Recommendations

Issue 1: Duplicate/nested evidence fields in Detailed Findings

The per-finding template now includes both:

• Evidence scope: Single account / Account+Region / OU / Organization-wide / Delegated admin / IaC only / Sampled
• Delegated admin?: <yes/no/service/account id>
• Organization-wide?: <yes/no/unknown>
• Effective-access qualifiers: <SCPs / permission boundaries / session policies / resource policies / none reviewed>
• Not Evaluable reason: <single account only / IaC only / missing delegated admin / missing Region denominator / missing account inventory / sampled export>

This is information-rich but risks overwhelming the output. Evidence scope already implies whether something is delegated-admin scoped or organization-wide. The separate Organization-wide? and Delegated admin? fields are redundant with the scope value.

Recommendation: Merge these fields. Keep a single Evidence scope line with a structured value like Organization-wide (delegated admin: 123456789012, org trail) and move the Not Evaluable reason inline into the status line: Status: Not Evaluable (single account only). This keeps findings concise while retaining all context.

Issue 2: Inconsistent capitalization in evidence scope values

The evidence scope values listed are mixed case:

• Single account (sentence case)
• Account+Region (title case)
• OU (abbreviation)
• Organization-wide (sentence case)
• Delegated admin (sentence case)
• IaC only (mixed abbreviation)

Recommendation: Use consistent lowercase for all values: single account, account+region, ou, organization-wide, delegated admin, iac only, sampled. This matters because agents parsing the output programmatically may case-match.

Issue 3: Step 2 is very dense — consider a structured checklist

Step 2 ("Establish Evidence Scope") is a single prose block listing ~12 items to capture. For a subagent executing this skill, it's easy to miss items.

Recommendation: Convert Step 2 into a bulleted checklist or table format matching the style of the benchmark-checklist.md. Example:

Capture:
- [ ] Organization ID and management account
- [ ] Delegated administrator accounts (service → account ID mapping)
- [ ] Accounts and OUs reviewed (include denominator)
- [ ] Regions reviewed (include opt-in/excluded)
- [ ] CloudTrail type: account trail / organization trail
- [ ] IAM Access Analyzer scope: ACCOUNT / ORGANIZATION
- [ ] Security Hub configuration: central / self-managed / not reviewed
- [ ] AWS Config aggregator scope: organization / account / not reviewed
- [ ] SCPs and effective access boundaries reviewed

Issue 4: No mention of CloudFormation StackSets or RAM

Organization-wide deployments often use CloudFormation StackSets (for multi-account resource provisioning) and AWS Resource Access Manager (for cross-account resource sharing). These can affect evidence scope — e.g., a StackSet may deploy a Config aggregator across all accounts, or RAM may share a KMS key organization-wide.

Recommendation: Add to Step 2's evidence capture: "CloudFormation StackSets and AWS Resource Access Manager shares reviewed (relevant for cross-account evidence scope)." Also add these to the discovery file patterns.

Issue 5: No mention of AWS Control Tower

Control Tower is a common landing-zone framework that enforces guardrails (SCPs), manages account factory, and centralizes logging. A reviewer encountering a Control Tower environment needs to understand that many CIS controls are managed at the Control Tower level rather than per-account.

Recommendation: Add a brief note: "If AWS Control Tower is detected (via aws_controltower_* resources or account factory configs), note that guardrails may enforce CIS-equivalent controls at the organization level."

Issue 6: "Not Evaluable from Single Account" sub-status risks inflating counts

The Executive Summary now has three separate "Not Evaluable" lines: general, from Single Account, and from IaC Only. A report could end up with 20+ controls as "Not Evaluable from Single Account" and only 5 actually evaluated. This makes the compliance percentage misleading.

Recommendation: Add a reporting guideline: "If more than 50% of controls are Not Evaluable, flag this in the Executive Summary as a data-gap warning and recommend a full organization-wide evidence collection before relying on the assessment."

Issue 7: Minor — "Delegated admin" column in SCP row of the AWS Organization Coverage table

The "AWS Organization Coverage" table has an SCP row where "Delegated Admin" is marked <n/a>. This is technically incorrect — for SCPs the management account is the relevant entity since only the management account can create/attach SCPs. Recommend changing to <management account> for clarity.

---

Overall Assessment

Criteria	Rating
Correctness	✓ Strong
Completeness	✓ Very good (minor gaps: StackSets, RAM, Control Tower)
Clarity	✓ Good, but Step 2 is dense
Consistency with sibling skills	✓ Consistent
False positive risk	✓ Reduced (new not-evaluable sub-statuses prevent over-claiming)
False negative risk	✓ Reduced (org trail/Access Analyzer/Security Hub central config now checked)

Status: PASS with minor suggestions

The PR is well-structured, technically accurate, and addresses a real gap — AWS organization-wide evidence is one of the most common areas where cloud security reviews produce unreliable results. The three-tier "Not Evaluable" system is a meaningful improvement over binary pass/fail for multi-account environments. The issues above are refinements, not blockers.

Priority Recommendations (for next iteration):

1. Format Step 2 as a checklist for easier subagent execution
2. Add StackSets and RAM to discovery patterns and evidence scope
3. Merge the redundant evidence scope fields in Detailed Findings for conciseness
4. Add Control Tower detection guidance for landing-zone environments

---

Review completed against commit 31eefd3. All 2 files reviewed, all changes analyzed.