Emit audit.app.process.ready and audit.app.process.not-ready events

[ameowlia]

Issue

We want cc to emit audit.app.process.ready and audit.app.process.not-ready events as an app passes and fails the readiness health check.

Context

We want to make it easier for app devs to know if their apps are "ready" or "not-ready" as determined by the readiness health check.

Related PRs

• TPS - Watch for readiness health check changes tps#49
• RuntimeSchema - add AppReadinessChangedRequest cc message runtimeschema#19
• Cloud controller - Add process readiness changed controller and event cloud_controller_ng#3554
• Running CF Audit Events List Docs - add readiness events to audit events list docs-running-cf#116
• CATs - update readiness health check test to test audit events cf-acceptance-tests#997
• CAPI - I did not submit a PR to capi release to bump all of these things. Let me know if I should. I thought it might be easier for whoever merges all of these things to do it then.

Steps to Test

1. Deploy the code from the PRs.
2. Make sure you have at least Diego 2.81.0 deployed.
3. Clone the code for the proxy app. It has good endpoints for testing health checks, including /eventuallyfail, /eventuallysucceed, and /flap.
4. Update the proxy manifest to have a readiness health check for the /flap endpoint (see diff below).
5. Push the proxy app
6. Run watch cf events APP and see the desired output.

diff --git a/src/example-apps/proxy/manifest.yml b/src/example-apps/proxy/manifest.yml
index bba43604..f3538084 100644
--- a/src/example-apps/proxy/manifest.yml
+++ b/src/example-apps/proxy/manifest.yml
@@ -4,5 +4,8 @@ applications:
     memory: 32M
     disk_quota: 32M
     buildpacks: [go_buildpack]
+    readiness-health-check-type: http
+    readiness-health-check-http-endpoint: /flap
+    readiness-health-check-interval: 10
     env:
       GOPACKAGENAME: proxy

Desired output

Every 2.0s: cf events potato                                                                                                                                                                              mammatus: Thu Dec 14 17:18:17 2023

Getting events for app potato in org system / space system as admin...
time                          event                         actor   description
2023-12-14T17:18:16.00+0000   audit.app.process.not-ready   web     index: 0, cell_id: c67943b4-79a1-450f-92d3-8e6ac141d4a0, instance: dc279be6-bc23-4a16-7b32-f83f
2023-12-14T17:17:36.00+0000   audit.app.process.ready       web     index: 0, cell_id: c67943b4-79a1-450f-92d3-8e6ac141d4a0, instance: dc279be6-bc23-4a16-7b32-f83f
2023-12-14T17:17:27.00+0000   audit.app.process.not-ready   web     index: 0, cell_id: c67943b4-79a1-450f-92d3-8e6ac141d4a0, instance: dc279be6-bc23-4a16-7b32-f83f
2023-12-14T17:16:47.00+0000   audit.app.process.ready       web     index: 0, cell_id: c67943b4-79a1-450f-92d3-8e6ac141d4a0, instance: dc279be6-bc23-4a16-7b32-f83f
2023-12-14T17:16:39.00+0000   audit.app.process.not-ready   web     index: 0, cell_id: c67943b4-79a1-450f-92d3-8e6ac141d4a0, instance: dc279be6-bc23-4a16-7b32-f83f

### Tasks
- [x] Docs
- [x] CATS
- [ ] CATS fails for windows. make this work

[ameowlia]

Example output...

Every 2.0s: cf events potato                                                                                                                                                                                                                mammatus: Thu Jan 11 16:58:57 2024

Getting events for app potato in org system / space system as admin...
time                          event                         actor   description
2024-01-11T16:58:45.00+0000   audit.app.process.ready       web     index: 1, cell_id: 8be86a8d-a448-440f-9c0b-7903bbf859ce, instance: 55d1a9c7-9ddf-4cb0-7334-3ff6
2024-01-11T16:58:39.00+0000   audit.app.process.not-ready   web     index: 1, cell_id: 8be86a8d-a448-440f-9c0b-7903bbf859ce, instance: 55d1a9c7-9ddf-4cb0-7334-3ff6
2024-01-11T16:58:15.00+0000   audit.app.process.ready       web     index: 0, cell_id: 39f91c3a-0233-48c5-911e-830a406a14d8, instance: 303cf47f-7872-4578-6d91-ba03
2024-01-11T16:58:09.00+0000   audit.app.process.not-ready   web     index: 0, cell_id: 39f91c3a-0233-48c5-911e-830a406a14d8, instance: 303cf47f-7872-4578-6d91-ba03
2024-01-11T16:57:08.00+0000   audit.app.process.ready       web     index: 1, cell_id: 8be86a8d-a448-440f-9c0b-7903bbf859ce, instance: 55d1a9c7-9ddf-4cb0-7334-3ff6
2024-01-11T16:57:04.00+0000   audit.app.process.scale       admin   instances: 2
2024-01-11T16:56:39.00+0000   audit.app.process.ready       web     index: 0, cell_id: 39f91c3a-0233-48c5-911e-830a406a14d8, instance: 303cf47f-7872-4578-6d91-ba03
2024-01-11T16:56:33.00+0000   audit.app.droplet.mapped      admin
2024-01-11T16:56:33.00+0000   audit.app.restart             admin
2024-01-11T16:56:33.00+0000   audit.app.revision.create     admin
2024-01-11T16:56:27.00+0000   audit.app.droplet.create      admin
2024-01-11T16:55:35.00+0000   audit.app.build.create        admin
2024-01-11T16:55:32.00+0000   audit.app.package.upload      admin
2024-01-11T16:55:31.00+0000   audit.app.package.create      admin
2024-01-11T16:55:29.00+0000   audit.app.update              admin
2024-01-11T16:55:29.00+0000   audit.app.process.scale       admin
2024-01-11T16:55:29.00+0000   audit.app.update              admin
2024-01-11T16:55:29.00+0000   audit.app.process.update      admin
2024-01-11T16:55:29.00+0000   audit.app.map-route           admin
2024-01-11T16:55:24.00+0000   audit.app.process.create      admin
2024-01-11T16:55:24.00+0000   audit.app.create              admin
2024-01-11T16:55:24.00+0000   audit.app.apply_manifest      admin

[Samze]

Just stumbled upon this, looks like cloudfoundry/cloud_controller_ng#3554 was merged and has been in a capi release for awhile 77bf902 . @ameowlia anything left to do here or can we close this issue?

[Samze]

Closing as this work is done

[sethboyles]

Reopening this because I want to double check audit events are the proper way to track ready and not_ready events. I thought they were intended to track attempts on actions, but from looking at the code it seems they are usually only recorded when an action is successful (they are usually created in the main transactions of update actions).

I'm looking at the audit.app.process.crash, audit.app.process.ready, and audit.app.process.not_ready events, which don't really feel like they align with the stated intent of audit events in our docs:

Audit events help Cloud Foundry operators monitor actions taken against resources (such as apps) via user or system actions.

Crash events are tracked both in Audit Events and Usage Events, which further confused the issue for me. But ready and not_ready seem more suited to Usage Events to me.

On the other hand, having a singular way to list events, rather than trying to stitch together the "who/why" from audit events and the "what" from usage events, seems more sensible from a usability standpoint...