Aggregator duty failures

[KaloyanTanev]

🐞 Bug Report

Description

We've observed increased amount of aggregator duty failures.

Has this worked before in a previous version?

Yes, pre-electra for sure. Since electra we've introduced some changes in how we match aggregation data which is likely causing the issue.

No.

🔬 Minimal Reproduction

Most of our canaries are failing some or all aggregations.

Client teams issues tracking

• Lodestar
  Issue: Update vc to submit beacon committee selections once per epoch ChainSafe/lodestar#8606
  PR: feat: update vc to submit beacon committee selections once per epoch ChainSafe/lodestar#8669
  Released: v1.39.0
• Lighthouse
  Issue: Update VC to submit beacon committee selections once per epoch sigp/lighthouse#8381
  PR: Call beacon_committee_selections only once per epoch sigp/lighthouse#8699
  Released: v8.1.0
• Teku
• Nimbus
  Issue: Update VC to submit beacon committee selections once per epoch status-im/nimbus-eth2#7718
  PR:
  Released:
• Prysm
• Vouch - N/A

[KaloyanTanev]

After some research, there were disparities on when VCs call POST /eth/v1/validator/beacon_committee_selections endpoint of Charon:

Teku        per epoch
Nimbus      per slot
Lodestar    per slot
Lighthouse  per slot
Prysm       per epoch
Vouch       not implemented

This caused the requests that the VCs calling the endpoint per epoch to hang, because Charon cannot yet aggregate signatures for all validators they ask.

To furthermore illustrate the issue, here is an example.
4-node Charon cluster, 2 Charons use Teku VC, 2 Charons use Nimbus VC. The cluster manages 64 validators.

1. At slot 0 of the epoch Teku sends request for beacon selection committees for 64 validators.
2. At slot 0 of the epoch Nimbus sends request for beacon selection committees for ~2 validators.
3. Charon reaches threshold for those 2 validators and returns successful response to Nimbus. Nimbus proceeds with successful aggregation partial signature further down the slot time.
4. Charon does not reach threshold for the other 62 validators and the response to Teku hangs.
5. Charon reaches threshold for all validators only at the last slot 31, as that is when Nimbus sends its last participation. By that time though many timeouts have elapsed. Because of that, Teku never participated in aggregations, as it never received response from the beacon committee selections call.