Cluster Status not healthy, DataPlane Agent error "invalid secret id"

Version/Environment (if relevant):

5.4 and beyond due to introduction of Dataplane Agent. 

Issue:

Nucleus services were manually restarted by an Admin, but ClusterStatus page is NOT listing healthy, and the Dataplane Agent crashloops with errors in its pod logs:

2023/01/24 14:48:43.599 INFO  [main] d.d.a.api.utils.BootstrapHelpers$ - logging in to vault via AppRole
2023/01/24 14:48:43.630 ERROR [main] d.d.a.api.utils.HttpRequestUtils$ - status code: 400, response body: {"errors":["invalid secret id"]}

2023/01/24 14:48:43.641 INFO  [main] d.d.agent.server.messaging.Agent$ - Failed to create injector for messaging modules. {}
com.google.inject.CreationException: Unable to create injector, see the following errors:

1) Error in custom provider, domino.dataplane.agent.api.utils.HttpException: {"errors":["invalid secret id"]}

  at domino.dataplane.agent.server.vault.VaultModule.providesVaultProvider(VaultModule.scala:31)
  while locating domino.dataplane.agent.server.vault.VaultProvider
    for the 1st parameter of domino.dataplane.agent.server.vault.VaultModule.providesRabbitMqCredentialsProvider(VaultModule.scala:62)
  at domino.dataplane.agent.server.vault.VaultModule.providesRabbitMqCredentialsProvider(VaultModule.scala:62)
  while locating domino.dataplane.agent.server.vault.RabbitMqCredentialsProvider
    for the 2nd parameter of domino.dataplane.agent.server.messaging.DataPlaneMessagingServerModule.providesConnectionEnvelope(DataPlaneMessagingServerModule.scala:104)
  at domino.dataplane.agent.server.messaging.DataPlaneMessagingServerModule.providesConnectionEnvelope(DataPlaneMessagingServerModule.scala:104)
  while locating domino.messaging.DominoMessagingConnectorEnvelope
    for the 1st parameter of domino.dataplane.agent.server.messaging.DataPlaneMessagingServerModule.providesMessagingConnector(DataPlaneMessagingServerModule.scala:41)
  at domino.dataplane.agent.server.messaging.DataPlaneMessagingServerModule.providesMessagingConnector(DataPlaneMessagingServerModule.scala:41)
  while locating domino.messaging.DominoMessagingConnector
    for the 1st parameter of domino.dataplane.agent.server.messaging.DataPlaneMessagingServerProvider.<init>(DataPlaneMessagingServerModule.scala:194)
  while locating domino.dataplane.agent.server.messaging.DataPlaneMessagingServerProvider
  at domino.dataplane.agent.server.messaging.DataPlaneMessagingServerModule.configure(DataPlaneMessagingServerModule.scala:32)
  while locating domino.dataplane.agent.server.messaging.DataPlaneMessagingServer

1 error
 at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:554)
 at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:188)
 at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:111)
 at com.google.inject.Guice.createInjector(Guice.java:87)
 at com.google.inject.Guice.createInjector(Guice.java:69)
 at domino.dataplane.agent.server.messaging.Agent$.main(Agent.scala:57)
 at domino.dataplane.agent.server.messaging.Agent.main(Agent.scala)
Caused by: domino.dataplane.agent.api.utils.HttpException: {"errors":["invalid secret id"]}

Root Cause:

There is a complex renewal and rotation mechanism for credentials involving Vault for communication between Nucleus and Dataplane Agent.  If there is a long period of time in which Nucleus and Dataplane Agent are not able to communicate with each other (like downtime, etc) then the Agent gets into an unrecoverable state.

Resolution:

Local only deploys:

Notes/Information: