Ideas for recovery if you accidentally delete a kubernetes resource

Version/Environment (if relevant):

The is relevant to Domino 4.x and above regarding objects/resources installed by the deployer/installer via Helm.

Issue:

A customer accidentally deleted a statefulset resource instead of the pod controlled by the statefulset.   Example of the accident:

kubectl -n <your namespace> delete sts hephaestus-buildkit

Root Cause:

The customer's command was accidentally issued due to miscommunication while troubleshooting a node level problem. Accidents happen, so theoretically other necessary kubernetes objects could be deleted, thus needing recovery.

Resolution:

Re-deploying or installing is one option, but you might have others too.

1. If you backed up your resources to yaml you can run:

kubectl apply -f /path/to/your/backup.yaml

2. Or you may be able to use Helm to "restore" the state of your cluster with just a few commands as long as you haven't accidentally deleted the actual helm release.  First check history for the helm "release" that added your deleted resource, in this case hephaestus 

helm -n <your namespace> history hephaestus

Second, choose the newest release number, like 6 in the example below

dom@xyz552 [/apps/domino] $ helm -n domino-platform history hephaestus
REVISION UPDATED STATUS CHART APP VERSION DESCRIPTION 
1 Tue Oct 18 21:43:49 2022 superseded hephaestus-0.1.20 0.1.20 Install complete 
2 Tue Oct 18 22:03:43 2022 failed hephaestus-0.1.20 0.1.20 Upgrade "hephaestus" failed: cannot patch "hephaestus-buildkit" with kind StatefulSet: StatefulSet.apps "hephaestus-buildkit" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden
3 Tue Oct 18 22:04:14 2022 superseded hephaestus-0.1.20 0.1.20 Rollback to 1 
4 Tue Oct 18 22:06:15 2022 superseded hephaestus-0.1.20 0.1.20 Upgrade complete 
5 Thu Mar 30 17:31:54 2023 superseded hephaestus-0.1.29 0.1.29 Upgrade complete 
6 Thu Mar 30 18:14:03 2023 deployed hephaestus-0.1.29 0.1.29 Upgrade complete                           

 and run

helm -n <your namespace> rollback hephaestus 6

kubectl -n <your namespace> get sts | grep hephaestus

Notes/Information:

It is possible that your organization has used a specific machine to perform the install/upgrade in the past where Helm might already be installed, but theoretically you should be able to install and use Helm anywhere you have kubectl access.  Install info for helm is at https://helm.sh/docs/intro/install/

References/Internal Records:

include any keys to related Jiras here (OTs, Bugs, FRs, etc.) 

• OT-2911