KubernetesConfigurationApplicationFailed error in the User Output logs when starting up a workspace or job

Version/Environment (if relevant):

Domino versions < 5.2.0

Issue:

This error appears in the User Output logs when either starting up a job or a workspace. The error appears even though the execution is successful and the job eventually runs or the workspace eventually starts up.

Failure in processing event on attempt 1 out of 1
domino.server.dispatcher.infrastructure.KubernetesConfigurationApplicationFailed: 
Applying Kubernetes configuration failed with status code 1: 
configmap/run-63961aa56991520b31292cd4 unchanged
networkpolicy.networking.k8s.io/run-63961aa56991520b31292cd4 unchanged
The Job "run-63961aa56991520b31292cd4" is invalid: spec.template: 
Invalid value: core.PodTemplateSpec

Root Cause:

This error occurs when something prevents the execution from starting on its first attempt. Then on the retry, you see the failure to rewrite the configmap.

Domino re-applies the same resource on every heartbeat until the workload is registered with Kubernetes. We do that in case the Kubernetes API server somehow fails to receive the Kubernetes API request for the initial kubectl apply command. However, if the re-generated resource yaml changes the PodTemplateSpec, then the kubectl apply command will fail as evident in the field is immutable error in the User Output logs. The execution does eventually succeed despite the errors. 

Overhead:core.ResourceList(nil), EnableServiceLinks:(*bool)(nil), 
TopologySpreadConstraints:[]core.TopologySpreadConstraint(nil)}}:
 field is immutable

Resolution:

This is a known issue that is resolved in Domino version 5.2.0. For reference: DOM-31875.

Notes/Information:

A similar error KubernetesConfigurationApplicationFailed can occur for workspaces with Spark clusters attached fail to launch. Please see: Workspaces with Spark clusters suddenly fail during assigning, no endpoints available for service "distributed-compute-operator-webhook-server"