Model API Build issues, builder node problems, replicatorStorage errors

Problem:

• There are problems publishing new Model API's and we see errors related to : replicatorStorage, cached, prepared or event-log folder
• We see errors like : Body: Unknown error caused by: domino.git.domain.models.GitActionFailure: Clone failed.
• We see errors 'repository not found <path to replicatorStorage>/[cached|prepared]/...
• We have a disk full on a build node (Models and Environment build may fail)

You have tried to follow instructions from https://tickets.dominodatalab.com/hc/en-us/articles/9027181072532-replicator-error-when-building-model-API-500-Internal-Server-Error-Unknown-error-caused-by-java-io-IOException-Unable-to-delete-directory-domino-shared-replicatorStorage-cached-directory- or https://tickets.dominodatalab.com/hc/en-us/articles/10448305288596-Model-APIs-New-builds-fail-with-Error-during-image-build-and-push-timed-out-waiting-for-replicator-to-prepare-resources-

Resolution:

Stop the replicator pod (remove build label from the node or nodes which are tagged as builders)

On any build node showing issues remedy by deleting the cached, prepared and event-log folder

Add the build label to all build nodes, which starts the replicator and the folders will be recreated.

Get a list of Domino build nodes:

kubectl get no -ldomino/build-node=true

Overwrite the build node’s label, which will prevent builds and the replicator from being scheduled to it during maintenance:

kubectl label no --overwrite <node> domino/build-node=false

Set is back to true after the maintenance:

kubectl label no --overwrite <node> domino/build-node=true

Root Cause:

Disparity in some Git repos, or partial cleanup may have created a issue and a full cleanup is most likely to work. Issues could also relate to disk full on the node in question.

Applies to:

Domino 3.6 to 5.2