Domino Run with Large Repo Fails to Start Due to JVM Heap Space Out of Memory Error

If your Domino run fails to start, and there is an external repo attached to the project of size 500 MB or larger, it is likely that the JVM heap space in the executor container of the run pod has been consumed by the executor's attempt to clone the repo.

If you have access to the support bundle, you will see entries similar to the following in the executor log file:

2022-06-02T14:48:13.025Z: INFO  [d.r.d.GitRepoUpdater] Cloning new copy of <gitlab-link>/YourRepo.git 
correlationId="cSVGlF3dDk" projectId="<project-id>" runId="<run-id>" tickId="<run-id>" thread="application-akka.actor.default-dispatcher-5"

2022-06-02T14:49:57.142Z: Uncaught error from thread [application-akka.actor.default-dispatcher-5]: 
Java heap space, shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[application]

2022-06-02T14:49:57.143Z: java.lang.OutOfMemoryError: Java heap space

By default the JVM Heap in the executor container is sized to 500MiB. This is the result of the default values associated with the following keys in Domino's Central Configuration, as JVM heap space is calculated by taking the difference between memoryRequestedMiB and nonHeapMemoryReserveMiB:

com.cerebro.domino.computegrid.kubernetes.executor.memoryRequestedMiB
com.cerebro.domino.computegrid.executor.nonHeapMemoryReserveMiB 

If these keys cannot be found in your Central Configuration, they will be using their default values of 1500 MiB for memoryRequestedMiB and 1000 MiB for nonHeapMemoryReserveMiB, which leaves 500 MiB for JVM heap space. You can allow more space for the JVM heap by increasing the the value for memoryRequestedMiB. Typically we advise increasing by the least expected amount and increasing it from there if needed. As seen in the following screenshot, 1600 or 1750 could be a good place to start.