Execution memory settings

There are three places to impact the memory of an execution run pod. Execution pods contain three separate containers, Run, Executor & Nginx. The Kubernetes Memory Requests can be resized in the following ways.

Run Container

This is set from the Hardware Tier definition page, Admin>>Advanced>>Hardware Tiers. Memory Requests set here will size the Run container memory where user executions occur.

Executor container

The memory request size for this container where the domino executor runs can be set by the central config setting,

com.cerebro.domino.computegrid.kubernetes.executor.memoryRequestedMiB

default value: 1500Mib

We have seen failures related to this setting when importing large projects. Imported projects in the 10s of GB range can cause run execution failures during the preparing phase of startup. If you are seeing this failure mode you can add additional memory here. Update this setting cautiously as it will use the additional memory resources on every execution.

Memory errors can express in a variety of ways, but an example of the execution logging we have seen related to this failure looks like this...

2021-02-11T08:18:48.718Z: Preparing working directory.
2021-02-11T08:19:18.904Z: Preparing working directory.
2021-02-11T08:19:44.888Z: Caught termination signal!
2021-02-11T08:19:44.888Z: -- killed by pod termination --
2021-02-11T08:19:44.964Z: Trapped error signal in start-run, exiting

Nginx container

The memory request for the nginx container in the execution can be similarly set in Central Config,

com.cerebro.domino.computegrid.kubernetes.apps.nginx.memoryRequestedMiB

default value: 50Mib

This setting is the total memory for nginx and should be set higher than client_max_body_size. The client_max_body_size controls the maximum size of individual files that can be uploaded to the execution.

com.cerebro.domino.computegrid.kubernetes.apps.nginx.clientBodyMaxSizeMiB

default value: 25mb

If you need to increase this setting then you will need to ensure the memory request setting for nginx is increased as well.

More information on updating values in central config is here.