UI 503 errors and Keycloak API errors due to Keycloak's default memory settings being insufficient

Version/Environment (if relevant):

Domino versions prior to 5.6 are susceptible since this has been seen in Keycloak version 15 and 17.

Issue:

After increased, or new usage, a customer began encountering 503 errors in the UI, and API calls directly to Keycloak were returning errors like:

OSError: Request Failed: msg=<html><head><title>Error</title></head><body>Internal Server Error</body></html>

Root Cause:

Symptoms like the above should typically lead to analysis of nucleus-frontend and Keycloak pods in terms of their status, restart count and logs. We found Keycloak (one or more) pods were crashing and restarting unexpectedly. Keycloak logs revealed the culprit, an OutOfMemoryError:

[0m[31m14:52:57,155 ERROR [org.jgroups.protocols.TCP] (Connection.Receiver [100.127.18.60:7600 - 100.126.63.5:57379]-19,ejb,keycloakv17-0) JGRP000030: keycloakv17-0: failed handling incoming message: java.lang.OutOfMemoryError: Java heap space

Resolution:

Domino is relying on Keycloak's default memory settings, and in some use-cases this may be insufficient. You can increase the Xmx (max heap space for the JVM) for Keycloak to fix this:

Via kubectl edit the keycloakvXX stateful set resource, where XX is the Keycloak version (15, 17, etc).
```
kubectl -n domino-platform edit sts keycloakv17
```
Find the env: section in the spec for the keycloak container

If it already has the JAVA_OPTS environment variable, append the following to its value:

-Xmx1536M .
Example syntax (word wrap between arguments on this line are fine, as depicted below)

- name: JAVA_OPTS
  value: -Djavax.net.ssl.trustStoreType=PKCS12 -Djavax.net.ssl.trustStore=/etc/ssl/certs/cacerts.p12 
         -Djavax.net.ssl.trustStorePassword=changeme -Xmx1536M

If there is no JAVA_OPTS variable, add the variable and give it the -Xmx1536M value.
Exiting the edit with a save will restart the pods in the stateful set.

Notes/Information:

This is targeted for resolution as part of a bug-submission in version 5.6, internal reference DOM-44649

Articles in this section

Version/Environment (if relevant):

Issue:

Root Cause:

Resolution:

Notes/Information:

Comments

Articles in this section

Version/Environment (if relevant):

Issue:

Root Cause:

Resolution:

Notes/Information:

Related articles