Version/Environment (if relevant):
Domino versions prior to 5.6 are susceptible since this has been seen in Keycloak version 15 and 17.
After increased, or new usage, a customer began encountering 503 errors in the UI, and API calls directly to Keycloak were returning errors like:
OSError: Request Failed: msg=<html><head><title>Error</title></head><body>Internal Server Error</body></html>
Symptoms like the above should typically lead to analysis of nucleus-frontend and Keycloak pods in terms of their status, restart count and logs. We found Keycloak (one or more) pods were crashing and restarting unexpectedly. Keycloak logs revealed the culprit, an OutOfMemoryError:
[0m[31m14:52:57,155 ERROR [org.jgroups.protocols.TCP] (Connection.Receiver [100.127.18.60:7600 - 100.126.63.5:57379]-19,ejb,keycloakv17-0) JGRP000030: keycloakv17-0: failed handling incoming message: java.lang.OutOfMemoryError: Java heap space
Domino is relying on Keycloak's default memory settings, and in some use-cases this may be insufficient. You can increase the Xmx (max heap space for the JVM) for Keycloak to fix this:
keycloakvXXstateful set resource, where
XXis the Keycloak version (15, 17, etc).
kubectl -n domino-platform edit sts keycloakv17
env:section in the spec for the
If it already has the
JAVA_OPTSenvironment variable, append the following to its value:
Example syntax (word wrap between arguments on this line are fine, as depicted below)
- name: JAVA_OPTS
value: -Djavax.net.ssl.trustStoreType=PKCS12 -Djavax.net.ssl.trustStore=/etc/ssl/certs/cacerts.p12
If there is no
JAVA_OPTSvariable, add the variable and give it the
Exiting the edit with a save will restart the pods in the stateful set.
This is targeted for resolution as part of a bug-submission in version 5.6, internal reference DOM-44649