After Domino 4.x to 5.x upgrade executions fail with "HTTP Error 400: Bad Request POSTing to..."

Version:

Domino 5.0.x, 5.1.x, 5.2.x

Issue:

After an upgrade from any Domino 4.x version to Domino 5.0.x~5.2.x executions may fail while trying to acquire a token.  The execution logs will have the following repeating pattern...

022-09-22T18:09:47.990Z: INFO:__main__:-- Preparing access token --
2022-09-22T18:09:47.997Z: ERROR:__main__:Attempt to refresh failed due to an HTTP Error HTTP Error 400: Bad Request POSTing to http://keycloak-http.domino-platform/auth/realms/DominoRealm/protocol/openid-connect/token
2022-09-22T18:09:52.538Z: Preparing working directory.
2022-09-22T18:10:22.886Z: Preparing working directory.
2022-09-22T18:10:53.212Z: Preparing working directory.
2022-09-22T18:11:23.571Z: Preparing working directory.
2022-09-22T18:11:48.086Z: INFO:__main__:-- Preparing access token --
2022-09-22T18:11:48.093Z: ERROR:__main__:Attempt to refresh failed due to an HTTP Error HTTP Error 400: Bad Request POSTing to http://keycloak-http.domino-platform/auth/realms/DominoRealm/protocol/openid-connect/token

Root Cause:

Background

The reason this occurs is that Domino 4.x makes use of a refresh token.  The refresh token is reissued when a user logs in and its valid for the timeframe set in keycloak by the SSO session Idle timeout field, typically 7 days.  After 7 days this token is no longer valid and a user who attempts to run scheduled jobs will have those jobs fail, as will executions started by API calls.  A login will refresh that token and resolve the problem for that individual user for the duration of the SSO session Idle Timeout.  Note that if a user explicitly logs out this will also expire the token and result in the same behavior.

To mitigate this behavior in Domino 4.x it's possible to use the

com.cerebro.domino.auth.refreshTokenInRun.enabled

Central Config (CC) flag and have it set to false to work around this.  We will no longer try to put these tokens in executions.  We basically just stop looking for the token file.  The nuance here is that any existing user with the field "key":"must:have" in their user_session_value entry within mongodb will continue to have the tokens enforced.  So these must be removed as well.  They can be removed globally with 

db.user_session_values.remove({"key": "must-have"})

That's the Domino 4.x story.

In Domino 5.x we have stopped using the refresh token.  We now use an offline token to resolve these issues.  This token never expires and is again issued when a user logs in without a valid token in place.  

Why this failure occurs after upgrade 
After an upgrade from 4.x-->5.x some of the artifacts in mongo are carried over through the upgrade, specifically in the user_session_values collection.  This creates a situation where a user has a field "key":"must-have" that forces a check of the token, now an offline token, and the token exists but is actually invalid because it is still the old refresh token.  
 

Resolution:

This CC key,com.cerebro.domino.auth.refreshTokenInRun.enabled, is no longer valid in 5.x because we no longer use refresh tokens, so cannot be used for mitigation.

So after a 4.x-->5.x upgrade we have found the simplest way to mitigate this for a single user is to have that user logout to expire all tokens and then log back in. 

To resolve the problem for all users remove the key value of JWT with the mongodb query:

db.user_session_values.remove({"key": "jwt"})

After this mongo query executes all users will have their keys removed and will need to log back in to be issued an offline token. Note that this will in fact cause scheduled jobs to fail even for users who have already logged out/in until the owners of those jobs log back in to Domino.  We have found that impersonation of the user by an admin is insufficient to get a valid key and the user must actually login.

The issue has been resolved in Domino 5.3 so if upgrading to 5.3 we automatically clean this up and you should not run in to this problem.

If you have any concerns with running mongo queries or any questions on this issue, reach out to Domino Support for assistance.

Notes:

Internal Domino ticket: DOM-39298