4.0 thru 4.6.1 Dispatcher restarts and connection problems (improper state transitions for workspaces/runs) during high-volume can be helped via Mongo index

Installs of 4.x prior to 4.6.2 with user-count greater than 200 can potentially encounter problems if many users are creating runs or workspaces simultaneously.  The symptom can be many things if Dispatcher component is struggling, but an example would be workspace or job states not changing properly, like the job never being listed as "finished".  Further evidence of this problem can be found by describing your Dispatcher pod and looking for frequent restarts and memory problems like OOMKills.

Resolution:

The resolution is to add a MongoDB index if Domino Account Managers or other Domino Engineers haven't already added it.  You can use db.sagas.getIndexes() , look to see if  sagaId returns.  If not, you should add it.   Additionally you can look for a message in the Admin toolkit:

To add the index, use the following steps:

• From the Domino Admin page, select Advanced then MongoDB

• Enter the following command and select RUN

• Successfully adding the index will produce the following output

{
    "createdCollectionAutomatically" : false,
    "numIndexesBefore" : 2,
    "numIndexesAfter" : 3,
    "ok" : 1,
    "operationTime" : Timestamp(1631807240, 1),
    "$clusterTime" : {
        "clusterTime" : Timestamp(1631807240, 1),
        "signature" : {
            "hash" : BinData(0,"cavksNg/n7n6v7f8cCH4KmHAS+g="),
            "keyId" : NumberLong("7007453384982659075")
         }

    }

}

Note that MongoDB won't recreate indexes, so this createIndex command is harmless whether or not the index already was created in the past. 

Rollback

In the event a rollback is required, you can remove the index with the following command

Final note,  sometimes when the hitting "RUN" in the UI the request won't land on the master mongo replicaset pod so you won't see the indication of success listed above.  Try a few more times.  If it never goes thru successfully then Technical Support can assist by referencing our internal document https://tinyurl.com/bddnhet7