Push Failed workspace syncing error in 5.3.0 & 5.3.1

Version/Environment (if relevant):

Applies to Domino version 5.3.0 and 5.3.1 and only applies to Jupyter and JupyterLab IDE's.

Issue:

Workspaces with a large quantity of files in /mnt are not able to sync.  Even if there are just a few files to sync they encounter a Push Failed error upon hitting "Sync to Domino" or "Sync All Changes".  This problem can be either intermittent or 100% reproducible depending on circumstances like duration of sync, number of files, etc.

Setup logs and support bundle's executor.log contain a stacktrace containing the messages like:

Upload failed: Failed: java.lang.IllegalStateException: Blob transfer (upload) failed because 1 operations failed: Check and upload blob 373201eed9ec04c6caa57da1622e6671fa2910a4 -> java.lang.IllegalArgumentException: No file with hash Sha1(373201eed9ec04c6caa57da1622e6671fa2910a4) exists in this local directory.

Root Cause:

A race-condition oriented bug is more prominent in 5.3.0 and 5.3.1, especially in workspaces involving Jupyter and JupyterLab when the /mnt directory contains a high number of files (umpteen thousand).

Resolution:

Until you upgrade to 5.3.2 you can workaround the problem by ensuring c.Application.log_level =30 is set in the running workspace's /home/ubuntu/.jupyter/jupyter_notebook_config.py and/or /home/ubuntu/.jupyterlab/jupyter_lab_config.py 

You can do this by modifying the Pre-run scripts for the Compute Environment associated with the breaking workspace.  After the build completes you can stop and restart the workspace (simpy stop, don't delete the workspace), then re-attempt the sync. 

These instructions force log level for jupyter/jupyterLab to "WARN" level.

Add these steps to the  "Pre-run" scripts section:

mkdir -p /home/ubuntu/.jupyter
printf '\nc.Application.log_level = 30\n' >> /home/ubuntu/.jupyter/jupyter_notebook_config.py 
mkdir -p /home/ubuntu/.jupyterlab
printf '\nc.Application.log_level = 30\n' >> /home/ubuntu/.jupyterlab/jupyter_lab_config.py

Notes/Information:

Internal reference https://dominodatalab.atlassian.net/browse/DOM-42742

In some installs customers might heavily config their compute environments, altering the default configuration used for jupyter or jupyterLab, or the start-up process, etc.  Take care to integrate the log_level setting into these in a case-by-case basis.