Version/Environment (if relevant):
Applies to Domino version 5.3.0 and 5.3.1 and only applies to Jupyter and JupyterLab IDE's. In a rare case this occurred in 5.1.2 also.
Workspaces with a large quantity of files in /mnt are not able to sync. Even if there are just a few files to sync they encounter a Push Failed error upon hitting "Sync to Domino" or "Sync All Changes". This problem can be either intermittent or 100% reproducible depending on circumstances like duration of sync, number of files, etc.
Setup logs and support bundle's executor.log contain a stacktrace containing the messages like:
Upload failed: Failed: java.lang.IllegalStateException: Blob transfer (upload) failed because 1 operations failed: Check and upload blob 373201eed9ec04c6caa57da1622e6671fa2910a4 -> java.lang.IllegalArgumentException: No file with hash Sha1(373201eed9ec04c6caa57da1622e6671fa2910a4) exists in this local directory.
A race-condition oriented bug is more prominent in 5.3.0 and 5.3.1, especially in workspaces involving Jupyter and JupyterLab when the /mnt directory contains a high number of files (umpteen thousand).
Until you upgrade to 5.3.2 you can workaround the problem by ensuring
c.Application.log_level =30 is set in the running workspace's
You can do this by modifying the Pre-run scripts for the Compute Environment associated with the breaking workspace. After the build completes you can stop and restart the workspace (simpy stop, don't delete the workspace), then re-attempt the sync.
These instructions force log level for jupyter/jupyterLab to "WARN" level.
Add these steps to the "Pre-run" scripts section:
mkdir -p /home/ubuntu/.jupyter
printf '\nc.Application.log_level = 30\n' >> /home/ubuntu/.jupyter/jupyter_notebook_config.py
mkdir -p /home/ubuntu/.jupyterlab
printf '\nc.Application.log_level = 30\n' >> /home/ubuntu/.jupyterlab/jupyter_lab_config.py
Internal reference https://dominodatalab.atlassian.net/browse/DOM-42742
In some installs customers might heavily config their compute environments, altering the default configuration used for jupyter or jupyterLab, or the start-up process, etc. Take care to integrate the log_level setting into these in a case-by-case basis.