Unable to sync due to illegal state exception

Issue:

You encounter issues/problem with filesync or syncing data back to dfs which might manifest as a hash mismatch

Root Cause:

Filesync expects the file object to be in a certain state, having a specific distinct sha1 hash, and due to some unknown reason the sha1 hash of the modified file isn't registered with filesync and causes a hash mismatch exception.   An example might be scripts or notebook in the run container are writing files during the sync attempt, but there many unknown reasons.

Resolution:

Often the executor log in your support bundle will contain stacktrace related to the domino.filesync.sync methods which are failing during the sync attempt, the "User log" might contain this info as well.  These will typically include the hash values.

One solution would be to calculate the sha1 checksum of all the files on dfs and compare those with the hash in the support bundle/User log.

Basically, you should try to identify the illegal hash from the logs and execute a simple find command to locate the problematic file : 
find . -type f -exec sha1sum {} + | grep <the hash value in log error>

Remove this problematic file from dfs once found.  

If the above doesn't return try to cd into the /mnt/.domino/repo directory and run

git log -p and review that output for the hash value in the log error.

This can provide clues on which files are being problematic.