My Workspace does not start because of a 'blob'

Issue

Blob transfer (download) failed because 1 operations failed: Download blob xyz -> domino.server.sync.S3BlobAccessException: S3ServiceException: errorCode="NoSuchKey" errorMessage="The specified key does not exist." xmlMessage="null"

Cause

All files are stored on the backend as blobs, the identifier; file-name is a sha1sum of the content. You can therefore match a file to a blob by running sha1sum against your files.

If the blob does not exist on the backend, either a failed sync has recorded it but it's not been uploaded. Check your Workspaces and ensure they've all synced and try again.

If you've not found the blob or a file with a matching sha1sum, i.e. try this in a terminal:

find . -type f -print0 | xargs -0 -I{} sha1sum {} |grep "xyz"

Another cause can be a fully deleted file (this is a red-box operation you can do on a file). This means that an identical file (with the same content) cannot be created on the Deployment! To revert a fully deleted file, you must locate the blob on the filestore, and remove a metadata file created in the blob location of the file. This requires a Domino Admin with access to the backend filestore!

Resolution

Get the assistance of a Domino Admin. Locate the UID of your Project (db.projects.find in MongoDB), locate the Project in the git-0 pod (requires backend access). Clone the 'bare' repository:

# kubectl exec into the git-0 pod

cd /tmp
git clone /var/opt/git/projectrepos/stub/UID.git # where the stub are the first 4 characters of the UID

Now you can search for the blob identifier (replace UID and stub in these commands):

grep -R "xyz" UID/*

This will tell you the filename and some metadata information.

If you can locate the correct file, use the UI, domino cli or a running Workspace to create this file and/or sync it OR upload a file with the same name and different content to get your Workspaces up and running.

Warning

Activity in the git-0 pod is potentially dangerous and can be harmful to all projects/files in your deployment! Take great care if and when you need to remove the temporary files you've created.

Ensure the git-backup jobs are successful to enable you to restore git-0 if it is damaged.