Why did my execution fail with the error 'No space left on device'?

If your execution has failed and you see a similar error to the following error in the logs,

then you have exceeded the disk space available to your execution.  From the user side this should be a relatively straightforward fix.  Remember that your executions execute inside a 'compute instance'. These instances have finite amounts of all resources, memory, cpu, and disk.  So to correct the problem as a user try the following steps...

1) Try a new hardware tier.

2) Reduce the amount of data you are using in your project and remember git repositories also count towards this.

3) Move project data to datasets.  

4) Make sure you don't have unexpected crash, dump, or core files in your workspace that could be taking up large amounts of space.  Occasionally jupyter or other IDEs or libraries will crash, leaving a huge file in your workspace files that you might not be immediately aware of.

If your deployment is 4.4 or above you can also set the disk size for you project in the Settings>>Hardware & Environment>>Workspace and Jobs Volume Size. 

Changes you make to the volume size will not impact existing workspaces. Instead, the changes will be applied to subsequent, new workspaces.

Estimating how much room you need can be a little tricky, it is a combination of your project files, any files being imported from an external repository and if you are generating lots of data in your run that should be taken in to consideration too. 

It is still highly recommended to move large project files to a dataset.

The Gory Details

The information below this point is mostly for administrators.

Domino v4.x and above

At v4.0 we have switched the architecture from docker to kubernetes and now each execution gets its own volume and is no longer impacted by caching and shared resource issues.  That said, these volumes have a finite size still and can still be filled.  From v4.0-v4.4 the default disk size is 10GB.  After v4.4 the default size is 15GB.  This default size can also be configured in the Central Config with the following setting...

com.cerebro.domino.computegrid.kubernetes.volume.volumesSizeInGB

ref: https://docs.dominodatalab.com/en/latest/admin_guide/eca4b2/manage-compute-resources/ 

Users can additionally change their individual disk size in the project settings of their project in v4.4 and above.

Domino v3.6 and below

When an execution starts, we copy the user's project files and the docker image for the user's Compute Environment to a partition on the executor so that they are available to the docker container that is started for the execution.

If the data moved to the executor instance for the execution exceeds available disk space the execution can fail on startup.

The product actually has quite a bit of logic built in to it to try and mitigate this, but it is not completely avoidable.  We poll executors for their disk status and if the partitions are less than 10% free we place the executor in to Maintenance Mode, taking it out of service.  Note that the most common scenario we see currently is that a disk is not quite at this threshold and an execution will fill the last 10%+ of the disk. 

Under the hood, we are caching project data and the environment images to try and reduce startup times.  Over time, if an executor is long lived and used by projects using a variety of different compute images that all get cached, the disk partition can also fill.

All of this data can add up over time to fill an executor disk.  We garbage collect and when there is disk pressure we are more aggressive in this.  But all of these mitigations cannot prevent an execution from filling the remaining disk space if the project's files and compute image exceed available space.  We do not check if an execution is too large for the remaining disk on an executor at startup because calculating the total project size, especially with external GIT repos, is not always possible.  

All of these strategies actually do a pretty good job and this type of failure is now no longer commonplace.  When it does happen though it can block an entire hardware tier as we look for the first available executor instance to run on.  If the executor disk is full but not yet been taken out of service for low disk space, new executions that land on it can fail repeatedly.  So that is why we recommend just switching hardware tiers for users who have no admin privileges as it usually unblocks the user to continue their work.

After this occurs, admins need to intervene and place the executor into maintenance mode.  Then terminate once any current executions finish, usually from the Domino Dispatcher UI.  Find the executor, select actions on the far right of the screen and once in the executor view, verify no executions are running, and then press terminate.  It will kill the instance and release the storage.

Garbage collection will actually often clean up disk space on these executors as well.  So you may not find that an executor will always need to be terminated, but they are ephemeral anyway, so there is no harm to just terminating as an expedient solution.  

If you find that this is becoming a more frequent occurrence it may just indicate that project and compute environment image sizes are outpacing the assigned disk space on your executors and you should consider increasing disk sizes.