Dealing with stalled runs

If your run is stuck in same run state for more than 30 minutes and your console output has not updated, it's likely you've encountered an unreachable executor. Most likely, the run is being executed on an undersized machine and attempting to use more than the available RAM.

To confirm, first, compare the size of your project, including all imported files, associated with the run and the machine on which the run is executed.

How to check files size?

1. Navigate to the Files tab
2. Scroll to the bottom of the file list

Note: don't forget to add the size of your imported projects.

How to check machine size?

1. Navigate to the Details page to the left of the runs list
2. Compare the Hardware Tier to the total file size

For example, if the machine you’re running has 3GB RAM and your project size is 10GB, you are using an undersized machine.

Using a smaller machine can cause heavy swap utilization*, which causes the machine to reach a crashed/unreachable state. It's unclear why but these machines are able to accept new runs, but then get stuck without sending a failure message to the end-user.

If this is the case, please reach us at support@dominodatalab.com and make sure to include the url of the stuck run.

*Swap utilization is when a hard disk, also called swap space, is used when the amount of virtual memory (RAM) is full. If the system needs more memory resources, it compensates with swap space. Compared to virtual memory, obtaining files from swap space can be very slow. So, if files in swap memory are frequently addressed, it can get very expensive.