If your run is taking longer than expected to start, then possible explanations may be found below. First, identify the run state that your run appears to be spending an unusual amount of time in:
Queued/Assigning
When a run is queued, this means that it is waiting to be assigned to a machine.
If you are working in a private on-premises deployment with a fixed number of machines:
A run stuck in the Queued state could mean that all of the available run slots are occupied by other runs. Please contact a SysAdmin, who can determine whether this is the case.
If you are working in our cloud or a private VPC deployment:
A run is stuck in the Queued state means that no machines in the chosen hardware tier are currently available. A new machine will need to spin up, which can take a few minutes. If your run remains queued after several minutes, it's possible that a capacity limit has been reached, or there may be some problem with the system. Please contact your system admins (or support@dominodatalab.com, if appropriate) for further assistance in diagnosing the issue.
System admins, please see possible explanations here.
Scheduled
This is different from user scheduled runs, this is a run state. When a run is in a scheduled state it means that the dispatcher has requested an executor to process a run. The executor has acknowledged this request but hasn’t started the run yet. If you see it in this state for more than a couple minutes, please contact us for further assistance.
Preparing
When the run is Preparing, this means that data is being downloaded to the machine where your code will run.
If you are experiencing a long wait in this stage:
First, check the size of your project as well as any projects from which you are importing files. Large projects, in terms of overall size (many GBs) and/or file count (thousands of files) can take some time to download and extract.
If you find the length of the Preparing step is inconsistent:
This could be explained by Domino's built-in file caching. If an existing copy of your project files is available on an executor machine you've previously used, Domino will attempt to re-use that data as a starting point for the Preparing stage, which can significantly speed up the sync.
If you think that the Preparing step is taking longer than it should, or if the length of time is prohibitive for your workflow:
Please contact us for further assistance.
Building
When the run is Building, this means that it is compiling a custom compute environment from a Dockerfile.
If this seems to be taking longer than expected:
Check your environment definition for any packages that you may be installing by mistake, or can otherwise remove.
If the length of this stage is inconsistent:
It could be due to built-in Docker caching, which is used when possible. If you re-run on a machine that has already built this environment, this stage will be skipped.
If you can't explain why this step is taking so long, or if the slowness is significantly impacting your workflow:
Please contact us for further assistance.
Running
This is the state where your script or interactive session runs. However, it is also the state where requirements.txt packages are pip installed and any code contained in the pre-setup and post-setup scripts of a custom compute environment is executed.
If you are waiting on a workspace, you might come across the message "There was a problem connecting to the notebook session. Click here to retry," or just "Starting" with a spinner:
or
This usually means that your requirements.txt or pre/post-setup script packages are still being installed.
If you don't expect these installations to take very long:
You can try clicking the "retry" link (if available) until the "Open Session" button appears. If the installation is still not completing, inspect the console output for debugging information (if you can no longer see the console output, try starting a new session or running a batch script).
If you do expect these installations to take a while:
Consider caching them.
If you don't have any requirements.txt or pre/post-setup script installations, or if they shouldn't be taking this long:
Please contact us for further assistance.
Comments
0 comments
Article is closed for comments.