Docker Layers - failed to register layer: max depth exceeded

Version: 

All docker versions

Issue: When attempting to launch a workspace you are presented with this error: "failed to register layer: max depth exceeded" in the logs. The error is also visible on the describe of a new pod (Under Events)

Warning Failed 12m (x7 over 26m) kubelet Failed to pull image "100.97.188.95:5000/dominodatalab/environment:63a051030e277d167bec6bfd-35": rpc error: code = Unknown desc = failed to register layer: max depth exceeded

What are docker layers:

Every docker image is made up of layers. A single layer is the consequence of a specific command written down in a dockerfile. Docker adds a new layer to the stack for each dockerfile instruction. The final docker image is built by combining all layers.

A hash represents each layer, docker computes a layer's hash based on contextual information such as changes and the parent layer chain. The maximum number of layers set by docker that a docker image can have has been set to 127 layers.

Why does the issue occur:

Docker has reached the number of layers that may be stacked in a container image, and it has hit that limit (127). This error typically occurs when the image contains too many layers or nested layers, which can occur as a result of complex dependencies or excessive use of Docker instructions.

Resolution:

To resolve the issue, you can simplify the dockerfile and can examine approaches like merging numerous instructions into a single layer, reducing superfluous dependencies, or utilising an alternative strategy to lower the overall layer count. At times it may also appear that the instructions you have, have not exceeded this limit, you must keep in mind that the image upon which you are building may have already used many layers previously.

1) Combine several commands into one via using operators such as &&, like this:

RUN apt-get update && \
    apt-get install -y package1 && \
    apt-get install -y package2 && \ 
    apt-get install -y package3 && \
    apt-get clean && \ 
    rm -rf /var/lib/apt/lists/*

2) Consolidate RUN instructions: Instead of utilising separate RUN instructions for each command, you may use shell scripting to integrate numerous commands into a single RUN instruction. As an example:

RUN apt-get update && apt-get install -y package1 package2 package3 && apt-get clean && rm -rf /var/lib/apt/lists/*

3) Use multi-stage builds: If your build process is complicated or you require numerous tools to generate your Docker image, you may use multi-stage builds. This method allows you to use one stage for constructing and another for the final image, eliminating the need for unneeded intermediate layers. Here's an example:

FROM builder AS build
# Build commands go here

FROM base_image
# Copy artifacts from the build stage
COPY --from=build /path/to/artifacts /app

If you still believe you have not reached the 127 layer limit even though you have built upon previous layer then you could also look at creating an entire new compute environment and copying over your existing instructions providing they don't exceed the 127 layer limit and build off this new environment.

Notes: 

Docker File instructions

https://docs.dominodatalab.com/en/latest/user_guide/813d89/use-dockerfile-instructions/

Docker Best Practices:

https://docs.docker.com/develop/develop-images/dockerfile_best-practices/

Docker File reference: 

https://docs.docker.com/engine/reference/builder/