Submitted originally by: joaquim.benseny
Hi,
is there any way of, when building an environment, accessing a file that has been uploaded to a specific project?
Use case: we have limited internet access through the environment build and we don't have direct access to a machine with access to our docker registry within domino. Therefore I wanted to do a quick test by downloading these files with a machine that has access to internet and upload them to domino from the files tab. Then I wanted to use these files inside my domino environment build.
Thanks
Joaquim
Comments
4 comments
Hi Joaquim,
It's possible to access project-specific files from an environment, as long as you do so in one of the setup scripts (pre-setup, post-setup, pre-run). At the point your code in these scripts runs, the docker container and file system containing your run will exist and the project folders will be mounted. This wouldn't work in the dockerfile however, since that file is executed when you build the compute environment, and therefore doesn't have access to any project-specific files.
This isn't really a best practice for a couple of reasons:
1) It breaks the Compute Environment model a bit by making them less portable (they only work in certain projects); this could confuse users who try to start this environment from a project that isn't setup correctly -- you'll probably want to write a friendly error message if the files don't exist to reduce this risk.
2) If you are doing a lot of work -- installing lots of packages, copying large amounts of data, etc -- then you will have to wait for that work to complete every time you start a run since Domino doesn't pre-cache scripts that are executed at run-time (unlike dockerfile installs/downloads, which only need to be re-done when the environment is re-built).
But if you're ok with these trade-offs it's not a bad approach; I've done it a couple of times for specific niche use cases.
Also, there's a somewhat similar alternative that might work well for your use case if accessing /mnt directly gets weird. You could use the Domino API to directly download files from the blobstore for a particular project to an environment. This *could* be done in the dockerfile, so would potentially allow you to cache these results better. That would require you to deal with securely storing and accessing the API key used to download the files (storing it in an environment variable, for instance) and is somewhat more convoluted, so if the simpler solution works, I'd use that.
Does that help?
-Zach
Submitted by: zach.ingraham
Hi Zach,
yes, your answer makes sense. We haven't tried as we got the domain we wanted finally whitelisted but will try next time once we want to have something else quickly.
Will let you know if this works
Submitted by: joaquim.benseny
FWIW, you can curl or wget files from a domino project into your Dockerfile. It's a bit hacky though.
I'm doing that at the moment but would probably prefer to just build a Docker image the normal way and store it in AWS ECR or something and then reference that...
Submitted by: msterling
@msterling is right on: If you have something you want to import into your environment at build time, one way to do it is to put it in a domino project and wget the file from that location using the URL that you would use to download the file from your project files page. Note that for this to work without adding your Domino credentials or API key, you can make the project temporarily public for the few minutes it takes to build the environment (if it's safe for you to do so -- note that anyone who is able to access the public Domino project URL would be able to see the file during those few minutes, so this is not the right approach for a sensitive file, especially if your Domino is open to the public internet).
There are other approaches that are possible, but this is one of the easiest ways to get an arbitrary file into your Docker image in Domino.
We are looking at ways to make it easier to import outside Docker images into Domino, though this is still under discussion. If you have suggestions or feedback about this please leave us a feature request here: https://community.dominodatalab.com/categories/feature-requests
Thanks!
Submitted by: katie.shakman
Please sign in to leave a comment.