How do we cache an environment on executors?

• 

  Jaclyn Patterson

  • March 15, 2021 01:41

  https://dominodatalab.atlassian.net/wiki/spaces/CS/pages/10780882/Bake+an+environment+into+an+AMI

  Copied from the link above.

  Bake an environment into an AMI
  

  Created by Mark Silverberg

  Last updated Jun 21, 2018 by Pulu Anau

  In private deployments, it is possible to "bake" compute environments' Dockerfiles on the executor template AMI so that the compute environment is cached, ready, and available for the user when they are ready.

  In short, doing this in AWS VPC environments will significantly reduce the amount of time a customer spends at the "Building" or "Pulling" run states

  Step-by-step guide

  
  

  1. Find the executor template (probably called domino-CUSTOMER-executor-template) and start it (it will likely be in the "stopped" state)
  2. Can use AWS console to find the IP address or in the deploy folder in the terraform.output file
  3. SSH into the executor template using the private key
  4. As root, run docker images to see what images are cached
  5. Run docker pull URL for the images you want to cache onto the executor; it's probably a good idea to go in and cache all global environments
  6. It's a good idea to run salt-call state.highstate before creating the AMI so that all necessary software and system updates are applied
  7. Create a new AMI and use the naming scheme of domino-CUSTOMER-executor-YYYYMMDD-HHMM . You can do this in two ways:
  8. If you have access to the AWS console, create an AMI based off of the executor template ec2 instance you just modified. 
  9. When creating the AMI, be sure to check "Delete on Termination" for all disks
  10. If you SSH into salt master (or central if there is no salt master server) you can do: VERBOSE=TRUE DRY_RUN=FALSE /domino/create_executor_amis.sh (you might have to chmod +x this file first)
  11. Wait for the AMI to be "available". You can run this from the salt master to see the current status: 
  12.  aws ec2 describe-images --image-ids ami-865ef0f9 --region us-east-1 | grep State
  13. Create a new hardware tier or update an existing one to use the new AMI and test thoroughly
  14. Once tested, update remaining hardware tiers to use the new AMI:
  15. db.executor_group_configuration.update({},{$set:{"executorImage":"NEW_AMI_ID"}},{multi:true})
  16. Depending on the scenario you may need to replicate the commands you performed on the template to the existing executors still running. Run commands on You can run commands from salt master (run test commands to ensure only executors will be invoked): salt "*exec-high*" cmd.run "df -h" 

  
  

  Submitted by: katie.shakman

  0

  Comment actions Permalink
• 

  Jaclyn Patterson

  • March 15, 2021 01:41

  https://dominodatalab.atlassian.net/wiki/spaces/CS/pages/9404419/Cache+a+Dockerfile+on+all+currently+running+executors

  Copied from the article linked above.

  
  

  Cache a Dockerfile on all currently running executors
  

  
  

  Created by Mark Silverberg

  Last updated May 05, 2017

  These instructions walk you through how to build/cache a Docker file on all of a Domino deployment's existing executors.

  Note: this is most useful for on-prem deployments where there is a fixed number of executors which are always running. This is also useful if you have snapped a new AMI for a customer with the latest Dockerfile cached and you want the existing executors to have the latest layer(s) too.

  Another note: this uses the "salt" command – essentially, Salt keeps a record of all the servers and their configs and we can tell it to run a command on all of the servers at the same time

  1. SSH into the Salt Master
  2. Go to the temp directory: cd /tmp
  3. Create a new file, Dockerfile, with the contents of your choosing, for example:
  4. Tell Salt to remove the previous /tmp/build folder and recreate it so we can have a clean slate
  5. salt '*exec*' cmd.run 'rm -rf /tmp/build && mkdir /tmp/build'
  6. Tell Salt to copy our local (on the Salt Master) Dockerfile to /tmp/build on each of the executors:
  7. salt-cp '*exec*' /tmp/Dockerfile /tmp/build/Dockerfile
  8. Tell Salt to build (and cache) the Dockerfile on each of the executors
  9. salt '*exec*' cmd.run 'docker build /tmp/build'
  10. All the executors should now have that Dockerfile cached and your users should spend very little time at the "Building" stage of this and other similar Compute Environments!

  
  

  Submitted by: katie.shakman

  0

  Comment actions Permalink
• 

  Jaclyn Patterson

  • March 15, 2021 01:41

  Note that we have a support article on this: Creating a new executor AMI

  Submitted by: nick.jablonski

  0

  Comment actions Permalink

Please sign in to leave a comment.