Submitted originally by: peter.valian
Hello, I'm new to the Domino platform so forgive me if this has been answered elsewhere.
I'm wondering if any one has had success in using Domino to spin up and scale jobs across multiple nodes (executors). From what I read in documentation, you can define hardware tiers to spin up a minimum set of executors but if I understand correctly I cannot have 1 job use more than 1 executor?
Can anyone correct me if I'm wrong or provide additional insight to work with this limitation. Looking for something like AWS EMR but managed by Domino.
Thanks!
Comments
3 comments
Hi @peter.valian ,
Welcome to Domino!
One way to scale horizontally in Domino is to set up a parent job that kicks off multiple child jobs on different executors. This is enabled by our Domino APIs, which allow you to interact programmatically with most major aspects of Domino functionality.
Domino API docs:
Domino API Python bindings:
We have a couple of projects available that demonstrate pipelining and spread-and-collect workflows in a Domino project, using the Domino APIs from a parent job to start child jobs.
Here is the pipeline example: https://try.dominodatalab.com/u/domino-admin/domino-pipeline/overview
And here is the spread-and-collect example: https://try.dominodatalab.com/jobs/domino-admin/spread-and-collect (This one should be receiving some updates soon, so you may notice some changes over the next few weeks.)
You may also be interested in using Spark on Domino -- here are a couple of articles to get you started:
Does this help? Please let us know if you still have questions or if you have suggestions for us.
Thanks,
Katie
Submitted by: katie.shakman
Hi,
The links above to the examples don't work for me. Are there updated links?
Thanks
Submitted by: timgalvin
Hi Tim,
Sorry about that, we migrated our "try" demo site and it looks like these projects didn't make it to the new version. I'm checking in on that now to get fresh links.
-Zach
Submitted by: zach.ingraham
Please sign in to leave a comment.