If you're following our Spark documentation but still getting errors when trying to run Spark, please check your version of Domino by going to your base url (example: https://demo.dominodatalab.com/) and adding /version to the end (https://demo.dominodatalab.com/version). If the output is not 4.5 or higher, you'll need to downgrade your version of Spark to 3.0.1 or earlier.
Other users have seen this issue come up as the following error:
Exception Traceback (most recent call last)
<ipython-input-5-66f9c693822e> in <module>
----> 1 sc = SparkContext(conf=conf)
/opt/conda/lib/python3.8/site-packages/pyspark/context.py in __init__(self, master, appName, sparkHome, pyFiles, environment, batchSize, serializer, conf, gateway, jsc, profiler_cls)
142 " is not allowed as it is a security risk.")
--> 144 SparkContext._ensure_initialized(self, gateway=gateway, conf=conf)
146 self._do_init(master, appName, sparkHome, pyFiles, environment, batchSize, serializer,
/opt/conda/lib/python3.8/site-packages/pyspark/context.py in _ensure_initialized(cls, instance, gateway, conf)
329 with SparkContext._lock:
330 if not SparkContext._gateway:
--> 331 SparkContext._gateway = gateway or launch_gateway(conf)
332 SparkContext._jvm = SparkContext._gateway.jvm
/opt/conda/lib/python3.8/site-packages/pyspark/java_gateway.py in launch_gateway(conf, popen_kwargs)
107 if not os.path.isfile(conn_info_file):
--> 108 raise Exception("Java gateway process exited before sending its port number")
110 with open(conn_info_file, "rb") as info:
Exception: Java gateway process exited before sending its port number
If you are already using the proper version of Spark or are on a 4.6+ deployment please verify your JAVA_HOME environment variable is set correctly.
Note: We will be updating our formal documentation shortly but wanted to make this information available for our users ASAP.