If you're following our Spark documentation but still getting errors when trying to run Spark, please check your version of Domino by going to your base url (example: https://demo.dominodatalab.com/) and adding /version to the end (https://demo.dominodatalab.com/version). If the output is not 4.5 or higher, you'll need to downgrade your version of Spark to 3.0.1 or earlier.
Other users have seen this issue come up as the following error:
Exception Traceback (most recent call last)
<ipython-input-5-66f9c693822e> in <module>
----> 1 sc = SparkContext(conf=conf)
/opt/conda/lib/python3.8/site-packages/pyspark/context.py in __init__(self, master, appName, sparkHome, pyFiles, environment, batchSize, serializer, conf, gateway, jsc, profiler_cls)
142 " is not allowed as it is a security risk.")
143
--> 144 SparkContext._ensure_initialized(self, gateway=gateway, conf=conf)
145 try:
146 self._do_init(master, appName, sparkHome, pyFiles, environment, batchSize, serializer,
/opt/conda/lib/python3.8/site-packages/pyspark/context.py in _ensure_initialized(cls, instance, gateway, conf)
329 with SparkContext._lock:
330 if not SparkContext._gateway:
--> 331 SparkContext._gateway = gateway or launch_gateway(conf)
332 SparkContext._jvm = SparkContext._gateway.jvm
333
/opt/conda/lib/python3.8/site-packages/pyspark/java_gateway.py in launch_gateway(conf, popen_kwargs)
106
107 if not os.path.isfile(conn_info_file):
--> 108 raise Exception("Java gateway process exited before sending its port number")
109
110 with open(conn_info_file, "rb") as info:
Exception: Java gateway process exited before sending its port number
If you are already using the proper version of Spark or are on a 4.6+ deployment please verify your JAVA_HOME environment variable is set correctly.
Note: We will be updating our formal documentation shortly but wanted to make this information available for our users ASAP.
Comments
0 comments
Please sign in to leave a comment.