Errors when setting up Spark: "Java gateway process exited before sending its port number"

If you're following our Spark documentation but still getting errors when trying to run Spark, please check your version of Domino by going to your base url (example: https://demo.dominodatalab.com/) and adding /version to the end (https://demo.dominodatalab.com/version). If the output is not 4.5 or higher, you'll need to downgrade your version of Spark to 3.0.1 or earlier. 

Other users have seen this issue come up as the following error: 

Exception                                 Traceback (most recent call last)
<ipython-input-5-66f9c693822e> in <module>
----> 1 sc = SparkContext(conf=conf)
/opt/conda/lib/python3.8/site-packages/pyspark/context.py in __init__(self, master, appName, sparkHome, pyFiles, environment, batchSize, serializer, conf, gateway, jsc, profiler_cls)
    142                 " is not allowed as it is a security risk.")
    143 
--> 144         SparkContext._ensure_initialized(self, gateway=gateway, conf=conf)
    145         try:
    146             self._do_init(master, appName, sparkHome, pyFiles, environment, batchSize, serializer,
/opt/conda/lib/python3.8/site-packages/pyspark/context.py in _ensure_initialized(cls, instance, gateway, conf)
    329         with SparkContext._lock:
    330             if not SparkContext._gateway:
--> 331                 SparkContext._gateway = gateway or launch_gateway(conf)
    332                 SparkContext._jvm = SparkContext._gateway.jvm
    333 
/opt/conda/lib/python3.8/site-packages/pyspark/java_gateway.py in launch_gateway(conf, popen_kwargs)
    106 
    107             if not os.path.isfile(conn_info_file):
--> 108                 raise Exception("Java gateway process exited before sending its port number")
    109 
    110             with open(conn_info_file, "rb") as info:
Exception: Java gateway process exited before sending its port number

If you are already using the proper version of Spark or are on a 4.6+ deployment please verify your JAVA_HOME environment variable is set correctly. 

Note: We will be updating our formal documentation shortly but wanted to make this information available for our users ASAP.