pip install pysparkand installing the latest version of Apache Spark leads to errors when callingpyspark.sql.DataFrame.show()methods ofDataFrameobjects.pip install pysparkand installing an older version of Apache Spark, i.e. having a version mismatch between PySpark and Apache Spark, leads to errors even when instantiating aSparkSession.pip install pyspark==3.3.4previously led to an error - the system was unable to build wheels for the package. Now, it seems to install that way, but behaves the same as in the previous case.- Trying to build the 3.3.4 PySpark package manually with
./build/mvnusing Bash from the appropriate directory led toCaused by: org.apache.maven.plugin.PluginExecutionException: Execution scala-compile-first of goal net.alchim31.maven:scala-maven-plu gin:4.4.0:compile failed.
Running this code after having installed this stuff as in case 3:
from pyspark.sql import SparkSession
from datetime import datetime, date
import pandas as pd
from pyspark.sql import Row
spark = SparkSession.builder.getOrCreate()
df = spark.createDataFrame([
Row(a=1, b=2., c='string1', d=date(2000, 1, 1), e=datetime(2000, 1, 1, 12, 0)),
Row(a=2, b=3., c='string2', d=date(2000, 2, 1), e=datetime(2000, 1, 2, 12, 0)),
Row(a=4, b=5., c='string3', d=date(2000, 3, 1), e=datetime(2000, 1, 3, 12, 0))
])
df.printSchema()
df.show()
leads to this:
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Traceback (most recent call last):
File "[python file path]", line 6, in <module>
spark = SparkSession.builder.getOrCreate()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "[python file path]", line 269, in getOrCreate
sc = SparkContext.getOrCreate(sparkConf)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "[python file path]", line 483, in getOrCreate
SparkContext(conf=conf or SparkConf())
File "[python file path]", line 197, in __init__
self._do_init(
File "[python file path]", line 282, in _do_init
self._jsc = jsc or self._initialize_context(self._conf._jconf)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "[python file path]", line 402, in _initialize_context
return self._jvm.JavaSparkContext(jconf)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "[python file path]", line 1585, in __call__
return_value = get_return_value(
^^^^^^^^^^^^^^^^^
File "[python file path]", line 326, in get_return_value
raise Py4JJavaError(
py4j.protocol.Py4JJavaError: An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext.
: java.lang.ExceptionInInitializerError
at org.apache.spark.unsafe.array.ByteArrayMethods.<clinit>(ByteArrayMethods.java:56)
at org.apache.spark.memory.MemoryManager.defaultPageSizeBytes$lzycompute(MemoryManager.scala:264)
at org.apache.spark.memory.MemoryManager.defaultPageSizeBytes(MemoryManager.scala:254)
at org.apache.spark.memory.MemoryManager.$anonfun$pageSizeBytes$1(MemoryManager.scala:273)
at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23)
at scala.Option.getOrElse(Option.scala:189)
at org.apache.spark.memory.MemoryManager.<init>(MemoryManager.scala:273)
at org.apache.spark.memory.UnifiedMemoryManager.<init>(UnifiedMemoryManager.scala:58)
at org.apache.spark.memory.UnifiedMemoryManager$.apply(UnifiedMemoryManager.scala:207)
at org.apache.spark.SparkEnv$.create(SparkEnv.scala:320)
at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:194)
at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:279)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:464)
at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62)
at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:238)
at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
at java.base/java.lang.Thread.run(Thread.java:1570)
Caused by: java.lang.IllegalStateException: java.lang.NoSuchMethodException: java.nio.DirectByteBuffer.<init>(long,int)
at org.apache.spark.unsafe.Platform.<clinit>(Platform.java:113)
... 25 more
Caused by: java.lang.NoSuchMethodException: java.nio.DirectByteBuffer.<init>(long,int)
at java.base/java.lang.Class.getConstructor0(Class.java:3784)
at java.base/java.lang.Class.getDeclaredConstructor(Class.java:2955)
at org.apache.spark.unsafe.Platform.<clinit>(Platform.java:71)
... 25 more
SUCCESS: The process with PID 21224 (child process of PID 9020) has been terminated.
SUCCESS: The process with PID 9020 (child process of PID 15684) has been terminated.
SUCCESS: The process with PID 15684 (child process of PID 4980) has been terminated.
Process finished with exit code 1
System environmental variables JAVA_HOME, HADOOP_HOME, SPARK_HOME are configured. The relevant binary directories are included in the Path system environmental variable.
PYTHON_SPARK is set to python.
EDIT: Great, and now Maven can't even attempt to build the package and throws the error
Error occurred during initialization of VM
Could not reserve enough space for 2097152KB object heap
Just great.
suspended an account I set up specifically for asking questions after I edited a relevant post.