Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Don't show spark UI and Application ID when using Spark 3.5.0 #2

Open
duc-dn opened this issue Jul 12, 2024 · 21 comments
Open

Don't show spark UI and Application ID when using Spark 3.5.0 #2

duc-dn opened this issue Jul 12, 2024 · 21 comments

Comments

@duc-dn
Copy link

duc-dn commented Jul 12, 2024 •
edited
Loading

Hi @askhatri, I built a snapshot master branch of LIVY with spark 3.5.0
After I replace them in helm chart, it worked well. However, when I checked LIVY UI, it doesn't show spark ui and application id.
I checked endpoint get /sessions, seem to not get info of spark session

{
    "id": 1,
    "name": null,
    "appId": null,
    "owner": null,
    "proxyUser": null,
    "state": "idle",
    "kind": "pyspark",
    "appInfo": {
        "driverLogUrl": null,
        "sparkUiUrl": null
    },
    "log": [
        "24/07/12 08:39:50 INFO LoggingPodStatusWatcherImpl: Application status for spark-c421e5052f844570bc681a897692dd91 (phase: Running)",
        "24/07/12 08:39:51 INFO LoggingPodStatusWatcherImpl: Application status for spark-c421e5052f844570bc681a897692dd91 (phase: Running)",
        "24/07/12 08:39:52 INFO LoggingPodStatusWatcherImpl: Application status for spark-c421e5052f844570bc681a897692dd91 (phase: Running)",
        "24/07/12 08:39:53 INFO LoggingPodStatusWatcherImpl: Application status for spark-c421e5052f844570bc681a897692dd91 (phase: Running)",
        "24/07/12 08:39:54 INFO LoggingPodStatusWatcherImpl: Application status for spark-c421e5052f844570bc681a897692dd91 (phase: Running)",
        "24/07/12 08:39:55 INFO LoggingPodStatusWatcherImpl: Application status for spark-c421e5052f844570bc681a897692dd91 (phase: Running)",
        "24/07/12 08:39:56 INFO LoggingPodStatusWatcherImpl: Application status for spark-c421e5052f844570bc681a897692dd91 (phase: Running)",
        "24/07/12 08:39:57 INFO LoggingPodStatusWatcherImpl: Application status for spark-c421e5052f844570bc681a897692dd91 (phase: Running)",
        "24/07/12 08:39:58 INFO LoggingPodStatusWatcherImpl: Application status for spark-c421e5052f844570bc681a897692dd91 (phase: Running)",
        "24/07/12 08:39:59 INFO LoggingPodStatusWatcherImpl: Application status for spark-c421e5052f844570bc681a897692dd91 (phase: Running)"
    ],
    "ttl": null,
    "idleTimeout": null,
    "driverMemory": null,
    "driverCores": 0,
    "executorMemory": null,
    "executorCores": 0,
    "archives": [],
    "files": [],
    "heartbeatTimeoutInSecond": 0,
    "jars": [],
    "numExecutors": 0,
    "pyFiles": [],
    "queue": null
}

I forward port of spark driver and can see spark UI of session.
In addition, when deploying with your image, I still see sparkUI and applicationID (build image following in Docker.md)
So Is this due to Livy Server (currently) not being compatible with Spark 3.5.0?
Thanks
@askhatri
Copy link
Owner

Hi @duc-dn , Which version of JRE/JDK are you using? Livy Server is currently not compatible with JDK 17. Spark 3.5.0 Docker images are built using JDK 17, and Livy is not compatible with JDK 17.

@duc-dn
Copy link
Author

duc-dn commented Jul 12, 2024 •
edited
Loading

Below image is the version that I built snapshot LIVY
image

I only change versions of SPARK and HADOOP in pom file
And spark image is: apache/spark:3.5.0-scala2.12-java11-python3-r-ubuntu
I find that they are the same java version

@askhatri
Copy link
Owner

Okay, I will test with Spark 3.5.0 and share my findings.

@duc-dn
Copy link
Author

duc-dn commented Jul 15, 2024 •
edited
Loading

Hi again. Do you have any updates about above problem?
Besides, I have a curious problem. I find docs in Livy

By default Livy is built against Apache Spark 2.4.5, but the version of Spark used when running Livy does not need to match the version used to build Livy. 
Livy internally handles the differences between different Spark versions.
The Livy package itself does not contain a Spark distribution. It will work with any supported version of Spark without needing to rebuild.

So follow you, when building Livy with spark 3.2.3, can it run with Spark 3.5.0?
I tried running basic command pyspark with Livy image: asifkhatri/livy:spark3.2.3, spark: apache/spark:3.5.0-scala2.12-java11-python3-r-ubuntu.
It works well and I can see spark ui link in LIVY UI.

@askhatri
Copy link
Owner

Hi @duc-dn , I am still working on it. Livy image: asifkhatri/livy:spark3.2.3 with spark: apache/spark:3.5.0-scala2.12-java11-python3-r-ubuntu might work. We need to build a new Livy image with spark 3.5.0 for spark: apache/spark:3.5.0-scala2.12-java11-python3-r-ubuntu.

@askhatri
Copy link
Owner

I have updated the Helm chart to support Spark 3.5.0.

You can use this repository for testing.

To build your own Docker image, refer to this commit in the incubator-livy repository.

Use the following command to build the Livy code:

mvn -e -Pthriftserver -Pscala-2.12 -Pspark3 -Phadoop3 -pl -:livy-python-api clean install

For the Docker image, use the following Dockerfile:

FROM apache/spark:3.5.0-scala2.12-java11-python3-r-ubuntu

ENV LIVY_VERSION=0.9.0-incubating-SNAPSHOT
ENV LIVY_PACKAGE=apache-livy-${LIVY_VERSION}_2.12-bin
ENV LIVY_HOME=/opt/livy
ENV LIVY_CONF_DIR=/conf
ENV PATH=$PATH:$LIVY_HOME/bin

USER root

COPY $LIVY_PACKAGE.zip /

RUN apt-get update && apt-get install -y unzip && \
    unzip /$LIVY_PACKAGE.zip -d / && \
    mv /$LIVY_PACKAGE /opt/ && \
    rm -rf $LIVY_HOME && \
    ln -s /opt/$LIVY_PACKAGE $LIVY_HOME && \
    rm -f /$LIVY_PACKAGE.zip

RUN mkdir /var/log/livy && \
    ln -s /var/log/livy $LIVY_HOME/logs

WORKDIR $LIVY_HOME

ENTRYPOINT ["livy-server"]

@duc-dn
Copy link
Author

duc-dn commented Jul 17, 2024

Hi @askhatri, I cloned this branch: https://github.com/askhatri/incubator-livy/commit/980a78db4ccae6e44afa960762b266e65ac38c4a and built in local using command: mvn -e -Pthriftserver -Pscala-2.12 -Pspark3 -Phadoop3 -pl -:livy-python-api clean install
However, I am facing errors:

[ERROR] Tests run: 19, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 29.218 s <<< FAILURE! - in org.apache.livy.rsc.TestSparkClient
[ERROR] testSparkSQLJob(org.apache.livy.rsc.TestSparkClient)  Time elapsed: 5.263 s  <<< ERROR!
java.util.concurrent.ExecutionException: 
java.lang.RuntimeException: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in stage 0.0 (TID 0) (192.168.1.13 executor driver): java.lang.NoClassDefFoundError: com/fasterxml/jackson/core/StreamReadConstraints
        at org.apache.spark.sql.catalyst.json.JSONOptions.buildJsonFactory(JSONOptions.scala:194)
        at org.apache.spark.sql.catalyst.json.JsonInferSchema.$anonfun$infer$1(JsonInferSchema.scala:83)
        at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2(RDD.scala:855)
        at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2$adapted(RDD.scala:855)
        at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:364)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:328)

@askhatri
Copy link
Owner

Hi @duc-dn , It appears that there are some test failures. I will work on fixing them. In the meantime, you can use the -DskipITs -DskipTests flag to skip the tests as a workaround.

@askhatri
Copy link
Owner

Hi @duc-dn, I have tested the build and found that there is a build error. You can check the detailed logs at the following link: log link.

@duc-dn
Copy link
Author

duc-dn commented Jul 18, 2024 •
edited
Loading

@askhatri, which version mvn and java do you use to build LIVY?
I'm using mvn 3.6.3, java 11.0.23. If not skipping testing, when testing with spark client, I faced errors in the log file
org.apache.livy.rsc.TestSparkClient-output.txt

@askhatri
Copy link
Owner

I am using "Apache Maven 3.3.9" and "Java version: 1.8.0_292". Same configuration that is used at https://github.com/apache/incubator-livy/actions/runs/9871334539/job/27259040026.

@duc-dn
Copy link
Author

duc-dn commented Jul 18, 2024

thanks @askhatri, let me check

@duc-dn
Copy link
Author

duc-dn commented Jul 18, 2024

@askhatri, I changed mvn 3.3.9 java1.8.0 but I faced same above error

java.lang.NoClassDefFoundError: com/fasterxml/jackson/core/StreamReadConstraints

So is this problem due to the version of jackson?? I don't know why you don't face this problem
I tried changing jackson version to 2.15.2 following on StackOverflow
log.txt

@askhatri
Copy link
Owner

I'm not sure about the Jackson version. This is new to me.

@duc-dn
Copy link
Author

duc-dn commented Jul 26, 2024 •
edited
Loading

Hi @askhatri, I found your latest update. Thanks
I still have the problem with auth livy from spark magic using LDAP.
I added additional config in livy.conf (configmap)

livy.server.auth.type=ldap
livy.server.auth.ldap.url=ldap://localhost:389
livy.server.auth.ldap.base-dn = ou=livy,dc=example,dc=org
livy.server.auth.ldap.username-domain = example.org
livy.server.auth.ldap.enable-start-tls=false
livy.server.auth.ldap.security-authentication=simple

In log of livy server. I don't see raise any error about LDAP, but livy server is restarted continuously

24/07/26 09:52:30 INFO LivyServer: LDAP auth enabled.
24/07/26 09:52:30 INFO WebServer: Starting server on http://livytest-livycluster-0.livytest-livycluster-headless.livy.svc.cluster.local:8998
24/07/26 09:52:31 DEBUG IOStatisticsContextIntegration: Created instance IOStatisticsContextImpl{id=3, threadId=28, ioStatistics=counters=();
gauges=();
minimums=();
maximums=();
means=();
}
24/07/26 09:52:35 DEBUG AuthenticationFilter: Got token null from httpRequest http://10.1.8.205:8998/version
24/07/26 09:52:35 DEBUG AuthenticationFilter: Request [http://10.1.8.205:8998/version] triggering authentication. handler: class org.apache.livy.server.auth.LdapAuthenticationHandlerImpl
24/07/26 09:52:38 DEBUG AuthenticationFilter: Got token null from httpRequest http://10.1.8.205:8998/version
24/07/26 09:52:38 DEBUG AuthenticationFilter: Request [http://10.1.8.205:8998/version] triggering authentication. handler: class org.apache.livy.server.auth.LdapAuthenticationHandlerImpl
24/07/26 09:52:45 DEBUG AuthenticationFilter: Got token null from httpRequest http://10.1.8.205:8998/version
24/07/26 09:52:45 DEBUG AuthenticationFilter: Request [http://10.1.8.205:8998/version] triggering authentication. handler: class org.apache.livy.server.auth.LdapAuthenticationHandlerImpl
24/07/26 09:52:46 DEBUG SparkKubernetesApp: getApplicationReport, applicationId: spark-0f59c651679748de9444a35a32fd1cd7, namespace: livy applicationTag: livy-batch-2-82hd2jb1
24/07/26 09:52:46 DEBUG DefaultSharedIndexInformer: Ready to run resync and reflector for v1/namespaces/livy/pods with resync 0
24/07/26 09:52:46 DEBUG DefaultSharedIndexInformer: Resync skipped due to 0 full resync period for v1/namespaces/livy/pods
24/07/26 09:52:46 DEBUG Reflector: Listing items (1) for v1/namespaces/livy/pods at v762663
24/07/26 09:52:46 DEBUG BatchSession: BatchSession 2 state changed from STARTING to FINISHED
24/07/26 09:52:46 DEBUG SparkKubernetesApp: spark-0f59c651679748de9444a35a32fd1cd7 FINISHED testbatch-a397cf90ee3f8789-driver.livy: 	node: docker-desktop 	hostname: null 	podIp: null 	startTime: 2024-07-26T08:54:20Z 	phase: Succeeded 	reason: null 	message: null 	labels: name=driver, spark-app-name=testbatch, spark-version=3.5.0, spark-role=driver, spark-app-selector=spark-0f59c651679748de9444a35a32fd1cd7, spark-app-tag=livy-batch-2-82hd2jb1 	containers: 		spark-kubernetes-driver: 			image: apache/spark:3.5.0-scala2.12-java11-python3-r-ubuntu 			requests: cpu=1, memory=1408Mi 			limits: memory=1408Mi 			command: [] [driver, --properties-file, /opt/spark/conf/spark.properties, --class, org.apache.spark.examples.SparkPi, local:///opt/spark/examples/jars/spark-examples_2.12-3.5.0.jar, 10000] 	conditions: 		PodCondition(lastProbeTime=null, lastTransitionTime=2024-07-26T08:55:55Z, message=null, reason=null, status=False, type=PodReadyToStartContainers, additionalProperties={}) 		PodCondition(lastProbeTime=null, lastTransitionTime=2024-07-26T08:54:20Z, message=null, reason=PodCompleted, status=True, type=Initialized, additionalProperties={}) 		PodCondition(lastProbeTime=null, lastTransitionTime=2024-07-26T08:55:54Z, message=null, reason=PodCompleted, status=False, type=Ready, additionalProperties={}) 		PodCondition(lastProbeTime=null, lastTransitionTime=2024-07-26T08:55:54Z, message=null, reason=PodCompleted, status=False, type=ContainersReady, additionalProperties={}) 		PodCondition(lastProbeTime=null, lastTransitionTime=2024-07-26T08:54:20Z, message=null, reason=null, status=True, type=PodScheduled, additionalProperties={})
24/07/26 09:52:48 DEBUG AuthenticationFilter: Got token null from httpRequest http://10.1.8.205:8998/version
24/07/26 09:52:48 DEBUG AuthenticationFilter: Request [http://10.1.8.205:8998/version] triggering authentication. handler: class org.apache.livy.server.auth.LdapAuthenticationHandlerImpl
24/07/26 09:52:55 DEBUG AuthenticationFilter: Got token null from httpRequest http://10.1.8.205:8998/version
24/07/26 09:52:55 DEBUG AuthenticationFilter: Request [http://10.1.8.205:8998/version] triggering authentication. handler: class org.apache.livy.server.auth.LdapAuthenticationHandlerImpl
24/07/26 09:52:55 DEBUG AuthenticationFilter: Got token null from httpRequest http://10.1.8.205:8998/version
24/07/26 09:52:55 DEBUG AuthenticationFilter: Request [http://10.1.8.205:8998/version] triggering authentication. handler: class org.apache.livy.server.auth.LdapAuthenticationHandlerImpl
24/07/26 09:52:55 INFO LivyServer: Shutting down Livy server.
24/07/26 09:52:55 DEBUG FileSystem: FileSystem.close() by method: org.apache.hadoop.fs.FilterFileSystem.close(FilterFileSystem.java:529)); Key: (root (auth:SIMPLE))@file://; URI: file:///; Object Identity Hash: e82c2ff
24/07/26 09:52:55 DEBUG FileSystem: FileSystem.close() by method: org.apache.hadoop.fs.RawLocalFileSystem.close(RawLocalFileSystem.java:895)); Key: null; URI: file:///; Object Identity Hash: 453d5f
24/07/26 09:52:55 DEBUG ShutdownHookManager: Completed shutdown in 0.004 seconds; Timeouts: 0
24/07/26 09:52:55 DEBUG ShutdownHookManager: ShutdownHookManager completed shutdown.

@duc-dn
Copy link
Author

duc-dn commented Jul 26, 2024

This is status of Pod. Can you please view this issue help me?
image

@askhatri
Copy link
Owner

I haven't integrated LDAP with Livy yet, but I'll attempt to do so and validate it. We'll need a running LDAP server and will configure Livy to use it with livy.server.auth.ldap.url=ldap://<LDAP_SERVER_HOST>:<LDAP_PORT>. Since the LDAP server might not be on the same pod as Livy, we can't use localhost. We'll need to replace <LDAP_SERVER_HOST> with the actual LDAP server's hostname or IP address.

@duc-dn
Copy link
Author

duc-dn commented Jul 27, 2024

Sorry I copied it wrongly as localhost. I deployed LDAP in same namespace with Livy Server
After, I executed livy pod and telnet to ldap and make sure it can communicate with ldap

@askhatri
Copy link
Owner

ok @duc-dn , I will also try the same from my end.

@duc-dn
Copy link
Author

duc-dn commented Aug 9, 2024 •
edited
Loading

Hi again @askhatri. Do you have any updates about LDAP?
Additionally, I am concerned about scale livy server therefore I tried to scale up 2 pod livy server
but faced an error when starting a spark session.
So can it scale up for livy server??

@askhatri
Copy link
Owner

askhatri commented Aug 9, 2024

Hi @duc-dn , I’m still working on the LDAP integration and will update you as soon as I have any progress to share. Currently, the Livy server doesn’t support scaled instances, so we’ll need to implement this high availability (HA) feature for Livy.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@duc-dn @askhatri