Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update apache-spark benchmarks to use Spark 3.2.0 and Scala 2.13.7 #327

Merged
merged 2 commits into from
Nov 22, 2021
Merged

Update apache-spark benchmarks to use Spark 3.2.0 and Scala 2.13.7 #327

merged 2 commits into from
Nov 22, 2021

Conversation

lbulej
Copy link
Member

@lbulej lbulej commented Nov 9, 2021

Benchmark code remains unchanged apart from relaxing imports for implicit conversions to Java collections and few other updates related to deprecations in Scala 2.13.

@lbulej
Update apache-spark benchmarks to use Spark 3.2.0 and Scala 2.13.7
f37413e 
Benchmark code remains unchanged apart from relaxing imports for
implicit conversions to Java collections and few other updates
related to deprecations in Scala.
@lbulej lbulej mentioned this pull request Nov 9, 2021
Open
5 tasks
@lbulej
Copy link
Member Author

lbulej commented Nov 16, 2021

FYI, I'm waiting with the merge because I would like to update the Hadoop Windows binaries found at https://github.com/cdarlint/winutils. So far there is no winutils.exe and hadoop.dll for Hadoop 3.3 which Spark 3.2 is built against. The latest available binaries are for Hadoop 3.2.2 and the version we currently have in Renaissance is for Hadoop 3.2.0. I don't know how much of a problem it is.

@farquet
Copy link
Collaborator

farquet commented Nov 22, 2021

Regarding the Windows support, can't we use the builds linked in the issue here? -> cdarlint/winutils#12

@lbulej
Copy link
Member Author

lbulej commented Nov 22, 2021

Regarding the Windows support, can't we use the builds linked in the issue here? -> cdarlint/winutils#12

Probably yes, the builds at https://github.com/kontext-tech/winutils are forked from the https://github.com/cdarlint/winutils repository and add 3.3.x versions, so I guess we can move on here.

@lbulej
Update winutils.exe and hadoop.dll for Hadoop 3.3.0
fd9db44 
Also updates the related comments so that version-specific details
are close to the code that needs to be updated.
@lbulej lbulej merged commit 21732ea into master Nov 22, 2021
@lbulej lbulej mentioned this pull request Nov 22, 2021
Closed
@lbulej lbulej deleted the topic/update-spark-scala branch November 23, 2021 08:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@farquet farquet farquet approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@lbulej @farquet