Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Spark 2.0.0 support #115

Open
wants to merge 22 commits into
base: master
Choose a base branch
from
Open

Spark 2.0.0 support #115

Changes from 1 commit
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
5f090fc
Spark 2.0.0 support
a-roberts Jun 20, 2016
a6c3403
Fix dependency for mllib
a-roberts Jun 20, 2016
27e640f
Add 1.6 target folder back in
a-roberts Jun 20, 2016
5473506
Add ML changes for Spark 2, note requires Spark in lib folder currently!
a-roberts Jun 30, 2016
b1d1947
Add config file template back in
a-roberts Jun 30, 2016
cd7eb12
Add ability to override scala version and cleanup in mllib project file
a-roberts Jun 30, 2016
95d31d3
Add QA 1.6 fixes code back in
a-roberts Jun 30, 2016
8cb9e62
Remove .rej file too
a-roberts Jun 30, 2016
9e26cfc
Comment dep on 2.0.0 preview
a-roberts Jun 30, 2016
9cc8cae
Use original config.py but with SPARK_HOME recognised, default local …
a-roberts Jul 1, 2016
44f7a5f
Back to 3.2.9 for json native
a-roberts Jul 1, 2016
cf6d671
Tidying up
a-roberts Jul 1, 2016
9e171d4
Add default to 2.0 for spark mllib version
a-roberts Jul 1, 2016
b35c294
Use GA Spark 2 not preview
a-roberts Aug 15, 2016
533f6a6
Ensure we can override Spark version for all projects
a-roberts Aug 15, 2016
1eb427c
Add a missing comma for ml project file
a-roberts Aug 15, 2016
70963cf
Just use mllib provided artifact and remove core part
a-roberts Aug 26, 2016
1d1441b
Use 2.0.0 to resolve artifacts for mllib
a-roberts Aug 26, 2016
c873532
Remove feature-noise from glm regression so we can build and run
a-roberts Aug 26, 2016
e15a40e
Remove sparkVersion statement in SparkTestsBuild.scala
a-roberts Aug 26, 2016
0ad07c7
Remove spark version from streaming project file also
a-roberts Aug 26, 2016
e6cbaf9
Lower defaults, feature-noise if Spark 1.x only
a-roberts Sep 27, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion config/config.py.template
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -375,7 +375,7 @@ MLLIB_PERF_TEST_RUNNER = "mllib.perf.TestRunner"
# * Set `USE_CLUSTER_SPARK = True` and `MLLIB_SPARK_VERSION = {desired Spark version, e.g. 1.5}`
# * Don't use PREP_MLLIB_TESTS = True; instead manually run `cd mllib-tests; sbt/sbt -Dspark.version=1.5.0-SNAPSHOT clean assembly` to build perf tests

MLLIB_SPARK_VERSION = 2.0
MLLIB_SPARK_VERSION = 2.0.0
Copy link

@harschware harschware Oct 11, 2016 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My environment is a little customized by the time I run this code (so line numbers don't match), but I am seeing this:

Loading configuration from <snip>/spark-perf/config/config.py
Traceback (most recent call last):
  File "./bin/../lib/sparkperf/main.py", line 40, in <module>
    config = imp.load_source("config", "", cf)
  File "", line 410
    MLLIB_SPARK_VERSION = 2.0.0


MLLIB_JAVA_OPTS = COMMON_JAVA_OPTS
if MLLIB_SPARK_VERSION >= 1.1:
Expand Down