Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Alternate means of configuration #10

Open
barend-xebia opened this issue Dec 9, 2024 · 0 comments
Open

Alternate means of configuration #10

barend-xebia opened this issue Dec 9, 2024 · 0 comments
Labels
zero priority Not sure if this is needed, delay implementation until there's an actual need.

Comments

@barend-xebia
Copy link
Contributor

At the moment, all configuration is read from the SparkConf, and keys are expected to have the spark. prefix that's required by spark-submit (but not SparkConf itself). In similar libraries elsewhere, it's often supported to read from multiple configuration sources, e.g.:

  1. Look for spark.com.xebia.data.spot.some-key in SparkConf.
  2. Look for com.xebia.data.spot.some-key in SparkConf.
  3. Look for spot.some-key in JVM System Properties.
  4. Look for SPOT_SOME_KEY in OS environment variables.

These should be tried in a documented sequence, with a documented conflict-resolution (such as first hit wins, or perhaps env-var as override).
@barend-xebia barend-xebia added the zero priority Not sure if this is needed, delay implementation until there's an actual need. label Dec 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
zero priority Not sure if this is needed, delay implementation until there's an actual need.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@barend-xebia