Is it possible to override the EMR URL programmatically?

[agrounds]

My company has different EMR clusters for different environments, so I sometimes need to run spark workloads on multiple EMR clusters simultaneously. Since sparkmagic uses a single static ~/.sparkmagic/config.json file to set the EMR URL, I have to manually edit this file to change the host each time I want to switch environments.

I'm hoping there's a better way to set the EMR URL, ideally on a per-notebook basis. I tried using the %%configure magic command, but it doesn't seem to support this. Is there another way?

[dosubot[bot]]

Answer generated by a 🤖

Answer

I understand that you're looking for a way to programmatically override the EMR URL in sparkmagic, as you're working with multiple EMR clusters across different environments. You're hoping to find a solution that allows you to set the EMR URL on a per-notebook basis, rather than having to manually edit the ~/.sparkmagic/config.json file each time you switch environments.

One potential solution to this issue is to use the _do_not_call_change_endpoint line magic function. This function allows you to change the endpoint URL programmatically. It takes the URL of the server as an argument and sets it as the new endpoint. Here's an example of how you can use this function:

%_do_not_call_change_endpoint -s <your_emr_url>

In the above command, replace <your_emr_url> with the URL of your EMR cluster. However, please note that you cannot change the endpoint if a session has already been started.

It's important to note that the function name _do_not_call_change_endpoint suggests that it's not intended for direct use. However, based on your requirements, it seems to be the only way to change the endpoint programmatically at the moment.

While this solution may not be ideal, it could serve as a workaround until a more user-friendly way to change the endpoint is implemented. If you find this solution helpful and believe it could benefit others, you might consider contributing to the sparkmagic project by making a pull request with this information.

Sources

• sparkmagic/sparkmagic/kernels/kernelmagics.py

---

This response is meant to be useful, save you time, and share context. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

[agrounds]

Sparkmagic honors the SPARKMAGIC_CONF_DIR and SPARKMAGIC_CONF_FILE environment variables when loading its configuration file. For my use case, I created two directories of notebooks and automatically set the SPARKMAGIC_CONF_FILE to the appropriate config file for each. This way, when starting jupyter lab in each directory, the correct sparkmagic config will be read and the correct endpoint will be used.