Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Get also mapreduce history service DelegationToken #197

Open
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

jdlesage
Copy link

This PR adds DelegationToken for the MapReduce History Server when submitting a new application

My use case
I call skein from a periodic job written in python to archive some files in HDFS using hadoop archive. The goal is to reduce the number of files in HDFS.
I call from a Skein Service the hadoop command hadoop archive that launches a Map Reduce job that creates the archive.
MapReduce jobs do some connections to the MapReduce History Server. It needs a delegation token to authenticate to this service. For this reason, I would like skein also to create a delegation token for this service.

Implementation details
The code is adapted from the hadoop mapreduce project. Unfortunately, I need some functions that are private. That's why I adapted the code in skein.
Original sourcefile: https://github.com/apache/hadoop/blob/branch-2.7.3/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java#L183
If the MapReduce History Server address is not present in the hadoop configuration files. The code does nothing.

How did I test it ?
I tested it by running a skein service that do a subprocess.check_output that triggers the hadoop archive command.

@jdlesage
Copy link
Author

The tests are failing because in the hadoop cluster used for test, the MR History Service is configured in mapred.xml but not started with the cluster
I did also a PR in hadoop-test-cluster project to start it also.

Introduce optional boolean in applicationspec

Change-Id: I5afa12e538629182dd7dbc6fe587925fb4d59557
@jdlesage
Copy link
Author

I added an option to request the new token. I consider it is safer because most of people doesn't care about map reduce history server, and I don't want to change the behavior in this case or break their apps.

Change-Id: Ife97067d5a0d57299092d97bf1863aef8dbfdfe0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant