-
Notifications
You must be signed in to change notification settings - Fork 260
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Document or improve scripts to make perf-java-flames work with docker #50
Comments
Looks like it's only creating the .data file and not .stacks or .collapsed
|
Can you try without |
I see that this is not particularly well documented... You could also try enabling |
I'm getting the same results without
Where is the resulting svg file supposed to turn up? |
In the same directory as the script is run but it should show a line with the name. |
Yeah it's definitely not showing up there. I'm going to try to create a Java FlameGraph with the manual steps. |
To eliminate some suspects:
|
|
Thanks! I managed to reproduce this issue locally and have a fix. I'll send a PR in a second, but it's very minor so if you can't wait you can go ahead and fix locally by applying the following:
Which just carries the JAVA_HOME through to the sudo command |
See PR #51 |
Please let me know if the fix helps your ultimate goal which is to get flame graphs. |
I suspect this fix will still fail for me, due to a bug in Java that requires me to dump the symbols as the user of the running Java process. But I will let you know! Also thanks for the jmaps link, but there's only one Java process on this machine. |
I added the fix but it still fails, even when running as root. New error though:
I get the same error when running as |
OK... Not seen that one before. For what it's worth my java executable has the exact same permissions. Running the script as root works for me if I setup the JAVA_HOME environment variable. We can work out an alternative, I think. The permissions game in the scripts is around 2 files, the map file and the perf.data file:
This will allow users to collect perf data for running processes.
Using the above I've setup boxes where users are allowed to perf profile their own Java processes with slightly modified scripts, essentially removing the sudo prefix everywhere and the file ownership manipulation. I've not attempted to merge these efforts back. |
Some security enforcement preventing alice from executing things? like seccomp? FWIW, my jmaps tool also works around the perf issue of needing the /tmp/perf*map files as owned by root. |
Apologies for going dark. I've been digging into this issue with the manual commands. The issue comes down to containers and namespacing. The Java process I'm trying to profile is running inside a container. The process is owned by a user inside the container, but only has a UID exposed to the host. Even spoofing a user with that UID on the host doesn't work when trying to dump symbols. And I can't do the profiling inside the container, because perf isn't installed and the version of Ubuntu running inside the container is too new for the underlying host kernel to have a supported perf package. My planned workaround (which I haven't verified, but I believe will work) is:
I think this will give me what I want. |
@alicegoldfuss Thanks for sharing your use case in more detail. I bow to your Linux Fu powers, sounds like you are on your way to cracking it, when you do please share the details. It would perhaps help to add this to the wiki. |
And the relevant blog post to go with the repo: |
Ah, looks like this person has come to the same conclusion as me! That's comforting :) |
@alicegoldfuss if nothing else I at least hope I've introduced you to the right person :-) |
Ah, right, containers and perf. I've been meaning to post a blog post too -- we've all probably been working on the same problem. :) Christopher's post is good, but he needs to let the JVM warm up a bit more -- too many "Interpreter" frames -- they haven't hit CompileThreshold yet. |
Ah, this is about containers. I also tried to get it working but only half-hearted. I would also be interested in getting this to work. Thanks for having the discussion here and the extra links, @alicegoldfuss, @nitsanw, and @brendangregg. |
Turned it into a blog post. Thanks everyone for your help: http://blog.alicegoldfuss.com/making-flamegraphs-with-containerized-java/ |
Fix #50 by carrying through observed JAVA_HOME to the sudo
Thanks a lot, @alicegoldfuss for documenting your findings! |
This is built on the following work: * jvm-profiling-tools/perf-map-agent#50 * http://batey.info/docker-jvm-flamegraphs.html * chbatey/perf-map-agent@60c50eb
I added transparent support for containers in |
To add to the "Why?" section of blog post by @alicegoldfuss, the reason seems to be this:
|
It’s a little unfortunate that this solution is Docker-specific (relies on
docker exec). Perhaps we can make it more general by using nsenter?
…On Wed, May 16, 2018 at 08:05 Ivan Babrou ***@***.***> wrote:
I added transparent support for containers in jmaps:
brendangregg/FlameGraph#171
<brendangregg/FlameGraph#171>.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#50 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ABm3vVdpi3fu_4QJGMKjWaEv31CJijXrks5ty7N-gaJpZM4Lsrjd>
.
|
@goldshtn I replied to your comment in the PR. Let's have PR related comments there. |
This is built on the following work: * jvm-profiling-tools/perf-map-agent#50 * http://batey.info/docker-jvm-flamegraphs.html * chbatey/perf-map-agent@60c50eb
This is built on the following work: * jvm-profiling-tools/perf-map-agent#50 * http://batey.info/docker-jvm-flamegraphs.html * chbatey/perf-map-agent@60c50eb
This is built on the following work: * jvm-profiling-tools/perf-map-agent#50 * http://batey.info/docker-jvm-flamegraphs.html * chbatey/perf-map-agent@60c50eb
I managed to run the attach script from the host namespace. I haven't properly integrated that as it needs some hacking of internals from tools.jar because as you say above the attach mechanism relies on well-known paths shared between attach and target JVM. Right now it will only work if the target process has PID 1 in the container. See jvm-profiling-tools/perf-map-agent/compare/jr/attach-to-container-from-host |
I'm trying to create a Java process FlameGraph with
perf-java-flames
. It seems to run successfully, but I can't find the resulting svg file.CentOS 7
3.10.0-327.36.3.el7.x86_64
cmake version 2.8.12.2
Up-to-date versions of perf-map-agent and the FlameGraph repo.
The text was updated successfully, but these errors were encountered: