[JENKINS-68126] Adding optional Pod/Container creation logs

[MattLud]

This adds back as an optional configuration to have pod/container logs stream to Jenkins during creation - it was removed in #1192

This provides feedback from the Kubernetes API on where/what the scheduling side is doing and can inform users if their pod isn't executed yet because of resource constraints, container image issues or other problems. Without it, users are left to dig up or contact cluster administrators to get more information.

Given we don't have a performance metric to track the load on the API server from this, making an optional configuration seems acceptable. From our brief testing of 300 simultaneous jobs being scheduled and executing, we did not see a degradation in enqueuing or pod spinup times.

Let me know if you'd prefer a new JIRA for this work.

• Make sure you are opening from a topic/feature/bugfix branch (right side) and not your main branch!
• Ensure that the pull request title represents the desired changelog entry
• Please describe what you did
• [x ] Link to relevant issues in GitHub or Jira
• Link to relevant pull requests, esp. upstream and downstream changes
• Ensure you have provided tests - that demonstrates feature works or fixes the issue

[amuniz]

IMO adding this back is not desired. I think the same outcome can be achieved relying on the Reaper.Listener extension point.

[MattLud]

@amuniz If so, can you offer pointers on implementing it?

I've put this behind a configuration element that is false by default so that current behavior is used. It's removal is going to cause lots of users asking why their CI jobs are not building.

[amuniz]

can you offer pointers on implementing it?

The Reaper.Listener extension will get all build pod events, so I think PodUtils.getTerminatedContainers(pod) could be used to detect terminated containers and log accordingly. It would go to the Jenkins log, not the pod launch logger (via TaskListener), so the user might or might not have access to it. In order to log it into the pod launch logger a more complex implementation would be needed, maybe keeping a map of SlaveComputer -> TaskListener and looking up for the right logger on each update, not sure how this would scale either.

I've put this behind a configuration element that is false by default

IMO, having a feature (even if it's disabled by default) that it's known to cause issues at scale is not a good thing.

I'll let other maintainers to chime in /cc @Vlatombe @jglick

[Vlatombe]

I have plans to re-introduce this feature but it needs to be scalable.

[danzehme]

Has there been any activity on this or any replacement that I can follow? Not being able to see why a pod failed to create in the logs of that job has been a big problem for us.