Stop memory consumption increasing linearly with time

[rahmans1]

Is your feature request related to a problem? Please describe.
For a wide range of datasets in simulation production eicrecon shows linear increase in memory consumption over time. The plot below demonstrates an example of this issues by comparing the memory consumption over time in eicrecon vs npsim for one of these datasets.

pythia8CCDIS.pdf

Describe the solution you'd like
We would prefer the rss to be under 2 GB in eicrecon for the duration of the run. This will allow us to achieve better throughput in Open Science Grid resources. The first step would be to identify why the memory consumption increases linearly without any obvious signs of memory leak.

Describe alternatives you've considered

Additional context

[wdconinc]

If we instrument the eicrecon-dis job here with prmon, does it show the same behavior? I.e. can we set something up that tracks this and flags degradation?

[wdconinc]

Entirely anecdotal, but I don't notice this on my own system which has a few differences from the containers we're running here, notably jana2@master and [email protected].

[veprbl]

I'm not able to reproduce with my local setup either (JANA2 2.2.1-rc1, podio 0.99).

    GB
1.896^                                     ##                                 
     |                   ::@:  ::::::::::::# ::@@:::::::::::::::::::  :::@::: 
     |          ::::@:@@:: @::::: : : : :: # : @ :: ::: :: ::: :: ::::: :@: ::
     |          :: :@:@ :: @:: :: : : : :: # : @ :: ::: :: ::: :: ::: : :@: ::
     |       ::::: :@:@ :: @:: :: : : : :: # : @ :: ::: :: ::: :: ::: : :@: ::
     |       :: :: :@:@ :: @:: :: : : : :: # : @ :: ::: :: ::: :: ::: : :@: ::
     |      ::: :: :@:@ :: @:: :: : : : :: # : @ :: ::: :: ::: :: ::: : :@: ::
     |      ::: :: :@:@ :: @:: :: : : : :: # : @ :: ::: :: ::: :: ::: : :@: ::
     |      ::: :: :@:@ :: @:: :: : : : :: # : @ :: ::: :: ::: :: ::: : :@: ::
     |      ::: :: :@:@ :: @:: :: : : : :: # : @ :: ::: :: ::: :: ::: : :@: ::
     |      ::: :: :@:@ :: @:: :: : : : :: # : @ :: ::: :: ::: :: ::: : :@: ::
     |    @@::: :: :@:@ :: @:: :: : : : :: # : @ :: ::: :: ::: :: ::: : :@: ::
     |    @ ::: :: :@:@ :: @:: :: : : : :: # : @ :: ::: :: ::: :: ::: : :@: ::
     |    @ ::: :: :@:@ :: @:: :: : : : :: # : @ :: ::: :: ::: :: ::: : :@: ::
     |   @@ ::: :: :@:@ :: @:: :: : : : :: # : @ :: ::: :: ::: :: ::: : :@: ::
     |   @@ ::: :: :@:@ :: @:: :: : : : :: # : @ :: ::: :: ::: :: ::: : :@: ::
     |   @@ ::: :: :@:@ :: @:: :: : : : :: # : @ :: ::: :: ::: :: ::: : :@: ::
     |   @@ ::: :: :@:@ :: @:: :: : : : :: # : @ :: ::: :: ::: :: ::: : :@: ::
     |::@@@ ::: :: :@:@ :: @:: :: : : : :: # : @ :: ::: :: ::: :: ::: : :@: ::
     |: @@@ ::: :: :@:@ :: @:: :: : : : :: # : @ :: ::: :: ::: :: ::: : :@: ::
   0 +----------------------------------------------------------------------->Gi
     0                                                                   635.0

@rahmans1 Can you run with valgrind --tool=massif --massif-out-file=massif.out?

[veprbl]

That was for single particle ^^

I ran a long job over pythia8CCDIS_10x100_minQ2=1000_beamEffects_xAngle=-0.025_hiDiv_1.hepmc3.tree.root

    GB
1.914^                                                                       :
     |      @@@ :::::::  :::::@@@@:@@@@:::::::::::@@::::::::::::::::#:::::::@@
     |  ::::@@ :: :: : :::: ::@@ @:@ @ : ::: : : :@ ::: :::: :: : ::#: :::::@@
     | :: : @@ :: :: : : :: ::@@ @:@ @ : ::: : : :@ ::: :::: :: : ::#: :::::@@
     | :: : @@ :: :: : : :: ::@@ @:@ @ : ::: : : :@ ::: :::: :: : ::#: :::::@@
     | :: : @@ :: :: : : :: ::@@ @:@ @ : ::: : : :@ ::: :::: :: : ::#: :::::@@
     | :: : @@ :: :: : : :: ::@@ @:@ @ : ::: : : :@ ::: :::: :: : ::#: :::::@@
     | :: : @@ :: :: : : :: ::@@ @:@ @ : ::: : : :@ ::: :::: :: : ::#: :::::@@
     | :: : @@ :: :: : : :: ::@@ @:@ @ : ::: : : :@ ::: :::: :: : ::#: :::::@@
     | :: : @@ :: :: : : :: ::@@ @:@ @ : ::: : : :@ ::: :::: :: : ::#: :::::@@
     | :: : @@ :: :: : : :: ::@@ @:@ @ : ::: : : :@ ::: :::: :: : ::#: :::::@@
     | :: : @@ :: :: : : :: ::@@ @:@ @ : ::: : : :@ ::: :::: :: : ::#: :::::@@
     | :: : @@ :: :: : : :: ::@@ @:@ @ : ::: : : :@ ::: :::: :: : ::#: :::::@@
     | :: : @@ :: :: : : :: ::@@ @:@ @ : ::: : : :@ ::: :::: :: : ::#: :::::@@
     | :: : @@ :: :: : : :: ::@@ @:@ @ : ::: : : :@ ::: :::: :: : ::#: :::::@@
     | :: : @@ :: :: : : :: ::@@ @:@ @ : ::: : : :@ ::: :::: :: : ::#: :::::@@
     | :: : @@ :: :: : : :: ::@@ @:@ @ : ::: : : :@ ::: :::: :: : ::#: :::::@@
     | :: : @@ :: :: : : :: ::@@ @:@ @ : ::: : : :@ ::: :::: :: : ::#: :::::@@
     | :: : @@ :: :: : : :: ::@@ @:@ @ : ::: : : :@ ::: :::: :: : ::#: :::::@@
     | :: : @@ :: :: : : :: ::@@ @:@ @ : ::: : : :@ ::: :::: :: : ::#: :::::@@
   0 +----------------------------------------------------------------------->Ti
     0                                                                   9.501

Can't see a leak, still.

[veprbl]

Re-running with npsim in the 24.05.0-stable container, I'm able to reproduce:

    GB
2.399^                                                                      : 
     |                                                          @@@@@@:@@:##::
     |                                           @@@@@@@@@@::@::@ @ @ :@ :# ::
     |                                   ::::@@:@@ @@@ @ @ ::@: @ @ @ :@ :# ::
     |                        ::::@@::@@@::: @ :@@ @@@ @ @ ::@: @ @ @ :@ :# ::
     |           ::@@:::::::::::: @@: @ @::: @ :@@ @@@ @ @ ::@: @ @ @ :@ :# ::
     |     ::::::: @ :: :: :: ::: @@: @ @::: @ :@@ @@@ @ @ ::@: @ @ @ :@ :# ::
     |  :::: :: :: @ :: :: :: ::: @@: @ @::: @ :@@ @@@ @ @ ::@: @ @ @ :@ :# ::
     |  :: : :: :: @ :: :: :: ::: @@: @ @::: @ :@@ @@@ @ @ ::@: @ @ @ :@ :# ::
     |  :: : :: :: @ :: :: :: ::: @@: @ @::: @ :@@ @@@ @ @ ::@: @ @ @ :@ :# ::
     |  :: : :: :: @ :: :: :: ::: @@: @ @::: @ :@@ @@@ @ @ ::@: @ @ @ :@ :# ::
     |  :: : :: :: @ :: :: :: ::: @@: @ @::: @ :@@ @@@ @ @ ::@: @ @ @ :@ :# :@
     |  :: : :: :: @ :: :: :: ::: @@: @ @::: @ :@@ @@@ @ @ ::@: @ @ @ :@ :# :@
     |  :: : :: :: @ :: :: :: ::: @@: @ @::: @ :@@ @@@ @ @ ::@: @ @ @ :@ :# :@
     |  :: : :: :: @ :: :: :: ::: @@: @ @::: @ :@@ @@@ @ @ ::@: @ @ @ :@ :# :@
     |:::: : :: :: @ :: :: :: ::: @@: @ @::: @ :@@ @@@ @ @ ::@: @ @ @ :@ :# :@
     |: :: : :: :: @ :: :: :: ::: @@: @ @::: @ :@@ @@@ @ @ ::@: @ @ @ :@ :# :@
     |: :: : :: :: @ :: :: :: ::: @@: @ @::: @ :@@ @@@ @ @ ::@: @ @ @ :@ :# :@
     |: :: : :: :: @ :: :: :: ::: @@: @ @::: @ :@@ @@@ @ @ ::@: @ @ @ :@ :# :@
     |: :: : :: :: @ :: :: :: ::: @@: @ @::: @ :@@ @@@ @ @ ::@: @ @ @ :@ :# :@
   0 +----------------------------------------------------------------------->Ti
     0                                                                   3.226

ms_print massif.out | grep IrtCherenkovParticleID

| | ->04.21% (82,426,288B) 0x2D6EDFC7: eicrecon::IrtCherenkovParticleID::process(std::tuple<gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::RawTrackerHitCollection const*>, gsl::not_null<edm4eic::MCRecoTrackerHitAssociationCollection const*> > const&, std::tuple<gsl::not_null<edm4eic::CherenkovParticleIDCollection*>, gsl::not_null<edm4eic::CherenkovParticleIDCollection*> > const&) const (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/libalgorithms_pid.so)
| |   ->04.21% (82,426,288B) 0x5FB7AFD5: JOmniFactory<eicrecon::IrtCherenkovParticleID_factory, eicrecon::IrtCherenkovParticleIDConfig>::Process(std::shared_ptr<JEvent const> const&) (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/EICrecon/plugins/DRICH.so)
| | ->09.26% (196,608,608B) 0x2D6EDFC7: eicrecon::IrtCherenkovParticleID::process(std::tuple<gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::RawTrackerHitCollection const*>, gsl::not_null<edm4eic::MCRecoTrackerHitAssociationCollection const*> > const&, std::tuple<gsl::not_null<edm4eic::CherenkovParticleIDCollection*>, gsl::not_null<edm4eic::CherenkovParticleIDCollection*> > const&) const (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/libalgorithms_pid.so)
| |   ->09.26% (196,608,608B) 0x5FB7AFD5: JOmniFactory<eicrecon::IrtCherenkovParticleID_factory, eicrecon::IrtCherenkovParticleIDConfig>::Process(std::shared_ptr<JEvent const> const&) (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/EICrecon/plugins/DRICH.so)
| | ->09.81% (210,377,160B) 0x2D6EDFC7: eicrecon::IrtCherenkovParticleID::process(std::tuple<gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::RawTrackerHitCollection const*>, gsl::not_null<edm4eic::MCRecoTrackerHitAssociationCollection const*> > const&, std::tuple<gsl::not_null<edm4eic::CherenkovParticleIDCollection*>, gsl::not_null<edm4eic::CherenkovParticleIDCollection*> > const&) const (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/libalgorithms_pid.so)
| |   ->09.81% (210,377,160B) 0x5FB7AFD5: JOmniFactory<eicrecon::IrtCherenkovParticleID_factory, eicrecon::IrtCherenkovParticleIDConfig>::Process(std::shared_ptr<JEvent const> const&) (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/EICrecon/plugins/DRICH.so)
| | ->10.69% (231,974,064B) 0x2D6EDFC7: eicrecon::IrtCherenkovParticleID::process(std::tuple<gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::RawTrackerHitCollection const*>, gsl::not_null<edm4eic::MCRecoTrackerHitAssociationCollection const*> > const&, std::tuple<gsl::not_null<edm4eic::CherenkovParticleIDCollection*>, gsl::not_null<edm4eic::CherenkovParticleIDCollection*> > const&) const (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/libalgorithms_pid.so)
| |   ->10.69% (231,974,064B) 0x5FB7AFD5: JOmniFactory<eicrecon::IrtCherenkovParticleID_factory, eicrecon::IrtCherenkovParticleIDConfig>::Process(std::shared_ptr<JEvent const> const&) (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/EICrecon/plugins/DRICH.so)
| | ->11.17% (244,414,800B) 0x2D6EDFC7: eicrecon::IrtCherenkovParticleID::process(std::tuple<gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::RawTrackerHitCollection const*>, gsl::not_null<edm4eic::MCRecoTrackerHitAssociationCollection const*> > const&, std::tuple<gsl::not_null<edm4eic::CherenkovParticleIDCollection*>, gsl::not_null<edm4eic::CherenkovParticleIDCollection*> > const&) const (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/libalgorithms_pid.so)
| |   ->11.17% (244,414,800B) 0x5FB7AFD5: JOmniFactory<eicrecon::IrtCherenkovParticleID_factory, eicrecon::IrtCherenkovParticleIDConfig>::Process(std::shared_ptr<JEvent const> const&) (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/EICrecon/plugins/DRICH.so)
| | ->12.47% (281,128,512B) 0x2D6EDFC7: eicrecon::IrtCherenkovParticleID::process(std::tuple<gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::RawTrackerHitCollection const*>, gsl::not_null<edm4eic::MCRecoTrackerHitAssociationCollection const*> > const&, std::tuple<gsl::not_null<edm4eic::CherenkovParticleIDCollection*>, gsl::not_null<edm4eic::CherenkovParticleIDCollection*> > const&) const (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/libalgorithms_pid.so)
| |   ->12.47% (281,128,512B) 0x5FB7AFD5: JOmniFactory<eicrecon::IrtCherenkovParticleID_factory, eicrecon::IrtCherenkovParticleIDConfig>::Process(std::shared_ptr<JEvent const> const&) (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/EICrecon/plugins/DRICH.so)
| | ->13.27% (305,147,528B) 0x2D6EDFC7: eicrecon::IrtCherenkovParticleID::process(std::tuple<gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::RawTrackerHitCollection const*>, gsl::not_null<edm4eic::MCRecoTrackerHitAssociationCollection const*> > const&, std::tuple<gsl::not_null<edm4eic::CherenkovParticleIDCollection*>, gsl::not_null<edm4eic::CherenkovParticleIDCollection*> > const&) const (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/libalgorithms_pid.so)
| |   ->13.27% (305,147,528B) 0x5FB7AFD5: JOmniFactory<eicrecon::IrtCherenkovParticleID_factory, eicrecon::IrtCherenkovParticleIDConfig>::Process(std::shared_ptr<JEvent const> const&) (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/EICrecon/plugins/DRICH.so)
| | ->13.54% (314,657,336B) 0x2D6EDFC7: eicrecon::IrtCherenkovParticleID::process(std::tuple<gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::RawTrackerHitCollection const*>, gsl::not_null<edm4eic::MCRecoTrackerHitAssociationCollection const*> > const&, std::tuple<gsl::not_null<edm4eic::CherenkovParticleIDCollection*>, gsl::not_null<edm4eic::CherenkovParticleIDCollection*> > const&) const (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/libalgorithms_pid.so)
| |   ->13.54% (314,657,336B) 0x5FB7AFD5: JOmniFactory<eicrecon::IrtCherenkovParticleID_factory, eicrecon::IrtCherenkovParticleIDConfig>::Process(std::shared_ptr<JEvent const> const&) (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/EICrecon/plugins/DRICH.so)
| | ->13.99% (325,824,296B) 0x2D6EDFC7: eicrecon::IrtCherenkovParticleID::process(std::tuple<gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::RawTrackerHitCollection const*>, gsl::not_null<edm4eic::MCRecoTrackerHitAssociationCollection const*> > const&, std::tuple<gsl::not_null<edm4eic::CherenkovParticleIDCollection*>, gsl::not_null<edm4eic::CherenkovParticleIDCollection*> > const&) const (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/libalgorithms_pid.so)
| |   ->13.99% (325,824,296B) 0x5FB7AFD5: JOmniFactory<eicrecon::IrtCherenkovParticleID_factory, eicrecon::IrtCherenkovParticleIDConfig>::Process(std::shared_ptr<JEvent const> const&) (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/EICrecon/plugins/DRICH.so)
| | ->14.33% (336,300,216B) 0x2D6EDFC7: eicrecon::IrtCherenkovParticleID::process(std::tuple<gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::RawTrackerHitCollection const*>, gsl::not_null<edm4eic::MCRecoTrackerHitAssociationCollection const*> > const&, std::tuple<gsl::not_null<edm4eic::CherenkovParticleIDCollection*>, gsl::not_null<edm4eic::CherenkovParticleIDCollection*> > const&) const (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/libalgorithms_pid.so)
| |   ->14.33% (336,300,216B) 0x5FB7AFD5: JOmniFactory<eicrecon::IrtCherenkovParticleID_factory, eicrecon::IrtCherenkovParticleIDConfig>::Process(std::shared_ptr<JEvent const> const&) (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/EICrecon/plugins/DRICH.so)
| | ->14.59% (343,495,320B) 0x2D6EDFC7: eicrecon::IrtCherenkovParticleID::process(std::tuple<gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::RawTrackerHitCollection const*>, gsl::not_null<edm4eic::MCRecoTrackerHitAssociationCollection const*> > const&, std::tuple<gsl::not_null<edm4eic::CherenkovParticleIDCollection*>, gsl::not_null<edm4eic::CherenkovParticleIDCollection*> > const&) const (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/libalgorithms_pid.so)
| |   ->14.59% (343,495,320B) 0x5FB7AFD5: JOmniFactory<eicrecon::IrtCherenkovParticleID_factory, eicrecon::IrtCherenkovParticleIDConfig>::Process(std::shared_ptr<JEvent const> const&) (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/EICrecon/plugins/DRICH.so)
| | ->15.12% (358,452,752B) 0x2D6EDFC7: eicrecon::IrtCherenkovParticleID::process(std::tuple<gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::RawTrackerHitCollection const*>, gsl::not_null<edm4eic::MCRecoTrackerHitAssociationCollection const*> > const&, std::tuple<gsl::not_null<edm4eic::CherenkovParticleIDCollection*>, gsl::not_null<edm4eic::CherenkovParticleIDCollection*> > const&) const (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/libalgorithms_pid.so)
| |   ->15.12% (358,452,752B) 0x5FB7AFD5: JOmniFactory<eicrecon::IrtCherenkovParticleID_factory, eicrecon::IrtCherenkovParticleIDConfig>::Process(std::shared_ptr<JEvent const> const&) (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/EICrecon/plugins/DRICH.so)
| | ->15.67% (375,047,008B) 0x2D6EDFC7: eicrecon::IrtCherenkovParticleID::process(std::tuple<gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::RawTrackerHitCollection const*>, gsl::not_null<edm4eic::MCRecoTrackerHitAssociationCollection const*> > const&, std::tuple<gsl::not_null<edm4eic::CherenkovParticleIDCollection*>, gsl::not_null<edm4eic::CherenkovParticleIDCollection*> > const&) const (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/libalgorithms_pid.so)
| |   ->15.67% (375,047,008B) 0x5FB7AFD5: JOmniFactory<eicrecon::IrtCherenkovParticleID_factory, eicrecon::IrtCherenkovParticleIDConfig>::Process(std::shared_ptr<JEvent const> const&) (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/EICrecon/plugins/DRICH.so)
| | ->16.70% (407,772,904B) 0x2D6EDFC7: eicrecon::IrtCherenkovParticleID::process(std::tuple<gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::RawTrackerHitCollection const*>, gsl::not_null<edm4eic::MCRecoTrackerHitAssociationCollection const*> > const&, std::tuple<gsl::not_null<edm4eic::CherenkovParticleIDCollection*>, gsl::not_null<edm4eic::CherenkovParticleIDCollection*> > const&) const (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/libalgorithms_pid.so)
| |   ->16.70% (407,772,904B) 0x5FB7AFD5: JOmniFactory<eicrecon::IrtCherenkovParticleID_factory, eicrecon::IrtCherenkovParticleIDConfig>::Process(std::shared_ptr<JEvent const> const&) (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/EICrecon/plugins/DRICH.so)
| | ->17.37% (428,545,040B) 0x2D6EDFC7: eicrecon::IrtCherenkovParticleID::process(std::tuple<gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::RawTrackerHitCollection const*>, gsl::not_null<edm4eic::MCRecoTrackerHitAssociationCollection const*> > const&, std::tuple<gsl::not_null<edm4eic::CherenkovParticleIDCollection*>, gsl::not_null<edm4eic::CherenkovParticleIDCollection*> > const&) const (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/libalgorithms_pid.so)
| |   ->17.37% (428,545,040B) 0x5FB7AFD5: JOmniFactory<eicrecon::IrtCherenkovParticleID_factory, eicrecon::IrtCherenkovParticleIDConfig>::Process(std::shared_ptr<JEvent const> const&) (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/EICrecon/plugins/DRICH.so)
| | ->17.77% (441,215,152B) 0x2D6EDFC7: eicrecon::IrtCherenkovParticleID::process(std::tuple<gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::RawTrackerHitCollection const*>, gsl::not_null<edm4eic::MCRecoTrackerHitAssociationCollection const*> > const&, std::tuple<gsl::not_null<edm4eic::CherenkovParticleIDCollection*>, gsl::not_null<edm4eic::CherenkovParticleIDCollection*> > const&) const (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/libalgorithms_pid.so)
| |   ->17.77% (441,215,152B) 0x5FB7AFD5: JOmniFactory<eicrecon::IrtCherenkovParticleID_factory, eicrecon::IrtCherenkovParticleIDConfig>::Process(std::shared_ptr<JEvent const> const&) (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/EICrecon/plugins/DRICH.so)
| | ->18.05% (447,386,632B) 0x2D6EDFC7: eicrecon::IrtCherenkovParticleID::process(std::tuple<gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::RawTrackerHitCollection const*>, gsl::not_null<edm4eic::MCRecoTrackerHitAssociationCollection const*> > const&, std::tuple<gsl::not_null<edm4eic::CherenkovParticleIDCollection*>, gsl::not_null<edm4eic::CherenkovParticleIDCollection*> > const&) const (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/libalgorithms_pid.so)
| |   ->18.05% (447,386,632B) 0x5FB7AFD5: JOmniFactory<eicrecon::IrtCherenkovParticleID_factory, eicrecon::IrtCherenkovParticleIDConfig>::Process(std::shared_ptr<JEvent const> const&) (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/EICrecon/plugins/DRICH.so)
| | ->18.26% (459,564,168B) 0x2D6EDFC7: eicrecon::IrtCherenkovParticleID::process(std::tuple<gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::RawTrackerHitCollection const*>, gsl::not_null<edm4eic::MCRecoTrackerHitAssociationCollection const*> > const&, std::tuple<gsl::not_null<edm4eic::CherenkovParticleIDCollection*>, gsl::not_null<edm4eic::CherenkovParticleIDCollection*> > const&) const (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/libalgorithms_pid.so)
| |   ->18.26% (459,564,168B) 0x5FB7AFD5: JOmniFactory<eicrecon::IrtCherenkovParticleID_factory, eicrecon::IrtCherenkovParticleIDConfig>::Process(std::shared_ptr<JEvent const> const&) (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/EICrecon/plugins/DRICH.so)
| | ->18.97% (482,290,312B) 0x2D6EDFC7: eicrecon::IrtCherenkovParticleID::process(std::tuple<gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::RawTrackerHitCollection const*>, gsl::not_null<edm4eic::MCRecoTrackerHitAssociationCollection const*> > const&, std::tuple<gsl::not_null<edm4eic::CherenkovParticleIDCollection*>, gsl::not_null<edm4eic::CherenkovParticleIDCollection*> > const&) const (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/libalgorithms_pid.so)
| |   ->18.97% (482,290,312B) 0x5FB7AFD5: JOmniFactory<eicrecon::IrtCherenkovParticleID_factory, eicrecon::IrtCherenkovParticleIDConfig>::Process(std::shared_ptr<JEvent const> const&) (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/EICrecon/plugins/DRICH.so)
| | ->19.79% (508,054,456B) 0x2D6EDFC7: eicrecon::IrtCherenkovParticleID::process(std::tuple<gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::RawTrackerHitCollection const*>, gsl::not_null<edm4eic::MCRecoTrackerHitAssociationCollection const*> > const&, std::tuple<gsl::not_null<edm4eic::CherenkovParticleIDCollection*>, gsl::not_null<edm4eic::CherenkovParticleIDCollection*> > const&) const (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/libalgorithms_pid.so)
| |   ->19.79% (508,054,456B) 0x5FB7AFD5: JOmniFactory<eicrecon::IrtCherenkovParticleID_factory, eicrecon::IrtCherenkovParticleIDConfig>::Process(std::shared_ptr<JEvent const> const&) (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/EICrecon/plugins/DRICH.so)
| | ->43.17% (532,159,880B) 0x2D6EDFC7: eicrecon::IrtCherenkovParticleID::process(std::tuple<gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::RawTrackerHitCollection const*>, gsl::not_null<edm4eic::MCRecoTrackerHitAssociationCollection const*> > const&, std::tuple<gsl::not_null<edm4eic::CherenkovParticleIDCollection*>, gsl::not_null<edm4eic::CherenkovParticleIDCollection*> > const&) const (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/libalgorithms_pid.so)
| |   ->43.17% (532,159,880B) 0x5FB7AFD5: JOmniFactory<eicrecon::IrtCherenkovParticleID_factory, eicrecon::IrtCherenkovParticleIDConfig>::Process(std::shared_ptr<JEvent const> const&) (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/EICrecon/plugins/DRICH.so)

The upper frame looks like:

->46.58% (574,210,296B) 0x4B9A578: TStorage::ObjectAlloc(unsigned long) (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/root-6.30.02-lzyvoi75d5zywtptko3ptcj42t22a5pp/lib/root/libCore.so.6.30.02)
| ->43.17% (532,159,880B) 0x96664A7: ChargedParticle::PIDReconstruction(CherenkovPID&) (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/irt-1.0.7-t5b4ycnb3lxj6yb543ck6qtbj5mzubsu/lib/libIRT.so)

[veprbl]

Reproducer:

mc cp S3/eictest/EPIC/EVGEN/DIS/CC/10x100/minQ2=1000/pythia8CCDIS_10x100_minQ2=1000_beamEffects_xAngle=-0.025_hiDiv_1.hepmc3.tree.root .
source /opt/detector/epic-24.05.0/bin/thisepic.sh
npsim --runType batch --filter.tracker edep0 --numberOfEvents 1000 --compactFile "$DETECTOR_PATH/$DETECTOR_CONFIG.xml" --inputFile pythia8CCDIS_10x100_minQ2\=1000_beamEffects_xAngle\=-0.025_hiDiv_1.hepmc3.tree.root --outputFile sim.edm4hep.root 
varlgrind --tool=massif --massif-out-file=massif.out eicrecon sim.edm4hep.root
ms_print massif.out

[c-dilks]

libIRT is leaky: eic/irt#36

IIRC, some deletes are needed, but they may need to be within #ifdef DISABLE_ROOT_IO blocks, since ROOT may also delete things.