jemalloc not work in spec06 tests

[jensen-yan]

when I use

# result-24 -> /nix/store/32vzkyzfncidv4754d6cpnsk15zk34cn-spec2006exe 
cd result-24/483.xalancbmk/run
riscv64-unknown-linux-gnu-nm 483.xalancbmk| grep -i jemalloc
# return null
riscv64-unknown-linux-gnu-nm 483.xalancbmk| grep -i malloc
return 
0000000000235c2e T __malloc
0000000000235c2e T malloc
...

It shows binary use malloc instead of jemalloc which have lower performance in spec06 checkpoints.
Especially for omnetpp (4% performance lower) and xalancbmk(9% performance lower)!

[eastonman]

https://github.com/jemalloc/jemalloc/wiki/Getting-Started
It seems the current build script just includes jemalloc path, but not use it.

[jensen-yan]

That is helpful for me, thanks.
But I still do not find a suitable way to fix this.
When I use

  shellHook = ''
    # ...
    export LDFLAGS_SUFFIX="${customJemalloc}/lib/libjemalloc.a -lm -pthread"
    export LIBS="$LIBS $LDFLAGS_SUFFIX"
"

in shell.nix, it linked jemalloc successfully.
But for omnetpp, it will define operator new and operator delete in it's cpp file,

+ LD -> build/471.omnetpp
/nix/store/djy0jdfvdqgn4wl5ys71r1fxki3c1mw4-riscv64-unknown-linux-gnu-binutils-2.43.1/bin/riscv64-unknown-linux-gnu-ld: /nix/store/09x1j6m3w9lnl6znh9hsaw8d8x143nqf-jemalloc-riscv64-unknown-linux-gnu-5.3.0/lib/libjemalloc.a(jemalloc_cpp.o): in function `operator new(unsigned long)':
(.text+0x12c): multiple definition of `operator new(unsigned long)'; /nfs/home/yanyue/tools/CPU2006LiteWrapper/471.omnetpp/build/src/libs/cmdenv/heap.o (symbol from plugin):(.text+0x0): first defined here
/nix/store/djy0jdfvdqgn4wl5ys71r1fxki3c1mw4-riscv64-unknown-linux-gnu-binutils-2.43.1/bin/riscv64-unknown-linux-gnu-ld: /nix/store/09x1j6m3w9lnl6znh9hsaw8d8x143nqf-jemalloc-riscv64-unknown-linux-gnu-5.3.0/lib/libjemalloc.a(jemalloc_cpp.o): in function `operator delete(void*)':
(.text+0x504): multiple definition of `operator delete(void*)'; /nfs/home/yanyue/tools/CPU2006LiteWrapper/471.omnetpp/build/src/libs/cmdenv/heap.o (symbol from plugin):(.text+0x0): first defined here

resulted linked error, I used weak attributes for jemalloc operator, but it's not sufficient.

    preBuild = ''
      # Add weak attribute to C++ operators, same as jemalloc_cpp.patch
      sed -i 's/void \*operator new/void __attribute__((weak)) *operator new/g' src/jemalloc_cpp.cpp
      sed -i 's/void operator delete/void __attribute__((weak)) operator delete/g' src/jemalloc_cpp.cpp
    '';

only operator delete[] is weak.

➜  CPU2006LiteWrapper git:(main) ✗ riscv64-unknown-linux-gnu-nm -C /nix/store/09x1j6m3w9lnl6znh9hsaw8d8x143nqf-jemalloc-riscv64-unknown-linux-gnu-5.3.0/lib/libjemalloc.a| grep -i "operator"
000000000000050c T operator delete[](void*)
000000000000051c W operator delete[](void*, std::nothrow_t const&)
0000000000000568 T operator delete[](void*, std::align_val_t)
0000000000000578 T operator delete[](void*, std::align_val_t, std::nothrow_t const&)
0000000000000542 T operator delete[](void*, unsigned long)
00000000000005ec T operator delete[](void*, unsigned long, std::align_val_t)
0000000000000504 T operator delete(void*)
0000000000000514 T operator delete(void*, std::nothrow_t const&)
0000000000000560 T operator delete(void*, std::align_val_t)
0000000000000570 T operator delete(void*, std::align_val_t, std::nothrow_t const&)
0000000000000524 T operator delete(void*, unsigned long)
0000000000000580 T operator delete(void*, unsigned long, std::align_val_t)
00000000000001ee T operator new[](unsigned long)
0000000000000372 T operator new[](unsigned long, std::nothrow_t const&)
0000000000000466 T operator new[](unsigned long, std::align_val_t)
00000000000004ce T operator new[](unsigned long, std::align_val_t, std::nothrow_t const&)
000000000000012c T operator new(unsigned long)
00000000000002b0 T operator new(unsigned long, std::nothrow_t const&)
0000000000000434 T operator new(unsigned long, std::align_val_t)
0000000000000498 T operator new(unsigned long, std::align_val_t, std::nothrow_t const&)    

I should use jemalloc_cpp.patch like

https://github.com/OpenXiangShan/CPU2006LiteWrapper/blob/main/jemalloc_cpp.patch

, but it's only useful for specific jemalloc version(da66aa391f853ccf2300845b3873cc8f1cf48f2d or older), not elegant

export RISCV=/riscv_toolchain/top
git clone https://github.com/jemalloc/jemalloc.git
cd jemalloc
cp /path/to/jemalloc_cpp.patch ./
git checkout -b apply-patch da66aa391f853ccf2300845b3873cc8f1cf48f2d
git apply jemalloc_cpp.patch
CC=$RISCV/bin/riscv64-unknown-linux-gnu-gcc CXX=$RISCV/bin/riscv64-unknown-linux-gnu-c++ LD=$RISCV/bin/riscv64-unknown-linux-gnu-ld ./autogen.sh --prefix=$RISCV --host=x86_64-linux-gnu

make && make install

Is there other way to fix it ?

[jensen-yan]

I also find some programs do not use jemalloc OpenXiangShan/CPU2006LiteWrapper#18

[jensen-yan]

I fix it using jemalloc patch(only support jemalloc v3.5), still running , when I finish performance testing, I will push.

[eastonman]

I believe these two lines should work

sed -i '/void/N;s/void[[:space:]]*\*[[:space:]]*operator new/void __attribute__((weak)) *operator new/g' src/jemalloc_cpp.cpp
sed -i '/void/N;s/void[[:space:]]*operator delete/void __attribute__((weak)) operator delete/g' src/jemalloc_cpp.cpp

[jensen-yan]

It looks great for me, I try it locally, it works great.
Checkpoint-generating is still running, when I get the performance results, I will push use your ways, thanks.

[jensen-yan]

添加jemalloc 之后分数差异更大了，都是和原始gcc12切片对比性能，
大机房地址/nfs/share/zyy/spec06_rv64gcb_O3_20m_gcc12.2.0-intFpcOff-jeMalloc/zstd-checkpoint-0-0-0
左边是原始切片，右边是nix 切片

上图是没有加jemalloc之前分数差异，基于89ec13 commit
下图是加上jemalloc 之后分数差异， 基于ed8e7a6321d2580169ac0ccf7ebe337a6ca37913 commit加上jemalloc之后测试

不确定是我nix 切片跑错了，还是jemalloc 影响，亦或者是重构代码导致，我会继续查询性能原因，暂时不推jemalloc patch.

[eastonman]

Did the nix-run even finish yet? coverage=1 does not mean it has finished. According to my previous experience, coverage is calculated based on m5out or something. So after warmup, the weight of that checkpoint has already been taken into account. This leads to problems like using stats from the warmup period to calculate scores.