Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

glibc: allow use ld-nix.so.conf #209753

Draft
wants to merge 1 commit into
base: staging
Choose a base branch
from

Conversation

Izorkin
Copy link
Contributor

@Izorkin Izorkin commented Jan 8, 2023

Description of changes

This PR allows libraries to be loaded via the `/etc/ld-nix.so.conf' configuration file. Loading of NSS modules is supported.

Example.
Add to nixos configuration:

  environment.etc."ld-nix.so.conf".text = ''
    ${pkgs.tcb.out}/lib
  '';

Generate cache:

sudo ldconfig -f /etc/ld-nix.so.conf -C /etc/ld-nix.so.cache

Checking loaded libraries:

ldconfig -p | grep tcb
        libtcb.so.0 (libc6,x86-64) => /nix/store/smm8bwymb7wm5lhw1l0fjb5s88dbsb7f-tcb-1.2/lib/libtcb.so.0
        libtcb.so (libc6,x86-64) => /nix/store/smm8bwymb7wm5lhw1l0fjb5s88dbsb7f-tcb-1.2/lib/libtcb.so
        libnss_tcb.so.2 (libc6,x86-64) => /nix/store/smm8bwymb7wm5lhw1l0fjb5s88dbsb7f-tcb-1.2/lib/libnss_tcb.so.2
LD_DEBUG=all passwd
...
      2800:     symbol=_nss_tcb_setsgent;  lookup in file=/nix/store/smm8bwymb7wm5lhw1l0fjb5s88dbsb7f-tcb-1.2/lib/libnss_tcb.so.2 [0]
      2800:     symbol=_nss_tcb_setsgent;  lookup in file=/nix/store/smm8bwymb7wm5lhw1l0fjb5s88dbsb7f-tcb-1.2/lib/libtcb.so.0 [0]
      2800:     symbol=_nss_tcb_setsgent;  lookup in file=/nix/store/56r5i30jq456xnlqk2kfhy4g0v2i2l51-glibc-2.35-224/lib/libc.so.6 [0]
      2800:     symbol=_nss_tcb_setsgent;  lookup in file=/nix/store/56r5i30jq456xnlqk2kfhy4g0v2i2l51-glibc-2.35-224/lib/ld-linux-x86-64.so.2 [0]
      2800:     /nix/store/smm8bwymb7wm5lhw1l0fjb5s88dbsb7f-tcb-1.2/lib/libnss_tcb.so.2: error: symbol lookup error: undefined symbol: _nss_tcb_setsgent (fatal)
      2800:     symbol=_nss_tcb_setspent;  lookup in file=/nix/store/smm8bwymb7wm5lhw1l0fjb5s88dbsb7f-tcb-1.2/lib/libnss_tcb.so.2 [0]
      2800:     binding file /nix/store/smm8bwymb7wm5lhw1l0fjb5s88dbsb7f-tcb-1.2/lib/libnss_tcb.so.2 [0] to /nix/store/smm8bwymb7wm5lhw1l0fjb5s88dbsb7f-tcb-1.2/lib/libnss_tcb.so.2 [0]: normal symbol `_nss_tcb_setspent'
      2799:
      2799:     file=libnss_tcb.so.2 [0];  dynamically loaded by /nix/store/56r5i30jq456xnlqk2kfhy4g0v2i2l51-glibc-2.35-224/lib/libc.so.6 [0]
      2799:     find library=libnss_tcb.so.2 [0]; searching
      2799:      search cache=/etc/ld-nix.so.cache
      2799:       trying file=/nix/store/smm8bwymb7wm5lhw1l0fjb5s88dbsb7f-tcb-1.2/lib/libnss_tcb.so.2
      2799:
      2799:     file=libnss_tcb.so.2 [0];  generating link map
      2799:       dynamic: 0x00007fe30f6b2cf0  base: 0x00007fe30f6af000   size: 0x0000000000004010
      2799:         entry: 0x00007fe30f6af000  phdr: 0x00007fe30f6af040  phnum:                 11
      2799:
      2799:     checking for version `GLIBC_2.3' in file /nix/store/56r5i30jq456xnlqk2kfhy4g0v2i2l51-glibc-2.35-224/lib/ld-linux-x86-64.so.2 [0] required by file /nix/store/smm8bwymb7wm5lhw1l0fjb5s88dbsb7f-tcb-1.2/lib/libnss_tcb.so.2 [0]
      2799:     checking for version `GLIBC_2.8' in file /nix/store/56r5i30jq456xnlqk2kfhy4g0v2i2l51-glibc-2.35-224/lib/libc.so.6 [0] required by file /nix/store/smm8bwymb7wm5lhw1l0fjb5s88dbsb7f-tcb-1.2/lib/libnss_tcb.so.2 [0]
      2799:     checking for version `GLIBC_2.4' in file /nix/store/56r5i30jq456xnlqk2kfhy4g0v2i2l51-glibc-2.35-224/lib/libc.so.6 [0] required by file /nix/store/smm8bwymb7wm5lhw1l0fjb5s88dbsb7f-tcb-1.2/lib/libnss_tcb.so.2 [0]
      2799:     checking for version `GLIBC_2.2.5' in file /nix/store/56r5i30jq456xnlqk2kfhy4g0v2i2l51-glibc-2.35-224/lib/libc.so.6 [0] required by file /nix/store/smm8bwymb7wm5lhw1l0fjb5s88dbsb7f-tcb-1.2/lib/libnss_tcb.so.2 [0]
      2799:     object=/nix/store/smm8bwymb7wm5lhw1l0fjb5s88dbsb7f-tcb-1.2/lib/libnss_tcb.so.2 [0]
      2799:      scope 0: passwd /nix/store/smm8bwymb7wm5lhw1l0fjb5s88dbsb7f-tcb-1.2/lib/libtcb.so.0 /nix/store/kxxs803rx6rckidqa3q5kqhhgf7428vj-linux-pam-1.5.2/lib/libpam.so.0 /nix/store/kxxs803rx6rckidqa3q5kqhhgf7428vj-linux-pam-1.5.2/lib/libpam_misc.so.0 /nix/store/56r5i30jq456xnlqk2kfhy4g0v2i2l51-glibc-2.35-224/lib/libc.so.6 /nix/store/mmxwcw9kacp494jz54dpzzxc71yc3m0n-audit-2.8.5/lib/libaudit.so.1 /nix/store/56r5i30jq456xnlqk2kfhy4g0v2i2l51-glibc-2.35-224/lib/ld-linux-x86-64.so.2
      2799:      scope 1: /nix/store/smm8bwymb7wm5lhw1l0fjb5s88dbsb7f-tcb-1.2/lib/libnss_tcb.so.2 /nix/store/smm8bwymb7wm5lhw1l0fjb5s88dbsb7f-tcb-1.2/lib/libtcb.so.0 /nix/store/56r5i30jq456xnlqk2kfhy4g0v2i2l51-glibc-2.35-224/lib/libc.so.6 /nix/store/56r5i30jq456xnlqk2kfhy4g0v2i2l51-glibc-2.35-224/lib/ld-linux-x86-64.so.2
      2799:
      2799:
...

cc @flokli @NickCao @SuperSandro2000 @trofi

Things done
  • Built on platform(s)
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
  • Tested, as applicable:
  • Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
  • Tested basic functionality of all binary files (usually in ./result/bin/)
  • 23.05 Release Notes (or backporting 22.11 Release notes)
    • (Package updates) Added a release notes entry if the change is major or breaking
    • (Module updates) Added a release notes entry if the change is significant
    • (Module addition) Added a release notes entry if adding a new NixOS module
    • (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes
  • Fits CONTRIBUTING.md.

@trofi
Copy link
Contributor

trofi commented Jan 8, 2023

NixOS allows presence of multiple glibc versions at the same time by running programs right out of store.

Writing library paths into /etc/ld-nix.so.conf will likely refer specific libc.so (and what not) via RUNPATH that does not match environment programs were built. Typical example is a:

  • build system against nixpkgs.v1, glibc.v1, libs.v1, populate /etc/ld-nix.so.conf
  • check out nixpkgs.v2 and start a program from there, glibc.v2, libs.v2, worst case it's not compatible with nixpkgs.v1 from previous step

Other nixpkgs mechanisms plagued by this are GIO_EXTRA_MODULES, QT plugins and opengl escape hatch via /run/opengl-driver. They all have this annoying property.

That usually causes libc load load failures of sorts with missing symbol errors (good, easy to diagnose). Worst case you get SIGSEGVs for simently broken ABI (bad, hard to debug). Things are a lot worse when programs load libraries a while after they start. Don't know if anything tries to load NSS as soon as possible at start. I would guess no, it's lazy. Otherwise every program in system would pay the cost of NSS.

Would anything prevent /etc/ld-nix.so.conf from being plagued by this scenario?

@flokli
Copy link
Contributor

flokli commented Jan 8, 2023

No, this will be a big footgun, i don't think it's a good idea.

@Izorkin
Copy link
Contributor Author

Izorkin commented Jan 8, 2023

I don't know so much about how glibc works. I was looking for a possible way to load the necessary modules for tcb to work.

@trofi
Copy link
Contributor

trofi commented Jan 8, 2023

System-wide plugins are very hard in NixOS. I'm not sure if nixpkgs even has a good example of those. One could say they should be infeasible by construction (a bit like global variables in haskell).

If we were to take your approach as is we would want to key path similar to /etc/ld-nix.so.conf off from glibc version instance. For example every program that is built against /nix/store/ayfr5l52xkqqjn3n4h9jfacgnchz1z7s-glibc-2.35-224 would need to know to read the config from something like /etc/ayfr5l52xkqqjn3n4h9jfacgnchz1z7s-glibc-2.35-224/ld-nix.so.conf (or even /run/ayfr5l52xkqqjn3n4h9jfacgnchz1z7s-glibc-2.35-224-ld-nix.so.conf without ever deleting entries there). No idea if it's easy to populate such directories later.

It will still have corner cases, like missing entries when you run non-active glibc. But it's arguably not as bad as loading incompatible libraries.

Would be nice to have an agreed design in this space. Your use case is a great example where it's badly needed.

@ofborg ofborg bot added the 10.rebuild-linux-stdenv This PR causes stdenv to rebuild label Jan 8, 2023
@ofborg ofborg bot requested review from edolstra and Ma27 January 8, 2023 22:52
@Izorkin Izorkin marked this pull request as draft January 9, 2023 07:28
@Izorkin
Copy link
Contributor Author

Izorkin commented Jan 9, 2023

Which variable can be used to pull the path to package?

@Izorkin
Copy link
Contributor Author

Izorkin commented Jan 9, 2023

This varaint not working:

diff --git a/pkgs/development/libraries/glibc/common.nix b/pkgs/development/libraries/glibc/common.nix
index 80ff28c2b1e..3fccbfb3000 100644
--- a/pkgs/development/libraries/glibc/common.nix
+++ b/pkgs/development/libraries/glibc/common.nix
@@ -115,6 +115,9 @@ stdenv.mkDerivation ({
       # Ensure that `__nss_files_fopen` can still be wrapped by `libredirect`.
       sed -i -e '/libc_hidden_def (__nss_files_fopen)/d' nss/nss_files_fopen.c
       sed -i -e '/libc_hidden_proto (__nss_files_fopen)/d' include/nss_files.h
+
+      substituteInPlace sysdeps/generic/dl-cache.h \
+        --replace '# define LD_SO_CACHE SYSCONFDIR "/ld.so.cache"' '# define LD_SO_CACHE "/etc/${lib.head (lib.splitString "-" (lib.elemAt (lib.splitString "/" "$out") 3))}/ld.so.cache"'
     ''
     # FIXME: find a solution for infinite recursion in cross builds.
     # For now it's hopefully acceptable that IDN from libc doesn't reliably work.
diff --git a/pkgs/development/libraries/glibc/dont-use-system-ld-so-cache.patch b/pkgs/development/libraries/glibc/dont-use-system-ld-so-cache.patch
index 0f852f6e632..40018de2769 100644
--- a/pkgs/development/libraries/glibc/dont-use-system-ld-so-cache.patch
+++ b/pkgs/development/libraries/glibc/dont-use-system-ld-so-cache.patch
@@ -36,16 +36,3 @@ index d29bdd69..c7124a65 100644

    unsigned int index = 0;
    static const char *system_dirs = SYSTEM_DIRS "\0";
-diff --git a/sysdeps/generic/dl-cache.h b/sysdeps/generic/dl-cache.h
-index df385dca..b907c214 100644
---- a/sysdeps/generic/dl-cache.h
-+++ b/sysdeps/generic/dl-cache.h
-@@ -35,7 +35,7 @@
- #endif
-
- #ifndef LD_SO_CACHE
--# define LD_SO_CACHE SYSCONFDIR "/ld.so.cache"
-+# define LD_SO_CACHE "/etc/ld-nix.so.cache"
- #endif
-
- #ifndef add_system_dir

Error:

error: list index 3 is out of bounds

       at /home/user/works/nixpkgs/pkgs/development/libraries/glibc/common.nix:120:129:

          119|       substituteInPlace sysdeps/generic/dl-cache.h \
          120|         --replace '# define LD_SO_CACHE SYSCONFDIR "/ld.so.cache"' '# define LD_SO_CACHE "/etc/${lib.head (lib.splitString "-" (lib.elemAt (lib.splitString "/" "$out") 3))}/ld.so.cache"'
             |                                                                                                                                 ^
          121|     ''
(use '--show-trace' to show detailed location information)

Is there another way or how to fix this variant?

@trofi
Copy link
Contributor

trofi commented Jan 9, 2023

Instead of $out there is a ${placeholder "out"} that allows you to embed self-reference paths. It should also survive content-addressed derivation conversions as well. Maybe that is good enough?

@Izorkin
Copy link
Contributor Author

Izorkin commented Jan 9, 2023

Weird, result with this variant # define LD_SO_CACHE "/etc${placeholder "out"}/ld.so.cache":

#ifndef LD_SO_CACHE
# define LD_SO_CACHE "/etc/1rz4g4znpzjwh1xymhjpm42vipw92pr73vdgl6xs1hycac8kf2n9/ld.so.cache"
#endif

I thought there would be an variant:

#ifndef LD_SO_CACHE
# define LD_SO_CACHE "/etc//nix/...-glibc-2.35-224/ld.so.cache"
#endif

This variant not working - # define LD_SO_CACHE "/etc/${lib.head (lib.splitString "-" (lib.elemAt (lib.splitString "/" ${placeholder "out"}) 3))}/ld.so.cache":

error: syntax error, unexpected DOLLAR_CURLY, expecting ')'

       at /home/user/works/nixpkgs/pkgs/development/libraries/glibc/common.nix:120:161:

          119|       substituteInPlace sysdeps/generic/dl-cache.h \
          120|         --replace '# define LD_SO_CACHE SYSCONFDIR "/ld.so.cache"' '# define LD_SO_CACHE "/etc/${lib.head (lib.splitString "-" (lib.elemAt (lib.splitString "/" ${placeholder "out"}) 3))}/ld.so.cache"'
             |                                                                                                                                                                 ^
          121|     ''
(use '--show-trace' to show detailed location information)

@Izorkin
Copy link
Contributor Author

Izorkin commented Jan 9, 2023

Also requires a method to locate the ld.so.cache file for the script:

   ldconfig -f /etc/ld-nix.so.conf -C /etc/.../ld.so.cache

@trofi
Copy link
Contributor

trofi commented Feb 15, 2023

I filed https://github.com/nixpkgs-architecture/issues/issues/15 to consider developing blessed way to handle plugins.

@Atry
Copy link
Contributor

Atry commented Aug 13, 2023

How do you think about #248777 ?

@Izorkin
Copy link
Contributor Author

Izorkin commented Aug 14, 2023

How do you think about #248777 ?

There is not enough free space on the virtual machine to rebuild the system and check.

@wegank wegank added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Mar 19, 2024
@Izorkin
Copy link
Contributor Author

Izorkin commented Oct 7, 2024

There are no new variants?
cc @flokli @trofi

@stale stale bot removed the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Oct 7, 2024
@trofi
Copy link
Contributor

trofi commented Oct 7, 2024

I'm not aware of a reasonable way of implementing plugins against immutable packages with uncontrolled locations. No insights in #295035 either. As there is no interest from others it sounds like you have to pave the path of designing something that works (which is not a great answer).

@Izorkin
Copy link
Contributor Author

Izorkin commented Oct 7, 2024

Sent an email to [email protected]. Maybe they will suggest some solution.

@wegank wegank added the 2.status: merge conflict This PR has merge conflicts with the target branch label Dec 10, 2024
@wegank wegank added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Jan 1, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
2.status: merge conflict This PR has merge conflicts with the target branch 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md 10.rebuild-darwin: 11-100 10.rebuild-linux: 501+ 10.rebuild-linux: 5001+ 10.rebuild-linux-stdenv This PR causes stdenv to rebuild
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants