lib.packagesFromDirectoryRecursive: use explicit recursion, support nested scopes

[nbraud]

Follow-up on #359941, blocked on it as this PR contains commits from it.
Blocked on #359898 and #361424.

• Extend lib.filesystem.packagesFromDirectoryRecursive with support for explicit recursion.
• Update packagesFromDirectoryRecursive to create nested scopes for each (sub)directory (not just the top-level one) when newScope is given.
• Apply recurseIntoAttrs by default when recursing into a directory.
• Add a test using scopes for packagesFromDirectoryRecursive, ensuring:
  • packages can depend on other packages in the same scope and in ancestor scopes (the scope's parent or grandparent) ;
  • packages can have mutual dependencies.
• Reject unknown parameters (breaking change!)
• Add an example showing how to use recurseIntoDirectory.

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• Tested nix-instantiate --eval --strict lib/tests/misc.nix
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 25.05 Release Note
• Fits CONTRIBUTING.md.

---

Add a 👍 reaction to pull requests you find important.

[nbraud]

Ran nixfmt over the new test fixtures

[nbraud]

Rebased following the parent PR being merged

[nbraud]

I'd prefer to land #359898 first, as there will be a merge conflict over the function's documentation, and having the improved documentation will make documenting this PR's changes simpler.

PS: Marked as draft again, since I'll wait for that to land before writing documentation in this PR.

[nbraud]

Rebased atop both blocking PRs, added documentation.

Still needs an example how to use recurseIntoDirectory

[nbraud]

Fixed up the release notes, putting those entries under the lib section

[nbraud]

@NixOS/ofborg-maintainers, where does one get details on that ofborg-internal-error ?

I'm guessing @ofborg choked on something which it wasn't previously recursing into, possibly the FreeBSD-specific packages? (for which it has no builder)

[nbraud]

Squashed fixup! commits, no diff

[nbraud]

Using recurseIntoAttrs by default will involve fixing some callers of packagesFromDirectoryRecursive in nixpkgs, so I am pushing it back to a later PR.

[nbraud]

nixpkgs-review result

Generated using nixpkgs-review.

Command: nixpkgs-review pr 359984

---

x86_64-linux

⏩ 1 package blacklisted:

• nixos-install-tools

✅ 1 package built:

• nixpkgs-manual

[nbraud]

This is only waiting for reviews, no functional change since the beginning of the month: only rebased to address the merge conflict and squash fixup commits, and moved the release notes entries. (One further change was tried and reverted, to be done in #369421)

@9999years if you re-review, I think we could get that merged soonish

[hsjobeki]

Nice changes. Thank you for the efforts that you already put into it.

[hsjobeki]

This function could be its own unit. For example defaultRecurseIntoDirectory. This makes it possible to unit-test and reuse this function. I think inline here is a little bit unhappy.

Also this inline documentation as plain comment will not render into any documentation.

If you pull this out you can write a nice doc-comment describing the behavior.
Place mutual references in the parent function-library-lib.filesets.packagesFromDirectoryRecursive
This will make it much easier for people to understand and implement their own function if needed.

/** 
  Create a new scope and mark it `recurseForDerivations`.
  This lets the packages refer to each other.
  
  See also:
  - [lib.makeScope](https://nixos.org/manual/nixpkgs/unstable/#function-library-lib.customisation.makeScope) and
  - [lib.recurseIntoAttrs](https://nixos.org/manual/nixpkgs/unstable/#function-library-lib.customisation.makeScope)
*/
defaultRecurseIntoDirectory = args: if args ? newScope then ...

[hsjobeki]

Suggested change

	# and recurseArgs can (optionally) hold data for its use ; see nixdoc above
	# and recurseArgs can (optionally) hold data for its use ; see comment above

[hsjobeki]

It seems we used a different heading in all the previous release notes:

Suggested change

	### Other notable changes {#sec-release-25.05-lib-notable-changes}
	### Additions and Improvements {#sec-release-25.05-lib-additions-improvements}