typst: add initial support for typst packages #369283

cherrypiejam · 2024-12-30T02:27:10Z

This PR adds initial support to build Typst documents with Typst packages. In particular, it adds a Typst package builder and exposes it at the top level to ease the process of defining new Typst packages. It further adds a new function to the Typst derivation for creating a new Typst environment in which the Typst compiler understands a specific set of packages, namely typst.withPackages. Moreover, three Typst packages are defined and collected as a single attribution set as an initial demonstration.

Only a few Typst packages are included in this PR for a reason. The Typst Universe (the official site for Typst packages) maintains a single git repo that contains all Typst packages. Each package entry does not necessarily map to the same version specified in their respective git repo published by their authors (if there is). This is because the package authors need to manually move their package and copy the source, instead of a pointer, to the Typst Universe repo. Breaking the Typst Universe repo down to per package requires separate places to store each of them, and due to the aforementioned reason, the exact source may disagree with the original git repo ¹. This is why, in my opinion, we might want to collect our own set of Typst packages instead of directly pulling from the Typst Universe repo. For gradual adaption, typst.withPackages also provides a method to override the predefined set of Typst packages, namely typst.withPackages.override, making it easier for end users overriding the existing set of Typst packages and integrating new packages into their project.

Things done

Add a 👍 reaction to pull requests you find important.

Assuming the author(s) also publish their package in another repo outside of the Typst Universe repo. ↩

SigmaSquadron

Welcome to Nixpkgs! This is truly amazing; I've wanted to build Typst packages for a very long time but never got around to it.

Note that the commits should be split as much as possible. Each new typstPackage should be added in a separate commit. The changes to the base typst package to wrap it should also be a separate commit.

pkgs/by-name/ty/typst/with-packages.nix

pkgs/build-support/build-typst-package.nix

pkgs/development/typst-packages/cetz.nix

SigmaSquadron

Regarding the actual package set, even though we can't import from Universe directly, it might still be a good idea to scrape it for information and build packages automatically. typst.toml gives us all of the necessary information to build each package.

pkgs/build-support/build-typst-package.nix

pkgs/development/typst-packages/polylux.nix

pkgs/development/typst-packages/oxifmt.nix

pkgs/development/typst-packages/cetz.nix

pkgs/by-name/ty/typst/with-packages.nix

drupol

Hello,

Thanks for tackling this, I have a few comments, let me know if you have questions.

@jtojnar : I know you're busy these days, but if you could have a look at this, it would be great.

Thanks!

pkgs/build-support/build-typst-package.nix

pkgs/by-name/ty/typst/package.nix

pkgs/by-name/ty/typst/with-packages.nix

cherrypiejam · 2025-01-05T03:08:18Z

Regarding the actual package set, even though we can't import from Universe directly, it might still be a good idea to scrape it for information and build packages automatically. typst.toml gives us all of the necessary information to build each package.

Ah that is great. Didn't know they already have that information included. I'll work on the script when I got time, but it will have to be waited for another week or so.

SigmaSquadron

Looks good to me, just one final open question for anyone who knows about packagesFromDirectoryRecursive.

SigmaSquadron · 2025-01-05T04:27:39Z

pkgs/top-level/all-packages.nix

+  typstPackages = callPackage ./typst-packages.nix {
+    callPackage = lib.callPackageWith (pkgs // { inherit buildTypstPackage; });
+  };


Suggested change

typstPackages = callPackage ./typst-packages.nix {

callPackage = lib.callPackageWith (pkgs // { inherit buildTypstPackage; });

};

typstPackages = lib.packagesFromDirectoryRecursive {

inherit callPackage;

directory = ../development/typst-packages;

}

Is this a good idea? It'd remove the need for typst-packages.nix, but I haven't seen packagesFromDirectoryRecursive in Nixpkgs before. (This is more of an open discussion and less of a review, though)

Have never used it before and happy to hear comments regarding it from other people

but I haven't seen packagesFromDirectoryRecursive in Nixpkgs before.

This is the mechanism in place for automatically discovering packages in pkgs/by-name.

Then I suppose we don't need it? typst-packages.nix is an expression defined in the top-level and itself is not a discoverable "package" (as it does not follow the naming convention <package-name/package.nix>). Alternatively we could rename it to a different path but I leave it here because all other *-packages.nix are defined in the same directory.

Also I was previously wondering how pkgs/by-name is discovered and joined to the pkgs set, and I guess this is it

Then I suppose we don't need it?

It's up to you to decide how you want this to work. To be honest, I still don't understand everything in your PR so, I let you do some progress and then I'll test it out.

FYI: You can see an example of external use of packagesFromDirectoryRecursive in https://github.com/drupol/pkgs-by-name-for-flake-parts/

Ok, I think not as many other *-packages.nix are defined directly in the top-level directory and I decide to follow this pattern. The just pushed change removes callPackage = lib.callPackageWith (pkgs // { inherit buildTypstPackage; }); as it is not needed.

drupol · 2025-01-05T22:32:54Z

pkgs/development/typst-packages/cetz.nix

+
+  preInstall = ''
+    rm -rf *
+    cp -r $BUILD_DIR/cetz/0.2.2/* .


I have the feeling that this is overly complex. Can't we simplify this?

Essentially we need to generate the typst package, and remove non-essential parts (e.g, cetz building scripts). Feels like there are many ways of implementing it but they are similar in principle

In that case I would avoid repeating the version more than once so it can be handled by rryan-tm bot at some point.

Looking at https://github.com/cetz-package/common/blob/26359dfa24a9a24b71d38e9bb856b670e71a9118/scripts/package#L84-L84 and https://github.com/typst/packages?tab=readme-ov-file#local-packages, maybe just use

installPhase = '' runHook preInstall just install "$out/share/typst/packages/local/" runHook postInstall '';

I do not see much of a point in buildTypstPackage in the first place.

There are a couple of things

Not all typst packages need to be generated from their source. If the package source is the typst package, they only need to be moved to the right location where typst-env assumes. buildTypstPackage, IMO, eases this process. In other words, trivial typst packages do not need to write their own install phase and understand the assumption that typst-env makes.

We want to seamlessly integrate nix into the compiling process of a Typst document. A Typst document can rely on two types of packages: one from the Typst universe (preview), and another is customized package (local). The Typst compiler, by design, understands packages from the Typst universe (it knows where to fetch the preview packages as long as if it has access to the Internet). As for local packages, Typst compiler shouldn't, and cannot, understand where to fetch, so it relies on the user to provide the source of local packages in a known place for the compiler to fetch. With that being said, when building Typst documents in nix, the Typst compiler should know the preview packages but cannot know since it doesn't have the access to the network (there's an alternative approach, will mention later). This is why we want to install the package to the preview directory to give Typst compiler this part of knowledge before the restricted (no network) build process. For the local packages, as mentioned earlier, the Typst compiler doesn't have the knowledge of where to fetch the source of local packages. This implies that the user must know where to fetch or what is the source. In other words, it is reasonable to leave the installation of local packages purely to the user.

This sounds like it might break ad-hoc package downloads https://github.com/typst/packages?tab=readme-ov-file#downloads

I believe my second point has answered this. Please let me know where doesn't make sense to you. This is all written in rush, so sorry in advance. Also, an alternative approach is to let Nix automatically fetch packages dependency of the target Typst document and wrap them into a derivation (e.g., Rust cargo builder, or Scala SBT builder). Thing is, the Typst compiler doesn't have a lock file (as the one used by the cargo builder for Rust), meaning that it is going to face the same issue of the SBT builder --- building twice and use the first build to retrieve the hash of all dependent packages. Alternatively, declaratively writing down required packages (the approach used in this PR) has a different issue --- nested dependency requires users to define all required packages in a single location, which can be extremely troublesome. To fix it, which is not yet to be implemented, we need to propagate dependencies from packages to the typst-env, which should be easy to implement.

In that case I would avoid repeating the version more than once so it can be handled by rryan-tm bot at some point.

Makes sense. Replaced with the version variable instead

pkgs/top-level/typst-packages.nix

drupol · 2025-01-06T06:21:18Z

Would it be a good idea to have some documentation for this ?

cherrypiejam · 2025-01-06T20:09:25Z

Would it be a good idea to have some documentation for this ?

I can write it, but where should I put it?

(Module updates) Added a release notes entry if the change is significant

Not sure if this change should be considered as significant

jtojnar · 2025-01-06T20:59:50Z

pkgs/by-name/ty/typst/with-packages.nix

+          cp -r ${typst}/share $out/share
+          mkdir -p $out/bin
+
+          makeWrapper "${lib.getExe typst}" "$out/bin/typst" --set TYPST_PACKAGE_CACHE_PATH $TYPST_LIB_DIR


This sounds like it might break ad-hoc package downloads https://github.com/typst/packages?tab=readme-ov-file#downloads

Please see my comment above

nix-owners bot requested a review from philiptaron December 30, 2024 02:28

NixOSInfra added the 12. first-time contribution This PR is the author's first one; please be gentle! label Dec 30, 2024

github-actions bot added 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin 10.rebuild-linux: 0 This PR does not cause any packages to rebuild on Linux labels Dec 30, 2024

cherrypiejam force-pushed the typst branch 3 times, most recently from b5ab425 to 0bcf7c9 Compare December 30, 2024 12:20

SigmaSquadron reviewed Dec 30, 2024

View reviewed changes

pkgs/by-name/ty/typst/with-packages.nix Outdated Show resolved Hide resolved

pkgs/build-support/build-typst-package.nix Outdated Show resolved Hide resolved

pkgs/development/typst-packages/cetz.nix Outdated Show resolved Hide resolved

cherrypiejam force-pushed the typst branch 2 times, most recently from 407f204 to b14cd40 Compare January 1, 2025 15:16

cherrypiejam requested a review from SigmaSquadron January 3, 2025 17:15

SigmaSquadron requested review from drupol, figsoda and kanashimia January 3, 2025 18:34

SigmaSquadron reviewed Jan 3, 2025

View reviewed changes

drupol requested changes Jan 3, 2025

View reviewed changes

wegank added the 2.status: merge conflict This PR has merge conflicts with the target branch label Jan 4, 2025

cherrypiejam force-pushed the typst branch from b14cd40 to 9f0d142 Compare January 5, 2025 02:51

ofborg bot removed the 2.status: merge conflict This PR has merge conflicts with the target branch label Jan 5, 2025

cherrypiejam requested review from drupol and SigmaSquadron January 5, 2025 03:10

SigmaSquadron reviewed Jan 5, 2025

View reviewed changes

cherrypiejam force-pushed the typst branch from 9f0d142 to e753a45 Compare January 5, 2025 12:00

cherrypiejam added 2 commits January 5, 2025 20:48

typst: add customized builder for typst packages

19ff78c

typst: add polylux package for typst

fdf3d6d

cherrypiejam force-pushed the typst branch from e753a45 to 7560346 Compare January 5, 2025 19:52

drupol reviewed Jan 5, 2025

View reviewed changes

cherrypiejam force-pushed the typst branch from 7560346 to 080f739 Compare January 6, 2025 20:07

jtojnar reviewed Jan 6, 2025

View reviewed changes

cherrypiejam added 3 commits January 6, 2025 23:21

typst: add cetz package for typst

12fd443

typst: add oxifmt package for typst

a762221

typst: add support to instantiate typst with a set of typst packages

e135bb1

cherrypiejam force-pushed the typst branch from 080f739 to e135bb1 Compare January 6, 2025 22:21

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

typst: add initial support for typst packages #369283

typst: add initial support for typst packages #369283

cherrypiejam commented Dec 30, 2024 •

edited

Loading

SigmaSquadron left a comment •

edited

Loading

SigmaSquadron left a comment

drupol left a comment

cherrypiejam commented Jan 5, 2025

SigmaSquadron left a comment

SigmaSquadron Jan 5, 2025

cherrypiejam Jan 5, 2025

drupol Jan 5, 2025

cherrypiejam Jan 5, 2025

drupol Jan 5, 2025

cherrypiejam Jan 5, 2025

drupol Jan 5, 2025

cherrypiejam Jan 6, 2025

drupol Jan 6, 2025

jtojnar Jan 6, 2025 •

edited

Loading

cherrypiejam Jan 6, 2025

cherrypiejam Jan 6, 2025

drupol commented Jan 6, 2025

cherrypiejam commented Jan 6, 2025

jtojnar Jan 6, 2025

cherrypiejam Jan 6, 2025

-  typstPackages = callPackage ./typst-packages.nix {
-    callPackage = lib.callPackageWith (pkgs // { inherit buildTypstPackage; });
-  };
+  typstPackages = lib.packagesFromDirectoryRecursive {
+    inherit callPackage;
+    directory = ../development/typst-packages;
+  }

typst: add initial support for typst packages #369283

Are you sure you want to change the base?

typst: add initial support for typst packages #369283

Conversation

cherrypiejam commented Dec 30, 2024 • edited Loading

Things done

Footnotes

SigmaSquadron left a comment • edited Loading

Choose a reason for hiding this comment

SigmaSquadron left a comment

Choose a reason for hiding this comment

drupol left a comment

Choose a reason for hiding this comment

cherrypiejam commented Jan 5, 2025

SigmaSquadron left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jtojnar Jan 6, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

drupol commented Jan 6, 2025

cherrypiejam commented Jan 6, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cherrypiejam commented Dec 30, 2024 •

edited

Loading

SigmaSquadron left a comment •

edited

Loading

jtojnar Jan 6, 2025 •

edited

Loading