Extending Pkl

[sgammon]

Hey there esteemed Pkl authors,

First of all, I'm a big fan of Apple and anything GraalVM, so this is very cool to see! Thank you for such an awesome project. I wanted to inquire about extending Pkl for custom use cases. Allow me to expand below; this is an idea for new functionality or at least an inquiry about how to do something the right way.

Use case

We are building some experimental dev-facing tools, that build code or lint code, etc. We are looking for a configuration format that allows some dynamic configuration ability, but still maintains speed and serialization to something like JSON. Pkl seems to fit the bill.

We want to provide a suite of "pre-defined" Pkl definitions, so that users can essentially adopt a "Pkl+"-like DSL, which we document and provide to consumers of this library or tool.

Approach

I see Pkl is written in Java, Kotlin, Groovy, etc., and we are very familiar with these languages. I don't know if Pkl's core code is designed for extension, or use as a library (with extension), beyond the documented runtime dependencies needed to load configurations, and so forth.

I also see it is built with Gradle; so are we. So maybe this is as simple as an includeBuild(...) statement, a Git submodule, and some dependencies.

Is there a supported approach to this already? If not, would the Pkl team be open to discussing such an idea?

Thank you for your hard work on this project and for consideration of this request 😄.

[bioball]

If your application is already written in Java, it should be very simple to import Pkl as a library into your application. In particular, you might be interested in either the pkl-core, or the pkl-config-java library.

When you use Pkl as a library, you can use its extension points by using custom module readers and resource readers. This lets you read("foo:whateverYouWant").

We want to provide a suite of "pre-defined" Pkl definitions, so that users can essentially adopt a "Pkl+"-like DSL, which we document and provide to consumers of this library or tool.

I think the most appropriate path here is to publish your Pkl definitions as a package, and this way your users can amends "package://path/to/your#/package.pkl", and get the most out of all of our tooling. Or am I missing something here?

[holzensp]

Adding to what @bioball said; what's in the + in Pkl+? Before considering language-level extensions, consider the benefit of having everything in your stack (the applications developed with your library, but also the infrastructure they run on) to be configured by one and the same Pkl configuration. There's value in being able to share configuration details across such boundaries. If you build an extension that's available in your library, but not in normal Pkl, deployment flows could become harder. You'd need to significantly guard against incompatible extensions "leaking" out of the app domain into other domains (or let your app code be the nexus for all things configuration).

Always happy to hear ideas here. If the things you're looking to + are compelling; they may just become part of Pkl-without-plus. Adversely, many things are likely doable in Pkl itself.

[sgammon]

@bioball / @holzensp Pkl is already extremely capable, but we want to provide a typed DSL they can follow; ideally, we want it to be embedded into a binary we re-distribute, so that the user invokes binary ... and it "just understands" the Pkl-defined DSL.

So far, here's what I'm understanding from the Pkl code and from suggestions above (thank you!):

• Pkl modules. We could distribute the DSL as a Pkl module, installed on the user's machine, with Pkl and with our tool. In this model, regular Pkl works because they are amending a Pkl module which can be located on-disk. The downside, of course, is that we'll need to make sure these files are all in the right position on disk, and that versions align enough with the user's copy of Pkl.

• Embedded JVM. We could distribute something like a jlink binary or a JAR (doesn't matter), and use pkl-core as a pkl-executor implementation on JVM. Custom module readers let us hook in at various points. In this model, the Pkl modules are distributed via Pkl's existing machinery (I haven't really used this yet), and our distributable knows how to read/interpret it via pkl-config-java; we would need to make sure that our binary and Pkl module align, but otherwise this should be simpler because the user's Pkl can rely on regular forward/backward compatibility semantics. One downside is that, as I understand it, the interpreted configuration would be dynamic here (instead of generated Kotlin, etc). Is that right?

• Embedded via Gradle. I tried the includeBuild(...) approach above, and have it working on my fork. This has helped a lot for testing with JPMS and other changes without needing to publish between changes. I can use the Pkl Gradle plugin, built from source. In this model, we would build our own native binary, similar to pkl-cli, with pkl-core embedded. We would probably use Pkl Core from a Context.newBuilder("pkl") just as the CLI/executor do. The benefit of this approach is that we could hook in to the GraalVM context to provide a custom FileSystem, which works nicely with other parts of our stack. Binary compatibility is also guaranteed for our "variant" of Pkl here.

Okay, I think the above summarizes the available approaches. It actually sounds to me like embedded JVM use is pretty close to optimal, so long as there is a way to hook into the org.graalvm...Context.Builder, which I added a simple plugin system for on my fork. With these hooks, I could use option 2 without any loss in control, and I definitely would prefer to stick to Pkl's code without source-level overrides if I can.

Adding to what @bioball said; what's in the + in Pkl+?

We want to build a build tool which uses Pkl as a DSL. Think Gradle's DSL but without the cumbersome precompile step. The tool is called dont, as it focuses on build avoidance. We are initially calling our DSL in Pkl "Dontfile syntax."

One thing we'd like to avoid, if possible, is the need for a user to include any boilerplate in a "Dontfile." The amends approach is valuable enough though that we're considering it anyway.

What do you guys think? Thank you for the advice here.

[bioball]

The path I'd recommend here is to distribute your Pkl DSL as a package. When you do that, your users can do:

amends "package://your.domain/[email protected]#/dont.pkl"

And your users can use all the standard tooling that we already provide for Pkl. They can import that module and get type checking and auto-completion, etc.

You can read more about packages and projects here: https://pkl-lang.org/main/current/language-reference/index.html#projects

Okay, I think the above summarizes the available approaches. It actually sounds to me like embedded JVM use is pretty close to optimal, so long as there is a way to hook into the org.graalvm...Context.Builder, which I added a simple plugin system for on my fork. With these hooks, I could use option 2 without any loss in control, and I definitely would prefer to stick to Pkl's code without source-level overrides if I can.

FWIW: we actually haven't explored the polyglot use-case yet. The golden path right now is to use the Evaluator API (you create an evaluator using EvaluatorBuilder, then use that for all evaluation needs). Maybe give that a shot first?