Handling external JavaScript-based libraries

[evmar]

In #1039 @theseanl is asking about module augmentation and externs generation, which requires a better understanding of how externs work at all, so I thought I'd write down a summary here.

When file A imports file B, tsickle rewrites them both to use goog.module/goog.require so Closure compiler can understand them. When you want to use an external library like react or jquery that is not written as a goog.module, what should tsickle do?

External libraries in Closure

The answer is that you first need to understand what Closure can do, and after you understand that, convince tsickle to emit whatever Closure wants.

There are two fundamental options:

1. You pass in the library as part of compilation so that the compiler includes it in the output.
2. You leave the library out of compilation, and handle it yourself.

For option 1 to work, you need the library to successfully pass compilation, and convince the compiler to put them in the output in the right order (perhaps by adding goog.module to the library source manually). In our experience this is possible for tiny libraries that you're willing to modify (like say "a uuid() function") and not possible for large libraries (like react).

So instead we generally recommend option 2. Here, your project has to bring in the library itself, e.g. via a separate <script> tag or by manually concatenating the library in front of the Closure compiled output, and then you need to convince Closure to produce a compiled bundle that references that script.

(If you're within Google you can read our massive doc go/tpl-js that tours the different ways different apps have tried to solve this, which are all minor variants of the above. It also has our proposal to fix it, more on this below. It's not too important here, it's just the above two paragraphs in more detail.)

External libraries as scripts

If you go with option 2, now you need to figure out how your code can refer to the external library. The simplest thing is when your library just produces some globals. E.g. after the <script> tag, imagine your library adds a function to window. This is relatively easy to make work in Closure, via externs, where you tell the compiler which global variables exist outside of your program.

In tsickle we take any script d.ts (and 'declare' statement within .ts) and generate externs from it. This ~mostly works. (It actually is still wrong: if a.ts does declare var x: string and b.ts does declare var x: number, TypeScript is happy to accept it but you get broken externs. This is an example of how subtle this all is, there are no good answers.)

External libraries as modules

More common these days is for an external library to be written as a module: in your TypeScript code you write an import statement. Closure basically has no real model for making this work -- if you import a library, it expects that library to be part of compilation.

So when tsickle sees a statement like import * as X from 'mylibrary', what should it do? All options are bad.

Our current design

Our current answer, which is not great but I am writing it down just so you can understand it, is to treat those imports the same as any other import. This means we let TypeScript resolve it to whatever file it thinks that import actually resolves to (which often means following node module resolution) and then we generate an import statement like goog.require('some.dotted.path.to.some.file');. That path often ends up under node_modules somewhere because that is usually where these libraries are defined.

But nothing defines the corresponding goog.module to satisfy that import, so this fails compilation. Within Google we sometimes write such a file by hand that attempts to glue things back together, see next section.

Finally, what happens to typings? The only reason the above import statement was even allowed by TypeScript is because there is a typings file somewhere that defines the library. This typings file is a module (contains an export statement), which we cannot translate into externs, because externs only let you define globals. As another weird hack, what we currently do is put all the types definitions into a hidden namespace with a name like tsickle_hidden_foo.your_library_here. This at least allows code that really wants to refer to those types to find them somehow, but is not otherwise linked into the rest of this system.

Gluing it back together

If there is a file (written by hand) that does something like

goog.module('the.name.under.node_modules');
/** @type {tsickle_hidden_foo.your_library} */
exports = the_actual_library;  // TODO: you figure this out

This glues all the systems back together. L1 makes the Closure compiler believe the import statement is satisfied, L2 shoves the TypeScript types into it, and in L3 you might be able to figure out what actual value should be used at runtime. In theory tsickle could autogenerate something like this maybe, but this whole area is really fragile already.

What should users do

It's all bad, I am sorry. As I wrote above we have made some proposals to Closure about how to make this better (which we haven't been able to convince the Closure team about yet -- briefly, my opinion is that we should make option 1 actually work) but they are very busy. Also there is some handling of node_modules within Closure itself (process_common_js_modules) that I have never taken the time to understand.