Spec Formalization: Human Readable Representation

[emkornfield]

In the short would it make sense to use protobuf's (or whatever binary format is chosen) as canonical text format for the human readable representation? It seems this would help ensure the IDL for both stays in sync while it goes through iterations. It also limits technologies that any library used to consume the spec needs to integrate with. The downside is not having some of the syntactic sugar of an ANTLR based grammar (although I suppose this could be integrated in protobuf as well, with something like custom annotations).

[jacques-n]

Having worked on this before way back when, I'd say that it is important that humans can easily produce (not just read) a text representation. Complex expression trees are very difficult to build by hand when using a graph of objects. I hate bringing in a parser but I don't really have any other ideas for how to make these human producible. When I did this before, I produced this antlr grammar. I'm sure it could be further simplified. I suggest people evaluating this try to explore a moderately complex query expression. FYI, when I first worked on the work linked above, I went all objects. It basically defeated the purpose of having something human readable/writable.

As a note, another point that I've struggled with is the expression of functions. Since there can be many add() implementations with different input types. Because the spec is defined to be specific about the actual type binding of each function, it means that add() in text representation is not enough. Best idea I've come up with is add_argument_types such as add_ii(foo,bar) if you're adding two integers (or whatever makes sense for each type).

[emkornfield]

Doing a quick search on ANTLR support in C++/Rust (there doesn't seem like there is a first class target)I'm a little concerned it might not be a great experience. I agree with the goals proposed, I'm just wondering if we can postpone heavy investment into the human readable/writeable phase until things are formalized?

If there is bandwidth to develop human readable along side binary representation, then great, but it seems like this requires a lot of infrastructure setup to guarantee compatibility between the two formats and across language bindings.

[jacques-n]

I think the link you had was about a C target, not C++. I believe C++ has a solid target. The Rust target is currently second class but relatively solid (really blocked by a missing feature in Rust last I checked). That being said, if there is an alternative mature parser/lexer framework I'm all for using it. The other note being that the grammar would be defined independently and then expressed in Antlr and would be hopefully simple enough that someone could reimplement it in another parsing system within a couple hours (I'd consider that a design goal).

WRT to the weight, I agree that it is a non-trivial effort. Part the reason that I didn't actually sketch any implementation is that I wanted things to start coalescing on the spec more before getting into this. That being said, it's close to the top of my priority list post spec agreement as I believe it will help drive innovation, especially in the space of frontends.

Of course, saying it's important now doesn't mean anything until myself or someone else actually volunteers time against it. :)

[jacques-n]

Another note is my expectation is that we'd have very few implementations of binary <> text conversion. (Maybe only one?). In most cases I'd expect people to work with one representation and simply use a conversion library if they want the other representation.

[emkornfield]

I think the link you had was about a C target, not C++. I believe C++ has a solid target.

Yes it is the C target. Its not clear from a casual perusal if the C++ target uses exceptions (which makes it harder to work with in Arrow) and the issues indicate some possible inefficiencies (which maybe isn't too important for text representation). ANTLR certainly seems like the most cross platform solution (quick web searches didn't really turn up a universal alternative).

[rdblue]

I'm all for having a human-readable representation, but it seems like adding one too early could easily side-track or delay the spec. I think serialization is less important because we can always come up with alternative serializations later. When we agree on what needs to be serialized, it is much easier to design how to do it.

[jacques-n]

+1