Awkward array as view over collection of C++ pointers?

[d-leroy]

Hello,

Is it possible to build awkward arrays (more specifically record arrays) in C++ from various pointers to C++ arrays, to obtain what would basically be a awkward array "view" aggregated from several arrays that could then be exposed through an embedded Python interpreter?

Let's say I have arrays A, B, and C. I would like to aggregate them into an awkward record array of the form {"A": A, "B": B, "C": C}, without copying data, to pass to the numba-jitted version of a Python function provided by the user.
Note that those C++ arrays can be wrapped as numpy arrays if that helps.

Is this possible at all, and if it is, how I should go about doing this effectively?

Thanks in advance for the help!

[agoose77]

With v2, Awkward Array is removing the mid-level C++ such that predominantly the kernels are left in C++. Whilst, for v1, there is a C++ layer, it will disappear.

So, if you want to create Awkward Arrays from C++, your best bet is probably using e.g. pybind11 to invoke Python code: https://pybind11.readthedocs.io/en/stable/advanced/embedding.html
This is what the new fastjet bindings do, e.g. https://github.com/scikit-hep/fastjet/blob/80c6ed177cc09667cb3f997a632edd21d342e906/src/fastjet/_singleevent.py#L69-L83

In principle you can just return Python buffers and pass them to the zip function, but in practice I think we probably don't generalise all the way down to non-NumPy arrays, so you'd want to wrap each pointer in a NumPy array

[jpivarski]

Well, actually, there are exceptions to exceptions and although we're moving the implementation of Awkward Arrays from C++ to Python, we're adding new ways of creating Awkward Arrays in C++.

This is from @ManasviGoyal's draft PR #1494: https://github.com/scikit-hep/awkward/blob/1ada7670af7c386478a5b818dc967a8c9e44f604/src/awkward/cpp-headers/GrowableBuffer.h

The difference here is that the new array-creating code is in a file of header-only template definitions, obliviating all the issues with linking against compiled shared libraries. Also, it's not the main codebase, the implementation of the arrays themselves (which was a maintenance burden for us, since it's so much easier to develop in Python). This just constructs arrays.

Thinking about @d-leroy's question, I guess he doesn't actually need LayoutBuilder to exist, he just needs to do the same sort of thing that LayoutBuilder does when jumping the border from C++ to Python. He needs to present his data to Python as a Py_buffer object, which can be done in pybind11 (for example) using py::buffer_info. I'm assuming the data will be owned by a C++ object—either the ownership will be managed by putting a Py_DECREF in the C++ object's destructor or through pybind11's return value policies. Or maybe they're long-lived objects and he'll just keep them around forever or handle ownership manually. Anyway, if it's not done right, Python can segfault as easily as C++ can.

Actually, if you're not worried about ownership safety at all, you can make a buffer entirely on the Python side with only two integers: the pointer value and length of the buffer. This SO question explains how to do it; a NumPy array is a buffer and can be used in ak.from_buffers (though the NumPy dtype will be ignored).

So assuming that he has Python buffer objects for each of the arrays that goes toward building an Awkward Array (the data of a NumpyArray, the offsets of a ListOffsetArray, etc.), then he just has to describe how they fit together with a Form (which can be JSON) and he has to know the length of the final array. That's the interface of ak.from_buffers.

It may be useful to experiment with ak.to_buffers to see what these things look like for a given array of a desired type. Also there's a tutorial on using them here. The from_buffers/to_buffers pair is slightly different between version 1 and the impending version 2, as are the Form constructors, so it's probably a good idea to learn and build on the version 2 interface. (The differences I'm talking about is where the functions are located, whether they have underscores between words of their names, the spelling and defaults of some arguments, etc.)

[agoose77]

Right - to add some context, one of the motivating factors for replacing the C++ layer with Python was that whilst it didn't offer performance gains (in fact, harmed performance), it did prove tricky to link against, and limited integration with other libraries like Jax (see https://indico.cern.ch/event/855454/contributions/4605044/ for more). That's why Jim refers to the new header-only library that is being built in cpp-templates.

As Jim and Ianna elaborate, @d-leroy just needs to pass their data to the Python constructors, whether that's ak.zip, ak.from_buffers, or even the actual layout objects themselves. If they want to provide a nice API that the user calls, they must either write a thin Python shim that collects these buffers and builds the layout object, or directly call awkward from C++ using the pybind11 Python API: https://pybind11.readthedocs.io/en/stable/advanced/embedding.html#adding-embedded-modules

Note that, as well as pybind11, there's also nanobind.

[ianna]

@d-leroy - I think, you can try the following:

1. Wrap C++ arrays as numpy arrays. This is just an example:

>>> A = np.array([1,2,3,4])
>>> B = np.array([1.1, 2.2, 3.3, 4.4, 5.5])
>>> C = np.array([1+0j, 2+2j, 3+3j])

2. Describe the Form of the future ak._v2.Array you want to get. Note, you need to provide a unique "form_key" for each content:

>>> form_str = """
...      {
...          "form_key": "node0",
...          "class": "RecordArray",
...          "contents": {
...              "A": {
...                  "class": "NumpyArray",
...                  "primitive": "int64",
...                  "form_key": "node1"
...              },
...              "B": {
...                  "class": "NumpyArray",
...                  "primitive": "float64",
...                  "form_key": "node2"
...              },
...              "C": {
...                  "class": "NumpyArray",
...                  "primitive": "complex128",
...                  "form_key": "node3"
...              }
...          }
...      }
...      """
>>> 
>>> form = ak._v2.forms.from_json(form_str)

3. Define a dictionary to associate the "form_key" with each numpy array:

>>> buffers = {"node1-data": A, "node2-data": B, "node3-data": C}

4. Pass it to ak._v2.from_buffers:

>>> array = ak._v2.from_buffers(form, 3, buffers)
>>> array
<Array [{A: 1, B: 1.1, C: 1+0j}, ..., {...}] type='3 * {A: int64, B: float6...'>