Broadcasting lists with different structures

[Durman]

Hello, I'm trying to adopt the library to deal with geometry data which include such types of elements as vertices. I have next shape of data which works perfectly.

ak.Array([[1],[2,3]]) + ak.Array([1])
# <Array [[2], [3, 4]] type='2 * var * int64'>

But, instead of scalar values I have vertices.

ak.Array([[[1,1,1]],[[2,2,2],[3,3,3]]]) + ak.Array([1,1,1])
# Error details: cannot broadcast RegularArray of size 2 with RegularArray of size 3 in add

Ok, it probably can't broadcast vertices because the array does not know that it contains vertices.

<Array [[[1, 1, 1]], [[2, ...], ...]] type='2 * var * var * int64'>

So I've tried to build the array in more clever way.

verts = ak.Array(ak.contents.ListOffsetArray(ak.index.Index64(np.array([0,1,3])), ak.contents.RegularArray(ak.from_iter([1,1,1,2,2,2,3,3,3], highlevel=False), 3)))
# <Array [[[1, 1, 1]], [[2, 2, 2], [...]]] type='2 * var * 3 * int64'>

But it seems does not fix anything.

verts + ak.Array([1,1,1])
# Error details: cannot broadcast RegularArray of size 2 with RegularArray of size 3 in add

The question is how to broadcast awkward list of vertices?

P.S. Also I've tried such variant:

verts + ak.Array(ak.contents.RegularArray(ak.from_iter([1,1,1,2,2,2], highlevel=False),3))

    numpy.add.__call__(
        <Array [[[1, 1, 1]], [[2, ...], ...]] type='2 * var * 3 * int64'>
        <Array [[1, 1, 1], [2, 2, 2]] type='2 * 3 * int64'>
    )
Error details: cannot broadcast nested list (in compiled code: https://github.com/scikit-hep/awkward-1.0/blob//src/cpu-kernels/awkward_RegularArray_broadcast_tooffsets.cpp#L18)

[agoose77]

Hi @Durman,

To start with, I commend you on going the distance in trying to figure this out. We consider our Content classes to be advanced features that most users of the library should never need to know about. You were pretty close with your first attempt!

The first difference here is that NumPy performs right broadcasting:

>>> np.broadcast_shapes(
...     (4, 3, 7,),
...           (7,)
... )
(4, 3, 7)

which aligns elements "to the right".

Awkward supports this broadcasting for entirely-regular (NumPy-like) arrays

>>> ak.from_numpy(np.zeros((4, 3, 7))) + ak.Array([1, 2, 3, 4, 5, 6, 7])
<Array [[[1, 2, 3, 4, 5, 6, 7], ..., [1, ...]], ...] type='4 * 3 * 7 * float64'>

But for any broadcast involving a ragged array, only left broadcasting is used. This would mean that, if NumPy supported this, one would see

>>> np.broadcast_shapes(
...     (4, 3, 7,),
...     (4,)
... )
(4, 3, 7)

So, in your case, Awkward Array is left broadcasting an array of length-2 against an array of length 3. That is an invalid broadcast. You want right broadcasting here, which you can perform yourself by constructing the appropriate slice to introduce regular length-1 dimensions

x = ak.Array([[[1, 1, 1]], [[2, 2, 2], [3, 3, 3]]])
y = ak.Array([1, 1, 1])
z = x + y[np.newaxis, np.newaxis, :]

where the : is superfluous, but helps readability.

As an aside, you might find the vector library useful here too. It provides some Awkward behaviour classes that add useful methods to arrays such as vector addition, dot products, etc.

[Durman]

Thanks, the answer is easier than I thought. I can't say that now I clearly understand how broadcasting rules work but I hope it will be improved through practice.)

Thanks for pointing to the vector library I'll have a look.

[agoose77]

We have some documentation on broadcasting here, though I can summarise it.

Awkward Array was originally written for Particle physicists to use in analyses. Most analyses ultimately work with data that looks like

for event in events:
    for particle in event.particles:
        do_something_with_particle()

As an array, this would look something like

[
    ["particle-1", "particle-2", ...],                # event 1
    ["particle-1", "particle-2", "particle-3", ...],  # event 2
    ...
]

It rarely if ever makes sense to do any kind of calculations that mix different events, because they are unrelated. So, we always want the outermost axis axis=0 to line up. NumPy's broadcasting rules, which align axis=-1, do the wrong thing as far as particle physicists are concerned. Hence, Awkward introduced "left" broadcasting, which always lines things up from left to right.