Selecting a name within multiple records: array[["pt", "eta", "phi"], "nominal"]

[maxgalli]

Hi,
first of all, thank you for the awesome job you're doing. I recently started using Awkward and I find it suitable and easy to use for most of the operations I usually perform during analysis.
I have a question regarding the possibility to get substructures from complex arrays.
Let's say I have an array whose type is the following:

22838 * {
"Muon_pt_1": {"Nominal": float64, "Up": float64, "Down": float64}, 
"Muon_pt_2": {"Nominal": float64, "Up": float64, "Down": float64}, 
"Electron_pt_1": {"Nominal": float64, "Up": float64, "Down": float64}, 
"Electron_pt_2": {"Nominal": float64, "Up": float64, "Down": float64}, 
"PV_x": float64, 
"PV_y": float64, 
"PV_z": float64
}

Is there a simple way to get an array containing e.g. all the Up values from the fields that have it?
Something like

my_array['*']['Up']

I took a look at the documentation but I wasn't able to find anything like this. Sorry if it's there and I missed it.

Thank you,

Massimiliano

[jpivarski]

You're right: there isn't an operation for that! There are a lot of ways to slice rows within rows, but this is a column within a column. Perhaps the following

my_array[["Muon_pt_1", "Muon_pt_2", "Electron_pt_1", "Electron_pt_2"], "Up"]

should be defined in such a way as to select the inner field "Up" from all four outer field names, by analogy with

another_array[:, 1]

for rows. It could be implemented here:

https://github.com/scikit-hep/awkward-1.0/blob/5625722d7da1e4c826783cf466c19e47bea15835/src/libawkward/array/RecordArray.cpp#L1621-L1630

Instead of applying the getitem_fields (slicing by ["Muon_pt_1", "Muon_pt_2", "Electron_pt_1", "Electron_pt_2"], which returns a new record of only these fields) and then applying the next tuple item ("Up") to the record that consists of those fields, the new feature would apply the rest of the tuple items (just "Up" in this case) to each of the fields ("Muon_pt_1", "Muon_pt_2", etc.) and then wrap the result of that up as a new record. It would be almost like moving the closing parenthesis of the getitem_fields call so that it includes the getitem_next, though it's not quite that simple.

So it could be done and it could be a new feature that you can request. The hard part is figuring out how to do the deprecation cycle: if people are relying on the current behavior, how do tell them that it's going to change so that they can fix their scripts?

Usually, new features are new function names or argument names, so they can't break any old code. When I want to change a named function or argument, I usually have to come up with a new name, so that the new name does the new thing and the old name has a deprecation warning and is eventually removed. (For example, renaming ak.from_arrayset → ak.from_buffers in #592.) But since this is a "__getitem__" thing, there's no name to change.

This my_array[["Muon_pt_1", "Muon_pt_2", "Electron_pt_1", "Electron_pt_2"], "Up"] would be a nice feature to have, and it would be more consistent with the way that my_array["Muon_pt_1", "Up"] works. I like it! But I think it would have to have a long deprecation cycle:

• in the first phase, uses of a multi-field selection followed by anything with a field or multi-field selection should raise a FutureWarning indicating when the change in behavior is going to happen and how to change code to keep the old behavior (users will need two square bracket selectors, one after the other);
• once the targeted date is reached, the new behavior becomes available for the first time. You wouldn't be able to use it until that day comes, because it would interfere with the old behavior.

That's not as nice as introducing a new name that people can use now, and it certainly doesn't help your short term problem of how to do it in your analysis. To do it now, you have to do it longhand:

ak.zip({field: my_array[field, "Up"] for field in ak.fields(my_array) if "Up" in ak.fields(my_array[field])})

(Note: the above would run in constant time, independent of the size of my_array, so it's not a performance problem. A for loop over columns is not slow like a for loop over rows, since we assume that arrays have no more than thousands of columns but can have billions of rows.)

About the "*" syntax, I'd rather not introduce something like that because there are no constraints on field names; "*" might be a valid field, and then how would you select it? Even with the above-described feature, you'd either have to know which fields might contain an "Up" or select them with

[x for x in ak.fields(my_array) if "Up" in ak.fields(my_array[x])]

But I'd like to keep this Discussion open in case there's a lot of interest in the

my_array[["Muon_pt_1", "Muon_pt_2", "Electron_pt_1", "Electron_pt_2"], "Up"]

selector.

[jpivarski]

Having mentioned the deprecation cycle, I should point out that this would also depend on having a formal roadmap: issue #616.

[jpivarski]

Despite all the big talk about a formal deprecation cycle, I noticed that the old behavior was never useful—any application of it could be rewritten in a simpler way. So I've redefined the old behavior as a bug that #619 fixes (immediately).

More detail here: #618 (comment).

[maxgalli]

Thank you for the detailed answer.
Yes, I agree that my_array[["Muon_pt_1", "Muon_pt_2", "Electron_pt_1", "Electron_pt_2"], "Up"] makes much more sense that using the '*'.
I will open a feature request later today and link this discussion to it.

[jpivarski]

For cross-linking completeness, this is issue #618.