How to remove the question mark in type after apply "ak.pad_none" to a full-value array?

[Star9daisy]

Hi developers, I want to keep a bunch of record arrays to have almost the same shape:

# In a jupyter notebook
records = ak.Array([[1, 2], [], [1]])
ak.to_regular(ak.pad_none(records, 3), None)

[[1, 2, None],
 [None, None, None],
 [1, None, None]]
--------------------
type: 3 * 3 * ?int64

The "3 * 3" part is the shape that I want to keep an array in. The "?int64" is also easy to understand: there maybe None type in this array. However, when the array is full of value or is exactly what I want, ak.pad_none will still change the data type inserting a question mark:

records = ak.Array([[1, 2, 3], [1, 2, 3], [1, 2, 3]])
ak.to_regular(ak.pad_none(records, 3), None)

[[1, 2, 3],
 [1, 2, 3],
 [1, 2, 3]]
--------------------
type: 3 * 3 * ?int64

Is there anyway to remove that "?" to indicate this array has exactly records of expected shape like "type: 3 * 3 * int64"?

[agoose77]

Having pad_none produce a different type according to the list-lengths of the array would introduce a data dependence on the operation result. That's not something we want to happen, as we rely on the lack of data dependence to be able to predict the result of Awkward operations ahead of time (used by https://github.com/dask-contrib/dask-awkward).

If you just want to drop the option type, because you know that the array does not contain any Nones, then you can use ak.drop_none(..., axis=-1). If you don't know whether your array will contain Nones then you need to use ak.fill_none, and choose the appropriate fill value.

[Star9daisy]

Morning @agoose77! Thank you for quick answer! I've realized that I could know whether an array should be padded in advance:

# In a jupyter notebook
required_length = 3  # axis=1
records = ak.Array([[1, 2], [], [1]])

is_padded = ak.all(ak.num(records) == 3)

array = ak.to_regular(ak.pad_none(records, required_length), None)
*shape_parts, dtype = str(array.type).split(" * ")
shape = " * ".join(shape_parts)
dtype = dtype if is_padded else dtype.replace("?", "")

print(f"shape: {shape}")
print(f"dtype: {dtype}")

shape: 3 * 3
dtype: int64

Now it's not a problem. 👍

[agoose77]

Morning!

I think it's worth adding more to this. In your case, you're adding None with pad_none; if there are no missing values already, then this introduces an option type. So, pad_none followed by drop_none is a "no-op" operation (add-then-remove-option).

Usually you want to avoid if branches that depend upon data like this; instead, if you need to drop the option later on, apply the appropriate operation. I.e., what do you expect to happen if records are missing? What do you want to do for that case vs non-missing data?

[Star9daisy]

Hi, I'm a developer from hep-ml-lab.

I'm working on parsing string like "jet.constituents.pt" to retrieve all transverse momenta from an array of records. The string is the default case, all constituents of all jets.

However, sometimes, "jet0.constituents:100.pt" (the first 100 constituents of the first jet) is needed. But there's no guarantee that there's a jet in an event. I could add a cut like "jet.size > 0" to ensure it but I want to tolerant such case: if there no jets, just pad a None, or if there's no enough constituents, pad None to fulfill the required length. So when checking the shape of this array, it's will show 1 * 100 and the dtype could give some hints for whether it's padded. I think 1 * 100 * ?float32 tell more than var * var * ?float32 in my use case.

There's indeed some operations after I get all pts like make jet images from them, or graphs from them. Usually, the array will be padded there. But the shape of the array before will contain much less information...

What's your thought about this? I'm at the early development stage and welcome any suggestions ❤️

[agoose77]

It sounds like you're inspecting the type-string to figure out which operations are permitted on an Array?

In general, this is perhaps not the most robust approach. It's usually preferable to write code that doesn't care about whether there are missing values or not.

If you want to pull out the first jet of each event, without handling the case that there is no such jet, you have some options:

maybe_jet0 = ak.firsts(events.jets, axis=1)

where maybe_jet0 is an array of jets or None values, of length len(events).

Another pattern, instead of using None, is to slice event.jets to a length-one-or-zero list:

up_to_jet0 = events.jets[:, :1]

Now, up_to_jet0 contains no Nones (assuming events.jets contains no Nones) and instead contains length-zero or length-one sublists, but most operations (slices, field access) will work just fine.

You can also filter events.jets to include only non-empty jets, as you point out:

at_least_one_jet = event.jets[ak.num(events.jets, axis=1) > 0]
jet0 = at_least_one_jet[:, 0]

But this maybe produce a jet0 array with a different length to events (a use case which normally motivates the first two solutions).

[Star9daisy]

In general, this is perhaps not the most robust approach. It's usually preferable to write code that doesn't care about whether there are missing values or not.

Oh my. That's exactly what I'm doing now 😂.

Another pattern, instead of using None, is to slice event.jets to a length-one-or-zero list

Here's my solution, almost the same! Considering all cases:

"jet0" -> is a "single" physics object: branch="jet", slices=[slice(0, 1)]
"jet:10" -> is a "collective" one: branch="jet", slices=[slice(0, 10)]
"jet0.constituents:100" -> is a combination of "collective" and "collective", a nested one: branch="jet.constituents", slices=[slice(0, 1), slice(0, 100)]. "jet0,jet1,jet2" -> is the ultimate case, a "multiple" physics object: branch=["jet", "jet", "jet"], slices=[[slice(0, 1)], [slices(1, 2)], [slice(2, 3)

With the branch and slices, I could fetch the observable values from events read by uproot.

where maybe_jet0 is an array of jets or None values, of length len(events).

I'm still getting familiar with awkward. I've implement some function working like ak.first before before I read it and understand its usage in the user guides this afternoon. I will refactor some parts of my codes in the near future because now I have to move forward to finish the paper about my little framework.

Thank you a lot !