How to convert a Record to an Array of it's fields?

[ruoyu0088]

Some JSON file use Object instead Array to save data rows, for example:

json_str = '''
{
"a":{"name":"name1", "value":10, "data":[1, 2, 3], "key":"check"},
"b":{"name":"name2", "value":20, "data":[2, 3, 4]},
"c":{"name":"name3", "value":30, "data":[9, 3, 4, 10]}
}
'''

from_json() returns a Record:

import awkward as ak
data = ak.from_json(json_str)
ak.to_list(data)

{'a': {'name': 'name1', 'value': 10, 'data': [1, 2, 3], 'key': 'check'}, 
'b': {'name': 'name2', 'value': 20, 'data': [2, 3, 4]}, 
'c': {'name': 'name3', 'value': 30, 'data': [9, 3, 4, 10]}}

but what I really want is:

import json
data2 = ak.from_json(json.dumps([val | {'id':key} for key, val in json.loads(json_str).items()]))
ak.to_list(data2)

[{'name': 'name1', 'value': 10, 'data': [1, 2, 3], 'key': 'check', 'id': 'a'}, 
{'name': 'name2', 'value': 20, 'data': [2, 3, 4], 'key': None, 'id': 'b'}, 
{'name': 'name3', 'value': 30, 'data': [9, 3, 4, 10], 'key': None, 'id': 'c'}]

Are there any method that can convert from data to data2 without modify the original JSON data?

[agoose77]

There is nothing wrong (and indeed, it is sometimes favourable) with using Python routines to manipulate your JSON data. When loading anything from Python or JSON, Awkward already has to perform a loop, so there is no harm in doing it yourself, particularly when it's cleaner to do so. Awkward is designed to deal with columnar data, i.e. many rows, few columns, whereas this scheme has many columns, single rows. When ak.from_json builds this array, it effectively creates multiple versions of the same layout — one for each key.

Rather than doing all of this work, we can directly use a JSON reading library (such as json, but also orjson etc.) to load the data:

import json

data = json.loads(json_str)

Then we can restructure it. Python has long guaranteed that iterating over the various dictviews such as keys and values will visit each item in the same order, so we can now use ak.with_field to add a new id column to our iterator of rows:

array = ak.with_field(data.values(), data.keys(), "id")

One could also do this by first creating the array from the values, and then calling with_field. The above code is slightly smaller, because with_field can convert a Python object to an Array if necessary.

[jpivarski]

I thought I'd add just a little more:

The one thing that you want to avoid doing is making a record with a large number of fields (more than 1000, say) because different fields are different array buffers, and working with lots of little arrays gives you none of the performance benefit of working with arrays. This should be avoided even as an intermediate step: it's better to iterate over data in Python than to make a million little arrays (which then get iterated over in Python).

Since each of your top-level fields points to structures that are all the same type (in your example), you'll want the same-type data to ask be in shared fields. The problem is just to turn the dict that you start with into a list (i.e. dropping top-level field names) before converting it into an Awkward Array. This is what dict.values does.

ak.from_iter(json.loads(json_str).values())

(Okay, so some of the values of your top-level fields differ in that they have a "check" subfield, but this week be read in as though "check" were a missing value on all of the other records, and you can work with it in Awkward Array as an option-type.)