net, sta, chan, and loc and Database schemas

[pavlis]

I came across an anomaly while testing the new version of Database and read/write_distributed_data that is the latest of a string of problems created by the ubiquitous miniseed net:sta:chan:loc attributes. I discovered the problem when testing the new Database code with the "getting_started" notebook in our tutorials repository. The notebook has this code box that showed the problem that caused me to write this page:

from mspasspy.util.converter import Trace2TimeSeries

#db=Database(dbclient,'getting_started')
# We have to redefine the client from obspy to use their so called bulk downloader
from obspy.clients.fdsn import RoutingClient
client = RoutingClient("iris-federator")
# This uses the (admitted obscure) approach of a set container of keepers to reduce the download time
keepers=set()
keepers.add(4)
keepers.add(7)
keepers.add(10)

cursor=db.source.find({},no_cursor_timeout = True)
count=0
start_offset=763.0-4*60.0
end_offset=1400.0+10*60.0
i=0
for doc in cursor:
    if i in keepers: 
        origin_time=doc['time']
        # We need the ObjectId of the source to provide a cross reference to link our 
        # waveform data to the right source.  Better to do this now than later as association 
        # can be a big challenge
        id=doc['_id']
        
        print('Starting on event number',i,' with origin time=',UTCDateTime(origin_time))
        stime=origin_time+start_offset
        etime=origin_time+end_offset
        strm=client.get_waveforms(
            starttime=UTCDateTime(stime),
            endtime=UTCDateTime(etime),
            network='TA',
            channel='BH?',
            location='*'
        )
        # The output of get_waveforms is an obspy Stream object. Stream objects are iterable 
        # so we work through the group, converting each to a MsPASS TimeSeries, and then saving 
        # the results to the database.  
        for d in strm:
            d_mspass=Trace2TimeSeries(d)
            # Here is where we save the id linked to the source collection. 
            # We use a MsPASS convention that such metadata have name collection_id
            d_mspass['source_id'] = id
            d_mspass=db.save_data(d_mspass)
            print('Saved data for sta=',d_mspass['sta'],' and chan=',d_mspass['chan'],"status=",d_mspass.live)
            count += 1
            if count>20:
                break
    else:
        print('Skipping event number ',i,' with origin time=',UTCDateTime(origin_time))
    i += 1
print('Number of waveforms saved=',count)

A separate issue is we need to change the method of pulling data with get_waveforms as the download can take hours, but what matters here is that the line calling save_data in the loop over strm does a nasty thing in my new implementation: it strips the net, sta, chan, loc attributes form the documents it saves in the wf_TimeSeries collection in this code. After some debugging I now know why. The following standardized code for converting Metadata to a python dict causes this problem:

    if "_id" in copied_metadata:
        copied_metadata.erase("_id")
    # Now remove any readonly data
    for k in copied_metadata.keys():
        if save_schema.is_defined(k):
            if save_schema.readonly(k):
                if k in changed_key_list:
                    newkey = "READONLYERROR_" + k
                    copied_metadata.change_key(k, newkey)
                    elog.log_error(
                        "Database.save_data",
                        "readonly attribute with key="
                        + k
                        + " was improperly modified.  Saved changed value with key="
                        + newkey,
                        ErrorSeverity.Complaint,
                    )
                else:
                    copied_metadata.erase(k)

    # this section creates a python dict from the metadata container.
    # it applies safties based on mode argument (see user's manual)
    if mode == "promiscuous":
        # A python dictionary can use Metadata as a constructor due to
        # the way the bindings were defined
        insertion_doc = dict(copied_metadata)
    else:
    ,,,,rest if the code for cautious and pedantic mode is below here 

The problem is that the keys net, sta, chan, and loc in this context are not marked as changed. As a result they all are erased.

Now this section has been trouble since the early days of MsPASS. Earlier versions of this same tutorials had to deal with the other way this would be handled. That is, earlier versions had net, sta, chan, and loc somehow marked as "changed" earlier in the algorithm. That is, the current code base has exactly this same algorithm but something else has changed that now doesn't mark net, sta, chan, and loc as changed. When they were marked changed we ended up with obnoxious READONLYERROR_net, READONLYERROR_sta, etc. and an elog entry for every call to save in this tutorial section. The workaround we used back then was to change the schema to "mspass_lite". That would still solve this particular problem too, but I think using that workaround for this particular fix alone is NOT a good idea. I would like to suggest we make a couple bigger changes that will make saving data more robust. This particular anomaly is NOT acceptable as this example runs for hours and products a useless database in which the waveform documents cannot be linked with channel metadata by any means. It would be way way too easy for an innocent user to make that mistake.

Items I recommend for discussion are:

1. I think the loop for readonly should be replaced by a standard function or method to strip all standard normalization generated attributes except "_id". That is, attributes with names like channel_net. That kind of attribute should ALWAYS be stripped from the document before it is saved. I would recommend we add a kwarg of normalizing_collections=["channel", "site", "source"] and erase any key implied by those names (e.g. "site_lat", "channel_vang", or "source_lat"). I emphasize things like "channel_id" must ALWAYS be treated specially as a standard normalization key.
2. I think the readonly loop should be moved to within the else block for "cautious" and "pedantic" mode.
3. I really think we should make net, sta, chan, and loc as optional but standard attributes in wf_TimeSeries and net, sta, chan as standard in wf_Seismogram. It is attributes like channel_sta, channel_net, etc. we should treat as readonly. net, sta, chan, and loc are so pervasive in seismology today we should not create problems when a workflow needs to carry them around.