UTC & leap seconds (again! again!!!)

[pvanlaake]

Topic for discussion

A Wise Man once said “If you think you know all about Time, you haven’t thought long enough”. Hence I beg your forgiveness for bringing up the UTC / leap seconds issue after the extended discussion (#304) and the recent inclusion in CF 1.12 (not to mention my missing the PR).

First of all, great job on the revamped section 4.4 and happy to see some new “calendars” added.

That immediately brings me to the first issue: Neither UTC nor TAI are calendars, they are clocks. So rather than than saying for utc “A Gregorian calendar with leap seconds as prescribed by UTC”, can this be changed to “Coordinated Universal Time (UTC) which includes leap seconds and where dates are represented using the Gregorian calendar”? And likewise for tai.

In the description of both calendars I would add that time zone offsets are not allowed and that a time zone indication, when given in a datetime, must be 0.

UTC

More concerning, from my perspective, is the definition of utc. As defined, it starts in 1958, but UTC did not exist until 1960 and was only officially adopted in 1963 (it was based on that same epoch as TAI, which is 1958-01-01 00:00:00 UT). The issue of leap seconds in UTC is not meaningfully addressed in the new text. Of course, there is section 4.4.3, but that really only deals with the consequences of having to consider leap seconds. I have seen the arguments being made in issue #542 but it feels unbalanced to have the detailed explanation of figure 4.1 but no basic information on leap seconds (such as where to find an authoritative list of the 27 leap seconds since 1972) or how it differs from the other calendars.

Further, prior to 1972 there were no leap seconds. The divergence of nearly 10 seconds between UTC and UT that accumulated between 1958-01-01 and 1971-12-31 was applied through three mechanisms: (1) a shifting in the length of the UTC second, at various rates over the period 1960-1971; (2) frequent small increments of multiples of 50ms; and finally (3) an anomalous jump of 0.107758 SI seconds to make the difference exactly 10 seconds at the onset of 1972-01-01. In other words, prior to 1972, the second used by UTC is not equal to the SI second. The introduction of the leap second was made exactly to align the UTC second with the SI second and to make time shifts to align the UTC clock with the celestial reality less burdensome (all of this before the advent of modern computer and communication infrastructure). You may find all the details here.

It is then a bit of a mystery to me why CF would want to dive head-first into this murky 1960-1971 business that many Wise Men have tried to obliterate through the introduction of the leap second. Is there a known use case for data sets using UTC that span the early years of that clock?

I would propose to side-step all of this complication by changing to definition of utc to begin at 1972-01-01 00:00:00, being the instant when UTC started using the SI second and leap seconds were introduced, with no time coordinates prior to that instant allowed.

units_metadata

I do not understand the purpose of this argument. The utc calendar was introduced to enable data producers that need second accuracy to record their observations correctly. When leap seconds are of concern, then why would the data producer use the standard or proleptic_gregorian calendar? Having this need for accuracy but not knowing whether or not the data collection equipment properly records and processes leap seconds does not seem like a real-world combination to me.

Why is there reference to the julian calendar here? Leap seconds are defined in the context of UTC and that uses the Gregorian calendar exclusively.

Finally, why an attribute units_metadata with a compound structure? Wouldn’t a simple attribute leap_seconds suffice?

[ChrisBarker-NOAA]

I'm all for making the docs more precise and clear, which I think is what much of this is about.

But is there any of this that might cause actual problems with using the new calendar?

To a few points:

“A Gregorian calendar with leap seconds as prescribed by UTC”, can this be changed to “Coordinated Universal Time (UTC) which includes leap seconds and where dates are represented using the Gregorian calendar”? And likewise for tai.

Sure -- that seems a bit more correct.

In the description of both calendars I would add that time zone offsets are not allowed and that a time zone indication, when given in a datetime, must be 0.

I don't think that's wise. As you point out about above, UTC is not a calendar per se -- a calendar specified days, and the time within in that is kind of a separate thing. However, for CF, they are a single concept a "calendar" is the full spec for how to interpret (and produce) datetime strings. So the difference between, say the standard calendar and utc is leap seconds, that's it -- there is just as much validity to specifying a timezone offset with the UTC calendar as any other. In that case, the time is not UTC, but it is relative to UTC -- so that's fine.

([UTC] was based on that same epoch as TAI, which is 1958-01-01 00:00:00 UT).

exactly, so it's valid from that date, yes? what's the problem here?

The issue of leap seconds in UTC is not meaningfully addressed in the new text. ... but no basic information on leap seconds (such as where to find an authoritative list of the 27 leap seconds since 1972) or how it differs from the other calendars.

I think that was pretty deliberate -- we try not to re-write specifications that others are responsible for. That being said, a bit more detail, and in particular a link to an external source with the full explanation would be great -- can you propose one?

It is then a bit of a mystery to me why CF would want to dive head-first into this murky 1960-1971 business that many Wise Men have tried to obliterate through the introduction of the leap second.

I'm guessing that was out of ignorance ....

I would propose to side-step all of this complication by changing to definition of utc to begin at 1972-01-01 00:00:00, being the instant when UTC started using the SI second and leap seconds were introduced, with no time coordinates prior to that instant allowed.

That sounds good to me -- but I'll let others that may know better weight in.

The goal here is that people can use a UTC timestamp, convert that to a CF-style "time-since" form, (and vice versa) and clearly define what that means. If that's not possible pre-1972, then yes, we shouldn't allow it.

When leap seconds are of concern, then why would the data producer use the standard or proleptic_gregorian calendar? Having this need for accuracy but not knowing whether or not the data collection equipment properly records and processes leap seconds does not seem like a real-world combination to me.

The issue is not that data producers don't know whether leap seconds are included in their timestamps -- the problem is that most of the software out there is not leap-second aware. So if a producer knows they have proper UTC timestamps, but their software (and that of their users) doesn't support leap seconds, then they need to use the standard calendar.

Practicality beats purity, and all that.

Why is there reference to the julian calendar here? Leap seconds are defined in the context of UTC and that uses the Gregorian calendar exclusively.

agreed -- we should remove that.

Finally, why an attribute units_metadata with a compound structure? Wouldn’t a simple attribute leap_seconds suffice?

because units_metadata was already introduced for temperature -- so keeping things a bit more consistent.

However, now that you mention is, I see in section 3.1.2:

"A variable must not have a units_metadata attribute if it has no units attribute or if its units do not involve a temperature unit."

That needs an update.

[pvanlaake]

Hi Chris, thanks for your reply. A few responses to your comments:

there is just as much validity to specifying a timezone offset with the UTC calendar as any other

Here I have to disagree. The U stands for Universal, UTC time is the same all over the world at a single instant, irrespective of where you are. Bangkok, London and Denver all have the same UTC time at the same instant.

UTC is managed by BIPM so I would suggest a reference to https://www.bipm.org/en/work-programme/time as the authority on UTC and TAI would suffice, perhaps with the addition that this is technical background information. On that page is a link to "time department database" where lots of reference material is available. Under "resources", then "time scales" are a number of files, including a list of the 27 leap seconds that have been applied. Relationship between TAI and UTC is instructive in the context of this discussion. The first few lines read:

 RELATIONSHIP BETWEEN TAI AND UTC
 ------------------------------------------------------------------------------- 
 Limits of validity(at 0h UTC)       TAI - UTC  
 
 1961  Jan.  1 - 1961  Aug.  1     1.422 818 0s + (MJD - 37 300) x 0.001 296s
       Aug.  1 - 1962  Jan.  1     1.372 818 0s +        ""
 1962  Jan.  1 - 1963  Nov.  1     1.845 858 0s + (MJD - 37 665) x 0.001 123 2s
 1963  Nov.  1 - 1964  Jan.  1     1.945 858 0s +        ""
 1964  Jan.  1 -       April 1     3.240 130 0s + (MJD - 38 761) x 0.001 296s
       April 1 -       Sept. 1     3.340 130 0s +        ""
       Sept. 1 - 1965  Jan.  1     3.440 130 0s +        ""
 1965  Jan.  1 -       March 1     3.540 130 0s +        ""
       March 1 -       Jul.  1     3.640 130 0s +        ""
       Jul.  1 -       Sept. 1     3.740 130 0s +        ""
       Sept. 1 - 1966  Jan.  1     3.840 130 0s +        ""
 1966  Jan.  1 - 1968  Feb.  1     4.313 170 0s + (MJD - 39 126) x 0.002 592s
 1968  Feb.  1 - 1972  Jan.  1     4.213 170 0s +        ""
 1972  Jan.  1 -       Jul.  1    10s            
       Jul.  1 - 1973  Jan.  1    11s
 1973  Jan.  1 - 1974  Jan.  1    12s	
...

So in 1961-01-01 the difference between TAI and UTC was 1.422818 SI seconds and increasing at a rate of 0.001296 SI seconds per day. That rate was adjusted three more times before 1972 and in addition there were steps of multiples of 0.05 SI seconds. Given that there were still only 86400 seconds in a day, the UTC second did not align with the SI second.

([UTC] was based on that same epoch as TAI, which is 1958-01-01 00:00:00 UT).

exactly, so it's valid from that date, yes? what's the problem here?

The values may have been coincident on that day, but they immediately started drifting apart.

Best, Patrick

[JonathanGregory]

Dear Patrick

Thanks for your comments and the discussion.

• You suggest we should change the first sentence in Sect 4.4.2 defining the utc calendar from "A Gregorian calendar with leap seconds as prescribed by UTC" to "UTC, which includes leap seconds and where dates are represented using the Gregorian calendar." (We don't need to spell out what "UTC" stands for in this sentence, because it is well-known and was anyway already defined earlier in Sect 4.4.2.) Actually, I don't see a difference of meaning between these two versions. Maybe the word "prescribed" is unclear? We could change "with leap seconds as prescribed by UTC" to "with the leap seconds that are included in UTC." Would that be better?

• I agree that it would be helpful to provide a link to Relationship between TAI and UTC. Thanks for suggesting it. We could also link Leap_Second.dat. Both of these files are available for download from BIPM under the CC BY-ND 4.0 licence, I see.

• You're right that we elided the complicated history of adjustments to UTC betwen its invention and 1972. I hadn't grasped that they had "rubber seconds" (as explained in wikipedia more concisely than by BIPM). Because of this, I think you're right that we should prohibit utc before 1972. In CF, a second is always an SI second.

• Like Chris, I don't think it's problematic to use time-zones with utc and tai, because these are CF calendars, not time-zones. A CF calendar defines an ordered set of valid datetimes with integer seconds, as stated in Sect 1.3 and at the start of Sect 4.4.2. We could remind the reader of this in the paragraph defining utc. It's important to be aware of, because the leap seconds are defined in UTC +0 and therefore occur in different hours when there's a non-zero time-zone offset. We should mention this at the end of Sect 4.4.1 as well.

• Chris, thanks for noticing that in Sect 3.1.2 we need to change "A variable must not have a units_metadata attribute if it has no units attribute or if its units do not involve a temperature unit." We overlooked it there. We did remember to update the conformance document, I'm glad to say, which now reads, "A variable must not have a units_metadata attribute if it has no units attribute, or if its units do not involve a temperature unit or a reference time unit."

• We included the julian calendar for completeness in the treatment of leap seconds, because it was easy to do, although there's unlikely to be a CF use-case. The Orthodox churches and some Berber communities (see e.g. Amazigh New Year) use the Julian calendar. Since they don't differ from everyone else in clock time, that means their version of the Julian calendar (in the CF sense) must include UTC leap seconds. Postscript: Acknowledging that leap seconds are unlikely to be needed with the Julian calendar, we could make units_metadata="leap_seconds: none" the default (instead of unknown) for calendar="julian". Then we need not recommend including units_metadata for that calendar. Do you think that would make more sense?

Best wishes and happy New Year (whenever it exactly occurs for you!)

Jonathan

[pvanlaake]

Dear Jonathan,

Happy New Year to you and all other collaborators on the Gregorian calendar! (For those of you on the Julian calendar, do not hesitate to fault me if I forget about your New Year in 9 days' time.)

I remain rather uncomfortable with some of the arguments made in previous comments. As I understand these arguments, they are based more on ease-of-use and every-day practice rather than standards and principles. UTC is one thing and one thing only. Time zones are referenced to UTC so to call a calendar utc and then allow, say, -6 to indicate that the time is actually local time 6 hours behind UTC comes across as a CF-specific quirk that will confuse people more than that it will help them. The fact that in Boulder, CO, there may have occurred an instant of time with the datetime 2015-06-30 17:59:60 -6 doesn't turn Mountain Daylight Time into UTC, so why create that impression through the calendar utc? (This example is not incidental. The conventions document has used an example from the UDUNITS documentation since at least version 1.7 which refers to Mountain Daylight Time, as observed in Boulder, CO, where, probably totally by coincidence, UCAR is located. Perhaps tellingly, the reference to MDT has disappeared in the most recent conventions text. So the explicit reference to daylight savings time has disappeared (and a good thing too!) but now we have the leap second.)

The first use-case that was made for a calendar using UTC would very unlikely be using the local time as reported by a local computer synchronising with an NTP server, and which might result in a datetime as indicated above. And that is probably the case with the large majority of data producers using the CF conventions. In climate and weather forecasting science, practitioners tend to know their clocks, and how to convert from their local time as reported by their instruments and processing environment into UTC. So, again, why the extra whistles and bells?

It should also be considered that utc as currently defined may lead to a confusion between a data producer using utc + timezone, and a data user interpreting that as UTC. Buyer beware!

The CF calendar utc uses the UTC clock and the Gregorian calendar: two separate things. The fact that these two separate entities are conflated into a single CF calendar is the root of the issue we are discussing now.

Best,
Patrick

[JonathanGregory]

Dear Patrick

Thanks for your further thoughts. You comment that the arguments "are based more on ease-of-use and every-day practice rather than standards and principles." I think that's partly true. Perhaps you've seen that the preceding discussions on the issue of leap seconds were extraordinarily long and difficult. It seemed to me that these arguments were often about what things ought to mean "in principle". Because of their different understanding of these principles, the participants in the debate could not agree, and perhaps sometimes could not understand one another properly.

In this case, we could not achieve an consensus about all the principles involved. This is a very unusual outcome for CF discussions, but it's not a show-stopper, because CF is a convention i.e. an agreement, rule, practice or custom. Even though we couldn't agree on what things ought to mean, we did manage to agree some workable conventions. You're not alone in being uncomfortable about some of our choices. That's regrettable, but perhaps can't be helped.

I think the important question to consider is whether there are any hazards with these conventions that are likely to cause data-producers to write incorrect metadata, or data-users to misinterpret the metadata. If there are, we should either clarify the standards document, or consider changing the conventions. This is consistent with one of the principles for design (Sect 1.2):

The conventions should minimise the possibility for mistakes by data-writers and data-readers.

You wrote about some possible hazards before, and I have responded with possible remedies. Thanks for those points. Here, you're suggesting that time zones in utc are a hazard:

It should also be considered that utc as currently defined may lead to a confusion between a data producer using utc + timezone, and a data user interpreting that as UTC.

Timezones have always been allowed in CF datetimes. This convention was iinherited from COARDS, which contains the example in Mountain Daylight Time which you refer to. As COARDS says, that example comes from the UDUNITS documentation. Another of the CF principles is

Conventions are provided to allow data-producers to describe the data they wish to produce, rather than attempting to prescribe what data they should produce.

If you have a data-logger which records local time and is leap-second-aware, the datetimes would most conveniently be expressed in CF using the utc calendar with a timezone offset in the units.

In the example you give, you might have a time coordinate of 64801 with units="seconds since 2015-6-30 0:0:0 -6" and calendar="utc". The convention says, "Subtracting the time zone offset from a given datetime converts it to the equivalent datetime with zero time zone offset." Therefore seconds since 2015-6-30 0:0:0 -6 means the same as seconds since 2015-6-30 6:0:0 in UTC. 64801 seconds since 2015-6-30 6:0:0 is the leap second 2015-6-30 23:59:60 in UTC, hence 2015-6-30 17:59:60 -6.

Where would the confusion arise? Do you think the data-user might ignore the timezone in the units, simply because the calendar is called utc? In that case, maybe we should adopt a different name for this calendar? What about civil, as a suggestion? Wikipedia, Royal Museums Greenwich and Time and Date all use "civil time" to mean local time which differs by an offset from UTC.

Best wishes

Jonathan

[ChrisBarker-NOAA]

I’m on vacation, so brief, but a few notes. Here I have to disagree. The U stands for Universal, UTC time is the same

all over the world at a single instant, irrespective of where you are. Bangkok, London and Denver all have the same UTC time at the same instant.

Of course, but the offset is with respect to UTC, and it is provided as part of the string, so there is no confusion about what it means. And this isn’t a CF-specific oddity — it’s part of the ISO 8601. As you say, CF is indeed focused on “on ease-of-use and every-day practice rather than standards and principles.” “Practicality beats purity” as the Python folks say. Indeed, that’s why UTC has not been supported in CF until now— very little software (and none of the most commonly used tools) support it properly. Note: we should try to use “time zone offset”, or “offset”, rather than “time zone”. Everyone in this discussion knows the difference, but when written down, we should make sure to minimize confusion. One of the practical challenges is that a LOT of people assume UTC means “time at the prime meridian”, I.e. with no time zone offset applied, which it does, but it’s also more specific than that, but that distinction is lost on many people. That rate was adjusted three more times before 1972 …

The values may have been coincident on that day, but they immediately

started drifting apart. Ah, thanks - then I agree— 1972+ makes better sense then.

What about civil, as a suggestion?

I think that would be even more confusing. UTC +- offset is precise and clear and very commonly used.

…

-CHB

[pvanlaake]

I believe we may be converging on an outcome that is both practical and standards-based; maybe even both practical and pure?

UTC +- offset is precise and clear and very commonly used

Correct, but that is local time or the common understanding of civil time as Jonathan pointed out. So rather than call it utc, I second Jonathan's suggestion to call it the civil calendar (I had been toying with the same idea when walking my dog a few days ago but came up with the local calendar name).

Rather than tossing out UTC as a calendar option, I would propose the following:

• utc calendar, starting 1972-01-01 on the Gregorian calendar, at that instant 10 seconds behind TAI. Time zone must be 00:00 if given. This satisfies the requirements of many data producers publishing global data sets or satellite imagery or basically anything that is not specifically tailored to a small geography.
• An additional civil (or local) calendar, based on UTC and the Gregorian or Julian calendar but using a time zone offset (and possibly supporting daylight savings time (on the data producer side!) through the correct use of tz offsets). Personally I prefer the local name as that conveys the fact that the data may be tied to a specific geography and requiring adjustment to align with other time series.

Would something like that work?

As a final note on standards versus practice: The CF conventions are far more influential than you make it sound. The decisions reached on issues such as calendars reverberate throughout the larger CF community and materialize in data sets that hang around for decades. The netcdf library itself was modified in response to CF conventions. I'm just putting in my 2 cents to propose something that may still be valid and logical in a few decades` time.

[davidhassell]

Hi,

On the 1958-1972 issue: I think we should keep the 1958 epoch. This change in approach before/after 1972 was indeed considered during the creation of the new text, and the wording we used was deliberately chosen to allow for any frequency of leap second application (e.g. daily), and an individual adjustment being any amount of seconds (e.g. negative amounts and non-integer amounts). I think that what we have now is entirely consistent with the messy situation that occurred in 1958-1972, so there is no need to change the minimum valid time. If you can deal with leap seconds applied after 1972, then dealing with the situation before 1972 (should have such data) is logically no different. (Whilst it currently seems unlikely, the leap second situation could get more messy again in the future, in which case the conventions will still hold, and we'dbe glad we didn't arbitrarily remove part of the valid time period.)

On the timezone/calendar-name issue: To me, the use of a timezone offset is merely an encoding choice in the units attribute string value in the dataset, just as choosing "seconds since" or "days since" is your units string is. Therefore I see no reason disallow their use, and no reason to change the CF calendar name. Also, if we changed the CF utc calendar name, surely we'd have to change the tai calendar name, too? (which I don't think we want!).

Cheers,
David

[larsbarring]

This is more of a general comment than a response to any of the previous comments in this discussion.

I think that we have to careful about what words we use in relation to all this. For example what do we mean by the word "calendar" (as a general concept) and when do we mean calendar as a CF construct. Is "time"/"timestamp" different from "date"/"datestamp" and from "datetime"/"datetimestamp"? Are they just the same, even across diffent calendars? What is a "clock", is it just a physical mechanism providing ticks and tocks at regular intervals,? And then what is a "regular interval" if you have two clocks?

These are maybe silly questions. But at the same time, CF is trying to make several different "clocks" and "calendars" come together into one concept; the CF calendar construct as a representation of time coordinates. From my own perspective, I think that Chris Little's presentation⁽¹⁾ at the 2024 CF Conventions Workshop was very useful, and in this context particularly viewgraphs 17-19.

I think that CF would be well served by general discussion about all this. For example, there seems to be a strong push towards having the "datetimestamp" labelling (strings such as "2025-01-07 12:00:00Z") as a vehicle for unifying points in time across different calendars. This can be a very useful and pragmatic solution that is legitimate and accepted in many situations. But in other situations such a solution cannot be justified or even make it invalid, whereas other pragmatic solutions exist. Trying to fold any such pragmatic solution into the CF Conventions is doomed to cause trouble and limit its usefulness.

(1) Little, C. (2024, September 17). OGC Temporal Domain Working Group Activities. 2024 CF Workshop, Swedish Meteorological and Hydrological Institute (SMHI), Norrköping, Sweden. https://doi.org/10.5281/zenodo.14194152.

[JonathanGregory]

Dear Lars

You're right that we have to be careful about words. In the world as a whole, the words you mention may be used in various ways. If there is inconsistency, we can't help it. Within the CF convention, however, we can be careful to use our words consistently. I hope that within the convention document we always mean "calendar" and "datetime" in the senses with which they're defined in Sect 1.3. On briefly looking, I believe that "date" separately always refers to just the year-month-day part of the datetime. A "time" coordinate represents a datetime, of course, not just a time of day; there's a lot in Sect 4.4 about it! In the standards document, "stamp" occurs only once, in Appendix B; we could delete it. The word "clock" occurs only once, in Sect 4.4.1; we could replace it. The phrase "regular interval" occurs only once, in App J, where it refers to distance rather than time.

Best wishes

Jonathan

[larsbarring]

Dear Jonathan,

I was more thinking of the different discussions, both here and in all the preceding discussions and issues on time/calendars/etc over the years. It is very easy to misunderstand or simply miss a point just because different understanding of concepts behind words. I am sure that we all have done it. Hence my general view is that CF has not done a very good job in what I mentioned as: "CF is trying to make several different "clocks" and "calendars" come together into one concept; the CF calendar construct as a representation of time coordinates." But this is a huge and difficult topic as we all are aware of. My point is that in order to make some progress in disentangling all this, it is useful to look at what others are doing. Hence my reference to Chris's presentation as food for thought.

[ChrisBarker-NOAA]

From my own perspective, I think that Chris Little's presentation⁽¹⁾ at

the 2024 CF Conventions Workshop was very useful, and in this context particularly viewgraphs 17-19.

I think that CF would be well served by general discussion about all this.

Is there a freely available summary of the new ISO 34000 ?

https://www.iso.org/standard/77019.html

Or do any of you have access to a copy? It seems that could be helpful to reference.

…

-CHB

[ChrisBarker-NOAA]

If you can deal with leap seconds applied after 1972, then dealing with the situation before 1972 (should have such data) is logically no

different. IIUC ( and I may not) it IS logically different — that is, rather than applying a “leap second” integer or not, at particular times, there was a slight adjustment to the size of every second in a range (did someone call them rubber seconds?). Which is a different approach, and would require different code to handle. @pvanlaake: could you please confirm? My concern about the addition of a UTC Calendar is that it is an attractive nuisance -- a lot of people *think* they are using UTC, when they are not (quite) -- and the software they are using may not support it properly. Allowing pre-1972 may make it even more likely for not-really-UTC data to be presented as UTC.

[davidhassell]

Ah, interesting. I though the changes were applied daily, rather than every second. Clarification would be very useful!

However, applying an increment annually, daily, hourly, even every second, is logically the same, surely?

[pvanlaake]

As I mentioned in my initial post in this thread, the second used by UTC in the period 1961 - 1972 is not the SI second. The table in my reply to Chris shows the rate at which the UTC second differs from the SI second used by TAI, expressed as a total amount per day but applied to the length of the second. The whole point of introducing leap seconds as of 1972 was to make UTC run on the SI second, which it previously didn't.

However, applying an increment annually, daily, hourly, even every second, is logically the same, surely?

Surely. As surely as applying an increment every second would make a mockery of claiming to use UDUNITS. And I do recall seeing some discussion about sub-second precision...

[davidhassell]

Hi Patrick,

I'm sorry, but I don't understand the UDUNITS analogy - could you expand (are you agreeing or disagreeing :))?

One of the reasons that the original discussions took so long was trying to understand the relationship between UTC time as stored in a CF dataset and elapsed time in the world (real or idealised) being measured.

Consider the difference between the UTC datestamps 1961-03-01 0:0:0 and 1961-03-01 0:0:1 (a rubber second), and the difference between the UTC datestamps 2016-12-31 0:0:0 and 2017-01-01 0:0:0 (includes the most recently applied leap second). Non-leap-second-aware software would calculate these differences as 1 SI second and 1 SI day respectively. Leap-second-aware software would give the differences as 0.999999985 SI seconds (a reduction of 15 ppb) and 1.000011574074074 SI days (86401 SI seconds), respectively.

I hope this illustrates why I think that the pre-1972 era is logically no different to the post-1972 era. Messier, for sure, but not really different!

Cheers,
David

[pvanlaake]

Hi David,

The "rubber second" is compressed (of shorter duration) relative to the SI second. So it is not a question of "applying an increment every second" - there is no such increment, it is a rate of divergence (my terminology) baked into the second that the UTC clock pre-1972 used. There is a computational solution, for sure, but at a significant cost of complicating code. And again, the leap second was introduced precisely because the pre-1972 system was considered too cumbersome.

Your suggestion has a very awkward fit with the concept of "second" as used otherwise in the conventions. Throughout the #304 thread, it is brought up on a few occasions that the "second" in CF is the SI second.

Is there a compelling need for CF support of the period 1961-1972? (1958-1960 is a real no-no because there was no UTC during that period and there are no published clock rates or step increments.) Can you identify a single data set that has true UTC datetimes that cover that period? Would it not be better to exclude this period until someone expresses that compelling need?

[davidhassell]

Hi Patrick,

Firstly, I'm more than happy to restrict ourselves to after 1972. It will also prevent anyone from the unlikely event of re-writing existing data from before 1972 using the utc calendar (which is almost certainly not what they want to do!).

This doesn't change my assertion that the text is "not wrong" for the pre-1972 period, but I think we're splitting hairs on different conceptual views of the same thing, so I'll stop going on about it, now :).

With regards use cases, as I recall the original use case was for a tai calendar in the remote sensing community, but we felt you can't introduce TAI with dealing with the concept of leap seconds, which we felt was best done by introducing a corresponding utc calendar to properly define it.

Cheers,
David

[ChrisBarker-NOAA]

“”” Non-leap-second-aware software would calculate these differences as 1 SI second and 1 SI day respectively. Leap-second-aware software would give the differences as 0.999999985 SI seconds “”” Well, “rubber seconds” aren’t the same as leap seconds, though they serve a similar purpose. With Leap seconds, seconds are always an integer. So we *could* say the “fully UTC compliant software could manage that” it can’t be handled with integer seconds. In fact, it looks like you’d need finer than microseconds, so pretty problematic. (And single precision float wouldn’t work either). And in any case when mapping between calendars, you couldn’t do it at all if the otgg bf er calendar only gandkk k ed integer seconds … So yes, it’s still a well defined transform, but from a practical perspective it’s a pretty different beast. Again, I’m concerned about the attractive nuisance — is there ANY software that can handle this correctly? So wouldn’t it be better to disallow it? Hmm — thinking now, it might actually work better to use the standard calendar in that case. You could use unaware software to transform back and forth, and it would be lossless, unlike leap seconds, where that almost works, except when there is an actual leap second, as most (all) non-leap second aware software doesn’t allow a 60th second.

[pvanlaake]

it might actually work better to use the standard calendar

This is also my thought. Allowing the inclusion of 23:59:60 into a datetime of a standard, proleptic_gregorian or julian calendar with units_metadata = "leap_seconds:utc" would solve many of the issues we have been discussing.

That opens up the issue of older software not being able to properly read newer data sets but that is a different discussion. (Software tends to follow new conventions. Should older software, i.e. of the kind that is no longer kept up-to-date, be actively "safe-guarded"? That is a rather unusual arrangement in the computing world and really holding back these conventions.)

[larsbarring]

CF is usually enhanced, expanded or changed based on concrete use cases. Over the years it has been argued, and accepted, that there are enough use cases to include the UTC and TAI calendar, where the latter was more of an complement to the UTC calendar.

However, concrete use cases are missing, even though there have been many references to that the satellite folks need it. And to that there are high-resolution measurement that may need this kind of precision. When I asked a well-placed EUMETSAT person I got the response that they reset their clocks at midnight (we did not go into any further details, so this is a bit of hearsay). And the only recent concrete high-frequency data collection example was from David @davidhassell, where they used UTC but was not even aware of the concept of leap seconds (David please correct me if I have mis-interpreted something). Way back I was involved in field measurements using sonic anemometers where the raw 20 Hz output was stored and the system clock was in the measurement hardware. But the final data was then reduced to 5- or 10-min covariances (etc), meaning that the second precision was not needed anymore. Similar systems today I imagine would either use NTP or GPS as clock.

All in all, without concrete uses cases we are not on firm ground, and I would really like to have at least one concrete use case to help us here.

[JonathanGregory]

Dear all

I think David is correct in asserting that the current text is not wrong, because in Sect 4.4.3 we say, for integer-second datetimes, "a time coordinate value expressed in seconds equals the number of valid (integer-second) datetimes after (not including) the reference datetime in the units up to (and including) the datetime that the time coordinate represents." That is, the time coordinate value is not the length of the time interval in SI seconds between the two instants. It is actually the count of the number of valid integer-second datetimes between them. These two quantities are equal except when a leap second intervenes in calendars where leap seconds are not valid datetimes, most importantly the standard calendar.

In that sense, rubber seconds are less problematic than leap seconds because, as Chris remarked, the standard calendar has the correct set of datetimes for that situation. There are always 86400 integer-second datetimes in a day in UTC between 1960 and 1972, so the encoding and decoding of time-coordinates by counting integer datetimes is easy!

However, the current text isn't adequate in Section 4.4.2, because we don't mention that seconds aren't SI seconds in UTC before 1972, or about the 0.1 s and 0.2 s jumps. As a curiosity, how did they deal with these in datetimes? If 0.1 s was inserted at the end of a day, for instance, did the clock jump from 23:59:60.1 to 0:0:0?

The simplest solution is to disallow the utc calendar before 1972, as Patrick suggested. I think we must also insert a further comment, however, about time-coordinates before 1972-1-1 0:0:0 UTC in all real-world calendars except tai, which is precise, and allowed back to 1958.

In other calendars, before 1972, when real-world datetimes are encoded as time-coordinates, the difference between two time-coordinates is in general not exactly equal to the length of the time-interval between them in SI seconds. Obviously it can't be, before the atomic clock was invented. The further back you go, the less accurate and universal the second was. For me, this underlines that the correct general intepretation of a real-world time-coordinate is as an encoded datetime. This encoding is additionally useful because, since 1972, and except for leap seconds, the difference between two time-coordinates equals the length of the intervening interval, but that's not the fundamental definition. Of course, in idealised model calendars, it's always exactly true.

Best wishes

Jonathan