Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Time mapper doesn't handle time zones from dsgrid #31

Open
daniel-thom opened this issue Dec 24, 2024 · 1 comment
Open

Time mapper doesn't handle time zones from dsgrid #31

daniel-thom opened this issue Dec 24, 2024 · 1 comment
Assignees
Labels
bug Something isn't working

Comments

@daniel-thom
Copy link
Collaborator

dsgrid creates geography records with time zones like EasternPrevailing, CentralPrevailing, etc: https://github.com/dsgrid/dsgrid-project-StandardScenarios/blob/main/dsgrid_project/dimensions/counties.csv

chronify uses pandas for time zone conversions and doesn't know about these. What should we do?

E               pytz.exceptions.UnknownTimeZoneError: 'PacificPrevailing'

env/lib/python3.11/site-packages/pytz/__init__.py:188: UnknownTimeZoneError
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> entering PDB >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PDB post_mortem (IO-capturing turned off) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> /Users/dthom/repos/secondary-dsgrid/env/lib/python3.11/site-packages/pytz/__init__.py(188)timezone()
-> raise UnknownTimeZoneError(zone)
(Pdb) up
> /Users/dthom/repos/secondary-dsgrid/timezones.pyx(170)pandas._libs.tslibs.timezones.maybe_get_tz()
(Pdb) 
> /Users/dthom/repos/secondary-dsgrid/timezones.pyx(145)pandas._libs.tslibs.timezones.maybe_get_tz()
(Pdb) 
> /Users/dthom/repos/secondary-dsgrid/env/lib/python3.11/site-packages/pandas/core/arrays/datetimes.py(911)tz_convert()
-> tz = timezones.maybe_get_tz(tz)
(Pdb) 
> /Users/dthom/repos/secondary-dsgrid/env/lib/python3.11/site-packages/pandas/core/indexes/datetimes.py(283)tz_convert()
-> arr = self._data.tz_convert(tz)
(Pdb) 
> /Users/dthom/repos/secondary-dsgrid/env/lib/python3.11/site-packages/pandas/core/indexes/accessors.py(132)_delegate_method()
-> result = method(*args, **kwargs)
(Pdb) 
> /Users/dthom/repos/secondary-dsgrid/env/lib/python3.11/site-packages/pandas/core/accessor.py(112)f()
-> return self._delegate_method(name, *args, **kwargs)
(Pdb) 
> /Users/dthom/repos/chronify/src/chronify/representative_time_range_generator.py(64)create_tz_aware_mapping_dataframe()
-> dft["timestamp_tmp"] = dft[timestamp_col].dt.tz_convert(tz)
(Pdb) ll
 54  	   def create_tz_aware_mapping_dataframe(
 55  	       self,
 56  	       df: pd.DataFrame,
 57  	       timestamp_col: str,
 58  	       time_zones: list[str],
 59  	   ) -> pd.DataFrame:
 60  	       """Create time zone-aware time mapping dataframe."""
 61  	       dfm = []
 62  	       for tz in time_zones:
 63  	           dft = df.copy()
 64  ->	           dft["timestamp_tmp"] = dft[timestamp_col].dt.tz_convert(tz)
 65  	           dft = self._handler.add_time_attribute_columns(dft, "timestamp_tmp")
@daniel-thom daniel-thom added the bug Something isn't working label Dec 24, 2024
@lixiliu
Copy link
Collaborator

lixiliu commented Jan 17, 2025

Either:
[1] Remove the use of these enums from dsgrid and rely instead on a tz library that pandas uses, which is dateutil: https://dateutil.readthedocs.io/en/stable/tz.html
[2] Convert the enums to pandas-friendly field values
[3] Build in Chronify capability to accept these enums.

These enums contain values for the pyspark API and can be used to go between pairs of standard time and prevailing time. They were useful for mapping industrial time. Let's see what @JensZack ends up implementing. I am inclined to go with the cleanest solution in that if these enums are no longer needed, we go with [1].

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants