timezone info is dropped if a list of dict is passed

[lucidfrontier45]

Polars version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of Polars.

Issue description

timezone info of datetime object is dropped if data is passed to the DataFrame constructor in a form of list of dict .

Reproducible example

import datetime
import zoneinfo

import polars as pl

dt1 = datetime.datetime(2023, 1, 1, 0, 0, 0, 0, zoneinfo.ZoneInfo("Asia/Tokyo"))
dt2 = datetime.datetime(2023, 1, 2, 0, 0, 0, 0, zoneinfo.ZoneInfo("Asia/Tokyo"))

# dict of list, this works fine, timezone info is kept
pl.DataFrame({"dt": [dt1, dt2]})

# list of dict, this doesn't work, timezone info is dropped
pl.DataFrame([{"dt": dt1}, {"dt": dt2}])

Expected behavior

The case of list of dict also keeps the timezone info as dict of list case.

Installed versions

---Version info---
Polars: 0.16.14
Index type: UInt32
Platform: Linux-5.15.90.1-microsoft-standard-WSL2-x86_64-with-glibc2.35
Python: 3.11.2 (main, Feb  9 2023, 09:23:29) [GCC 11.3.0]
---Optional dependencies---
numpy: 1.24.2
pandas: 1.5.3
pyarrow: <not installed>
connectorx: <not installed>
deltalake: <not installed>
fsspec: <not installed>
matplotlib: 3.7.1
xlsx2csv: <not installed>
xlsxwriter: <not installed>```

</details>

[rMazeiks]

I also just ran into this. As a workaround, it seems you can specify the schema manually:

pl.DataFrame([{"dt": dt1}, {"dt": dt2}], schema={"dt": pl.Datetime("us", "UTC")})

[lucidfrontier45]

@rMazeiks

Thank you so much. But the negative side of that workaround is that I have to specify schema for all columns otherwise unspecified columns will be ignored.

---

[edit]

use schema_overrides is a better option.

[cmdlineluser]

This also came up recently in another form via pl.from_dicts #12638 #12151

I think they are all the same underlying issue.

[tim-x-y-z]

+1

This threw me off big time

[scur-iolus]

It would be great if this could be fixed in v1. Related issues: 6130, 6619, 12151, 12638, 13033, 14211.

Here's a test which still fails with 1.0.0-beta.1:

import datetime
from zoneinfo import ZoneInfo

import polars as pl


def test_constructor_considers_timezone() -> None:
    """A timezone aware datetime should not become naive when parsed by Polars.

    There should be no need to add the timezone to pl.Datetime in the provided schema.
    """
    tz = "Europe/Paris"
    dt = datetime.datetime(2024, 5, 2, 0, 0, tzinfo=ZoneInfo(tz))
    df = pl.DataFrame([{"dt": dt}], schema={"dt": pl.Datetime(time_zone=tz)})
    assert df[0, "dt"].tzinfo == dt.tzinfo
    assert df[0, "dt"] == dt
    df = pl.DataFrame([{"dt": dt}], schema={"dt": pl.Datetime})
    assert df[0, "dt"].tzinfo == dt.tzinfo  # fails with v1.0.0b1
    assert df[0, "dt"] == dt
    df = pl.DataFrame([{"dt": dt}], schema={"dt": pl.Datetime(time_zone="*")})
    assert df[0, "dt"].tzinfo == dt.tzinfo  # raises a PanicException with v1.0.0b1
    assert df[0, "dt"] == dt