API: how to handle time zone (DST) transitions in the distant future?

[jorisvandenbossche]

Historically, by using pytz by default for time zones, tz-aware data had wrong offsets for DST periods after 2038 (e.g. see #33061 and #28207, and documented at https://pandas.pydata.org/pandas-docs/version/3.0/user_guide/timeseries.html#working-with-time-zones (scroll down to the 5th(!) warning box), sidenote: we should update this).

The same happens if you were using dateutil.tz time zones, and the short summary is that pytz and dateutil are only parsing the Version 1 TZif files, those don't include a "rule" for future times, and therefore typically include transitions times extrapolated to the future, but because of 32bit are limited to dates up to 2038.

This issue was fixed by switching to zoneinfo by default (#34916) in pandas 3.0, since zoneinfo supports the newer versions of TZif files, and can use a rule-based logic for determining the offset for distant dates.

However, then after improving the performance for zoneinfo timezones to also use our internal fast binary search of the transition points (#64379) in pandas 3.0.3, this changed this behaviour again, because now we have to pre-populate our array of transition dates (to use for the binary search), and the linked PR decided to do that up to 2100 (meaning we only get correct offsets up to 2100, and not for later dates, see the example below).
In #65705, I am extending that cut-off from 2100 to 2262.

While having tz-aware data so far in the future is a bit a debatable use case, having an arbitrary cut off after which it behaves differently is not ideal.

Options:

• Decide we are OK with an arbitrary cut-off if it is far enough in the future (e.g. 2262) and document this
• Populate the transition data dynamically based on the range of the input data (eg by default fill up to year 2100, but if the input data max year is 2150, we recalculate it up to that date and update the cached data)
• Use our fast-path for a "normal" range (eg up to 2100) to cover most use cases, and then for dates after that, use the zoneinfo python API directly as a slower fallback

--

Code illustration:

import pandas as pd

import dateutil.tz
import zoneinfo
import pytz

arr = pd.to_datetime([
    "2000-01-01", "2000-07-01",
    "2036-01-01", "2036-07-01",
    "2038-01-01", "2038-07-01",
    "2100-01-01", "2100-07-01",
    "2200-01-01", "2200-07-01"
], utc=True)

With pytz / default in pandas < 3:

>>> print(arr.tz_convert("Europe/Brussels"))   # pandas < 3
>>> print(arr.tz_convert(pytz.timezone("Europe/Brussels")))
DatetimeIndex(['2000-01-01 01:00:00+01:00', '2000-07-01 02:00:00+02:00',
               '2036-01-01 01:00:00+01:00', '2036-07-01 02:00:00+02:00',
               '2038-01-01 01:00:00+01:00', '2038-07-01 01:00:00+01:00',
               '2100-01-01 01:00:00+01:00', '2100-07-01 01:00:00+01:00',
               '2200-01-01 01:00:00+01:00', '2200-07-01 01:00:00+01:00'],
              dtype='datetime64[ns, Europe/Brussels]', freq=None)

With dateutil:

>>> print(arr.tz_convert(dateutil.tz.gettz("Europe/Brussels")))
DatetimeIndex(['2000-01-01 01:00:00+01:00', '2000-07-01 02:00:00+02:00',
               '2036-01-01 01:00:00+01:00', '2036-07-01 02:00:00+02:00',
               '2038-01-01 01:00:00+01:00', '2038-07-01 01:00:00+01:00',
               '2100-01-01 01:00:00+01:00', '2100-07-01 01:00:00+01:00',
               '2200-01-01 01:00:00+01:00', '2200-07-01 01:00:00+01:00'],
              dtype='datetime64[us, tzfile('/usr/share/zoneinfo/Europe/Brussels')]', freq=None)

With zoneinfo in pandas 2.3 or 3.0.0 - 3.0.2:

>>> print(arr.tz_convert("Europe/Brussels"))  # pandas >= 3.0
>>> print(arr.tz_convert(zoneinfo.ZoneInfo("Europe/Brussels")))
DatetimeIndex(['2000-01-01 01:00:00+01:00', '2000-07-01 02:00:00+02:00',
               '2036-01-01 01:00:00+01:00', '2036-07-01 02:00:00+02:00',
               '2038-01-01 01:00:00+01:00', '2038-07-01 02:00:00+02:00',
               '2100-01-01 01:00:00+01:00', '2100-07-01 02:00:00+02:00',
               '2200-01-01 01:00:00+01:00', '2200-07-01 02:00:00+02:00'],
              dtype='datetime64[us, Europe/Brussels]', freq=None)

With default zoneinfo in pandas 3.0.3:

>>> print(arr.tz_convert("Europe/Brussels"))
DatetimeIndex(['2000-01-01 01:00:00+01:00', '2000-07-01 02:00:00+02:00',
               '2036-01-01 01:00:00+01:00', '2036-07-01 02:00:00+02:00',
               '2038-01-01 01:00:00+01:00', '2038-07-01 02:00:00+02:00',  # 2038 is now correct, but 2100 and above not
               '2100-01-01 01:00:00+01:00', '2100-07-01 01:00:00+02:00',
               '2200-01-01 01:00:00+01:00', '2200-07-01 01:00:00+02:00'],
              dtype='datetime64[us, Europe/Brussels]', freq=None)