porting-py2-to-py3

Porting Python 2 to Python 3

Core principle: the port lands on the main branch as Python-2-compatible commits, behind CI gates that only ratchet forward; the interpreter switch is the last, smallest step. Code changes are cheap. What is expensive is production risk, calendar (soak windows, other teams' reviews) and the attention of the team that keeps shipping on the same tree.

Before you start

• Read references/target-version.md to choose the target. Default in late 2026: 3.13, with 3.14 as a go/no-go once the suite is green. Never pick an EOL version, and never assume a framework version "won't run" on a newer interpreter: measure it (Phase 0, step 5).
• Read references/tooling.md for the converter, the interpreter flags, the pins of the last py2-compatible versions, and which tools are dead (do not recommend caniusepython3, pylint --py3k on pylint 2.x, futurize --stage2, or the future runtime).
• Read references/catalog.md when fixing a specific idiom. It classifies every change as M (a tool rewrites it) or J (decide per site with the stated procedure).
• references/case-studies.md has the tactics behind this workflow (Instagram, Dropbox, Yelp, Mercurial).
• OPTIONAL PRE-PHASE: removing-dead-code (sibling skill). Every function, route, job and dependency removed is one fewer to port and verify.

Phase 0: safety net and measurements (on main)

Done when: the environment is reproducible, CI publishes coverage and fails on regression, golden tests exist for every persisted invariant, and the spike has answered which interpreter the current framework chain runs on.

1. Freeze what production really runs. pip freeze inside the production image becomes a versioned lockfile. Base images maintained elsewhere often install unpinned packages that the repo never names (SQLAlchemy, DB drivers, memcache clients). Those are dependencies too. If the base image belongs to another team, build your own from python:2.7.18-slim plus the lockfile now: the py3 image needs the same layout, and the switch stops depending on their calendar.
2. Pick the target version with the decision rule in references/target-version.md. Write it into the project's README and agent instructions file in the same PR.
3. Coverage as a gate. Run the suite with coverage (5.5 on py2, 7.x on py3) in CI, publish the number, set fail_under to the current value. Single-source code has lines only one interpreter can execute (if six.PY2:, an except ImportError fallback for a backport), so exclude them per leg with # pragma: py2 only / # pragma: py3 only and the .coveragerc in references/tooling.md, section 5; otherwise the number is capped below 100 and the gate cannot ratchet to the end. Start collecting coverage from production or staging now (see removing-dead-code for the multi-service design); the window must be long before Phase 2 needs it.
4. Golden tests for cross-version invariants. Record outputs under py2 as fixtures and assert byte-equality under both interpreters: IDs derived from hashes or sys.maxint, signed tokens, encrypted payloads, cache keys, serialized exports (JSON, CSV, XLSX, PDF), URLs built from user data, anything parsed by another system. These are the tests that catch b'...' leaks and rounding drift before a deploy does.
5. One-day spike. Build a py3 image on the target with the py2-compatible framework chain pinned (references/tooling.md, section 6) and import the chain in isolation, since the application still has py2 syntax at this point: python3 -c "import flask, flask_sqlalchemy, sqlalchemy, <driver>, <rest layer>, <event loop lib>" (Django: django.setup() plus migrate against sqlite) plus a minimal app factory smoke test. Expect the ceiling to differ by stack: the Flask 1.1 chain runs on 3.13, Django 1.11 stops at 3.9 (references/tooling.md, section 6). Check wheels for the target with pip download --only-binary=:all: --python-version 3XY --platform manylinux2014_x86_64 -r requirements.txt before building; a C extension without a wheel for the target either builds from source in the image or forces a split pin (step 6). If the chain imports on the target, the cutover changes only the interpreter. If it does not, cut over on the newest version it runs on and make the framework upgrade the next project. One variable at a time. The -t tests stage of the compile gate only becomes meaningful after Phase 1.
6. Dependency inventory. For every package: last version with 2.7, first without, whether the pinned version imports on the target, and whether a wheel exists for the target. Internal packages without py3 builds go to their owners now; they are the calendar risk. Archived upstreams get a fork (references/tooling.md, section 7). A package that cannot straddle both interpreters (its last py2 release has no build for the target: pandas 0.24, numpy 1.16, often psycopg2 2.8 and mysqlclient 1.4 without source builds) gets a split pin by environment marker, pkg==old; python_version<"3" and pkg==new; python_version>="3", code restricted to the API subset both versions share, and golden tests for its outputs. From this point: no new package without py3 support.
7. Baseline the census. python3 scripts/py2ism_census.py SRC --save py2ism-baseline.json and add --check to CI. Teammates cannot add Python 2 idioms from now on.