windows-encoding-fixes

Windows: encodings, console, cross-platform

Experience from two coding-kit installation bug reports on Windows 10 (research.db id=141, 146). Each pitfall — with symptom, cause, fix. Apply to ANY script/file that must also work on Windows.

1. stdout encoding: cp1251 kills Russian output

Symptom: UnicodeEncodeError: '\u2713' ... codec can't encode — fails when printing ✓/✗/Cyrillic. Shows up when redirecting output (script > log 2>&1) and in a cp1251 console.

Cause: the Windows console defaults to cp1251; Python 3.12 takes the console encoding when redirecting, and Unicode does not fit into it.

Fix (required in every CLI script):

try:
    sys.stdout.reconfigure(encoding="utf-8", errors="replace")
    sys.stderr.reconfigure(encoding="utf-8", errors="replace")
except Exception:  # noqa: S110,BLE001 — reconfigure is optional
    pass

In coding-kit — the single memory/scripts/_compat.py: fix_encoding() instead of copying.

Fix for bash wrappers:

export PYTHONIOENCODING="${PYTHONIOENCODING:-utf-8}"

System-level fix (recommended for every Windows box):

[Environment]::SetEnvironmentVariable("PYTHONUTF8", "1", "User")

2. Writing files: CRLF breaks md5 checks

Symptom: Path.write_text() on Windows writes \r\n; the canonical form is \n. Any md5 check of mirrors/copies falsely fails with «files diverged».

Fix when writing: newline="\n" + byte comparison:

dst.write_text(content, encoding="utf-8", newline="\n")
# freshness check:
if dst.read_bytes() == content.encode("utf-8"): ...

Fix when checking (bash, md5 with normalization):

first=$(tr -d '\r' < "${MIRRORS[0]}" | md5sum | cut -d' ' -f1)

3. PowerShell 5.1: UTF-8 without BOM is read as cp1251

Symptom: a bootstrap .ps1 ignores --check, outputs mojibake, fails on the dash «—» in comments.

Cause: Windows PowerShell 5.1 decodes a .ps1 without BOM as cp1251; Cyrillic and dashes break parsing (down to the argument logic).