Standard Library and Idioms

Python’s “batteries included” library is genuinely large. This page covers the modules that come up in almost every project, then the idioms that separate code that works from code that reads as Python.

pathlib — filesystem paths

pathlib.Path replaced string path manipulation and most of os.path. Paths are objects, / joins them, and the same code works on Windows.

from pathlib import Path

p = Path("/home/user/docs/report.txt")

p.name         # => 'report.txt'
p.stem         # => 'report'
p.suffix       # => '.txt'
p.parent       # => Path('/home/user/docs')
p.parts        # => ('/', 'home', 'user', 'docs', 'report.txt')

Path("data") / "raw" / "file.csv"          # => Path('data/raw/file.csv')
p.with_suffix(".md")                        # => Path('/home/user/docs/report.md')
p.with_name("other.txt")

Queries and operations:

p.exists(), p.is_file(), p.is_dir()
p.stat().st_size                 # bytes
p.resolve()                      # absolute, symlinks resolved
Path.cwd(), Path.home()

Path("out").mkdir(parents=True, exist_ok=True)
p.rename("new.txt")
p.unlink(missing_ok=True)        # delete a file
Path("dir").rmdir()              # only if empty; use shutil.rmtree otherwise

Reading and writing without opening a handle:

text = p.read_text(encoding="utf-8")
p.write_text("hello", encoding="utf-8")
data = p.read_bytes()

Globbing:

list(Path("src").glob("*.py"))         # this directory
list(Path("src").rglob("*.py"))        # recursive
[f for f in Path(".").iterdir() if f.is_file()]

Tip

Convert to str only when calling an API that demands it — most standard-library functions accept a Path directly. Keep paths as Path objects everywhere else.

os and sys

os is the operating-system interface; sys is the interpreter itself.

import os

os.environ["HOME"]                 # KeyError if unset
os.environ.get("API_KEY", "")      # safe
os.getenv("PORT", "8000")

os.cpu_count()
os.getpid()
os.makedirs("a/b", exist_ok=True)
os.listdir(".")
os.walk("src")                     # (dirpath, dirnames, filenames) recursively

import sys

sys.argv          # ['script.py', 'arg1'] — argv[0] is the script
sys.exit(1)       # raises SystemExit; 0 = success
sys.stdin, sys.stdout, sys.stderr
sys.version_info  # (3, 12, 7, 'final', 0)
sys.platform      # 'linux' | 'darwin' | 'win32'
sys.path          # the import search path

Writing errors and diagnostics to stderr keeps stdout clean for piped data:

print("warning: retrying", file=sys.stderr)

Caution

Reading secrets from os.environ is standard practice, but never log the whole environment and never commit a .env file. Prefer os.environ["NAME"] for required settings so a missing one fails loudly at startup rather than silently becoming None.

json

import json

# Python -> JSON
json.dumps({"a": 1})                                 # => '{"a": 1}'
json.dumps(data, indent=2, sort_keys=True)
json.dumps(data, ensure_ascii=False)                 # keep non-ASCII readable
json.dumps(data, default=str)                        # fallback for unknown types

# JSON -> Python
json.loads('{"a": 1}')                               # => {'a': 1}

# Files: dump/load take a file object
with open("config.json", encoding="utf-8") as f:
    config = json.load(f)

with open("out.json", "w", encoding="utf-8") as f:
    json.dump(data, f, indent=2)

The type mapping is exact and lossy in one direction: JSON objects become dict, arrays become list, null becomes None, true/false become bools. Tuples serialise as arrays and come back as lists.

from datetime import datetime, date

class Encoder(json.JSONEncoder):
    def default(self, obj):
        if isinstance(obj, (datetime, date)):
            return obj.isoformat()
        return super().default(obj)

json.dumps({"when": datetime.now()}, cls=Encoder)

Caution

json.dumps raises TypeError on datetime, Decimal, set, and custom objects. Dict keys are coerced to strings, so {1: "a"} round-trips as {"1": "a"}. Catch json.JSONDecodeError (a ValueError subclass) when parsing anything you did not produce.

re — regular expressions

import re

re.search(r"\d+", "abc 123")        # first match anywhere -> Match or None
re.match(r"\d+", "abc 123")         # anchored at the START -> None here
re.fullmatch(r"\d+", "123")         # the whole string must match
re.findall(r"\d+", "1 and 22")      # => ['1', '22']  (list of strings)
re.finditer(r"\d+", text)           # iterator of Match objects
re.sub(r"\s+", " ", messy)          # replace all
re.split(r"[,;]", "a,b;c")          # => ['a', 'b', 'c']

Always use raw strings (r"...") so backslashes reach the regex engine unchanged.

Working with a match:

m = re.search(r"(?P<year>\d{4})-(?P<month>\d{2})", "on 2024-05-01")
if m:
    m.group(0)          # => '2024-05'   the whole match
    m.group(1)          # => '2024'
    m.group("year")     # => '2024'      named group
    m.groupdict()       # => {'year': '2024', 'month': '05'}
    m.span()            # => (3, 10)

Compile a pattern you use repeatedly, and pass flags:

pattern = re.compile(r"^error:\s*(.+)$", re.MULTILINE | re.IGNORECASE)
for m in pattern.finditer(log):
    print(m.group(1))

Useful flags: re.IGNORECASE, re.MULTILINE (^/$ match line boundaries), re.DOTALL (. matches newlines), re.VERBOSE (whitespace and # comments allowed in the pattern).

Caution

findall returns strings when there are no groups, a list of strings with one group, and a list of tuples with several. That inconsistency causes real bugs — use finditer when you need reliability.

Danger

Never parse HTML, XML, or CSV with regex. Use html.parser/lxml, xml.etree.ElementTree (with defusedxml for untrusted input), and the csv module. Also beware catastrophic backtracking: nested quantifiers such as (a+)+b against attacker-controlled input are a denial-of-service vector.

See the regex section for the pattern language itself.

datetime

from datetime import datetime, date, time, timedelta, timezone, UTC

date.today()                       # => date(2024, 5, 1)
datetime.now()                     # local time, NAIVE (no timezone)
datetime.now(UTC)                  # aware UTC  (UTC alias added in 3.11)
datetime.now(timezone.utc)         # same thing, all versions

Aware vs naive is the thing to get right: a naive datetime has no tzinfo and does not identify a real instant. Store and compute in UTC; convert to local time only for display.

from zoneinfo import ZoneInfo          # stdlib since 3.9

now = datetime.now(UTC)
paris = now.astimezone(ZoneInfo("Europe/Paris"))

Danger

Comparing or subtracting a naive and an aware datetime raises TypeError. Mixing them is the root of most date bugs — pick “aware everywhere” and stick to it.

Parsing and formatting:

datetime.strptime("2024-05-01 14:30", "%Y-%m-%d %H:%M")     # parse
dt.strftime("%Y-%m-%d %H:%M:%S")                            # format
dt.isoformat()                      # => '2024-05-01T14:30:00+00:00'
datetime.fromisoformat("2024-05-01T14:30:00+00:00")

Common strftime codes: %Y 4-digit year, %m month, %d day, %H 24-hour, %M minute, %S second, %f microseconds, %z UTC offset, %A weekday name, %B month name.

Arithmetic uses timedelta:

tomorrow = date.today() + timedelta(days=1)
delta = end - start                  # a timedelta
delta.total_seconds()                # => 3600.0
timedelta(hours=1, minutes=30)

Caution

timedelta has no months or years — they are not fixed durations. Use dateutil.relativedelta (third-party) or calendar.monthrange for calendar arithmetic.

For elapsed-time measurement use time.perf_counter() (monotonic, high resolution), never datetime.now(), which can jump when the clock is adjusted.

collections

defaultdict — supplies a default for missing keys by calling a factory.

from collections import defaultdict

groups = defaultdict(list)
for word in words:
    groups[word[0]].append(word)     # no key check needed

Caution

Merely reading d[missing] on a defaultdict inserts the key. Use d.get(k) when you want a pure read.

Counter — tallying, with the ranking built in.

from collections import Counter

c = Counter("mississippi")
c                     # => Counter({'i': 4, 's': 4, 'p': 2, 'm': 1})
c.most_common(2)      # => [('i', 4), ('s', 4)]
c["z"]                # => 0  — missing keys return 0, and are NOT inserted
c.total()             # => 11  (3.10+)

Counter(words) + Counter(more_words)     # counters support arithmetic

deque — a double-ended queue with O(1) appends and pops at both ends.

from collections import deque

dq = deque([1, 2, 3])
dq.appendleft(0)      # O(1) — a list's insert(0, x) is O(n)
dq.popleft()          # O(1)
dq.rotate(1)

recent = deque(maxlen=100)    # a bounded ring buffer: oldest is dropped

Use a deque for queues, BFS frontiers, and sliding windows. Use a list for stacks (append/pop are already O(1) at the end).

Also here: namedtuple (see data structures), ChainMap for layered lookups, and collections.abc for the abstract base classes (Iterable, Mapping, Sequence) used in type hints and isinstance checks.

subprocess

Run external programs. Pass arguments as a list, not a string.

import subprocess

result = subprocess.run(
    ["git", "status", "--porcelain"],
    capture_output=True,
    text=True,            # decode stdout/stderr as str instead of bytes
    check=True,           # raise CalledProcessError on a non-zero exit
    timeout=30,
    cwd="/path/to/repo",
)
result.returncode    # => 0
result.stdout        # => '...'
result.stderr

Without check=True a failing command is silently ignored — you must inspect returncode yourself.

try:
    subprocess.run(["false"], check=True)
except subprocess.CalledProcessError as exc:
    print(exc.returncode, exc.stderr)

Danger

Never use shell=True with any string you did not fully construct yourself. subprocess.run(f"ls {user_input}", shell=True) lets ; rm -rf / execute. The list form does not involve a shell at all and needs no quoting or escaping. If you genuinely need shell features such as pipes, prefer wiring two Popen objects together, or shlex.quote every interpolated value.

subprocess.Popen gives you a handle on a still-running process for streaming or concurrency; run is the right default for everything else.

argparse

Command-line parsing, with --help, type conversion, and validation generated for you.

tool.py

import argparse

def main():
    parser = argparse.ArgumentParser(
        prog="tool",
        description="Process some files.",
    )
    parser.add_argument("paths", nargs="+", help="files to process")
    parser.add_argument("-o", "--output", default="out.txt", help="output file")
    parser.add_argument("-n", "--count", type=int, default=1, help="repetitions")
    parser.add_argument("-v", "--verbose", action="store_true", help="chatty output")
    parser.add_argument("--mode", choices=["fast", "safe"], default="safe")

    args = parser.parse_args()

    if args.verbose:
        print(f"processing {len(args.paths)} files in {args.mode} mode")

if __name__ == "__main__":
    main()

python3 tool.py a.txt b.txt -o result.txt --mode fast -v
python3 tool.py --help

Key parameters: nargs ("?" optional, "*" zero or more, "+" one or more), action (store_true, append, count, version), type (any callable, including pathlib.Path), required, dest, metavar. Subcommands come from parser.add_subparsers().

For a small script, sys.argv is fine. Past two flags, use argparse — the generated --help alone pays for it. Third-party click and typer are popular for larger CLIs.

EAFP vs LBYL

Two styles for handling things that might not work.

LBYL — Look Before You Leap. Check first.

if os.path.exists(path):
    with open(path) as f:      # the file may vanish between the two lines
        ...

EAFP — Easier to Ask Forgiveness than Permission. Try it and handle failure. This is the Pythonic default.

try:
    with open(path) as f:
        ...
except FileNotFoundError:
    ...

EAFP wins because it is atomic — no race between the check and the action — and because it avoids duplicating the condition the operation already tests internally. LBYL is still right when the check is cheap and the failure is expected and frequent, since exceptions cost more than a comparison.

# EAFP with dicts
try:
    value = d[key]
except KeyError:
    value = default

# but the built-in is better still
value = d.get(key, default)

Duck typing is the same idea applied to types: care what an object does, not what it is.

def render(obj):
    try:
        return obj.render()          # anything with .render() works
    except AttributeError:
        return str(obj)

Pythonic idioms

Unpacking instead of indexing.

first, second = pair
head, *tail = items
a, b = b, a

enumerate and zip instead of index arithmetic.

for i, item in enumerate(items, start=1): ...
for name, score in zip(names, scores, strict=True): ...

Comprehensions instead of append loops.

names = [u.name for u in users if u.active]
by_id = {u.id: u for u in users}

in for membership instead of chained or.

if status in {"ok", "done", "complete"}:     # a set literal: O(1)
    ...

Truthiness for emptiness.

if not items: ...          # not `if len(items) == 0:`

The conditional expression for small either/or values.

label = "yes" if flag else "no"

The walrus operator to bind and test at once (3.8+).

if (n := len(data)) > 100:
    print(f"{n} is too many")

while (line := f.readline()):
    process(line)

with for anything that must be closed. join for building strings. sorted(key=...) rather than manual comparison. collections rather than reinventing a counter.

Iterate the object, not its indices.

for x in xs: ...                    # yes
for i in range(len(xs)): ...        # only if you truly need i

_ for values you are discarding.

for _ in range(3): ...
name, _, extension = filename.partition(".")

Named booleans over cryptic call sites.

resize(image, width=100, keep_aspect=True)

String formatting

Three generations exist; use f-strings unless you cannot.

name, value = "x", 3.14159

f"{name} = {value:.2f}"                     # f-string: preferred (3.6+)
"{} = {:.2f}".format(name, value)           # str.format: when the template is data
"%s = %.2f" % (name, value)                 # %-formatting: legacy

str.format still earns its place when the template comes from elsewhere — a config file or translation catalogue — since it separates template from data:

TEMPLATE = "Hello, {name}! You have {count} messages."
TEMPLATE.format(name="Ada", count=3)

string.Template is the safest choice for user-supplied templates, because it supports only simple $name substitution and cannot execute expressions or reach into attributes.

Logging is the one place to keep %-style arguments rather than f-strings:

logger.info("user %s logged in from %s", user_id, ip)     # formatted only if emitted
logger.info(f"user {user_id} logged in from {ip}")        # formats even when filtered out

The lazy form also lets log aggregators group messages by their template.

Key points

• pathlib.Path for every path; / joins and read_text/write_text cover most I/O.
• Always pass encoding="utf-8" when opening text files.
• subprocess.run([...], check=True) — argument lists, never shell=True with user input.
• Store datetimes as timezone-aware UTC; zoneinfo handles conversion.
• defaultdict, Counter, and deque remove a lot of hand-written bookkeeping.
• EAFP over LBYL: try the operation, catch the specific exception, avoid the race.
• f-strings everywhere except logging, where %s arguments stay lazy.