Python 3.11¶
Python 3.11 was released on October 24th, 2021
artilcle link: https://testdriven.io/blog/python311/
Aganda¶
- Faster CPython
- Improved type hints
- Better error messages
- Exception notes
- TOML library
Faster CPython¶
Python 3.11 is 10 - 60% faster than Python 3.10. On average, it's 25% faster.
Faster Startup¶
In terms of startup, rather than Reading pycache -> Unmarshall -> Heap allocated code object -> Evaluate, Python 3.11 uses a new process. Core modules essential for the startup of the Python interpreter are now frozen in the interpreter -- their code is statically allocated. The new flow is Statically allocated code object -> Evaluate. The latter is 10 - 15% faster.
Faster Runtime¶
Every time you call a function in Python, a frame gets created. It holds information about function execution. The core developers simplified their creation process, internal information, and memory allocation. Another improvement occurred with inline function calls. From now on, most of the Python function calls consume no C stack space.
Maybe the most important improvement in this regard is PEP 659: Specializing Adaptive Interpreter. This PEP is one of the key elements for fast Python. Its main idea is that Python code has regions where types rarely change. Within those regions, Python can optimize operations by using more specialized types when it sees that some operation is always done only for certain types of data. For example, instead of using general multiplication, it can use multiplication for integers if only integers are being used. Another example is the direct call of underlying C implementations for common built-in functions like len and str instead of going through the internal calling convention
You can observe this by checking generated bytecode:
import dis
from random import random
def dollars_to_pounds(dollars):
return 0.87 * dollars
dis.dis(dollars_to_pounds, adaptive=True)
# 5 0 RESUME 0
#
# 6 2 LOAD_CONST 1 (0.87)
# 4 LOAD_FAST 0 (dollars)
# 6 BINARY_OP 5 (*)
# 10 RETURN_VALUE
for _ in range(8):
dollars_to_pounds(random() * 100)
dis.dis(dollars_to_pounds, adaptive=True)
# 5 0 RESUME_QUICK 0
#
# 6 2 LOAD_CONST__LOAD_FAST 1 (0.87)
# 4 LOAD_FAST 0 (dollars)
# 6 BINARY_OP_MULTIPLY_FLOAT 5 (*) # <-- CHANGED!
# 10 RETURN_VALUE
As you can see, after 8 calls of dollars_to_pounds with a float, the bytcode is optimized. Instead of using BINARY_OP, it's using specialized BINARY_OP_MULTIPLY_FLOAT, which is faster when multiplying floats. If you'd run the same thing with an older Python version there wouldn't be any difference.
Improved Type Hints¶
there are improvements on the type hinting front.
Self Type¶
Python finally has support for a Self type. So, now you can easily type class methods and dunder methods:
from typing import Self
from dataclasses import dataclass
@dataclass
class Car:
manufacture: str
model: str
@classmethod
def from_dict(cls, car_data: dict[str, str]) -> Self:
return cls(manufacture=car_data["manufacture"], model=car_data["model"])
print(Car.from_dict({"manufacture": "Alfa Romeo", "model": "Stelvio"}))
# Car(manufacture='Alfa Romeo', model='Stelvio')
TypedDict NotRequired¶
Another improvement is the NotRequired type for typed dictionaries:
from typing import TypedDict, NotRequired
class Car(TypedDict):
manufacture: str
model: NotRequired[str]
car1: Car = {"manufacture": "Alfa Romeo", "model": "Stelvio"} # OK
car2: Car = {"manufacture": "Alfa Romeo"} # model (model is not required)
car3: Car = {"model": "Stelvio"} # ERROR (missing required field manufacture)
Literal String Type¶
There's also a new LiteralString type that allows literal strings and strings created from other literal strings. This can be used for type checking when executing SQL and shell commands to add another layer of safety for preventing injection attacks.
def run_query(sql: LiteralString) -> ...
...
def caller(
arbitrary_string: str,
query_string: LiteralString,
table_name: LiteralString,
) -> None:
run_query("SELECT * FROM students") # ok
run_query(query_string) # ok
run_query("SELECT * FROM " + table_name) # ok
run_query(arbitrary_string) # type checker error
run_query( # type checker error
f"SELECT * FROM students WHERE name = {arbitrary_string}"
)
Better Error Messages¶
One of the exciting new features is more descriptive tracebacks. Python 3.10 saw some improvements made by introducing better and more descriptive errors. Python 3.11 takes it a step further with improved descriptions of the exact error position. Let's take a look at an example.
Error in Python 3.10:
return sum(grades) / len(grades)
ZeroDivisionError: division by zero
Error in Python 3.11:
return sum(grades) / len(grades)
~~~~~~~~~~~~^~~~~~~~~~~~~
ZeroDivisionError: division by zero
Exception Notes¶
add_note was added to BaseExceptions. This allows you to add additional context to an exception after it was created. For example:
try:
raise ValueError()
except ValueError as exc:
exc.add_note("When this happened my dog was barking and my kids were sleeping.")
raise
# raise ValueError()
# ValueError
# When this happened my dog was barking and my kids were sleeping.
TOML Library¶
Python now has a library for parsing TOML files called tomllib. Its usage is very similar to that of the built-in json library.
[tool.poetry]
name = "example"
version = "0.1.0"
description = ""
authors = []
[tool.poetry.dependencies]
python = "^3.11"
[tool.poetry.dev-dependencies]
[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"
You can load the file like so:
import pprint
import tomllib
with open("pyproject.toml", "rb") as f:
data = tomllib.load(f)
pp = pprint.PrettyPrinter(depth=4)
pp.pprint(data)
"""
{'build-system': {'build-backend': 'poetry.core.masonry.api',
'requires': ['poetry-core>=1.0.0']},
'tool': {'poetry': {'authors': [],
'dependencies': {'python': '^3.11'},
'description': '',
'dev-dependencies': {},
'name': 'example',
'version': '0.1.0'}}}
"""