Chapter 10: HTTP, APIs, and WebSockets

The Application Layer in Practice

Most network communication today happens over HTTP — the protocol that powers the web, REST APIs, microservices, and increasingly, machine-to-machine communication. WebSockets extend HTTP to provide full-duplex, persistent connections for real-time applications.

This chapter covers HTTP from the programmer’s perspective: making requests, building APIs, and implementing real-time communication with WebSockets.

HTTP Fundamentals

Request and Response

HTTP follows a request-response model:

Client → Server:
  GET /api/users HTTP/1.1
  Host: api.example.com
  Accept: application/json
  Authorization: Bearer token123

Server → Client:
  HTTP/1.1 200 OK
  Content-Type: application/json
  Content-Length: 42

  [{"id": 1, "name": "Alice"}]

HTTP Methods

Method	Purpose	Idempotent	Safe
GET	Retrieve a resource	Yes	Yes
POST	Create a resource	No	No
PUT	Replace a resource entirely	Yes	No
PATCH	Partially update a resource	No	No
DELETE	Remove a resource	Yes	No
HEAD	Like GET but response has no body	Yes	Yes
OPTIONS	Describe communication options	Yes	Yes

Status Codes

Range	Category	Common Codes
1xx	Informational	100 Continue, 101 Switching Protocols
2xx	Success	200 OK, 201 Created, 204 No Content
3xx	Redirection	301 Moved, 302 Found, 304 Not Modified
4xx	Client Error	400 Bad Request, 401 Unauthorized, 403 Forbidden, 404 Not Found
5xx	Server Error	500 Internal Error, 502 Bad Gateway, 503 Service Unavailable

HTTP Clients with requests

The requests library is Python’s standard for synchronous HTTP:

import requests

# GET request
response = requests.get("https://httpbin.org/get")
print(response.status_code)  # 200
print(response.json())       # Parsed JSON

# POST with JSON body
data = {"name": "Alice", "email": "alice@example.com"}
response = requests.post("https://httpbin.org/post", json=data)

# Custom headers and authentication
response = requests.get(
    "https://api.example.com/data",
    headers={"Accept": "application/json"},
    auth=("username", "password"),
    timeout=10,
)

Session Management

Use a Session to persist settings across requests (connection pooling, cookies, headers):

import requests

session = requests.Session()
session.headers.update({"Authorization": "Bearer token123"})

# All requests through this session include the auth header
r1 = session.get("https://api.example.com/users")
r2 = session.get("https://api.example.com/orders")

Error Handling

import requests

try:
    response = requests.get("https://api.example.com/data", timeout=10)
    response.raise_for_status()  # Raises HTTPError for 4xx/5xx
    data = response.json()
except requests.ConnectionError:
    print("Failed to connect")
except requests.Timeout:
    print("Request timed out")
except requests.HTTPError as e:
    print(f"HTTP error: {e.response.status_code}")
except requests.JSONDecodeError:
    print("Invalid JSON response")

Async HTTP with aiohttp

For high-performance applications that need to make many HTTP requests concurrently, aiohttp is the async equivalent of requests:

Async Client

import asyncio
import aiohttp

async def fetch(session: aiohttp.ClientSession, url: str) -> dict:
    async with session.get(url) as response:
        return await response.json()

async def main():
    async with aiohttp.ClientSession() as session:
        # Fetch 10 URLs concurrently
        urls = [f"https://httpbin.org/get?id={i}" for i in range(10)]
        tasks = [fetch(session, url) for url in urls]
        results = await asyncio.gather(*tasks)
        for r in results:
            print(r["url"])

asyncio.run(main())

Full example: code/async_http_client.py

Async HTTP Server

aiohttp also provides an HTTP server framework:

from aiohttp import web

store = {}

async def get_item(request: web.Request) -> web.Response:
    key = request.match_info["key"]
    if key in store:
        return web.json_response({"key": key, "value": store[key]})
    raise web.HTTPNotFound(text=f"Key '{key}' not found")

async def set_item(request: web.Request) -> web.Response:
    key = request.match_info["key"]
    body = await request.json()
    store[key] = body["value"]
    return web.json_response({"key": key, "value": body["value"]}, status=201)

async def list_items(request: web.Request) -> web.Response:
    return web.json_response(store)

app = web.Application()
app.router.add_get("/items", list_items)
app.router.add_get("/items/{key}", get_item)
app.router.add_post("/items/{key}", set_item)

if __name__ == "__main__":
    web.run_app(app, host="0.0.0.0", port=8080)

REST API Design Patterns

When building APIs, follow these conventions:

Resource Naming

GET    /api/users           → List all users
GET    /api/users/42        → Get user 42
POST   /api/users           → Create a new user
PUT    /api/users/42        → Replace user 42
PATCH  /api/users/42        → Update user 42 partially
DELETE /api/users/42        → Delete user 42
GET    /api/users/42/orders → List orders for user 42

Pagination

async def list_users(request):
    page = int(request.query.get("page", 1))
    per_page = int(request.query.get("per_page", 20))
    offset = (page - 1) * per_page

    users = all_users[offset:offset + per_page]
    return web.json_response({
        "data": users,
        "page": page,
        "per_page": per_page,
        "total": len(all_users),
    })

Rate Limiting

import time
from collections import defaultdict

rate_limits = defaultdict(list)

def check_rate_limit(client_ip: str, max_requests: int = 60, window: int = 60) -> bool:
    now = time.time()
    requests = rate_limits[client_ip]
    # Remove old entries
    rate_limits[client_ip] = [t for t in requests if now - t < window]
    if len(rate_limits[client_ip]) >= max_requests:
        return False
    rate_limits[client_ip].append(now)
    return True

WebSockets

What Are WebSockets?

HTTP is a request-response protocol — the client always initiates communication. WebSockets upgrade an HTTP connection to a persistent, full-duplex channel where both client and server can send messages at any time.

WebSockets are ideal for:

Real-time chat and messaging
Live dashboards and monitoring
Collaborative editing
Gaming
Financial data streaming

The WebSocket Handshake

The connection starts as a regular HTTP request with an Upgrade header:

Client → Server:
  GET /ws HTTP/1.1
  Host: example.com
  Upgrade: websocket
  Connection: Upgrade
  Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==
  Sec-WebSocket-Version: 13

Server → Client:
  HTTP/1.1 101 Switching Protocols
  Upgrade: websocket
  Connection: Upgrade
  Sec-WebSocket-Accept: s3pPLMBiTxaQ9kYGzzhZRbK+xOo=

After this handshake, the connection switches from HTTP to the WebSocket protocol — a lightweight framing layer over TCP.

WebSocket Server with websockets

import asyncio
import websockets
import json

connected = set()

async def handler(websocket):
    connected.add(websocket)
    try:
        async for message in websocket:
            data = json.loads(message)
            response = json.dumps({"echo": data, "clients": len(connected)})
            # Broadcast to all connected clients
            await asyncio.gather(
                *[ws.send(response) for ws in connected]
            )
    finally:
        connected.discard(websocket)

async def main():
    async with websockets.serve(handler, "0.0.0.0", 8765):
        print("WebSocket server on ws://0.0.0.0:8765")
        await asyncio.Future()  # Run forever

asyncio.run(main())

WebSocket Client

import asyncio
import websockets
import json

async def client():
    async with websockets.connect("ws://localhost:8765") as ws:
        # Send a message
        await ws.send(json.dumps({"type": "hello", "name": "Alice"}))

        # Receive response
        response = await ws.recv()
        print(f"Received: {response}")

        # Listen for messages
        async for message in ws:
            print(f"Server: {message}")

asyncio.run(client())

Full example: code/websocket_chat.py

HTTP/2 and HTTP/3

HTTP/2

HTTP/2 improves on HTTP/1.1 with:

Multiplexing — multiple requests over a single TCP connection
Header compression (HPACK) — reduces overhead
Server push — server can proactively send resources
Binary framing — more efficient parsing

Python support via httpx or h2:

import httpx

# httpx supports HTTP/2 natively
async with httpx.AsyncClient(http2=True) as client:
    response = await client.get("https://example.com")
    print(response.http_version)  # "HTTP/2"

HTTP/3

HTTP/3 replaces TCP with QUIC (UDP-based), providing:

Faster connection setup (0-RTT)
No head-of-line blocking
Built-in encryption (TLS 1.3)
Connection migration (works across network changes)

HTTP/3 adoption is growing rapidly, especially for mobile and real-time applications.

Building a Complete API Client

Here’s a pattern for a robust, reusable API client:

import aiohttp
import asyncio
from typing import Any

class APIClient:
    def __init__(self, base_url: str, token: str):
        self.base_url = base_url.rstrip("/")
        self.token = token
        self.session: aiohttp.ClientSession | None = None

    async def __aenter__(self):
        self.session = aiohttp.ClientSession(
            headers={"Authorization": f"Bearer {self.token}"},
            timeout=aiohttp.ClientTimeout(total=30),
        )
        return self

    async def __aexit__(self, *args):
        if self.session:
            await self.session.close()

    async def get(self, path: str, **kwargs) -> Any:
        async with self.session.get(f"{self.base_url}{path}", **kwargs) as r:
            r.raise_for_status()
            return await r.json()

    async def post(self, path: str, data: dict, **kwargs) -> Any:
        async with self.session.post(f"{self.base_url}{path}", json=data, **kwargs) as r:
            r.raise_for_status()
            return await r.json()

# Usage
async def main():
    async with APIClient("https://api.example.com", "token123") as api:
        users = await api.get("/users")
        new_user = await api.post("/users", {"name": "Bob"})

Key Takeaways

HTTP is the dominant application-layer protocol, built on request-response over TCP
requests is the standard synchronous HTTP client; aiohttp provides async HTTP client and server capabilities
Follow REST conventions: resource-based URLs, proper HTTP methods, meaningful status codes
WebSockets upgrade HTTP to full-duplex persistent connections for real-time communication
HTTP/2 adds multiplexing and header compression; HTTP/3 uses QUIC for faster, more reliable transport
Always implement timeouts, error handling, and session management in HTTP clients
Use connection pooling (via sessions) to avoid the overhead of per-request TCP + TLS handshakes

← Previous: Asynchronous Networking and Protocols

Table of Contents

Next: Network Monitoring and Packet Analysis →

Gaëlle Candel