Rate-Limiting Strategies for High-Volume Batch Uploads

A fleet-wide overnight sync can slam an MRO gateway with tens of thousands of logbook records in a burst, and a gateway that buckles under that load risks dropping airworthiness data on the floor. The fix is not to reject bulk traffic but to shape it: admit what the backend can durably persist and make the client wait for the rest. This page covers token-bucket and sliding-window rate limiting at the gateway edge, a control layer within the Secure API Gateway Architecture that keeps continuing-airworthiness records — the kind mandated by EASA Part-M M.A.305 — from being silently lost under peak load.

Prerequisites & inputs

Rate limiting operates on authenticated, identified traffic. Each incoming request already carries a verified client identity (from the certificate and token layers) and a scope, so the limiter can budget per client rather than globally — the reliability feed for D-AIMA gets its own bucket separate from a subcontractor’s APU-overhaul exporter. The limiter needs three tuned inputs per client: a sustained rate (records per second the backend can commit), a burst allowance (how far above sustained a short spike may go), and a window length for the counting strategy.

Critically, the limiter’s job is admission, not deletion. A request that exceeds the budget is answered with 429 Too Many Requests and a Retry-After hint, never dropped. That contract lets a well-behaved client pace itself and retry, which is exactly the cooperative behaviour a bulk uploader should already implement. The gateway limiter here is the coarse first defence; the fine-grained flow control inside a worker pool is handled downstream by Backpressure and Rate Limiting in Asyncio Ingestion Pipelines.

Shaping the load

Gateway rate-limitingBatchuploadTokenbucketSlidingwindowAdmittedload

Step-by-step implementation

  1. Choose the model per traffic shape. A token bucket absorbs bursts gracefully: tokens refill at the sustained rate and a full bucket lets a short spike through. A sliding window enforces a hard ceiling over a rolling interval and is better where a strict cap must never be exceeded. Bulk uploads usually want the bucket for its burst tolerance.
  2. Refill lazily. Do not run a background timer per client. On each request compute elapsed time since the last refill and add elapsed * rate tokens, capped at capacity. This is O(1) per request and needs no scheduler.
  3. Charge per unit of work. A request carrying 500 records should cost more than one carrying five. Charge tokens proportional to batch size so a single fat request cannot starve the backend.
  4. Return an honest Retry-After. When the bucket is empty, compute how long until enough tokens accrue and hand that back so the client waits the right amount instead of hammering in a tight loop.
  5. Emit metrics. Every admit and throttle decision is logged and counted so operators can see when a client is chronically over budget and needs its sustained rate renegotiated rather than throttled nightly.

The following limiter combines a lazily-refilled token bucket with a sliding-window ceiling and returns a structured decision the gateway can turn into a 200 or a 429.

from __future__ import annotations

import logging
import time
from collections import deque
from dataclasses import dataclass, field
from typing import Deque

logging.basicConfig(level=logging.INFO, format="%(asctime)s | %(levelname)s | %(message)s")
logger = logging.getLogger("mro.gateway.ratelimit")


@dataclass
class Decision:
    admitted: bool
    retry_after_s: float
    reason: str


@dataclass
class ClientLimiter:
    """Per-client token bucket plus a sliding-window hard ceiling.

    Continuing-airworthiness records must not be lost under load; see
    EASA Part-M M.A.305 (aircraft continuing-airworthiness record).
    """

    rate_per_s: float          # sustained refill rate
    capacity: float            # burst allowance
    window_s: float            # sliding-window length
    window_max: int            # max records admitted per window
    _tokens: float = field(default=0.0)
    _last_refill: float = field(default_factory=time.monotonic)
    _events: Deque[tuple[float, int]] = field(default_factory=deque)

    def __post_init__(self) -> None:
        self._tokens = self.capacity

    def _refill(self, now: float) -> None:
        elapsed = now - self._last_refill
        self._tokens = min(self.capacity, self._tokens + elapsed * self.rate_per_s)
        self._last_refill = now

    def _window_count(self, now: float) -> int:
        while self._events and now - self._events[0][0] > self.window_s:
            self._events.popleft()
        return sum(count for _, count in self._events)

    def admit(self, registration: str, record_count: int) -> Decision:
        now = time.monotonic()
        self._refill(now)

        if self._window_count(now) + record_count > self.window_max:
            oldest = self._events[0][0] if self._events else now
            wait = max(0.0, self.window_s - (now - oldest))
            logger.warning("THROTTLED_WINDOW reg=%s n=%d wait=%.2fs", registration, record_count, wait)
            return Decision(False, round(wait, 2), "sliding_window_exceeded")

        if self._tokens < record_count:
            deficit = record_count - self._tokens
            wait = deficit / self.rate_per_s
            logger.warning("THROTTLED_BUCKET reg=%s n=%d wait=%.2fs", registration, record_count, wait)
            return Decision(False, round(wait, 2), "bucket_empty")

        self._tokens -= record_count
        self._events.append((now, record_count))
        logger.info("ADMITTED reg=%s n=%d tokens_left=%.1f", registration, record_count, self._tokens)
        return Decision(True, 0.0, "ok")

Worked example

The exporter for D-AIMA is configured with rate_per_s=50, capacity=200, window_s=1.0, and window_max=120. At the start of an overnight run the bucket is full at 200 tokens. The first request carries 150 ATA 49 APU-overhaul records for part APS3200-77; the bucket has room, 150 tokens are charged, and the window count becomes 150 — but wait, that exceeds window_max of 120, so the sliding window throttles it with Retry-After covering the remaining window. The client resubmits in two batches of 75.

The first batch of 75 is admitted (ADMITTED reg=D-AIMA n=75 tokens_left=125) and recorded in the window. The second batch of 75 pushes the window count to 150, over the 120 ceiling, so it is answered THROTTLED_WINDOW with a sub-second Retry-After. The client waits, the window slides, and the second batch flows a moment later. No APU record is ever discarded; each is either admitted or explicitly deferred with a precise wait.

Edge cases & compliance gotchas

  • Retries must not double-count. A client that retries after a 429 must send the same records, not append them again. Key each batch with an idempotency token tied to the source document hash so a replayed batch is deduplicated rather than counted twice against the window.
  • Distributed gateways drift. Two gateway replicas each holding a local bucket admit twice the intended rate. For a hard fleet-wide ceiling, back the counters with a shared store (a Redis sliding window, for example); the in-process limiter here is correct only for a single node or as a per-node sub-budget.
  • monotonic versus wall clock. Refill math must use a monotonic clock. If you compute elapsed time from the wall clock and an NTP step jumps it backwards, elapsed goes negative and the bucket stalls or over-fills — a subtle way to lose an upload window.
  • Throttling is not authentication. A limiter shapes volume; it does not decide who may push. Enforce identity first with Mutual TLS for Aviation Maintenance Data APIs so the budget is charged to a proven client and an unauthenticated flood is rejected at the handshake, never reaching the bucket at all.

Effective rate limiting turns a destructive traffic spike into an orderly queue. By pacing bulk uploads with a burst-tolerant bucket, capping them with a rolling window, and always answering overflow with an honest deferral instead of a dropped record, the gateway keeps every airworthiness entry accounted for while protecting the backend it must never overwhelm.

Related pages