Fast OCR in Production: RapidOCR, ONNX Runtime and Bounded Queues

A 24-megapixel photo arrives with incorrect EXIF orientation, and the target number occupies only two percent of the frame. OCR may answer in a fraction of a second on a clean benchmark and then saturate an API when twenty similar images arrive together. The model explains only part of the outcome. Resize policy, concurrency, input limits and response handling matter just as much.

RapidOCR with ONNX Runtime is a useful foundation for lightweight, local CPU OCR. Treat it as a vision pipeline rather than a magic function that turns every image into correct text.

What the pipeline actually runs

A complete OCR pipeline normally chains three networks:

1. Detection locates regions that resemble text and returns quadrilaterals.
2. Orientation classification determines whether a line should be rotated, commonly by 180 degrees.
3. Recognition converts each rectified region into characters with a score.

RapidOCR orchestrates these stages and uses ONNX Runtime for CPU inference. Common PP-OCRv4 mobile distributions include a detector of roughly 4.5 MB, a small orientation classifier and a recognizer of roughly 10 MB. The v4 recognizer commonly receives a line resized to 3 × 48 × 320, while the classifier works with 3 × 48 × 192.

Those sizes describe weight files, not total process memory. Add ONNX Runtime, OpenCV or Pillow, decoded pixels, intermediate tensors and session workspaces. Four processes will generally load four copies of the chain. Measure real memory after warm-up with representative images.

The recognition vocabulary is another model decision. A Chinese–English–digit model does not have the same dictionary or errors as an English or Latin-specialized model. For numeric bibs, restricting acceptable values with an event rule is often more effective than silently accepting every character in a broad dictionary.

The RapidOCR API documentation lists input dimensions, detection thresholds and the three stages. Defaults are a starting point, not calibration for your cameras.

Speed numbers are reference points, not promises

The PP-OCRv4 publication reports a 15.8 MB pipeline, an Hmean of 62.24 percent and 76 ms per image in its end-to-end evaluation. That measurement used OpenVINO on an Intel Gold 6148. It does not include a web request’s JPEG decoding, EXIF correction, network transfer and JSON serialization. The official PP-OCRv4 report states its hardware and inference engine.

In another benchmark published by RapidOCR, the 4.5 MB PP-OCRv4 ONNX detector records 0.2256 seconds per image on its test set. Its 10 MB recognizer records 0.6836 seconds per test image, with 83.23 percent exact matches and 93.55 percent correct characters. These values cannot be added directly because the datasets, units and number of detected lines differ. They illustrate how far protocols can move a headline number. The complete tables appear in the RapidOCR model comparison.

A practical expectation is narrower: a reasonably sized image can often be processed interactively on a modern CPU, while latency changes substantially with resolution, number of text regions, processor and concurrency. A dense page costs more than a crop containing one number.

Measure at least:

• file read and decode time;
• preprocessing separately from inference;
• detection, orientation and recognition durations;
• time waiting in a queue;
• total p50, p95 and p99 latency;
• sustained throughput, errors and peak memory;
• quality on the operational corpus, not only printed text.

A useful test includes warm-up, several image sizes, images without text, dense scenes and request bursts. Reporting only an average hides the blocking that appears in p95.

Why resize, and how far

Decoding a 6000 × 4000 image already creates about 72 MB of uncompressed RGB data. Feeding it unchanged into detection increases memory and compute even though the networks will resize their input again.

Bounding the longest dimension to 1,500 or 2,000 pixels makes load more predictable. Apply EXIF orientation first, preserve aspect ratio and convert to the expected color representation. The bound has a cost: a small distant number may become unreadable.

A two-pass strategy is often a better compromise:

• analyze a bounded image to locate probable regions;
• recognize crops taken from the original at higher resolution, with a margin around the text.

If another stage already knows the location—such as a detected bib region—sending only that crop saves more work than changing the inference engine. Preserve the geometric transform so returned coordinates still map to the source image.

Deploy a reproducible pipeline

A CPU container is enough to begin. Include required system libraries, locked Python dependencies and preferably the ONNX files themselves. Downloading weights during the build prevents the first live request from depending on the network or paying initialization alone.

Several rules make the service reproducible:

• pin RapidOCR and ONNX Runtime versions plus model checksums;
• store explicit paths for detector, classifier, recognizer and dictionary;
• attach those revisions to metrics and results;
• build without user credentials or machine-specific caches;
• run synthetic inference during the build and warm the model at startup;
• expose readiness only after models are loaded;
• apply CPU and memory limits, then test what happens at the limit;
• stop gracefully by refusing new work before terminating in-flight work.

A health endpoint returning “OK” does not prove that inference works. A startup probe can load the models, while readiness checks worker state without running expensive OCR each time. A less frequent synthetic test can exercise the complete chain.

Create one engine per process and keep it resident. Rebuilding it for every image adds unnecessary latency. Increasing the process count without a bound multiplies memory and may reduce throughput as internal threads compete.

Keep OCR from blocking the API

Inference is synchronous CPU work. Placing it directly inside an async route does not make it asynchronous; the event loop may remain occupied until the calculation finishes. Three architectures cover most uses.

Interactive response with bounded concurrency

For one image, run the engine in a thread pool or synchronous worker behind a global semaphore. Choose a small limit from load tests. When every slot is occupied, wait briefly, then return 429 Too Many Requests or 503 Service Unavailable with Retry-After.

slots = Semaphore(MAX_IN_FLIGHT)

async def recognize(upload):
    image = await decode_with_limits(upload)
    if not await slots.acquire(timeout=0.2):
        raise Busy(retry_after=1)
    try:
        return await run_in_worker_thread(ocr_engine, image)
    finally:
        slots.release()

The limit protects latency and memory. It should not mechanically equal the number of cores because ONNX Runtime uses internal threads. The ONNX Runtime threading guide explains intra-op and inter-op controls. Keep processes × concurrent requests × ONNX threads consistent with the CPUs actually allocated.

Durable asynchronous queue

For a batch, validate the file, create an idempotent job ID and return 202 Accepted. A separate worker consumes a bounded durable queue, writes the result and signals completion. The client polls status or receives an event.

This design absorbs bursts without holding HTTP connections for minutes. It also supports priority, cancellation and restart recovery. The queue still needs a hard capacity. An infinite queue turns visible overload into invisible delay.

Micro-batches

Recognizing several text lines may benefit from a small batch, while grouping arbitrary full-size photos delays the first result. Set both a maximum batch size and a short time window. Dispatch when either is reached.

Retries, deadlines and deduplication

A call can succeed internally and fail before the client receives the response. Without a stable identifier, retrying performs the work again.

Build a key from the image content, pipeline revision and meaningful parameters. The cache must distinguish a valid empty result from a technical failure. A new model revision produces a new key.

Retry selectively:

• never retry an invalid format or oversized image;
• retry temporary unavailability with exponential backoff and jitter;
• cap attempt count and total duration;
• send persistent failures to a review queue;
• use a circuit breaker when the worker is unavailable;
• propagate a deadline so expired work does not consume CPU.

A client timeout alone is insufficient. The service needs a way to cancel queued work and avoid filling the queue with requests whose callers have already left.

Strengths and limits

RapidOCR with ONNX Runtime offers compact weights, local CPU execution, no mandatory third-party API call, text coordinates and configurable stages. ONNX also makes it easier to select a runtime for a target platform.

Its limits are those of lightweight general OCR. Small characters, blur, perspective, reflections, folded fabric and curved text reduce quality. Model confidence is not a calibrated business probability. A highly confident system may read the wrong number perfectly—for example, a sign in the background.

The standard classifier mainly corrects 0- or 180-degree lines; it does not rectify every rotation or perspective. Unsupported alphabets, fonts and languages add errors. A GPU is not automatically faster for small dynamic-shape models because transfer, initialization and small batches can cancel its advantage.

For a bib, plate or short identifier, validate against domain rules: format, length, permitted range, candidate roster and event context. Preserve raw text, score, region and alternatives. The bib-number organization guide shows how to use this evidence without turning it into proof of identity.

Pre-production checklist

Before opening traffic, verify that:

• allowed formats, byte size and pixel count are checked before full allocation;
• EXIF orientation is corrected and unnecessary sensitive metadata is removed;
• dependencies and weights are pinned, included in the image and checksum-verified;
• the model loads once, warms up and has measured memory use;
• concurrency and queue capacity are bounded, with explicit overload responses;
• timeouts, cancellation, idempotency, caching and retry rules are tested;
• queue metrics, p50/p95/p99, throughput, memory, errors and empty results are visible;
• the validation corpus represents real angles, materials, devices and lighting;
• thresholds are selected on that corpus and uncertain cases enter review;
• restarts have been tested with queued and active jobs.

The live event upload workflow adds reliable ingestion and incomplete-state handling. The person crop quality guide explains why the content sent to a model often matters more than a few milliseconds saved during inference.

Fast production OCR is not only a small model. It is a compact model, controlled input, bounded concurrency and preserved uncertainty. That combination keeps an API responsive when images stop arriving one at a time.