Hawkeye: Reading Illegible Text

Abstract

While modern Vision-Language Models (VLMs) have been remarkably successful in general scene text recognition, they consistently struggle with the long-tail distribution of text that is degraded by motion blur or low resolution. These limitations are particularly apparent in egocentric applications like smart glasses, where text is often captured in noisy environments. In this paper, we introduce the BSTRB (Blurry Scene Text Recognition Benchmark), a novel benchmark of 9,936 noisy image samples with high-fidelity ground-truth annotations derived from 1,699 real-world video sequences. To support this, we present Hawkeye4B, a VLM fine-tuned on BSTRB to read and reason about illegible text by leveraging deep contextual priors and pixel-level visual alignment. Our results show that Hawkeye4B matches or exceeds text transcription performance on much larger closed-weight VLMs, making it practical for real-time deployment. We characterize the text resolution at which current VLMs begin to fail.

Results at a Glance

Held-out results from the paper. Differences are paired against each model’s own Qwen3-VL base, with clustered bootstrap confidence intervals and Holm correction.

+10.1pp

BSTRB video, Hawkeye 4B vs. base

BoW F1 0.627 → 0.728; Hawkeye 8B +6.9 pp. Photos: +6.0 and +5.3 pp.

0.744

Highest BSTRB video score of 24 models

Hawkeye 8B, ahead of Qwen3-VL-235B (0.726), Kimi K2.6 (0.723) and GPT-6 Astra (0.709).

+13.0pp

Word inclusion on smart-glasses captures

Hawkeye 8B vs. base on 37 objects captured with Meta Ray-Ban glasses.

+7.2pp

Snellen rows of 7 px or smaller

Hawkeye 4B vs. base on 50 charts; Hawkeye 8B +7.5 pp.

0/ 10

General OCR benchmarks with a decrease

No significant BoW F1 decrease relative to the base model at either size.

14.3% → 7.2%

Repetition loops on text of 20 px or less

Hawkeye 4B vs. base on BSTRB video; 8B: 8.5% → 5.2%.

The BSTRB Benchmark

Smartphone recordings of everyday objects, each captured while the camera moves away, so that one transcription is paired with text that shrinks from roughly 20 px to below 5 px.

Six frames from one BSTRB video: a box of text shrinks from 19.6 px to 4.5 px mean text height as the camera moves away; a VLM's transcription errors grow from 0 to over 45. — Six frames from one recording. Mean text height decreases from 19.6 px to 4.5 px, and the number of transcription errors by GPT-5.2 increases from 0 to more than 45.

Split	Video frames	Videos	Photos	Labels
train	6,704	1,128	695	automatic (GPT-5)
validation	333	56	36	human-verified
test	1,325	223	241	human-verified

Objects. Packaging, books, medicine, signage and screens, recorded at home.
Sampling. Six frames per recording, one transcription per recording.
Splits. No recording appears in more than one split; validation was used for model selection.
Privacy. Neutral file identifiers, image metadata removed, gated access with a data-use agreement.

Grid of held-out BSTRB clips, one object per row, at six frame positions, each labelled with its median text height. — Held-out recordings at the six sampled frame positions, with the median text height of each frame.

Composition and text density

Automatic training labels for photographs are frequently incomplete; Hawkeye is therefore trained on video frames only.

Bar charts of object, scene and text categories in BSTRB. — Composition by object, scene and text category.

Words per image for BSTRB and other OCR benchmarks. — Words per image in BSTRB and in other OCR benchmarks.

Results

The improvement over the base model is largest for text below 20 px; on video frames, Hawkeye 8B attains the highest BoW F1 of the evaluated models.

BoW F1 by mean text height on held-out BSTRB video frames (bins of about 105 frames, 95% bootstrap CIs shown for Hawkeye). Each side of 20 px has its own vertical scale. Select a legend entry to show or hide a model; hover or focus a bin for exact values.

Benchmark (held-out)	Hawkeye 4B − Qwen3-VL-4B	Hawkeye 8B − Qwen3-VL-8B
BSTRB video, BoW F1	+10.1 pp (.627 → .728)	+6.9 pp (.675 → .744)
BSTRB photos, BoW F1 (241)	+6.0 pp	+5.3 pp
10 general OCR sets, BoW F1	4 better, 6 n.s., 0 worse	2 better, 8 n.s., 0 worse
Smart-glasses captures (37 objects)	BoW F1 +14.7, aligned F1 +16.6 pp; word inclusion n.s.	word inclusion +13.0, BoW F1 +11.5, aligned F1 +12.3 pp
Snellen charts, text ≤7 px (50 charts)	+7.2 pp	+7.5 pp
Repetition loops, BSTRB video ≤20 px	14.3% → 7.2%	8.5% → 5.2%

Highlighted cells are significant. pp: percentage points.

Statistical methodology

Test set. Results as reported in the paper, on the full held-out video set (1,331 frames, 224 videos).
Public split. One video is omitted for privacy reasons, which changes every score by at most 0.001.
Statistics. Paired bootstrap, clustered by video (BSTRB), object (smart-glasses captures) or chart (Snellen), with Holm correction.

All Models and Benchmarks

BoW F1 of all 24 models on held-out BSTRB and ten general OCR benchmarks. Select a column header to sort the models by that benchmark.

Values are BoW F1 on each benchmark’s evaluation set (n images); outlined cells mark the best model per benchmark. Hover a cell for the exact value.

Change Relative to the Base Model

Difference between each Hawkeye model and its own Qwen3-VL base on every benchmark, with 95% confidence intervals. Filled markers are significant after Holm correction; open markers are not.

Hawkeye 4BHawkeye 8Bnot significant

Smart-Glasses Captures

91 images of 37 objects captured with Meta Ray-Ban smart glasses, outside the BSTRB distribution. Scores are paired over images against the same base model.

Degenerate outputs are empty or repetitive responses, removed for both models. Differences are paired bootstrap estimates with Holm correction.

Conventional OCR Engines

EasyOCR and Tesseract fall far behind the vision-language models on degraded frames. Google Vision runs only on the sharp reference frames, where it is competitive.

Top: BoW F1 by mean text height on held-out BSTRB video frames. Bottom: BoW F1 on the sharp reference frames, with 95% bootstrap confidence intervals.

Small Text: Snellen Charts

On clean, synthetic Snellen-style charts, models diverge at 7 px and below: Hawkeye improves over its base by 7.2 pp (4B, 95% CI 4.5–9.9) and 7.5 pp (8B, 4.7–10.2) on these rows.

One of the 50 evaluation charts: nine rows of capital letters whose height decreases from 48 px at the top to 4 px at the bottom.

A Snellen chart is the eye-test chart used in optometry: rows of capital letters whose size decreases from top to bottom, read until the letters can no longer be made out. Here each chart is rendered as an image with known letter heights, from 48 px down to 4 px, so that reading accuracy can be measured separately for each height on clean text, free of blur, lighting or perspective.

Row	Height	Letters

At 4 px. Hawkeye 4B reads 0.48 of characters, against 0.24 for its base and 0.27–0.40 for the API models shown.
From 5 px. Most models are near ceiling; there the base 4B is slightly above Hawkeye 4B (0.976 vs. 0.947).

Rows	4BΔ pp [95% CI]	8BΔ pp [95% CI]
Loading.

Character accuracy per row on 50 charts, by optotype height; Hawkeye minus base, paired and clustered by chart, Holm-corrected. Hover or focus a height for exact values.

Readings of the Two Smallest Rows

Model

Letters are compared position by position with the reference; incorrect or extra letters are shown in red and missing letters as a red dot.

Latency and Cost

On one local GPU, Hawkeye reads a crop in under a second (median), comparable to Qwen3-VL-235B served through an API and 10–13 times faster than the reasoning API models.

Assumptions GPU rental (RTX 6000 Ada) $ per hour Reading rate crops per second Use hours per day

Model	AccuracyBSTRB video BoW F1	VRAMpeak, GB	Price per 1M tokensUSD, input / output	First tokenseconds	Decode speedtokens per second	Latencyseconds per crop	CostUSD per 1,000 crops	Cost per dayUSD, continuous use
Measurements are being added.

* Local models: GPU rental price divided by measured throughput at batch size 1 (batching lowers per-token and per-crop cost). API models: list prices.

Measurement details

Inputs: 70 crops from the demonstration recordings; every model receives identical inputs.
Local models: One NVIDIA RTX 6000 Ada, batch size 1, Hugging Face transformers, bfloat16.
API models: Streamed through a LiteLLM proxy with the evaluation protocol and output budgets of the paper.
Latency: Medians per crop; hidden reasoning tokens are included in API latency and cost.
First token: Where an API returns the response in a single chunk, it equals total latency and no decode speed is given.
Decode speed: Generated tokens after the first, divided by the time after the first token, pooled over crops.
Price per token: Local input: GPU price / prefill throughput; local output: GPU price / decode throughput.
Cost: Local: mean GPU time per crop × GPU price. API: token counts × list prices (October 2026).
Cost per day: Reading rate × hours per day; above about one crop per second a second GPU is needed.
Repetition loops: Loops on dense text raise the mean GPU time, and hence the local cost, above what the median latency suggests.
VRAM: Peak allocated GPU memory: 8.7 GB for the 4B models, 16.7 GB for the 8B models.
Accuracy: BoW F1 on held-out BSTRB video frames as reported in the paper; Claude Opus 5.5 was not part of that evaluation.

Accuracy on the Demonstration Recordings

The same 70 crops, scored against reference transcriptions of the eight recorded objects with order-independent metrics averaged over crops.

Model	BoW F1	Word inclusion	Edit-tolerant BoW F1
Hawkeye 4B local	0.273	0.231	0.282
Hawkeye 8B local	0.253	0.241	0.267
GPT-6 Astra API	0.215	0.205	0.223
Kimi K2.6 API	0.190	0.175	0.222
Qwen3-VL-235B API	0.186	0.162	0.213
Claude Opus 5.5 API	0.159	0.125	0.212
Qwen3-VL-8B (base) local	0.190	0.187	0.197
GPT-5 API	0.157	0.124	0.187
Qwen3-VL-4B (base) local	0.176	0.173	0.184

Scoring details

References. Tea bag, honey, Lysol, Bounty and Band-Aid packages: transcribed by the authors.
References. Book, supplement label and web page: words read by both GPT-6 Astra and Claude Opus 5.5 on close-up photographs.
Edit-tolerant F1. A word matches an unmatched reference word at normalized edit distance ≤ 0.2.
Splits and merges. A word split or merged at one space (“Sue Bee” / “SueBee”) counts as one match.
Recall bound. Each reference covers the whole object, while a crop usually shows part of it, equally for all models.
Scope. Crops were assigned to objects by inspection; eight objects and 70 crops illustrate the demonstration and are not a benchmark result.

Attention Analysis

The base model attends mainly to the largest print, whereas Hawkeye also attends to smaller text, which it then transcribes.

Loading the attention viewer.

Mean attention of the base models and Hawkeye at 4B and 8B (brighter indicates more attention). Frames were selected among those on which Hawkeye outperforms its base model at both sizes.

Attention at individual decoding steps, zoomed to the word being written, for base and Hawkeye at 4B and 8B. — Attention at individual decoding steps, cropped around the generated word. The cross marks the attention peak; the box marks the word as located by OCR.

Methodology

Attention to image patches is taken from the last four layers and averaged over heads; content-free attention-sink patches are excluded. Across all analysed frames, the attention peak of both models falls on the word being generated far more often than chance, and Hawkeye does not cover a larger share of the text regions on average. The difference lies in how much of the text it goes on to transcribe. The examples shown are illustrative.

Reading trace of base Qwen3-VL-4B and Hawkeye 4B on a web page photographed with smart glasses. — Reading trace on a web page captured with smart glasses. The base model transcribes a single title; Hawkeye transcribes the sidebar and the post.

Method

An open model, fine-tuned on degraded video frames and run in a single pass.

Pipeline: scene keyframe, object mask, cropped object, image tiles, visual tokens, transcription.

1

Initialization

Hawkeye 4B and 8B are initialized from Qwen3-VL-4B-Instruct and Qwen3-VL-8B-Instruct.

2

Training

Supervised fine-tuning on BSTRB video frames, followed by reinforcement learning (GRPO) with a bag-of-words F1 reward.

3

Inference

The input is upscaled three times with bicubic interpolation and transcribed in a single pass, without chain-of-thought or tool use.

Implementation details

Training. Supervised fine-tuning on 6,734 BSTRB training frames (vision encoder unfrozen, 420 steps), followed by GRPO with LoRA (rank 64) and a bag-of-words F1 reward against the reference transcription (300 steps).
Inference. An OCR system prompt, the instruction, and the image upscaled ×3 with bicubic interpolation (the processor limits it to approximately 1 MP); greedy decoding with up to 4,096 new tokens.
Evaluation. All models receive the same prompt and image; scores are computed per image after normalizing case, punctuation and whitespace.

Usage

The hawkeye package implements the inference protocol used in the paper.

# pip install "hawkeye[infer] @ git+https://github.com/byron123t/hawkeye"
from hawkeye import Hawkeye

reader = Hawkeye("byron123t/Hawkeye-4B")      # or "byron123t/Hawkeye-8B"
print(reader.read("photo.jpg"))

Models. Hawkeye-4B and Hawkeye-8B on Hugging Face (Apache-2.0); each model card includes a version using transformers directly.
Dataset. BSTRB on Hugging Face (CC BY-NC 4.0); access is granted on request under a data-use agreement.
Code. Training, evaluation and analysis in the GitHub repository.

BibTeX

@article{tang2026hawkeye,
  title   = {Hawkeye4B: Reading Illegible Text with Vision Language Models},
  author  = {Tang, Brian and Shin, Kang G.},
  journal = {arXiv preprint arXiv:XXXX.XXXXX},
  year    = {2026}
}