# OCR and visible-text automation

Recognize, assert, and interact with visible text in images, web pages, mobile apps, and desktop applications.

Canonical HTML: https://shafthq.github.io/docs/integrations/ocr
Guide index: https://shafthq.github.io/llms.txt

# OCR and visible-text automation

:::warning[Preview: module not released]
`shaft-ocr` is available in `SHAFT_ENGINE` source but is not included in the
current published SHAFT release. Do not add the dependency below to a released
BOM project yet. Build the current source reactor only when you are evaluating
this preview, and wait for a containing release before using it in a released
project.
:::

Use `io.github.shafthq:shaft-ocr` in a source-build preview when you need to
recognize text from pixels. The module runs Tesseract locally through bundled
native JavaCPP libraries; it does not require a system Tesseract installation
or a cloud OCR service.

## Add the module

After a containing release is published, import its SHAFT BOM, then add
`shaft-engine` and `shaft-ocr` without module versions:

```xml
 
 
 
 io.github.shafthq 
 shaft-bom 
 ${shaft.version} 
 pom 
 import 
 
 
 

 
 
 io.github.shafthq 
 shaft-engine 
 
 
 io.github.shafthq 
 shaft-ocr 
 
 
```

Without `shaft-ocr`, OCR calls fail with a message that names the missing optional dependency.

## Target visible text

Create exact or partial OCR targets through the usual locator namespace. Use the target with WebDriver, Appium, or Playwright element actions:

```java
var exactText = SHAFT.GUI.Locator.hasOcrText("Checkout");
var partialText = SHAFT.GUI.Locator.containsOcrText("Check");

driver.element().click(exactText);
driver.element().hover(partialText);
driver.element().doubleClick(exactText);
```

The default target requires one match. Select a zero-based occurrence when the same text appears more than once:

```java
driver.element().click(
 SHAFT.GUI.Locator.containsOcrText("Save").occurrence(1)
);
```

SHAFT maps recognized screenshot coordinates to the active backend. WebDriver uses the browser viewport, Appium uses touch input, and Playwright uses its page mouse. Add `shaft-sikulix` to use the same OCR targets for desktop screen actions:

```java
new SHAFT.GUI.SikuliX().element()
 .click(SHAFT.GUI.Locator.hasOcrText("Calculator"));
```

## Assert recognized text

Assert an element screenshot with the same native string assertion syntax used elsewhere in SHAFT:

```java
driver.element().assertThat(By.id("receipt"))
 .ocrText()
 .contains("Payment complete");
```

Assert an encoded image or image file directly:

```java
SHAFT.Validations.assertThat()
 .image(Path.of("test-data/receipt.png"))
 .ocrText()
 .contains("Total");

SHAFT.Validations.verifyThat()
 .image(imageBytes)
 .ocrText()
 .contains("Order number");
```

OCR recognition attaches the source image and recognition details to the report. OCR targeting also attaches the selected match, including its text, confidence, and bounds.

## Process PDF documents

Use `PdfFileManager` to read native text and recognize text from scanned or mixed pages in the same PDF. SHAFT processes every page, keeps native positioned text when it is available, and calls `shaft-ocr` with a whole-page render only for pages that need pixel recognition.

```java
var document = new PdfFileManager("test-data/invoice.pdf").process();

String text = document.fullText();
var firstPage = document.pages().getFirst();

System.out.println(firstPage.source()); // NATIVE, OCR, or HYBRID
System.out.println(firstPage.confidence());
System.out.println(firstPage.tables());
```

Each page result contains page, block, paragraph, line, and word geometry. It also reports any applied orientation or deskew correction, inferred tables, confidence, and warnings. Result lists are immutable. Table inference uses aligned word geometry; validate irregular, borderless, or merged-cell results before using them as structured data.

Request exports explicitly. SHAFT writes each export through a temporary sibling file and then moves it into place:

```java
Files.createDirectories(Path.of("build"));

var result = new PdfFileManager("test-data/scanned-invoice.pdf").process(
 PdfExportRequest.to(PdfExportFormat.SEARCHABLE_PDF, Path.of("build/invoice-searchable.pdf")),
 PdfExportRequest.to(PdfExportFormat.HOCR, Path.of("build/invoice.hocr")),
 PdfExportRequest.to(PdfExportFormat.TSV, Path.of("build/invoice.tsv")),
 PdfExportRequest.to(PdfExportFormat.JSON, Path.of("build/invoice.json"))
);

result.exports().forEach(export ->
 System.out.println(export.output() + " " + export.sha256())
);
```

Existing output files are rejected by default. Call `replacingExisting()` on an export request when replacement is intentional. Searchable export of a signed PDF is also rejected because changing the document invalidates its signatures; call `allowingSignatureInvalidation()` only when that consequence is acceptable.

Process independent PDFs as an ordered batch:

```java
var requests = List.of(
 PdfDocumentRequest.of(Path.of("test-data/one.pdf")),
 PdfDocumentRequest.of(Path.of("test-data/two.pdf"))
);

PdfBatchResult batch = PdfFileManager.processAll(
 requests,
 new PdfBatchOptions(4, 256L * 1024 * 1024, false)
);

batch.items().forEach(item ->
 System.out.println(item.source() + " successful=" + item.successful())
);
```

The batch keeps request order and records item failures without discarding successful results. Set `failFast` to `true` to stop before later requests can publish exports; fail-fast execution is serial for that reason.

Set per-call recognition and safety limits through `PdfDocumentOptions`:

```java
var options = PdfDocumentOptions.defaults()
 .withRenderDpi(240)
 .withResourceLimits(100L * 1024 * 1024, 250, 20_000_000)
 .withPageTimeout(Duration.ofSeconds(60))
 .withAllureEvidence(false);

var document = new PdfFileManager("test-data/archive.pdf").process(options);
```

PDF processing attaches a JSON document summary and page-level recognition details to Allure by default. Those details can contain recognized document text, geometry, tables, warnings, and the source path. Disable them with `withAllureEvidence(false)` when the document is sensitive. `shaft.ocr.document.maximumAllureArtifactBytes` controls whether explicit export files are attached or represented by a size and checksum manifest; it does not cap page-level JSON details.

:::warning
SHAFT accepts PDF input only; it does not add Tabula or an ML table runtime. It rejects encrypted PDFs, inputs and page counts above their limits, and individually oversized rendered pages. Concurrent raster work is throttled by the batch byte budget. Treat OCR page timeouts as caller-side bounds: a native OCR library call may finish in its background thread after the timed operation has returned.
:::

## Tune recognition

Start from `OcrOptions.defaults()` when asserting an image or element. Tune a target directly when interacting with visible text:

```java
var options = OcrOptions.defaults()
 .withLanguages("English", "Arabic")
 .withMinimumConfidence(0.80)
 .withPreprocessingMode(OcrPreprocessingMode.GRAYSCALE)
 .withPageSegmentationMode(OcrPageSegmentationMode.SPARSE_TEXT)
 .within(new OcrRectangle(0, 0, 900, 500));

SHAFT.Validations.assertThat()
 .image(Path.of("test-data/bilingual-receipt.png"))
 .ocrText(options)
 .contains("الإجمالي");
```

Available preprocessing modes are `AUTO`, `NONE`, `GRAYSCALE`, `BINARY`, and
`INVERT`. `AUTO` derives an Otsu threshold from the image, composites alpha onto
white, and preserves pixel coordinates. Page segmentation modes cover automatic
text, a single block, line, or word, and sparse text.

## Configure language models

English and Arabic are the default languages. Pass Tesseract three-letter model
codes or supported human-readable names for other languages. SHAFT downloads
missing models on first use from a pinned `tessdata_fast` revision, verifies
their integrity, and stores them in the user cache.

Configure provisioning through the typed property namespace:

```java
SHAFT.Properties.ocr.set()
 .cacheDirectory("build/shaft-ocr-models")
 .downloadEnabled(false)
 .documentRenderDpi(300)
 .documentMaximumPages(500)
 .documentMaximumInFlightRasterBytes(256L * 1024 * 1024);
```

Use the matching `shaft.ocr.*` keys in `custom.properties` or as system properties when code configuration is not appropriate. Document options passed to `process(...)` override the defaults for that call.

:::warning
When downloads are disabled, every requested language model must already exist
in the configured cache and pass integrity verification. SHAFT fails before
recognition if a model is missing or altered.
:::

## Preview: managed OCR setup

:::warning[Not released]
The selection-aware managed setup workflow below is not yet available on
`SHAFT_ENGINE` `main` or in a published SHAFT release. Do not run these commands
or compile against these overloads until a containing release is available.

The planned provider pins every supported Tesseract language to one
`tessdata_fast` revision and SHA-256 checksum. It also applies the setup
artifact safety ceiling. Create and review an immutable plan before install:

```bash
shaft-cli setup plan --profile OCR --mode MANAGED \
 --language eng --language ara \
 --output /absolute/path/ocr-plan.json

# Review the JSON and copy its digest, then repeat every policy option.
shaft-cli setup install --plan /absolute/path/ocr-plan.json \
 --approve sha256: 
shaft-cli setup verify --profile OCR \
 --language eng --language ara
```

Omit `--language` for the baseline `eng` and `ara` bundle. Repeat it with
three-letter Tesseract codes such as `fra` and `deu` to provision another exact
set. Java callers will be able to pass

```java
new SetupSelection(List.of("fra", "deu"))
```

to the selection-aware `SHAFT.Infrastructure.plan`,
`SHAFT.Infrastructure.status`, `SHAFT.Infrastructure.verify`, and
`SHAFT.Infrastructure.install` overloads (or the equivalent low-level service
overloads). CLI install recovers the selection from the reviewed actions; repeating
`--language` is optional and must match when supplied. A custom cache must be
absolute. The provider will prefer it only when the complete requested set
verifies there, otherwise it will use the platform-native shared setup cache
when that complete set verifies.
:::

## Choose OCR for pixel-only text

Prefer semantic locators when the application exposes a stable DOM, accessibility tree, or native element hierarchy. Use OCR for canvases, remote desktops, streamed applications, rendered documents, screenshots, inaccessible native surfaces, and other cases where text exists only as pixels.

OCR accuracy depends on image resolution, contrast, font rendering, language models, and segmentation. Restrict the region, choose a suitable preprocessing mode, and raise the confidence threshold when the screen contains unrelated text.

## Related

- [Visual testing](./visual.md)
- [Desktop and video automation](./desktop-and-video.md)
- [Modular dependencies](../features/modules.md)
- [Mobile testing](../testing/mobile.md)