The Optimizer

What is the Optimizer?

The Optimizer creates versioned schema blueprints that improve extraction accuracy. It runs asynchronously in the background, iteratively refining your schema based on training documents and their correct values.

Blueprints and the optimizer are available from the Standard plan. On Free, POST /v1/optimize and a blueprint_id on POST /v1/process are refused with 402 naming the plan that has them.

How the Optimizer Works

1. Submit training data → Upload training documents with correct values, from the portal or via POST /v1/optimize
2. DeepAgent optimizes → Runs automatically: analyzes the training data and enhances your schema descriptions, iterating until it hits your target accuracy or budget
3. Use in production → Reference the blueprint in processing requests

Quick Start

Step 1: Prepare Training Data

Collect sample documents with their correct (ground truth) values. The API needs at least 2; we recommend 8 or more (see below). Below is a set of documents and the ground truth for that dataset.

Documents:

• invoice_001.pdf
• invoice_002.pdf
• invoice_003.pdf
• invoice_004.pdf

Ground Truth:

• ground_truth.json

Note: We believe that clean data leads to high accuracy. If you have the data and are having trouble putting a dataset together, we encourage you to submit a support request at https://www.deepread.tech/contact.

Step 2: Create Optimization Job

From the portal — create a new optimization request at dashboard/optimizer and upload your documents and ground truth. Optimization starts automatically: the page shows live progress (iteration, current accuracy, cost) while it runs, and we email you when it finishes. No manual step, and nothing to wait on a business day for.

From the API — POST /v1/optimize does the same thing:

Send your documents as base64 in documents_inline, in the same order as ground_truth_data:

curl -X POST https://api.deepread.tech/v1/optimize \
  -H "X-API-Key: $DEEPREAD_API_KEY" \
  -H "Content-Type: application/json" \
  --data-binary @- <<EOF
{
  "name": "Invoice blueprint",
  "document_type": "invoice",
  "documents_inline": [
    "$(base64 < invoice_001.pdf | tr -d '\n')",
    "$(base64 < invoice_002.pdf | tr -d '\n')"
  ],
  "ground_truth_data": [
    {"vendor": "Acme Corp", "total": 1250.00},
    {"vendor": "Globex", "total": 890.50}
  ]
}
EOF

Response:

{
  "job_id": "550e8400-e29b-41d4-a716-446655440000",
  "blueprint_id": "660e8400-e29b-41d4-a716-446655440001",
  "status": "pending",
  "message": "Optimization started",
  "warning": "Only 2 documents provided. With fewer than 8 documents the held-out validation is very small, so the accuracy comparison is a rough signal and tuned versions are unlikely to prove themselves. Add more labelled documents for a reliable optimization."
}

How many documents? We recommend 8 or more labelled documents. Part of your set is held back to validate the tuned version on documents it has never seen — with fewer than 8 docs that held-out set is only 1-2 documents, so the result is a rough signal and the tuned version is unlikely to prove itself. Runs with fewer documents still execute, but the response carries a warning like the one above.

* Provide either documents_inline or training_documents.

Poll GET /v1/blueprints/jobs/{job_id} for progress — it returns status, iteration, current_accuracy, validation_accuracy and total_cost.

Submitting from the portal is limited to 10 optimization requests per hour. Each run is bounded by max_iterations and max_cost_usd, so set max_cost_usd to the ceiling you're willing to spend on a run.

Step 3: Use the Optimized Schema

The result of the optimization request is a schema blueprint. Copy the blueprint_id from the respective optimization request on the optimizer page in our portal — or take it from the POST /v1/optimize response — and use it when calling the /process endpoint.

Reference the blueprint id in your processing requests:

curl -X POST https://api.deepread.tech/v1/process \
  -H "X-API-Key: $DEEPREAD_API_KEY" \
  -F "file=@new-invoice.pdf" \
  -F "blueprint_id=660e8400-e29b-41d4-a716-446655440001"

The API automatically uses the optimized schema that belongs to the blueprint_id.

Example of an Optimized Schema:

{
  "type": "object",
  "properties": {
    "vendor": {
      "type": "string",
      "description": "Utility company name. Often appears in header or top-left corner. Common formats: 'XYZ Electric', 'ABC Power Co.'"
    },
    "total": {
      "type": "number",
      "description": "Total amount due including all charges and taxes. Usually found at the bottom of the bill in bold or highlighted box. Look for labels like 'Amount Due', 'Total', 'Balance Due'."
    },
    "due_date": {
      "type": "string",
      "description": "Payment due date in YYYY-MM-DD format. Typically near the total amount or in a payment summary section. May appear as 'Due Date', 'Pay By', or 'Payment Due'."
    }
  }
}

Notice how the descriptions include details such as:

• Location hints: "top-left corner", "bottom of the bill"
• Format examples: "YYYY-MM-DD format"
• Common variations: "Amount Due', 'Total', 'Balance Due'"

What Gets Optimized?

The optimizer improves field descriptions by adding:

1. Location hints: Where the field appears in the document
2. Format examples: Expected data formats and patterns
3. Common labels: Alternative names the field might have
4. Edge case handling: How to handle ambiguous cases

Before optimization:

{
  "due_date": {
    "type": "string",
    "description": "Due date"
  }
}

After optimization:

{
  "due_date": {
    "type": "string",
    "description": "Payment due date in YYYY-MM-DD format. Typically near the total amount or in a payment summary section. May appear as 'Due Date', 'Pay By', or 'Payment Due'."
  }
}