Running RolmOCR on Vast.ai

RolmOCR from Reducto is an open-source document OCR model built on Qwen2.5-VL-7B. Unlike traditional OCR tools like Tesseract that rely on character-level pattern matching, RolmOCR uses vision-language understanding to interpret documents semantically. This means it handles rotated text, complex layouts, and mixed content without manual preprocessing, and can extract structured data directly rather than just raw text. This guide demonstrates extracting structured pricing data from invoice images.

Prerequisites

Vast.ai account with credits
Vast.ai CLI installed (pip install vastai)

Hardware Requirements

GPU RAM: 16GB minimum, 60GB recommended for larger documents and wider context
Disk: 80GB
Static IP: Required for stable endpoint

Setting Up the CLI

pip install vastai
vastai set api-key YOUR_API_KEY

Finding an Instance

vastai search offers 'compute_cap >= 750 geolocation = US gpu_ram >= 60 num_gpus = 1 static_ip = true direct_port_count >= 1 verified = true disk_space >= 80 rentable = true'

Deploying the Instance

First, generate a secure API key to protect your endpoint:

VLLM_API_KEY=$(openssl rand -hex 32)
echo "Save this API key: $VLLM_API_KEY"

Deploy with vLLM using the v1 engine (required for RolmOCR):

INSTANCE_ID=<your-instance-id>
vastai create instance $INSTANCE_ID \
    --image vllm/vllm-openai:latest \
    --env "-p 8000:8000 -e VLLM_USE_V1=1 -e VLLM_API_KEY=$VLLM_API_KEY" \
    --disk 80 \
    --args --model reducto/RolmOCR

Installing Client Dependencies

pip install --upgrade openai datasets pydantic

Loading Sample Data

For this example, we’ll use a public invoice dataset from Hugging Face. Streaming mode avoids downloading the entire dataset:

from datasets import load_dataset

streamed_dataset = load_dataset(
    "katanaml-org/invoices-donut-data-v1",
    split="train",
    streaming=True
)
subset = list(streamed_dataset.take(3))

Encoding Images

The API expects base64-encoded images. This helper function also resizes images to reduce payload size and improve processing speed:

import base64
import io
from PIL import Image

def encode_pil_image(pil_image):
    max_size = 1024
    ratio = min(max_size / pil_image.width, max_size / pil_image.height)
    new_size = (int(pil_image.width * ratio), int(pil_image.height * ratio))
    resized_image = pil_image.resize(new_size, Image.Resampling.LANCZOS)

    img_byte_arr = io.BytesIO()
    resized_image.save(img_byte_arr, format='JPEG', quality=85)
    return base64.b64encode(img_byte_arr.getvalue()).decode("utf-8")

Defining the Output Schema

Define a Pydantic model for the expected output. vLLM uses this schema with guided decoding-constraining the model’s token generation to only produce valid JSON matching your structure. This eliminates malformed responses and post-processing validation, which is critical for OCR pipelines where you need reliable field extraction:

from pydantic import BaseModel

class Invoice(BaseModel):
    invoice_number: str
    invoice_amount: str

json_schema = Invoice.model_json_schema()

Calling the API

Create the OpenAI client and a function to extract invoice data. The guided_json parameter passes your schema to vLLM’s constrained decoding engine, guaranteeing the response will be parseable JSON with exactly the fields you defined:

from openai import OpenAI

VAST_IP_ADDRESS = "your-ip"
VAST_PORT = "your-port"
VLLM_API_KEY = "your-api-key"

client = OpenAI(
    api_key=VLLM_API_KEY,
    base_url=f"http://{VAST_IP_ADDRESS}:{VAST_PORT}/v1"
)

def ocr_invoice(img_base64):
    response = client.chat.completions.create(
        model="reducto/RolmOCR",
        messages=[
            {
                "role": "user",
                "content": [
                    {
                        "type": "image_url",
                        "image_url": {"url": f"data:image/png;base64,{img_base64}"},
                    },
                    {
                        "type": "text",
                        "text": "Return the invoice number and total amount as JSON: {invoice_number: str, invoice_amount: str}",
                    },
                ],
            }
        ],
        extra_body={"guided_json": json_schema},
        temperature=0.2,
        max_tokens=500
    )
    return response.choices[0].message.content

Processing Invoices

Loop through the sample invoices and compare extracted values against the ground truth labels:

import json

for sample in subset:
    img_base64 = encode_pil_image(sample['image'])
    result = json.loads(ocr_invoice(img_base64))

    ground_truth = json.loads(sample["ground_truth"])
    expected = {
        "invoice_number": ground_truth["gt_parse"]["header"]["invoice_no"],
        "invoice_amount": ground_truth["gt_parse"]["summary"]["total_gross_worth"]
    }

    print(f"Expected: {expected}")
    print(f"Extracted: {result}")

Example output:

Expected: {'invoice_number': '40378170', 'invoice_amount': '$8,25'}
Extracted: {'invoice_number': '40378170', 'invoice_amount': '$8.25'}

Expected: {'invoice_number': '61356291', 'invoice_amount': '$ 212,09'}
Extracted: {'invoice_number': '61356291', 'invoice_amount': '$212.09'}

Expected: {'invoice_number': '49565075', 'invoice_amount': '$96,73'}
Extracted: {'invoice_number': '49565075', 'invoice_amount': '$96,73'}

The model accurately extracts invoice data with minor formatting differences (comma vs decimal point) that can be normalized downstream.

AI/ML frameworks

Serving infrastructure

Migration guides

AI agents

MCP

Text generation

Image generation

Video generation

Audio & transcription

Information extraction

Serverless deployments

Templates

Virtual desktops & VMs

Graphics & GPU programming

Other

Running RolmOCR

Running RolmOCR on Vast.ai

Prerequisites

Hardware Requirements

Setting Up the CLI

Finding an Instance

Deploying the Instance

Installing Client Dependencies

Loading Sample Data

Encoding Images

Defining the Output Schema

Calling the API

Processing Invoices

Additional Resources

​Running RolmOCR on Vast.ai

​Prerequisites

​Hardware Requirements

​Setting Up the CLI

​Finding an Instance

​Deploying the Instance

​Installing Client Dependencies

​Loading Sample Data

​Encoding Images

​Defining the Output Schema

​Calling the API

​Processing Invoices

​Additional Resources

Running RolmOCR on Vast.ai

Prerequisites

Hardware Requirements

Setting Up the CLI

Finding an Instance

Deploying the Instance

Installing Client Dependencies

Loading Sample Data

Encoding Images

Defining the Output Schema

Calling the API

Processing Invoices

Additional Resources