convertapi-haystack

Haystack converters for ConvertAPI, the file conversion API.

• ConvertAPIDocumentConverter turns PDF, Word, Excel, PowerPoint and HTML files into Markdown Documents that keep their headings and tables, ready for splitting and embedding. It can also extract plain text, with OCR for scanned PDFs and images.
• ConvertAPIFileConverter converts files and web pages to another format and returns ByteStreams: Word, Excel and PowerPoint files to pixel-perfect PDFs, HTML and web pages to PDF, PDF to Word or images, and the rest of ConvertAPI's 300+ conversions.

The conversion runs on ConvertAPI's servers, so there is no office suite, browser or OCR engine to install next to your pipeline. Both components work in Pipeline and AsyncPipeline, and with Haystack 2.23 or later, 3.x included.

Installation

pip install convertapi-haystack

Sign up at convertapi.com, copy your API token from the Authentication page, and set it as an environment variable:

export CONVERTAPI_TOKEN="your-api-token"

Usage

Index files as Markdown

ConvertAPIDocumentConverter returns one Document per file. Markdown keeps the structure that matters for retrieval: headings stay headings and tables stay tables.

from pathlib import Path

from haystack import Pipeline
from haystack.components.preprocessors import DocumentSplitter
from haystack.components.writers import DocumentWriter
from haystack.document_stores.in_memory import InMemoryDocumentStore

from haystack_integrations.components.converters.convertapi import ConvertAPIDocumentConverter

document_store = InMemoryDocumentStore()

pipeline = Pipeline()
pipeline.add_component("converter", ConvertAPIDocumentConverter())
pipeline.add_component("splitter", DocumentSplitter(split_by="passage", split_length=3))
pipeline.add_component("writer", DocumentWriter(document_store=document_store))
pipeline.connect("converter.documents", "splitter.documents")
pipeline.connect("splitter.documents", "writer.documents")

pipeline.run(
    {"converter": {"sources": [Path("report.docx"), Path("figures.xlsx"), Path("slides.pptx"), Path("paper.pdf")]}}
)
print(document_store.count_documents())

For plain text, use ConvertAPIDocumentConverter("txt"). It reads PDF, Word, PowerPoint, RTF, ODT and HTML files, and PNG, JPG and TIFF images with OCR. Scanned PDFs are OCRed automatically.

Convert Office files and web pages to PDF

ConvertAPIFileConverter sends local files and ByteStreams to ConvertAPI, and has ConvertAPI open http(s):// URLs in a browser. Each output keeps its source's metadata (file_path, or url for web pages).

from pathlib import Path

from haystack_integrations.components.converters.convertapi import ConvertAPIFileConverter

converter = ConvertAPIFileConverter(to_format="pdf")
result = converter.run(sources=[Path("contract.docx"), Path("budget.xlsx"), "https://haystack.deepset.ai"])
for pdf in result["output"]:
    print(pdf.meta, pdf.mime_type, len(pdf.data))

Set to_format to any format ConvertAPI converts to, for example docx, png or txt, or to a PDF tool such as ocr or pdfa for PDF input. Some conversions return several files, for example one PNG per page; their ByteStreams follow each other in page order.

The output plugs into any Haystack component that reads ByteStreams. This pipeline renders web pages to PDF and then turns the PDFs into Markdown:

from haystack import Pipeline

from haystack_integrations.components.converters.convertapi import (
    ConvertAPIDocumentConverter,
    ConvertAPIFileConverter,
)

pipeline = Pipeline()
pipeline.add_component("to_pdf", ConvertAPIFileConverter("pdf"))
pipeline.add_component("to_markdown", ConvertAPIDocumentConverter())
pipeline.connect("to_pdf.output", "to_markdown.sources")

result = pipeline.run({"to_pdf": {"sources": ["https://example.com"]}})
print(result["to_markdown"]["documents"][0].content)

Web pages as Markdown

To convert a page's HTML without rendering it, fetch it with LinkContentFetcher and pass the ByteStreams on:

from haystack import Pipeline
from haystack.components.fetchers import LinkContentFetcher

from haystack_integrations.components.converters.convertapi import ConvertAPIDocumentConverter

pipeline = Pipeline()
pipeline.add_component("fetcher", LinkContentFetcher())
pipeline.add_component("converter", ConvertAPIDocumentConverter())
pipeline.connect("fetcher.streams", "converter.sources")

result = pipeline.run({"fetcher": {"urls": ["https://haystack.deepset.ai/overview/intro"]}})
print(result["converter"]["documents"][0].content[:500])

Async

Both components have run_async, which sends up to concurrency_limit requests at a time (5 by default), and work in AsyncPipeline:

import asyncio
from pathlib import Path

from haystack_integrations.components.converters.convertapi import ConvertAPIDocumentConverter


async def main() -> None:
    converter = ConvertAPIDocumentConverter(concurrency_limit=10)
    result = await converter.run_async(sources=sorted(Path("docs").glob("*.pdf")))
    print(len(result["documents"]))


asyncio.run(main())

Configuration

Conversion parameters. parameters passes options to ConvertAPI with every request, named as on the converter's page in the list of conversions:

ConvertAPIDocumentConverter(parameters={"PageRange": "1-20", "Images": "describe"})  # AI image descriptions
ConvertAPIDocumentConverter("txt", parameters={"OcrLanguage": "de"})
ConvertAPIFileConverter("pdf", parameters={"Pdfa": True})  # PDF/A from Word files

For Markdown, ConvertAPIDocumentConverter removes images by default (Images=remove), because embedded base64 images bloat Documents, and removes HTML tags that Markdown cannot express while keeping their text (UnsupportedTags=Bypass); parameters overrides both. If a conversion returns several text files, for example one per page with SplitPages, the Document joins them with form feeds (\f), which DocumentSplitter treats as page breaks.

Input formats. A local file's format is its extension. A ByteStream's format is its MIME type, else the extension of file_path or file_name in its metadata, else the extension of its url. Set from_format to override it, for example from_format="file" for ConvertAPI's universal converter.

Regions. Set api_base_url to a regional endpoint, such as https://eu-v2.convertapi.com, to keep files in that region.

Failures. A file that cannot be converted is logged as a warning and skipped, so the rest of the batch still goes through. Set raise_on_failure=True to get a ConvertAPIError (or the OSError/ValueError for a file that cannot be read) instead. ConvertAPI's error message is included, for example Unsupported conversion odt -> md.

Retries. Requests are retried up to max_retries times (3 by default) when ConvertAPI answers 408, 429 or 503, honoring Retry-After, or cannot be reached. Other errors are not retried, so a file is never converted, and charged for, twice. timeout (960 seconds by default) leaves a margin above ConvertAPI's own 900-second conversion limit.

Data handling. The components always send StoreFile=false: ConvertAPI converts the file in memory and returns the result in the response, and nothing is written to disk.

Development

pip install hatch
hatch run test:unit        # unit tests, no network
hatch run test:types       # mypy
hatch run fmt              # ruff

The integration tests call the live API and use about 20 conversions per run:

CONVERTAPI_TOKEN="your-api-token" hatch run test:integration

scripts/with-token.sh runs them with the token read from a file, never printed. Its default file is ConvertAPI's internal test token; set CONVERTAPI_TOKEN_FILE to use your own.

License

convertapi-haystack is distributed under the Apache-2.0 license.