Warning
This repo is under construction.
pyonb is two things:
- a Python SDK for document extraction via the Hyland OnBase REST API (work in progress)
- a suite of APIs wrapped around open-source Optical Character Recognition (OCR) tools, designed for local deployment, for converting PDFs to structured text including:
pyonb requires Docker and Docker Compose.
-
Rename
.env.sampleto.env. -
Edit
.envwith the correctHOST_DATA_FOLDERlocation, e.g.:
HOST_DATA_FOLDER="/absolute/path/to/documents/folder"
# e.g. for unit tests on GAE:
# HOST_DATA_FOLDER="/gae/pyonb/tests/data/single_synthetic_doc"- Set OCR service ports, e.g.:
OCR_FORWARDING_API_PORT=8110
MARKER_API_PORT=8112
SPARROW_API_PORT=8001
DOCLING_API_PORT=8115Important
For GAE usage, set OCR service ports and UCLH proxy details:
http_proxy=
https_proxy=
HTTPS_PROXY=
HTTP_PROXY=- Start the OCR API Server (e.g. using marker and docling):
docker compose --profile marker --profile docling up -d- Open FastAPI Swagger at http://127.0.0.1:8110/docs to view and execute endpoints.
Use the following POST endpoints to execute the chosen OCR tool on a PDFs:
- marker - POST
/marker/inference_single - docling - POST
/docling/inference_single
- View the JSON response:
- Alternatively to Swagger, use Postman to construct, save and make your API requests.
- Clone the repo:
git clone https://github.com/SAFEHR-data/pyonb.git- Create a virtual environment (we suggest using uv) and install dependencies:
uv venv --python3.12
source .venv/bin/activate
uv sync- Copy the
tests/.env file to root directory to use with tox:
cp /tests/.env.tests .env- Start the Docker services:
docker compose --profile marker --profile docling up -d- Run tests using tox:
tox -e py312NB: this may take a few minutes to perform the inference tests. Some may fail depending on which OCR tools you choose to raise. For example, with --profile marker --profile docling the Sparrow API will not be raised,
so the associated tests will fail.
To run unit tests individually, adapt the following:
tox -e py312 -- tests/api/test_routers.py::test_inference_single_file_upload_marker- Arman Eshaghi
- Tom Roberts (tom.roberts@ucl.ac.uk)
- Kawsar Noor
- Lawrence Lai
- Stefan Piatek
- Richard Dobson
- Steve Harris
- Sarah Keating
This work was funded by the National Institute for Health and Care Research (NIHR, award code NIHR302495).
This project is developed in collaboration with the Centre for Advanced Research Computing, University College London.