Unity Catalog is the industry’s only universal catalog for data and AI.
- Multimodal interface supports any format, engine, and asset
- Multi-format support: It is extensible and supports Delta Lake, Apache Iceberg and Apache Hudi via UniForm, Apache Parquet, JSON, CSV, and many others.
- Multi-engine support: With its open APIs, data cataloged in Unity can be read by many leading compute engines.
- Multimodal: It supports all your data and AI assets, including tables, files, functions, AI models.
- Open source API and implementation - OpenAPI spec and OSS implementation (Apache 2.0 license). It is also compatible with Apache Hive's metastore API and Apache Iceberg's REST catalog API. Unity Catalog is currently a sandbox project with LF AI and Data Foundation (part of the Linux Foundation).
- Unified governance for data and AI - Govern and secure tabular data, unstructured assets, and AI assets with a single interface.
The first release of Unity Catalog focuses on a core set of APIs for tables, unstructured data, and AI assets - with more to come soon on governance, access, and client interoperability. This is just the beginning!
This is a community effort. Unity Catalog is supported by
- Amazon Web Services
- Confluent
- Daft (Eventual)
- dbt Labs
- DuckDB
- Fivetran
- Google Cloud
- Granica
- Immuta
- Informatica
- LanceDB
- LangChain
- LlamaIndex
- Microsoft Azure
- NVIDIA
- Onehouse
- PuppyGraph
- Salesforce< BB1B /a>
- StarRocks (CelerData)
- Tecton
- Unstructured
Unity Catalog is proud to be hosted by the LF AI & Data Foundation.
Let's take Unity Catalog for spin. In this guide, we are going to do the following:
- In one terminal, run the UC server.
- In another terminal, we will explore the contents of the UC server using a CLI. An example project is provided to demonstrate how to use the UC SDK for various assets as well as provide a convenient way to explore the content of any UC server implementation.
You have to ensure that your local environment has the following:
- Clone this repository.
- Ensure the
JAVA_HOMEenvironment variable your terminal is configured to point to JDK11+. - Compile the project using
build/sbt package
In a terminal, in the cloned repository root directory, start the UC server.
bin/start-uc-serverFor the rest of the steps, continue in a different terminal.
Let's list the tables.
bin/uc table list --catalog unity --schema defaultYou should see a few tables. Some details are truncated because of the nested nature of the data.
To see all the content, you can add --output jsonPretty to any command.
Next, let's get the metadata of one of those tables.
bin/uc table get --full_name unity.default.numbersYou can see that it is a Delta table. Now, specifically for Delta tables, this CLI can print snippet of the contents of a Delta table (powered by the Delta Kernel Java project). Let's try that.
bin/uc table read --full_name unity.default.numbersFor trying with DuckDB, you will have to install it (at least version 1.0).
Let's start DuckDB and install a couple of extensions. To start DuckDB, run the command duckdb in the terminal.
Then, in the DuckDB shell, run the following commands:
install uc_catalog from core_nightly;
load uc_catalog;
install delta;
load delta;If you have installed these extensions before, you may have to run update extensions and restart DuckDB
for the following steps to work.
Now that we have DuckDB all set up, let's try connecting to UC by specifying a secret.
CREATE SECRET (
TYPE UC,
TOKEN 'not-used',
ENDPOINT 'http://127.0.0.1:8080',
AWS_REGION 'us-east-2'
);You should see it print a short table saying Success = true. Then we attach the unity catalog to DuckDB.
ATTACH 'unity' AS unity (TYPE UC_CATALOG);Now we are ready to query. Try the following
SHOW ALL TABLES;
SELECT * from unity.default.numbers;You should see the tables listed and the contents of the numbers table printed.
To quit DuckDB, run the command Ctrl+D or type .exit in the DuckDB shell.
You can read Delta Uniform tables from Spark via Iceberg REST Catalog APIs, operate on volumes and functions from the CLI, and much more. See the full tutorial for more details.
- Open API specification: The Unity Catalog Rest API is documented here.
- Compatibility and stability: The APIs are currently evolving and should not be assumed to be stable.
- To create a tarball that can be used to deploy the UC server or run the CLI, run the following:
This will create a tarball in the
build/sbt createTarball
targetdirectory. See the full deployment guide for more details.
- Install JDK 11 by whatever mechanism is appropriate for your system, and set that version to be the default Java version (e.g., by setting env variable JAVA_HOME)
- To compile all the code without running tests, run the following:
build/sbt clean compile
- To compile and execute tests, run the following:
build/sbt clean test - To execute tests with coverage, run the following:
build/sbt jacoco
- To update the API specification, just update the
api/all.yamland then run the following:This will regenerate the OpenAPI data models in the UC server and data models + APIs in the client SDK.build/sbt generate
Java code adheres to the Google style, which is verified via build/sbt javafmtCheckAll during builds.
In order to automatically fix Java code style issues, please use build/sbt javafmtAll.
Follow the instructions for Eclipse or IntelliJ to install the google-java-format plugin (note the required manual actions for IntelliJ).
The build script checks for a lower bound on the JDK but the current SBT version imposes an upper bound. Please check the JDK compatibility documentation for more information
Create a virtual environment:
# Create virtual environment
python -m venv uc_docs_venv
# Activate virtual environment (Linux/macOS)
source uc_docs_venv/bin/activate
# Activate virtual environment (Windows)
uc_docs_venv\Scripts\activateInstall the required dependencies:
pip install -r requirements-docs.txtThen serve the docs with
mkdocs serve