FastFlight is a framework built on Apache Arrow Flight, designed to simplify high-performance data transfers while improving usability, integration, and developer experience.
It addresses common challenges with native Arrow Flight, such as opaque request formats, debugging difficulties, complex async management, and REST API incompatibility. FastFlight makes it easier to adopt Arrow Flight in existing systems.
✅ Typed Param Classes – All data requests are defined via structured, type-safe parameter classes. Easy to debug and
validate.
✅ Service Binding via param_type – Clean and explicit mapping from param class → data service. Enables dynamic
routing and REST support.
✅ Async & Streaming Ready – async for support with non-blocking batch readers. Ideal for high-throughput
systems.
✅ REST + Arrow Flight – Use FastAPI to expose Arrow Flight services as standard REST endpoints (e.g., /stream).
✅ Plug-and-Play Data Sources – Includes a DuckDB demo example to help you get started quickly—extending to other
sources (SQL, CSV, etc.) is straightforward.
✅ Built-in Registry & Validation – Automatic binding discovery and safety checks. Fail early if service is
missing.
✅ Pandas / PyArrow Friendly – Streamlined APIs for transforming results into pandas DataFrame or Arrow Table.
✅ CLI-First – Unified command line to launch, test, and inspect services.
FastFlight is ideal for high-throughput data systems, real-time querying, log analysis, and financial applications.
pip install "fastflight[all]"or use uv
uv add "fastflight[all]"# Start both FastFlight and REST API servers
fastflight start-all --flight-location grpc://0.0.0.0:8815 --rest-host 0.0.0.0 --rest-port 8000This launches both gRPC and REST servers, allowing you to use REST APIs while streaming data via Arrow Flight.
# Example REST API call to DuckDB demo service
curl -X POST "http://localhost:8000/fastflight/stream" \
-H "Content-Type: application/json" \
-d '{
"type": "fastflight.demo_services.duckdb_demo.DuckDBParams",
"database_path": ":memory:",
"query": "SELECT 1 as test_column",
"parameters": []
}'FastFlight provides a command-line interface (CLI) for easy management of Arrow Flight and REST API servers.
# Start only the FastFlight server
fastflight start-flight-server --flight-location grpc://0.0.0.0:8815
# Start only the REST API server
fastflight start-rest-server --rest-host 0.0.0.0 --rest-port 8000 --flight-location grpc://0.0.0.0:8815fastflight start-all --flight-location grpc://0.0.0.0:8815 --rest-host 0.0.0.0 --rest-port 8000Important: When using the /stream REST endpoint, ensure the type field is included in the request body for
proper service routing.
# Development setup (both servers in one container)
docker-compose --profile dev up
# Production setup (separated services)
docker-compose up
# Background mode
docker-compose up -d# Run both servers
docker run -p 8000:8000 -p 8815:8815 fastflight:latest start-all
# Run only FastFlight server
docker run -p 8815:8815 fastflight:latest start-flight-server
# Run only REST API server
docker run -p 8000:8000 fastflight:latest start-rest-serverSee Docker Guide for complete deployment options and configuration.
For comprehensive examples, see the examples/ directory which includes:
- Multi-Protocol Demo:
examples/multi_protocol_demo/- Complete demonstration of FastFlight with both gRPC and REST interfaces - Benchmark Tools:
examples/benchmark/- Performance measurement and analysis comparing sync vs async operations
from fastflight import FastFlightBouncer
from fastflight.demo_services.duckdb_demo import DuckDBParams
# Create client
client = FastFlightBouncer("grpc://localhost:8815")
# Define query parameters
params = DuckDBParams(
database_path=":memory:",
query="SELECT 1 as test_column, 'hello' as message",
parameters=[]
)
# Fetch data as Arrow Table
table = client.get_pa_table(params)
print(f"Received {len(table)} rows")
# Convert to Pandas DataFrame
df = table.to_pandas()
print(df)import asyncio
from fastflight import FastFlightBouncer
async def stream_data():
client = FastFlightBouncer("grpc://localhost:8815")
async for batch in client.aget_record_batches(params):
print(f"Received batch with {batch.num_rows} rows")
# Process batch incrementally
asyncio.run(stream_data())- Data Service Developer Guide – Guide for implementing custom data services
- CLI Guide – Detailed CLI usage instructions
- Docker Deployment – Container deployment and Docker Compose guide
- Error Handling – Comprehensive error handling and resilience patterns
- Technical Details – In-depth implementation details and architecture
- FastAPI Integration – REST API integration guide
FastFlight supports extending to custom data sources. See Data Service Developer Guide for implementation details.
✅ Structured Ticket System (Completed)
✅ Async & Streaming Support (Completed)
✅ REST API Adapter (Completed)
✅ CLI Support (Completed)
✅ Enhanced Error Handling & Resilience (Completed)
🔄 Support for More Data Sources (SQL, NoSQL, Kafka) (In Progress)
🔄 Performance Benchmarking Tools (In Progress)
🔄 Production Monitoring & Observability (Planned)
Contributions are welcome! If you have suggestions or improvements, feel free to submit an Issue or PR. 🚀
This project is licensed under the MIT License.
🚀 Ready to accelerate your data transfers? Get started today!