rxtls is a high-throughput, fault-tolerant Certificate Transparency log processor designed for hyperscale environments. It provides efficient processing of CT logs with dynamic backpressure handling, adaptive rate limiting, and comprehensive observability.
- High Throughput: Process CT logs efficiently with a worker pool architecture
- Fault Tolerance: Automatic retries, backpressure handling, and graceful shutdown
- Dynamic Rate Limiting: Adaptive rate control based on success/failure patterns
- Observability: Prometheus metrics for monitoring and alerting
- Configurable: CLI flags for customizing behavior and CT log sources
- Versatile: Download raw certificates or extract domains from CT logs
The system consists of several key components:
- Manages a pool of workers
- Distributes work using least-loaded worker selection
- Implements graceful shutdown
- Provides statistics and metrics
- Process work items from their queues
- Implement backpressure handling
- Track success/failure metrics
- Support CPU affinity for optimal performance
- Dynamic rate adjustment based on success/failure
- Token bucket implementation for smooth rate limiting
- Backpressure integration
- Atomic operations for thread safety
- Prometheus integration for monitoring
- Queue pressure tracking
- Success/failure rate monitoring
- Resource utilization metrics
The tool provides several subcommands:
# List available CT logs
rxtls list
# Download certificates from CT logs
rxtls download
# Extract domains from certificates in CT logs
rxtls domains
# Fetch and save the CT logs list to a local file
rxtls fetch-logs
# Direct processing with URI (legacy mode)
rxtls --ct-uri https://ct.example.com/log# Use local logs list instead of fetching from internet
rxtls --local-logs [command]
# Customize worker pool size
rxtls --workers 8
# Set initial rate limit
rxtls --rate-limit 1000
# Enable debug logging
rxtls --debug
# Configure Prometheus metrics port
rxtls --metrics-port 9090# Basic download with interactive log selection
rxtls download
# Specify output directory
rxtls download --output /path/to/output
# Configure concurrency
rxtls download --concurrency 10
# Adjust buffer size
rxtls download --buffer 262144
# Enable compression
rxtls download --compress
# Enable high-speed mode
rxtls download --turbo# Basic domain extraction with interactive log selection
rxtls domains
# Specify output directory
rxtls domains --output /path/to/domains
# Configure concurrency
rxtls domains --concurrency 10
# Adjust buffer size
rxtls domains --buffer 32768
# Enable compression
rxtls domains --compress
# Enable high-speed mode
rxtls domains --turbo--ct-uri: CT log URI to process (default: from config)--workers: Number of worker goroutines (default: runtime.NumCPU())--rate-limit: Initial rate limit in requests/second (default: 100)--debug: Enable debug logging--metrics-port: Prometheus metrics port (default: 9090)--local-logs: Use local logs list instead of fetching from internet
RXTLS_CONFIG: Path to config fileRXTLS_LOG_LEVEL: Log level (debug, info, warn, error)RXTLS_METRICS_PORT: Prometheus metrics port
The following Prometheus metrics are exposed:
rxtls_worker_queue_size: Current size of worker queuesrxtls_worker_queue_pressure: Queue pressure (0-1)rxtls_worker_processed_total: Total processed itemsrxtls_worker_errors_total: Total errorsrxtls_rate_limit_current: Current rate limitrxtls_rate_limit_success_total: Total successful requestsrxtls_rate_limit_failure_total: Total failed requests
- Go 1.21 or later
- Make (optional, for build scripts)
# Build binary
go build
# Run tests
go test ./...
# Run benchmarks
go test -bench=. ./...The codebase includes comprehensive tests:
- Unit tests for all components
- Integration tests for the full pipeline
- Benchmarks for performance testing
- Race condition detection enabled
GNU Affero General Public License v3 - see LICENSE file for details