Conduit is a domain-specific language (DSL) for creating node-based workflows in Rust. It enables you to build complex data processing pipelines with a simple, declarative syntax.
- Quick Start
- Basic Syntax
- Node Components
- Arrow Notation
- Module Implementation
- Anonymous Nodes
- Node Sharing and Chaining
- Examples
- Contributing
- License
Add Conduit to your Cargo.toml:
[dependencies]
conduit = { git = "https://github.com/milewski/conduit-challange.git", version = "0.1.0" }
conduit-derive = { git = "https://github.com/milewski/conduit-challange.git", version = "0.1.0" }use conduit::Engine;
use conduit_derive::Node;
#[derive(Node)]
pub struct Logger {
pub left: Input<String>,
pub right: Input<String>,
}
impl ExecutableNode for Logger {
fn run(&self) {
println!("{} {}", self.left.read(), self.right.read());
}
}
fn main() {
let workflow = r#"
logger {
left <- "hello"
right <- "world"
}
"#;
let mut engine = Engine::new();
engine.run_pipeline(workflow);
// you should see hello world printed to the console
}A node in Conduit is defined using the following syntax:
node_name module_name {
attribute_1 <- 123
attribute_2 <- "abc"
}
Each node consists of three main components:
-
Node Name (optional): An identifier used to reference the node from other nodes. If omitted, the node is considered "anonymous" and cannot be referenced elsewhere.
-
Module Name: Specifies the Rust module that implements the node's functionality. Modules are Rust files that define how inputs are processed and outputs are generated.
-
Attributes: Define the inputs and outputs of the node using arrow notation.
The direction of arrows indicates data flow:
<-(left arrow): Assigns a value to an input attribute->(right arrow): Directs output to another node
Values can be:
- Numbers:
123 - Strings:
"abc" - Booleans:
true,false - References to other nodes:
node_nameornode_name::output_port - Inline node definitions (nested nodes)
Modules are implemented as Rust structs. Here's an example of a file reading module:
#[derive(Node)]
pub struct ReadFile {
pub input: Input<String>,
pub output: Output<Vec<u8>>,
}
impl ExecutableNode for ReadFile {
fn run(&self) {
if let Ok(data) = fs::read(self.input.read()) {
self.output.write(data)
}
}
}Nodes without names are called anonymous nodes and can be defined in two ways:
module_a {
attribute_1 <- 123
}
module_b {
attribute_1 <- 123
}
module_a {
attribute_1 <- module_b {
attribute_1 <- 123
}
}
When defining inline nodes, by default the output field is used. For other output fields:
module_a {
attribute_1 <- module_b::another_port {
attribute_1 <- 123
}
}
You can share a node's output among multiple nodes:
file file_reader { input <- "./my-file.txt" }
module_a {
attribute_1 <- file
}
module_b {
attribute_1 <- file::output
}
Note: The ::output suffix is optional when the output field is named "output".
This approach enables efficient resource sharing and potential parallel execution based on the dependency graph.
You can also chain nodes using forward notation:
file file_reader {
input <- "./my-file.txt"
output -> module_b {
output -> write_file {
destination <- "output.text"
}
}
}
In this syntax:
output -> module_bis equivalent tooutput -> module_b::input- Arrows point in the direction of data flow
The true power of Conduit emerges when you create reusable nodes and chain them together to build complex workflows that can be executed from both Rust and your favorite language via FFI.
Check out the examples directory for complete workflow examples.
Contributions are welcome! Please feel free to submit a Pull Request.
This project is licensed under the MIT License - see the LICENSE file for details.