A simple framework for writing testable ETL systems. Takes concepts from Kiba but reorganises them to use simple objects, composition and dependency injection. This means your ETL stays simple, reliable and testable.
DSLs are cool, but composable, testable objects are cooler.
Add this line to your application's Gemfile:
gem 'setl'And then execute:
$ bundle
Or install it yourself as:
$ gem install setl
- Define an object that is responsible for sourcing the data. Must respond to
#eachandyieldto the provided block. - Define your transformations. These are simple objects that will receive one of the things provided by the source object.
- Provide your transforms to a controller object. Setl provides
Setl::Controllerand you can simply add your transforms into this object. Can actually be any object that responds to#call - Define a destination object. Must respond to
#calland sends output of each transformed row of data to a destination.
source = [{id: 1, name: 'foo'},{id: 2, name: 'bar'}]
destination = lambda {|d| puts d.inspect }
transform = lambda do |d|
d[:name].upcase!
d
end
Setl::ETL.new(source, destination).process(transform)
#=> {id: 1, name: 'FOO'}
#=> {id: 2, name: 'BAR'}See the examples folder for some more extensive, and realistic, implementations.
- Fork it ( https://github.com/[my-github-username]/setl/fork )
- Create your feature branch (
git checkout -b my-new-feature) - Commit your changes (
git commit -am 'Add some feature') - Push to the branch (
git push origin my-new-feature) - Create a new Pull Request