Define the Kiba Pro offering #20
Comments
|
Would this multi threaded implementation take use of something like Celluloid? |
|
@dhonig I have started several prototypes and I'm not sure yet what the first published implementation will rely on (it could be with Celluloid, or just concurrent-ruby, or even more simple depending on my findings). I need to investigate further! May I ask: would you have a preference to see a Celluloid-based solution, or the contrary? Any insight is most welcome. |
|
I'm familiar with the Celluloid idea of actors from Erlang and Akka...I think it could be a big win to use actors here. But I'm all for the simplest and best solution without introducing another heavy weight dependency. Not too familiar with concurrent-ruby but after a quick glance, I still think celluloid actors might be a simpler way forward. |
|
I do like actors too, especially since here it seems to make a lot of sense given how the processing occurs. I will keep experimenting with Celluloid indeed, and see if I get enough control. Thanks for those extra details, appreciated! |
|
Any chance of automated rollback or versioning? You can roll your own, but it would be great to have a common way of handling it as part of the ETL process for frequently updated data. |
|
@mroby-dresources can you email me (thibaut.barrere+kiba@gmail.com) so we can discuss this in more depth? Thanks! |
|
https://github.com/grosser/parallel This would help easily with multi processor and multi threading. |
|
Bumping this. Love the current features but looking at rolling some multi-threading support myself since it looks like progress has stalled on that work? |
|
@standingwave work hasn't stalled actually! I now have an "alpha" quality multi-threaded kiba-pro runner which I'm currently packing up for a first release. Hit me up via email (thibaut.barrere+kiba@gmail.com) and I'll add you to the "alpha" as soon as it's ready (a bit of extra testing is required, and packaging as well, but I'm getting there). |
|
After several projects with clients, the first component will be a flexible & fast SQL insert destination, as I saw this provided more leverage than multithreading usually. Closing this for now, and I'll update the README when the Pro bundle is available. |
|
@thbar The multithreaded version = Elixir right ? |
|
@dhonig Ruby as well! While I do have plans to ship Elixir versions of Kiba Pro components, I also plan to release multithreading-enabled features in the Ruby version. |
|
@thbar exciting! We have used Kiba on several projects now and an Elixir version seems like a great fit! |
|
@dhonig glad you find Kiba useful :-) And yes, I'm coding more with Elixir on ETL related tasks, and will definitely publish something at some point around that! |
As first mentioned on my blog I plan to launch a Kiba Pro offering. Please comment here or email me to discuss more details, manifest your interest and support, this will be greatly appreciated.
Sustainable OSS
Especially after maintaining activewarehouse-etl for quite a bit of time, I'm willing to find a sustainable way to keep a high quality ETL solution for Ruby for the years to come, without killing my solopreneur-business.
If I'd paraphrase Mike Perham's statement with Sidekiq-Pro (see Sidekiq-Pro FAQ), "Kiba Pro is (well, will be) an extension which adds a few nice features to Kiba. Kiba is open source and free for all to use but unfortunately it takes a lot of my time to maintain and support. Kiba Pro is a way for you to purchase really useful functionality while also ensuring Kiba will be supported for years to come".
A recurring yearly subscription similar to Sidekiq Pro would secure some of my time to maintain and improve the solution and avoid the fate of activewarehouse-etl.
Potential features
Features could include:
Interested? Please chime in!
I'm pushing this publicly for transparency and also to find the "good" feature set, the one that larger companies would be willing to support my work on Kiba while getting extra value.
Do not hesitate to email me if your company could be interested in supporting richer features.
The text was updated successfully, but these errors were encountered: