Closed
Description
Add several higher level abstractions to the Work Queue Python interface, something like the follows:
wq.map( f, a ) -> [ f(a[0]), f(a[1]), ... ]
wq.allpairs( f, a, b ) -> [ f( a[0], b[0] ), f( a[0], b[1] ), . . . ]
wq.treereduce( f, a ) -> f( f( a[0], a[1] ), f( a[2], a[3] ) )
These should work by generating functions in the PythonTask interface, and then submitting and waiting as needed.
For the first implementation, each element of the array should be a single task, which should be pretty straightforward.
Make that a PR with a nice writeup in the manual.
The second iteration will be more tricky. Consider the case where each element of the array is very fast to compute. It won't make sense to have the overhead of a separate function for each element. Instead, multiple elements will have to be grouped within a task in order to get an acceptable runtime. Let's talk about that more once version #1 is in place.