Tags: tuplex/tuplex
Tags
Release notes for Tuplex v0.3.6 =============================== [1] Features ------------ - Update to LLVM16. - Support building Tuplex for newer LLVM versions, up to LLVM v17. - Remove typed pointers in LLVM IR code generation, use explicit LLVM types instead. - Enable Apple Silicon support for Python3.9+. - Add script support for building arm64 wheels for Apple Silicon (`scripts/build_macos_wheels.sh`) - Add Github actions to build and test wheels for Intel Mac, Ubuntu 22.04. - Add fallback for platforms for generated CSV parser without SSE4.2 instructions. - Add support for Python 3.11. - Use Github actions to build wheels for Python 3.8, 3.9, 3.10, 3.11 under both manylinux and macos. [2] Bug Fixes ------------- - Explicitly shut down AWS SDK (when built with) to avoid race condition of AWS threads in destructors. - Fix ResolveTask issue when parsing single column CSV file with heterogeneous data. - Refactor decoding of exceptions in ResolveTask into function. - Fix Protobuf issue when using `-DBUILD_WITH_ORC` where ORC libraries may have been build with different Protobuf version than Tuplex causing issues when delocating the wheel under macOS (Intel/ARM). - Fix exception handling in individual tasks, which did not unlock partitions in case of error leading to crash. - Fix `toPythonString()` conversion in `Row`. - Fix serialization bug for lists, tuples together with options. [3] Improvements ---------------- - Explicitly mark export symbols in runtime shared object. - Change test script for exceptions to use `pytest.parametrize`. - Add dwarf/elf lib for backward under macOS to print stacktrace. - Change to batch processing for result-set conversion for any strategy and explicitly check signals to avoid fine grained locking of GIL. - Lower parameters/test data size for exception (merge) testing for faster CI. - Change to pick up compatible zstd version in cmake to avoid conflicts with pre-installed, older zstd library. - Add cache step for macOS in Github action to speed up brew installs. - Fix issue in `setup.py` when Python executable is not in `/usr/local`. - Remove Ubuntu 18.04 support, update Azure CI runner to use Ubuntu 22.04. - Deprecate Python 3.6. and 3.7 support. - Use newer ANTLR4 version 13.1 and require Java 11 as dependency. - Support setting Python version when creating Lambda runner in `scripts/create_lambda_zip.sh`. - Update dependencies in tuplex/ci containers, build Python version specific containers. - Refactor code generation logic for iterators. - Update to recent pybind11 version 2.11.1 from 2.10.4.
Release notes for v0.3.5: This is a bugfix release, addressing issues for the 2nd notebook example and testing the WebUI. [1] Features: [2] Bug Fixes: - Fix hashmap free issue preventing second example from working. - Add error message when users try to import Tuplex but it is not compiled. Allows to use helper functions without having to have Tuplex compiled, - Improve WebUI testing and error reporting [3] Credits: Leonhard Spiegelberg
Release notes for v0.3.4: This is a maintenance release which includes various refcount fixes for v0.3.3. [1] Features: [2] Bug Fixes: - Fix refcount in FunctionGlobals test - Redesigned internal GIL helper functions to work properly with python debug version - Add an explicit python garbage collection run before collect/take to free up/compact space. - Fix 2 bugs related to 1.) hashmap thread data-race 2.) introduce non-schema-conformant resolvers triggering fallback case. - Update docker file and add examples - Add new examples directory - Update Docker CI image, notably add both asan and tsan toolset - Add option to use environment variable to easily modify build type for wheel - Update script for macOS, identical aws version for mac now used like in Linux wheels - Fix bug regarding data-race because of wrongly moved hash entries (solves crash in notebook) - Add new GENERALCASEVIOLATION exception to deal with non-conforming resolvers. Necessary, because resolvers that use tracing to determine typing may be not conforming. [3] Credits: Leonhard Spiegelberg
Release notes for Tuplex v0.3.3: ================================ The Tuplex team is thrilled to announce our next release of the framework to the public! With this release, for the first time we offer a pip package for macOS (Intel). Please find details about the changes in our detailed changelog from v0.3.2 to v0.3.3 below: [1] Features: - Add support for boolean operations `and` and `or` as well as `in` and `not in` to tracer. (#113) - Add isnan, isinf, and isclose for builtin math module (#105) - Introduce correct merging behavior for exceptions in ParallelizeOperator, new structure to keep group of partitions and more correct merge-in-order behavior. (#101) - Add serialization/deserialization support for list of tuples, optional list, optional tuple (#92) Changes to make Tuplex more easily reproducible and obtain all figures of the SIGMOD'21 paper. Adds new master script to drive experimentation. (#71) - Add `webui` option when constructing Context as alias to `webui.enable` - Update scripts to be compatible with Python 3.10 [2] Bug Fixes: - Replace pybind steal calls with borrow - Fix several refcnt issues for none, true, false - Fix for key with optional tuplex. prefixing (#69) - Fix hashing for rows with more complex types - Fix running tests on a machine with multiple users by prefixing the scratch dir with the user name - Improve python interpreter detection in cmake - Remove unused function from test to find python stdlib - Fix for missing module, or import failure - Add another mod import fallback - Add support for typing.Optional[...] type hint and catch errors when extracting type annotations in CSV - Add another code extract fallback using new ast.unparse feature for python 3.9+ - Fix bug94 by making fallback partitions immortal as well - Fixes for key setting and aggregateByKey over arbitrary rows (#112) - Fix pure version based python finding - Fix nbconvert issue with 7.0a versions by restricting to <7.0 - Fix when multiple llvm versions are present, tries to find llvm9 and llvm6 then - Remove boto from setup.py, because its dependencies are broken - Add version fix for empty test - Add extraction of function source code in shell (#107) - Prevent deadlock from TraceVisitor.cc when unsupported feature is encountered - Fix temp dot file bug (#110) - Fix webui for Ubuntu caused by pymongo bug - Fix when undefined symbol is present in UDF and symbol name coincides with supported builtin names. (#95) - Fix macOS wheel issues due to outdated deployment target (#87) [3] Credits: Ben Givertz, Rhea Goyal, Yunzhi Shao, Leonhard Spiegelberg
Release notes for Tuplex v0.3.2: Features: - Add support for for and while loops including unrolling, type speculation and break/continue keywords - Add preliminary support for reading/writing Apache ORC files - Add support for builtin iterators iter, zip, enumerate, next, reversed - Add support for is keyword - Add list functionality to both Posix and AWS S3 file systems - Package WebUI with auto-start (requires working installation of MongoDB) - Add option to package experimental Lambda runner with pip package/wheel - Add auto-setup functionality for deploying Lambda runner to AWS - Add support for running Tuplex in Google Colab Notebooks - Add MacOS wheel - Switch to PyBind11 from Boost Python Bug Fixes: - Auto-unpacking of dictionaries with string keys works now in fallback mode as well - Test suite now works also when invoked using multiple processes/threads - Update various links to software in dockerfiles, install scripts - Fix reference count issues when invoking parallelize on lists - Fix issue with file output deleting the output directory, changed to validate directory first - Fix bug where non-conforming rows will crash Tuplex when the majority type is a simple tuple containing var length fields due to a missing type check in parallelize. - Fix building Tuplex on Ubuntu 20.04, add missing dependencies - Fix auto-complete bug when using Tuplex in shell mode - Fix calling len on empty lists - Fix bug in aggregateByKey where source code was not correctly extracted - Free local partitions when python context object is destroyed Committers: Ben Givertz, Yash Gotmare, Zain Ruan, Malte Schwarzkopf, Yunzhi Shao, Leonhard Spiegelberg, Rahul Yesantharao