Releases: fybrik/fybrik
Releases · fybrik/fybrik
v1.3.3
v1.3.2
What's Changed
- Clean up the write FybrikApplication before the creating the read FybrikApplication by @revit13 in #1994
- Remove option to overwrite base images. by @revit13 in #2018
- Adding go-toolset Dockerfile by @lappanna in #2008
- Storage manager documentation by @shlomitk1 in #2024
- Avoid redundant createAsset calls if asset already exists in openmeta… by @revit13 in #2028
- Change OPA server to use TLS instead of mutual TLS in notebook test by @revit13 in #2027
- Remove redundant AddToScheme call by @roytman in #2029
- Support all connection types from a specific catalog provider by @shlomitk1 in #2031
- Bump actions/setup-go from 3 to 4 by @dependabot in #2032
- small fixes by @roytman in #2033
- Updates the instructions how to run manager locally by @roytman in #2036
- updated the available modules list on the website by @aradhalevy in #2025
- fixed the arch pptx file by @aradhalevy in #2034
- Removing go-toolset , revert #2008 by @lappanna in #2038
- Design document for the Network Policies based isolation by @roytman in #1944
- Minor fix to chaining-sample.md by @revit13 in #2040
New Contributors
Full Changelog: v1.3.1...v1.3.2
Contributors
roytman, revit13, and 4 other contributors
Assets 2
v1.3.1
The fix release
- Keeps blueprint annotations upon update.
What's Changed
- Add a mapping between Fybrik and airbyte-module versions by @revit13 in #2015
- Add instructions for deployment of localstack on openshift in samples. by @revit13 in #2016
- Keep blueprint annotations upon update by @shlomitk1 in #2019
Full Changelog: v1.3.0...v1.3.1
Contributors
revit13 and shlomitk1
Assets 2
v1.3.0
Highlights
-
The taxonomy compilation commands were moved to a separate repository. The fybrik builds use a docker container to compile taxonomy. That allowed us to resolve several third-party dependencies conflicts and update:
- goLang to 1.19
- Open Policy Agent (OPA) to v0.48.0
- Helm to v3.11.1
- K8s apimachinery, apiserver, cli-runtime, and client-go to v0.26.0
- K8s controller-runtime to v0.13.1
These updates resolved a number of vulnerabilities.
-
Helm uninstallof previous fybrik releases left “orphan” leader election ConfigMap and Lease. This behavior is fixed by the current release. -
This release allows deployment of Fybrik Internals CRs (plotter and blueprints) and user-faced CRs (fybrikmodules, fybrikstorageaccount) into separate namespaces.
-
Implement
StorageManager- support S3 by minio API
- support mySQL
-
When no policy could be applied to a request, previous Fybrik versions returned
Deny. Now it was changed toAllowand can be configured. -
Improved error messages propagating from connectors to fybrik application, and added separate handling to the 403/404 response codes.
-
Documentation fixes.
What's Changed
- run test when PR is pushed to master by @cdoron in #1827
- Use python modules to set variables in mkdocs. by @revit13 in #1820
- Disable tests for draft mode. by @revit13 in #1826
- Avoid deploying Open-metadata catalog in integration tests by @revit13 in #1835
- update-openmetadata-connector-version-to-0.4.0 by @cdoron in #1841
- fix typos by @roytman in #1842
- Remove replace directives in go.mod by @ityuhui in #1843
- Storage management design by @shlomitk1 in #1837
- Bump actions/setup-python from 4.3.0 to 4.4.0 by @dependabot in #1854
- Add python modules to the backward compatibility guide by @roytman in #1836
⚠️ Set go version to 1.18 by @ityuhui in #1667- Use retryable http client in fybrikUtils.GetHTTPClient. by @revit13 in #1839
- Mac support for or_tools by @zivnevo in #1840
- Connection taxonomy by @shlomitk1 in #1853
- update arrow-flight-module version to 0.10.0 by @cdoron in #1871
- Add a step to set swap space to github build.yml workflow. by @revit13 in #1830
- Bump golang.org/x/text and golang.org/x/net to resovle CVE-2022-27664 and CVE-2022-32149 by @ityuhui in #1873
- add dummy change in the autogenerated files, to prevent the warnings by @roytman in #1874
- remove the dummy change in the autogenerated files by @roytman in #1875
- Bump helm to v3.10.3 and K8s components to 0.25 by @roytman in #1856
- Bump github.com/containerd/containerd from 1.6.6 to 1.6.12 by @dependabot in #1876
- Make TLS tests work with openmetadata connector by @revit13 in #1833
- address a problem encountered on an IBM cloud VM by @cdoron in #1862
- bump base image to ubi-minimal:8.7 by @ityuhui in #1877
- add Super quick start (a script to boot up Fybrik in a single command) by @aradhalevy in #1869
⚠️ Change FybrikStorageAccount CRD by @shlomitk1 in #1855- Combine jobs in build.yml workflow. by @revit13 in #1846
- update link to OneClickDemo by @aradhalevy in #1879
- Bump actions/setup-python from 4.4.0 to 4.5.0 by @dependabot in #1881
- A skeleton for handling multiple implementations of storage allocation/deletion based on the storage type by @shlomitk1 in #1878
- Upgrade OPA server, Vault server and controller-gen versions. by @revit13 in #1870
⚠️ bump goLang version to 1.19 by @roytman in #1880- bump crdoc to 0.6.2 by @roytman in #1905
⚠️ Promote supported K8s versions to 1.23 - 1.25 by @roytman in #1901- Control testing verbosity option by @shlomitk1 in #1904
- Support StorageManager for S3 using minio SDK by @shlomitk1 in #1903
- Upgrade OPA to 0.48.0 by @roytman in #1910
- Remove hard-coded storage type selection by @shlomitk1 in #1907
- Fix format selection by @shlomitk1 in #1913
- Fixes install_or_tools.sh by @roytman in #1911
- add requirements for OneClickDemo by @aradhalevy in #1921
- Fix status update by @shlomitk1 in #1924
- Prevent writing csp logs in /tmp by @shlomitk1 in #1916
- Constrain sink interface to match storage protocol by @zivnevo in #1927
- Use install script inside Dockerfile by @zivnevo in #1932
- FybrikStorageAccount API Version - Change to app.fybrik.io/v1beta2 by @cdoron in #1936
- fix the workflow targets by @roytman in #1920
- bump json-schema-generator to v0.3.0 by @roytman in #1937
- Fix version generation in site/main.py. by @revit13 in #1930
- Sinlge taxomony json generation by @tomersolomon1 in #1946
- New optimizer docker image by @shlomitk1 in #1942
- add katalog documentation to quickstart + sample (on master) by @cdoron in #1951
- Add new connection types to the taxonomy layer in connection.yaml. by @revit13 in #1915
- added documentation about installing mkdocs tools by @aradhalevy in #1945
- Optimizer: Move code for generating high-level constraints to a separate file by @zivnevo in #1960
- Reconcile tools versions by @aradhalevy in #1943
- fix dev version fybrik installation instructions by @roytman in #1953
- add code for transformation library by @cdoron in #1959
- Redirect taxonomy references to new repository by @Vicky-Goldin in #1957
- Adds OpenShift specific values to the vault deployment by @roytman in #1961
- StorageManager API, server and client implementation by @shlomitk1 in #1965
- Bump helm.sh/helm/v3 from 3.10.3 to 3.11.1 by @roytman in #1968
- Fixing illegal asset name in the Katalog by @tomersolomon1 in #1955
- Propagate messages from the connectors by @shlomitk1 in #1966
- Bump go-restful and golang-jwt to resolve PRISMA-2022-0227 and PRISMA-2022-0270 by @ityuhui in #1974
- Bump github.com/containerd/containerd from 1.6.15 to 1.6.18 by @dependabot in #1977
- Bump golang.org/x/net from 0.5.0 to 0.7.0 by @dependabot in #1978
- Update Lease based leader election by @roytman in #1979
- Remove external.json after running 'make generate' by @Vicky-Goldin in #1982
- Validate create asset response. by @revit13 in #1981
- remove redundant RBAC definitions by @roytman in #1985
- rename leader election ID to prevent conflicts by @roytman in #1986
- added documentation to module dev page by @aradhalevy in #1908
- Propagate error messages from connectors to fybrik application by @shlomitk1 in #1989
- address problem which eliminates rows with missing values by @cdoron in #1991
- Configurable namespaces for CRs by @shlomitk1 in #1990
- fix python publish workflow by @cdoron in #1992
- Fix setting of ADMIN_CRS_NAMESPACE in notebook/write-flow/setup.sh by @revit13 in #1995
- Fix notebook read flow and set TEST_VERBOSITY in build.yml by @revit13 in #1996
- Allocation of MySQL storage by @shlomitk1 in #1973
- helm charts for egeria-connector by @cdoron in #1997
⚠️ Allow policy by default by @tomersolomon1 in #1956- Modify the convention of release name generation by @shlomitk1 in https://github.com/fybrik/fybrik/...
Contributors
ityuhui, roytman, and 8 other contributors
Assets 2
v1.2.1
Highlights
- fixes security alerts by updating third party dependencies
- fixes documentations
- fixes Optimizer deployment on MAC
What's Changed
- cherry-pick from #1841 into releases/1.2.1 by @fybrik-bot in #1882
- cherry-pick from #1842 into releases/1.2.1 by @fybrik-bot in #1883
- cherry-pick from #1854 into releases/1.2.1 by @fybrik-bot in #1884
- Remove replace directives in go.mod (#1843) by @revit13 in #1885
- Add python modules (#1836) by @revit13 in #1886
- Set go version to 1.18 (#1667) by @revit13 in #1887
- Mac support for or_tools (#1840) by @revit13 in #1888
- update arrow-flight-module version to 0.10.0 (#1871) by @revit13 in #1889
- Bump golang.org/x/text and golang.org/x/net (#1873) by @revit13 in #1890
- Bump helm to v3.10.3 and K8s components to 0.25 (#1856) by @revit13 in #1891
- Bump github.com/containerd/containerd from 1.6.6 to 1.6.12 (#1876) by @revit13 in #1892
- address a problem encountered on an IBM cloud VM (#1862) by @revit13 in #1893
- bump base image to ubi-minimal:8.7 (#1877) by @revit13 in #1894
- add Super quick start (a script to boot up Fybrik in a single command… by @revit13 in #1895
- update link to OneClickDemo after merge (#1879) by @revit13 in #1896
- Bump actions/setup-python from 4.4.0 to 4.5.0 (#1881) by @revit13 in #1897
- Upgrade OPA server, Vault server and controller-gen versions. (#1870) by @revit13 in #1898
- cherry-pick from #1880 into releases/1.2.1 by @fybrik-bot in #1902
Full Changelog: v1.2.0...v1.2.1
Contributors
revit13
Assets 2
v1.2.0
What's Changed
- Configurable discoveryBurst and discoveryQPS
- Removing wait from helm uninstall command
- Adding TLS support to the katalog and OPA connectors and Vault
- Support OpenMetadata catalog as a default opensource catalog option
- The Web site documentation improvements
- Resolve the security alerts
Full Changelog: v0.1.0...v1.2.0
v1.1.0
Highlights
- Added liveness and readiness probes to the manager
- Added an option to provide the Secret information to the data connector, when vault is turned off
- Replaced the
waitflag in the helm operations by a configurable polling interval - TLS configuration was improved
- Number of code and documentation fixes.
What's Changed
- Added version check when installing tools by @Raviv-S in #1624
- Fix attribute instance name by @shlomitk1 in #1681
- Fix a misleading statement in the manager-deployment.yaml` file by @revit13 in #1676
- Bump actions/setup-python from 4.1.0 to 4.2.0 by @dependabot in #1663
- TLS support requirements by @roytman in #1668
- Refactor TLS code. by @revit13 in #1637
- give access to vault secrets to pods in the fybrik-system namespace by @cdoron in #1687
- Replace the wait flag in the helm operations by a configurable polling interval by @revit13 in #1666
- Remove any reference to protos and GRPC by @revit13 in #1686
- Minor fix to hack/tools/common.sh by @revit13 in #1690
- Remove the property creationTimestamp from the yaml file of the CRDs by @Mohammad-nassar10 in #1688
- Fix "connection closed before server preface receive" error in notebo… by @revit13 in #1693
- Issue #1589: documentation for optimizer by @zivnevo in #1633
- Change tls container port in the connectors deployment. by @revit13 in #1698
- Provide the Secret information to data connector, when vault is turne… by @revit13 in #1684
- Update site/docs/concepts/connectors.md regarding control plane secur… by @revit13 in #1699
- Bump bluwy/substitute-string-action from 1 to 2 by @dependabot in #1701
- Optimizer documentation by @shlomitk1 in #1616
- Preserve logging format in manager by @shlomitk1 in #1697
- Fix dirSize limits by @roytman in #1706
- Add liveness and readiness probes to the manager by @shlomitk1 in #1703
- Adding hyperlinks for missing request objects to REST APIs by @rohithdv in #1694
- git command does not return the latest stable version by @rohithdv in #1707
- Explanations on how to run openmetadata on kind by @cdoron in #1700
- increase livenessProbe.initialDelaySeconds for localstack installation by @roytman in #1709
Full Changelog: v1.0.1...v1.1.0
Contributors
roytman, cdoron, and 7 other contributors
Assets 2
v1.0.1
Highlights
Version 1.0.1 of Fybrik is a major milestone. New functionality was added, performance improvements were made, documentation was improved, and bugs were squashed.
Fybrik CRDs apiVersion was promoted to v1.beta1 and its backward compatibility will be supported.
Please note: Fybrik version 1.0.0 does not provide backward compatibility to version 0.7 and versions 0.6.x ( There was no version 0.8 nor version 0.9)
Backward Compatibility
The following require changes when you upgrade from version 0.7 to version 1.0.0.
Changes to infrastructure.json. Metrics instances separate from the attributes were added so they can be referenced from multiple infrastructure attributes.
Due to a lack of backward compatibility, the previous Fybrik version should be uninstalled before the installation of this version.
Features Added
- Soft policies - i.e. IT config policies that enable an IT admin to indicate how to optimize the use of the infrastructure
- Optimizer that determines the optimal data path based on governance decisions, soft and hard IT config policies, workload context, and dataset context
- Fybrik deployment now includes a constraint-solving engine
- Known limitations:
- Optimization is done for a given workload and a given dataset. Optimization across datasets in workload is future work, as is optimization across workloads
- Documentation in fybrik website has not yet been completed for soft policies nor for how to configure the optimizer. Will be released soon in a patch of version 1.0.0.
- Increased security between Fybrik manager and its connectors. Communications with data catalog connectors and policy connectors might be via Transport Layer Security (TLS) and optionally with mutual TLS PR 1587
Key Issues Fixed
- Fybrik stores configuration and temporary files in the local file-system, which prevented in the past to configure the managers pod file system as read-only. This release fixed this limitation PR 1512
- Improved performance by removing excess reconciles in FybrikApplication, and removing blueprint polling in single cluster deployements PR 1613; PR 1611
- Improve performance by removing redundant code and duplicate HELM operation PR 1561
What's Changed
- Deploy solver by @shlomitk1 in #1508
- add documentation for the module chaining sample. by @cdoron in #1504
- Minor fix to site/docs/samples/notebook-write.md by @revit13 in #1523
- Add a temporary solution for deployment kubebuilder on M1 CPU by @roytman in #1526
- Added sample for delete-module by @Raviv-S in #1521
- add link to chaining sample by @cdoron in #1534
- Fixing wrong handling of actions with write flows by @zivnevo in #1537
- Documenting helm values passed to modules by @shlomitk1 in #1528
- Use a fixed fybrik/optimizer image from ghcr.io repository by @shlomitk1 in #1532
- Fixes to notebook sample. by @revit13 in #1541
- Bump actions/setup-python from 3.1.2 to 4.0.0 by @dependabot in #1538
- Bump stretchr/testify from v1.7.0 to v1.7.2 to fix CVE-2022-28948 by @ityuhui in #1542
- Add backward compatability test by @roytman in #1540
- Remove redundant code form helmer by @roytman in #1561
- Document the high-availability method used by the manager by @revit13 in #1564
- Test optimization goals by @shlomitk1 in #1522
- Bandwidth-like optimiztion goals by @zivnevo in #1558
- Fix issue #1570 - wrong handling of 0 distances by @zivnevo in #1572
- Pass multiple datasets to the solver by @shlomitk1 in #1566
- Remove VAULT_ADDR and VAULT_API_ADDR from vault values.yaml file. by @revit13 in #1545
⚠️ Add metrics to infrastructure.json by @shlomitk1 in #1562- optimizer to consider multiple datapath lengths by @zivnevo in #1575
- issue #1574 - normalize attribute values to scale by @zivnevo in #1576
- fix wrong optimization score for distance by @zivnevo in #1580
- fallback plan for optimizer crashes by @zivnevo in #1582
- Bug fix (referencing nil pointer) by @shlomitk1 in #1583
- Define LOGGING env vars in the connectors. by @revit13 in #1578
- Bump github.com/distribution/distribution to fix security vulnerabilities by @ityuhui in #1584
- Fix publication links by @roytman in #1588
- support attr with multiple instance types by @zivnevo in #1586
- Documentation: Using OpenAPI Generator to Generate by @cdoron in #1577
- Update kubectl version in site/docs/get-started/quickstart.md by @revit13 in #1543
- Define local charts mount path as env var. by @revit13 in #1567
- Add VAULT_ENABLED setting to manager/README.md. by @revit13 in #1591
- Remove extra reconciles of FybrikApplication by @shlomitk1 in #1597
- Provide a clear error message when storage accounts are not defined by @shlomitk1 in #1603
- Use
/dataas data root directory instead of/tmp. by @revit13 in #1512 - Bump actions/setup-python from 4.0.0 to 4.1.0 by @dependabot in #1607
- Bump containerd to fix security vulnerability by @ityuhui in #1605
- Local cluster information retrieval by @shlomitk1 in #1611
- Migrate to Ginkgo v2 by @ityuhui in #1612
- Mention Python Vault package in site/docs/contribute/modules.md. by @revit13 in #1601
- Fix inconsistency in credential var that holds vault secret path by @revit13 in #1469
- Avoid polling blueprints for a single cluster setup by @shlomitk1 in #1613
- Support for mutual TLS connection between the manager and connectors by @revit13 in #1587
- Fixes to 1.0 release by @shlomitk1 in #1614
- Change webhook cert dir. by @revit13 in #1623
- Add the wait flag to the helm uninstall command by @roytman in #1627
- Specify emptyDirs size limits by @revit13 in #1641
- fix print the ref of the last commit by @roytman in #1639
- Error parsing PRETTY_LOGGING when manager is started from a command line by @roytman in #1645
- Update manager/testdata/blueprint-read.yaml by @revit13 in #1644
- Fix helm pull, when it works with localChartsMountPath by @roytman in #1649
- Prevent update/delete errors in logs by @shlomitk1 in #1646
⚠️ Code refactoring: apis cleanup by @shlomitk1 in #1653⚠️ add v1beta1 support by @roytman in #1651- Bump docker/docker to v20.10.17 by @ityuhui in #1657
- Fix handling of TLS CA certificates. by @revit13 in #1662
Full Changelog: v0.7.0...v1.0.1
Contributors
ityuhui, roytman, and 6 other contributors
Assets 2
v0.6.8
v0.6.7
Fix security vulnerabilities in the third party dependencies
What's Changed
- add build CI tests to the releases branches by @roytman in #1552
- update dependencis according to #1441 by @roytman in #1551
- Bump stretchr/testify from v1.7.0 to v1.7.2 to fix CVE-2022-28948 by @roytman in #1555
Full Changelog: v0.6.6...v0.6.7
Contributors
roytman