Releases: sqlfluff/sqlfluff
[3.4.0] - 2025-04-17
Highlights
This release brings a couple of breaking changes:
- We have dropped support for Python 3.8
- We have a new rule:
LT15
(layout.newlines
). This rule allows you to control the number of consecutive
blank lines both inside, and outside of queries.
We also have a host of dialect improvements, bugfixes, and general code clean-up. This release has 79 contributions in it!
Also, welcome to the thirteen new contributors! Thanks for your contributions!
Whatβs Changed
- Clickhouse: support more complex types and functions in table definition (#6804) @LSturtew
- feat: Add possibility to set dbt project_dir as environment variable (#6737) @svdimchenko
- Add support for
ZEROFILL
columns in MySQL. (#6735) @joaostorrer - RF02: Add config for ignoring external references found in subqueries (#6791) @keraion
- Fix
CREATE TABLE
statements in Redshift. (#6808) @joaostorrer - Improve support for
CREATE USER
statements in Oracle. (#6809) @joaostorrer - Add support for the WITH clause for an OPENROWSET in T-SQL (#6810) @rayz90
- Add hybrid tables to Snowflake dialect (#6812) @moreaupascal56
- Trino Dialect - Parse UNNEST (...) WITH ORDINALITY (#6813) @rileymcdowell
- Adds support for
varchar(max)
in Redshift. (#6806) @joaostorrer - BigQuery: Make select replace
AS
not optional (#6780) @keraion - CI: Update add-to-release-notes to use ubuntu-latest (#6798) @keraion
- CI: mirror image deployment to ghcr.io (#6790) @keraion
- add ERROR_INTEGRATION to AlterPipeSegment (#6796) @moreaupascal56
- Housekeeping: snowflake use CommentEqualsClauseSegment everywhere (#6795) @moreaupascal56
- Allow quoted charset identifiers for MySQL/MariaDB (#6781) @pprkut
- Fix reserved keywords in Postgres. (#6782) @joaostorrer
- Adds support for user defined datatypes in Redshift. (#6783) @joaostorrer
- improve alter table statement handling for clickhouse dialect (#6787) @LSturtew
- Athena: Support some Trino-based v3 functions (#6788) @keraion
- SparkSQL/Databricks: Support dot sign operator (#6789) @keraion
- RF02: do not trigger on snowflake lambda anonymous parameters (#6689) @shervinmathieu
- Adding additional object to grant statement in snowflake dialect (#6779) @SteveFrensch
- Adds support for
+=
,-=
,*=
,/=
and%=
operators in tsql. (#6756) @joaostorrer - ST11: Fix quoted table name comparisons (#6768) @keraion
- Formatter Stub Types (#6761) @alanmcruickshank
- CP02, LT01: Fix duplicated elements generated from a
FixPatch
ordering conflict (#6769) @keraion - Added support for geospatial data for MySQL (#6765) @AidanHarveyNelson
- Adds support for identity clause in
CREATE TABLE
andALTER TABLE
statements in Oracle. (#6767) @joaostorrer - Fix
PivotSegment
to accept alias inFunctionSegment
in Oracle. (#6766) @joaostorrer - New Rule: LT15 (#6641) @WittierDinosaur
- Additional cleanup for python 3.9 (#6758) @keraion
- feat(snowflake): allow reference variables in create task (#6759) @mrlannigan
- Always use utf-8 encoding for toml files (#6760) @keraion
- Fix rule failures for MySQL/MariaDB with backticked identifiers (#6702) @pprkut
ON CONFLICT
indentation for Postgres (#6660) 8000 @WillMatthews- Update release instructions (#6751) @alanmcruickshank
- Add support for Oracle PL/SQL (Procedures, Functions, Packages, Triggers, Types, Cursors, Loops) (#6635) @joaostorrer
- Tighter typing of Rule configs (#6750) @alanmcruickshank
- Fix: column aliases should find the topmost alias (#6755) @keraion
- AM08: Handle any-cased
UNNEST
(#6757) @keraion - Python 3.9+ Type simplifications (#6745) @alanmcruickshank
- Snowflake dynamic table fixes (#6748) @ulixius9
- Update TSQL Execute Statement for Expressions (#6744) @nathanthorell
- Duckdb: Adding InsertStatementSegment for Duckdb specifics (#6740) @nathanthorell
- Drop support for Python 3.8 (#6743) @WittierDinosaur
- Fix tox config so that it supports spaces in the path name (#6726) (#6727) @maxgrenderjones
- BigQuery: support UNION ALL BY NAME and other new set operator modifiers (#6722) @Robin-C
- Fill in gap in parsing
create user rsa_public_key=...
/ snowflake (#6731) @juli4nb4dillo - Adds support for partition options in
ALTER TABLE
statements in MySQL. (#6742) @joaostorrer - Update oracle reserved keywords. (#6730) @joaostorrer
- Add support in TSQL for Open Symmetric Key (#6736) @nathanthorell
- Cleaning: Remove Todo set tag from Snowflake ALTER TABLE statement (#6738) @moreaupascal56
- SparkSQL: Make colon optional in
STRUCT
datatype (#6739) @NormallyGaussian - Redshift: Support GROUP BY ALL (#6714) @WittierDinosaur
- Fix tests with Click 8.2 (#6706) @cjwatson
- Remove
USER
from reserved words in Postgres. (#6707) @joaostorrer - Add support for
key: value
syntax in functions in Postgres. (#6708) @joaostorrer - Fill in Gap in Snowflake
DROP DYNAMIC TABLE
(#6673) @LoganPrice344 - Fill in Gap in Snowflake
ALTER TABLE
Parsing (#6654) @LoganPrice344 - Fill in Gap in Snowflake
CREATE AUTHENTICATION POLICY
Parsing (#6685) @LoganPrice344 - Adds support for
JoinClauseSegment
inDeleteStatementSegment
in Postgres (#6692) @joaostorrer - Fix check constraints in
ALTER TABLE
statements in Oracle. (#6661) @joaostorrer - Add support for Obevo annotations. (#6662) @joaostorrer
- Add support for
DEFAULT
keyword as function parameter in tsql. (#6663) @joaostorrer - Fill in gap in
DECLARE
Syntax (#6665) @LoganPrice344 - AL03: Lint for missing aliases on subqueries (#6666) @keraion
- IN TABLE/CTE and implicit Array, Tuple support for ClickHouse (#6667) @pheepa
- ST11: Fix rule name in documentation for
noqa
(#6668) @keraion - Fill in Gap in Snowflake
ALTER TABLE
Parsing (#6671) @LoganPrice344 - Allows Snowflake
CREATE ROW ACCESS POLICY
to Parse Quoted Policy Names (#6648) @LoganPrice344 - Add support for
EVENT
statements in MySQL. (#6646) @joaostorrer - Fill in Gap in Snowflake
Exception
Parsing (#6645) @LoganPrice344 - Snowflake: Add TITLE parameter support in CREATE STREAMLIT statement (#6642) @kawashiro
- TSQL: Add support for OPENQUERY (#6640) @nathanthorell
- Snowflake: Add support for CORTEX SEARCH SERVICE (#6639) @sfc-gh-amauser
- Add support for
WithCompoundStatementSegment
in procedures in Postgres (#6632) @joaostorrer - Add support for
SET NEW.<column>
statements in MySQL. (#6634) @joaostorrer - feat: Add missing parquet file format keywords to SF dialect (#6638) @jjlkant
[3.3.1] - 2025-02-05
Highlights
This is a bugfix release resolving a several issues from 3.3.0 and previously.
- Dialect improvements for Snowflake, Oracle, SQLite, BigQuery, TSQL, Databricks, SparkSQL, Hive, Trino & Postgres.
- Rule improvements for AL04, AM08, RF01, RF02, ST03, ST05, ST09, ST10, ST11.
This release also includes the first contributions from thirteen new contributors! Welcome to the project and thank you for your contributions. ππππππ
Whatβs Changed
- Update acronym ETL/ELT in README.md (#6500) @galenseilis
- Fill in Gap for Snowflake
CREATE
Syntax Parsing (#6514) @LoganPrice344 - Add support for PostGIS
geometry(point)
datatype. (#6543) @joaostorrer - Allow double equals in hive dialect (#6623) @TheCleric
- Bump Python version in the docs and metadata (#6626) @WittierDinosaur
- Added support for Variable Substitution for
Snowflake
Target_lag
(#6567) @LoganPrice344 - Sqlite: Added
set_clause_list
toUPDATE
statement (#6605) @keraion - Oracle: Support for
NOMAXVALUE
andNOMINVALUE
(#6606) @keraion - RF02: Handle additional edge cases with subqueries (#6607) @keraion
- TSQL Add Set Context_Info and Replicate function (#6613) @nathanthorell
- Add Support for Snowflake Variables (#6618) @LoganPrice344
- Add support for BigQuery
LOAD DATA
statement (#6612) @VMois - CI: Update to black 25.1 and pre-commit hooks (#6615) @keraion
- Add support for psql
\gset
and\gexec
meta commands. (#6555) @joaostorrer - TSQL improve support for CREATE USER (#6608) @nathanthorell
- Add support for Postgres
ALTER FOREIGN TABLE
statement. (#6597) @joaostorrer - Add zip and zip_strict dbt jinja2 functions (#6564) @boushphong
- TSQL: fix for hex default constraint (#6586) @nathanthorell
- Add support for Oracle SQL*Plus substitution variable
&
. (#6573) @joaostorrer - ST11: Table and wildcard reference fixes (#6572) @keraion
- Adding support in TSQL for Create Login (#6570) @nathanthorell
- ST05: Ordering and rule conflict fixes (#6578) @keraion
- TSQL: Add
WITH ROLLUP
option forGROUP BY
(#6579) @keraion - ST03: Handle quotes in CTE names (#6591) @keraion
- ST09: Handle
lt_eq_gt
operator (#6592) @keraion - AL04: Check for duplicates in subquery aliases and table references (#6593) @keraion
- TSQL: adds drop and add period to alter table (#6590) @nathanthorell
- [Starrocks] Support for generated columns (#6581) @maver1ck
- Fix TSQL OPENROWSET for ADLS use (#5114) (#6584) @rayz90
- Add support for Postgres
CREATE OPERATOR
syntax. (#6540) @joaostorrer - Update postgres bare functions. (#6545) @joaostorrer
- Add Postgres
DROP FOREIGN TABLE
statement. (#6548) @joaostorrer - Add support for Postgres full text search operator
!!
. (#6541) @joaostorrer - Fix Snowflake Dynamic Table Parsing (#6566) @ulixius9
- add variable use in tsql TemporalQuerySegment (#6565) @nathanthorell
- Add support for Postgres functions with column referenced type and argmode between parameter name and datatype. (#6542) @joaostorrer
- add MariaDB to README.md (#6559) @robertsilen
- mypyc: workaround for ABC inheritance (#6552) @keraion
- Add DuckDB to RF01 filter for dialects with dot-access (#6554) @brunobeltran
- Fill in Gap in Snowflake
ALTER TAG
Syntax (#6531) @LoganPrice344 - Fill in Gap in
CREATE ROW ACCESS POLICY
Syntax Parsing (#6550) @LoganPrice344 - [docs] Update CONTRIBUTING.md virtual env setup instructions (#6538) @sarahmccuan
- Added logic to parse named row fields using reference operator (.) for trino dialect (#6536) @prabh-me
- SparkSQL dialect: Allow datatypes for column definition in create view statements (#6518) @KikeSenpai
- Fill in Gap for
ALTER_TABLE_COLUMN
Syntax Parsing (#6526) @LoganPrice344 - fix: databricks describe volume recognition (#6529) @benfdking
- Fill in Gap in Snowflake
ALTER ROW ACCESS POLICY
Syntax Parsing (#6534) @LoganPrice344 - Fill in Gap for Snowflake
ALTER warehouse
Syntax (#6516) @LoganPrice344 - [docs] Update missing convention whitespace for CV10 (#6528) @sarahmccuan
- Fill in gaps for
GRANT
syntax (#6520) @LoganPrice344 - Snowflake: allow multiple predecessor tasks in CREATE/ALTER TASK (#6510) @mrebaker
- AM08: fix edge cases (#6506) @rogalski
- ST10: Fix edge cases (#6505) @rogalski
- Snowflake - Fill in gaps in
GRANT
command parsing (#6503) @korverdev - Add support for Snowflake's
CREATE OR ALTER
syntax (#6497) @korverdev - TSQL dialect - Allow for multiple columns in ALTER TABLE DROP COLUMN statements (#6501) @korverdev
[3.3.0] - 2024-12-10
Highlights
This release brings a few more significant changes. Especially given the introduction of several new rules, we highly recommend testing this release on your project before upgrading to make sure they are configured appropriately for your project style guide. As always, we have tried to make sure that the defaults for all new rules are both widely applicable, and fairly light touch. While all have been tested on some existing larger codebases which the maintainers have access to - do still report any bugs you might find on GitHub in the usual manner.
- We've dropped the
appdirs
package as a dependency (as an abandoned project) and instead addedplatformdirs
instead. Users should not notice any functionality changes beyond the different dependency. - TWO new dialects: Impala and StarRocks.
- FIVE new rules:
AM08
(ambiguous.join_condition
), which detectsJOIN
clauses without conditions (i.e. without anON
orUSING
clause). These are often typos and can result in significant row count increases if unintended.CV12
(convention.join_condition
), which is related toAM08
and detects cases where users have used aWHERE
clause instead of aJOIN ... ON ...
clause to do their join conditions. The join condition is a form of metadata and should communicate to the end user how the table should be joined. By mixing this information into theWHERE
clause it makes the SQL harder to understand.LT14
(layout.keyword_newline
), which allows certain keywords to trigger line breaks in queries. Primarily this forces the mainSELECT
statement clauses likeWHERE
,GROUP BY
etc. onto new lines. This rule has been designed to be highly configurable, but with sensible light-touch defaults. Check out the docs to adapt it to the conventions of your project.ST10
(structure.constant_expression
), some SQL users include redundant expressions in their code (e.g.WHERE tbl.col = tbl.col
). These conditions always evaluate to a constant outcome (i.e. always evaluate asTRUE
orFALSE
) as so add no functionality or meaning to the query. This rule catches them.ST11
(structure.unused_join
), which detects unused joins in SQL statements, and is designed to catch tables that were once used, but where the column references have since been removed and now the table is unnecessary.
Beyond these changes, we've seen a whole host of dialect improvements to almost all of the supported dialects and several bugfixes which are combined into this release.
We also welcome TWELVE new contributors to the project in this release. Thanks to all of them for their hard work πππ.
Whatβs Changed
- New Rule LT14: Keyword line positioning (#6213) @keraion
- New Rule ST11: Detect unused tables in join (#5266) @danparizher
- Sno 8000 wflake Create Table allow inline foreign key with on delete β¦ (#6486) @WobblyRobbly
- Fix minor linting error in CI (#6483) @alanmcruickshank
- Snowflake: alter table on delete and update support (#6473) @WobblyRobbly
- New Rules AM08 + CV12: Detect implicit cross joins (#6239) @rogalski
- New Rule ST10: const expression checker (#6392) @rogalski
- DuckDB: Support MAP data type (#6478) @WittierDinosaur
- Hive: Add 'ALTER VIEW' query grammar (#6479) @mrebaker
- Standardise json operator spacing between dialects (#6447) @WittierDinosaur
- fixes #6463: Set Variable Parsing for SparkSQL and Databricks (#6464) @fstg1992
- Teradata: support REPLACE VIEW and LOCKING ... FOR ... syntax (#6467) @V-D-L-P
- Rule names in warnings logic (#6459) @LuigiCerone
- Bigquery: Support column level key definitions (#6465) @keraion
- Add Implicit Indents to Qualify (#6438) @WittierDinosaur
- Postgres: Fix Select statement ordering (#6446) @WittierDinosaur
- fixes #6457: databricks dialect alter table foo drop column bar (#6461) @fstg1992
- Switch from
appdirs
toplatformdirs
(#6399) @alanmcruickshank - Impala: support CREATE TABLE AS SELECT (#6458) @mrebaker
- Databricks Dialect: Backticked function identifiers now parsable (#6453) @fstg1992
- Issue #6417: Leading -- MAGIC Cells don't break parsing of notebooks (#6454) @fstg1992
- Add "target_path" configuration to the dbt templater (#6423) @wircho
- Sparksql: Fix ordering of create table options (#6441) @WittierDinosaur
- Dialect: Impala (#6445) @mrebaker
- RF02: Allows for lambda functions in Databricks (#6444) @keraion
- SQLite: Support any order of VARYING/NATIVE in CHAR types (#6443) @keraion
- Snowflake: Allow literals in match_by_column_name (#6442) @WittierDinosaur
- Trino: Remove TemporaryTransientGrammar (#6440) @WittierDinosaur
- Mysql: Fix parsing of system variables (#6439) @WittierDinosaur
- Sparksql: Fix hint function for proper spacing (#6437) @WittierDinosaur
- Snowflake: Support ORDER BY boolean (#6435) @WittierDinosaur
- TSQL: allow
NEXT VALUE FOR
use as expression (#6431) @timz-st - Prework for introducing mypyc (#6433) @rogalski
- Fix pre-commit on main branch (#6432) @rogalski
- Initial support for Starrocks dialect (#6415) @maver1ck
- Databricks: Parse Table Valued Functions (#6417) @fstg1992
- Snowflake: Support
PARTITION_TYPE
forCREATE EXTERNAL TABLE
(#6422) @ninazacharia-toast - Fix docs for CP04 config and add test cases (#6416) @alanmcruickshank
- Fix: Parse violations not being shown when run
fix
command with--show-lint-violations
(#6382) @joaopamaral - RF01: refine support for dialects with dot access syntax (#6400) @rogalski
- Add new
TYPE
property to Snowflake users (#6411) @mroy-seedbox - Document the config API (#6384) @alanmcruickshank
- Error handling for trying to render callable builtins #5463 (#6388) @alanmcruickshank
- SQLite : Add
CREATE VIRTUAL TABLE
Statement (#6406) @R3gardless - Updated README with Table Of Contents (#6407) @27Jashshah
[3.2.5] - 2024-10-25
Highlights
This release is mostly bugfixes and dialect improvements. Notably:
- Whitespace handling improvements to
LT01
<02
. - Better error messages around trying to iterate on missing jinja variables.
- Better case sensitivity for
AL09
. - Improved handling of jinja context in inline config directives.
- Enabling
AM02
for Trino and Snowflake. - Handling potential collisions between
ST02
<01
. - Preventing false positives in AL05 with arrays.
There's also a bunch of documentation improvements in this release, including guides on how to troubleshoot SQLFluff and how to write custom rules. Check out https://docs.sqlfluff.com for more details.
We also saw five new contributors to the project this month. Welcome to the project, and thanks for taking the time to contribute! πππ
Whatβs Changed
- Guides for custom rules and for troubleshooting (#6379) @alanmcruickshank
- Documentation and small overhaul of parametrized rule test cases (#6380) @alanmcruickshank
- TSQL: add missing unreserved keyword NULLS (#5212) (#6390) @simonhoerdumbonde
- Introducing SQLFluff Guru on Gurubase.io (#6373) @kursataktas
- Improve heuristics for inline config (#6391) @rogalski
- Postgres: Handle expressions that occur in
IN
functions (#6393) @keraion - Snowflake: Support bracketed lambda functions without datatypes (#6394) @keraion
- LT01: Add default config for
match_condition
to touch (#6395) @keraion - Snowflake: Allow for additional
CONNECT BY
expressions that may usePRIOR
(#6396) @keraion - Details on debugging and setup for diff-quality (#6381) @alanmcruickshank
- Fix edge case in Jinja reindents (#6383) @rogalski
- Support identifier clause for Databricks (#6377) @PaulBurridge
- Enable AM02 for snowflake and trino by default. (#6369) @mchen-codaio
- Postgres: Support identifiers in
ALTER DATABASE SET
(#6376) @keraion - SparkSQL: Improved lexing and parsing of file literals (#6375) @keraion
- Fix Snowflake alter share (#6372) @greg-finley
- Resolve collision between ST02 and LT01 (#6366) @alanmcruickshank
- Prevent false positives with AL05 and array functions (#6365) @alanmcruickshank
- Handle iteration and getting undefined jinja variables (#6364) @alanmcruickshank
- Update documentation with new dialects (#6337) @mchen-codaio
- enable default values when creating databricks tables (#6362) @VictorAtIfInsurance
- Postgres : Add
ALTER AGGREGATE
Statement (#6353) @R3gardless - Revise AL09 (self aliasing) - stricter case sensitivity (#6333) @alanmcruickshank
- Make dbt
RelationEmulator
safely callable (#6358) @mroy-seedbox
New Contributors
- @VictorAtIfInsurance made their first contribution in #6362
- @mchen-codaio made their first contribution in #6337
- @PaulBurridge made their first contribution in #6377
- @kursataktas made their first contribution in #6373
- @simonhoerdumbonde made their first contribution in #6390
[3.2.4] - 2024-10-14
Highlights
This release is almost all dialect fixes and bugfixes. Notably also, this release brings official python 3.13 support too (although most users should not realise any differences).
We also see two new contributors to the project. Welcome @R3gardless & @brandonschabell! πππ
Whatβs Changed
- Utilize a deepcopy of the config object when parsing files (#6344) @brandonschabell
- Snowflake supports other literals in system functions (#6355) @alanmcruickshank
- Snowflake: Un-reserve CURRENT_USER (#6354) @alanmcruickshank
- tsql: handle additional primary/foreign key options in constraints (#6347) @keraion
- Add
DROP COLUMN
support for multiple dialects (#6348) @keraion - TSQL: allow
UPDATE
to be a function name (#6349) @keraion - tsql: allow both on delete and on update in a
reference_constraint
(#6346) @keraion - Postgres : Allow Extensions with Special Characters in Name (#6345) @R3gardless
- Fix
tox
command in test/fixtures/dialects/README.md (#6342) @R3gardless - Revise dbt warnings when a file fails to compile (#6338) @alanmcruickshank
- Postgres: Add
CREATE FOREIGN DATA WRAPPER
statement (#6335) @keraion - Trino: Add some support to
json_query
functions (#6336) @keraion - Handle deprecation warning of "fork" (#6332) @alanmcruickshank
- Python 3.13 support and make it default for test coverage (#6269) @alanmcruickshank
- ST06 - Fix union of CTE/Subquery (#6298) @rogalski
- Refactor timestamp grammar (#6331) @greg-finley
[3.2.3] - 2024-10-10
Highlights
This is another release of dialect improvements and rule bugfixes. Notably:
- More robust algorithms for the indentation of Jinja template tags in
LT02
. - The
github-annotation-native
format option now has groups for each filename.
There's also a refactor of where we guides and howtos in the docs. Keep an eye on that section going forward for more information about best practice and troubleshooting for SQLFluff.
Even in this small PR, we've seen two new contributors. Welcome @nspcc-cm & @rogalski to the project!
Whatβs Changed
- BigQuery: Support Tuple syntax in other locations (#6328) @keraion
- Trino: Fix rule interactions with lambda functions (#6327) @keraion
- Resolve some more edge cases in LT02 (#6324) @alanmcruickshank
- RF05 - fine tuning for snowflake dialect (#6297) @rogalski
- Indentation:
UPDATE
andRETURNING
clauses (#6314) @keraion - Postgres: Fix lexing some JSON operators (#6323) @keraion
- Add support for
grant monitor on user ...
in Snowflake dialect (#6322) @mroy-seedbox - Exclude templated casts from CV11 (#6320) @alanmcruickshank
- Snowflake allow double-quoted comments (#6318) @greg-finley
- Databricks materialized view (#6319) @greg-finley
- Allow double quotes to be escaped by writing twice (#6316) @alanmcruickshank
- Resolve an oscillation bug with LT02 (#6306) @alanmcruickshank
- Create a "Guides and Howtos" section of the docs. (#6301) @alanmcruickshank
- Add groups to the
github-annotation-native
format option. (#6312) @alanmcruickshank - Snowflake: Support CTEs and multiple orders of CopyOptions in COPY INTO (#6313) @keraion
- Allow expressions in ORDER BY for clickhouse (#6311) @alanmcruickshank
- Snowflake: support temp UDFs (#6309) @rogalski
- fix: tsql create function syntax corrections (#6289) @nspcc-cm
[3.2.2] - 2024-10-07
Highlights
This is a hotfix release to resolve an issue with the JJ01 rule when running in parallel mode.
Whatβs Changed
- Hotfix for JJ01 (#6304) @alanmcruickshank
- Add note on 3.0.x to main docs page. (#6302) @alanmcruickshank
[3.2.1] - 2024-10-06
Highlights
This release is primarily housekeeping, bugfixes and dialect improvements. More specifically:
- Resolving regressions in
JJ01
, filename extension handling and the treatment of unfixable/unparsable files, which have been noticed with recent releases. - Resolving bugs in
LT07
<12
which relate to jinja whitespace control. - More robust support for arbitrary methods on the
ref
andsource
macros for the dbt templater.
There's also dialect improvements for BigQuery, TSQL, MySQL, MariaDB, Snowflake, DuckDB, Databricks, Postgres, Teradata, Exasol & Vertica.
We also saw six new contributors merge their first pull request as part
of this release. Welcome to the project! πππ
Whatβs Changed
- Postgres: Support walrus operator named arguments (#6299) @keraion
- TSQL: handle nested joins, RF01 better aliasing (#6300) @keraion
- Exclude Macros - Allow multiple paths. (#6221) @culpgrant
- Dededuplicate rule ignore docs (#6296) @alanmcruickshank
- Bugfix for LT07 with consumed newlines. (#6294) @alanmcruickshank
- Bugfix for LT12 with jinja whitespace consumption (#6292) @alanmcruickshank
- RF02: Ignore
DECLARE
variables in BigQuery (#6295) @keraion - Bugfix for JJ01 in parallel mode (#6293) @alanmcruickshank
- Allow arbitrary attributes & methods for
ThisEmulator
(#6254) @mroy-seedbox - RF05: Add
table_aliases
option (#6273) @keraion - BigQuery: Add support for concatenating in
EXECUTE IMMEDIATE
(#6287) @keraion - BigQuery: Add support for
SET
with system variables (#6288) @keraion - Plugins: Migrate example plugin to
pyproject.toml
(#6286) @keraion - TSQL: Add DATETRUC to date_part_function_name list (#6283) @paysni
- MySQL Alter table convert to character set (#6277) @greg-finley
- Remove dependency on coveralls. (#6284) @alanmcruickshank
- Test dbt Templater Plugin with dbt 1.9.0 (#6280) @edgarrmondragon
- AM06: Ignore array expressions in BigQuery (#6276) @keraion
- Add mariadb to issue labeler (#6278) @greg-finley
- BigQuery: Add
GROUPING SETS
clause (#6275) @keraion - Snowflake: Support
ARRAY
types (#6272) @keraion - Move most of the config validation settings out into rule bundles. (#6262) @alanmcruickshank
- Revise warnings with fixing unfixable files. (#6257) @alanmcruickshank
- Strict mypy on
sqlfluff.core
(#6246) @alanmcruickshank - DuckDB: Add
DROP MACRO
(#6270) @keraion - Added Support for Databricks SQL Notebook Cells (#6267) @gabepesco
- dbt templater
pyproject.toml
nits (#6268) @alanmcruickshank - Add UTF8 support for identifiers in Vertica dialect (#6183) @troshnev
- Almost all of
util
up to strict typing (#6263) @alanmcruickshank - Update link to diff-cover docs (#6256) @alanmcruickshank
- MySQL: AL09 handle double quoted identifiers (#6249) @keraion
- fix: complex file extensions discovery (#6228) @Clepech
- fix RF06 issue in postgres naked identifier regex (#6247) @fvankrieken
- Strict typing for
sqlfluff.core.linter
(#6240) @alanmcruickshank - DuckDB: Fixed
DatatypeSegment
references (#6244) @keraion - Postgres: Support
SET
with double quoted identifiers (#6243) @keraion - Consolidate Teradata tests (#6241) @greg-finley
- RF02: Ignore alias references that are self-inner (#6242) @keraion
- Add additional CREATE TABLE support for Databricks (#6216) @pahunter90
- Complete the support for PIVOT in Snowflake dialect (#6217) @fpsebastiam
- Exasol - allow function calls in values clause (#6226) @stephnan
- Snowflake: Support defining virtual columns (#6237) @babak-l1
- Teradata order of VOLATILE and MULTISET (#6233) @greg-finley
- Remove duplicate timing columns from the timing records (#6229) @Tenzer
- Fix time travel clauses in Snowflake dialect (#6230) @fpsebastiam
[3.2.0] - 2024-09-18
Highlights
This release brings a few minor breaking changes, both for the core project and for the dbt templater. For the main project:
-
Resolving an issue with the spacing of functions (LT01), which involved a change to how functions are parsed. If your project relies on the specific parsing of functions, the bracketed arguments are now wrapped in a
function_contents
object. We recommend that you examine the new parsing structure using this new release in testing first. -
RF06
(references.quoting
) is now case sensitive when removing quotes which are detected as unnecessary. This rule has also been re-enabled by default for Snowflake and Postgres where it had previously been disabled (for the reason that in the past it hadn't been appropriately case sensitive). Treatment for totally case-insensitive dialects like DuckDB and SparkSQL have also been included. Please check the new documentation for this rule (which is much more explicit now), for details related to your dialect. -
Patterns equivalent to those from
.sqlfluffignore
can now be included in.sqlfluff
andpyproject.toml
files. -
Using the
python
templater, users now have an option to include variables which include a dot in the path, like{{ foo.bar }}
using a specialsqlfluff
context variable. -
Significant changes under the hood to the handling of configuration files. Most of these should not be visible to end users, but for anyone integrating SQLFluff into a larger project and relying on native file loading may need to refactor their project for this release. Most notably here, for maintainers of plugins, the
ConfigLoader
class has been deprecated, and plugins should instead call the config loading functions directly. See the example plugin for details. -
Documentation, especially for dialects, has been significantly improved. Documentation for
CP02
(capitalisation.identifiers
) has also been clarified to make it's implication for references and aliases more clear. -
During testing, to isolate the effect of specific rules, there's a new CLI option
--disable-noqa-except
which allows allnoqa
options to be ignored except the ones provided in this option.
For the dbt templater:
-
Support for dbt 1.1-1.3 has been removed. All have been in End of Life (EOL) support by dbtlabs for almost two years. They are also poorly supported by other projects and tools.
-
The dbt templater has been migrated to use
pyproject.toml
. -
Handling of errors and exceptions raised within dbt has had an overhaul. Users may see a slightly different presentation of errors, but the overall stability should be more robust.
In addition to those changes, there have been too many dialect contributions and bugfixes to mention specifically. We've also seen six people make their first contributions to the project as part of preparing for this release! πππ.
Whatβs Changed
- Handle multi-processing dbt exceptions much better. (#6138) @alanmcruickshank
- Support variables with dot in python templater (#5872) @timchurch
- Add postgres normalization operator support (#6211) @fnimick
- Fix patch will anchor on first buffer insertion point (#6212) @keraion
- Allow ignore patterns in other config files. (#6130) @alanmcruickshank
- Strict typing in
config
andhelpers
. (#6206) @alanmcruickshank - TSQL: Support multiple options in
SET
statement (#6205) @keraion - DuckDB: Support
CREATE TYPE
statement (#6204) @keraion - Update Slack link (#6203) @greg-finley
- Add quoted literal checking for Snowflake TARGET_LAG in dynamic tables. (#6201) @mvastarelli
- Databricks: Support
COMMENT ON
statement (#6196) @keraion - DuckDB: Support
STRUCT
datatype (#6198) @keraion - Deprecate the
ConfigLoader
(#6177) @alanmcruickshank - DuckDB: Support
CREATE MACRO
/CREATE FUNCTION
(#6194) @keraion - DuckDB: Support functions with walrus operators (#6193) @keraion
- Add volume syntax support for Databricks (#6179) @TheCleric
- Handle errors better in AL09 (#6186) @alanmcruickshank
- Add support for managed locations to databricks dialect schemas (#6182) @TheCleric
- MYSQL: Create Table Optional AS (#6109) @WittierDinosaur
- More dialect documentation (#6165) @alanmcruickshank
- Better documentation on how to cross reference rules and fix a few. (#6162) @alanmcruickshank
- RF06: Case Sensitivity (#6173) @alanmcruickshank
- SparkSQL/Databricks: Support for
VARIANT
type (#6167) @keraion - sparksql: Allow
INSERT OVERWRITE
after a CTE (#6172) @keraion - postgres: Add
SET CONSTRAINTS
statement (#6171) @keraion - TSQL: Fix
MERGE
without a target alias (#6170) @keraion - TSQL: add
OFFSET
andFETCH
(#6169) @keraion - postgres: Add support for
SUBSCRIPTION
statements (#6168) @keraion - Duckdb: Add support for list comprehensions (#6166) @keraion
- Update Docs and tests for CP02 (#6163) @alanmcruickshank
- Cached property in RF06 rather than DIY (#6164) @alanmcruickshank
- CI: Update
util.py
for dbt templaterpyproject.toml
(#6160) @keraion - Auto generate dialect docs (#6153) @alanmcruickshank
- Deprecate support for dbt 1.1-1.3 (#6159) @WittierDinosaur
- ST08: Ignore
DISTINCT
s with subqueries (#6146) @keraion - Duckdb: Fix Create View coverage (#6158) @WittierDinosaur
- Snowflake: Support Password Policies (#6154) @WittierDinosaur
- Postgres: Allow negative integers in sequences (#6111) @WittierDinosaur
- Postgres: Add SHOW Statement (#6110) @WittierDinosaur
- Migrate dbt templater to pyproject.toml (#6155) @keraion
- Snowflake: Add DEFAULT option for function parameters (#6145) @keraion
- Snowflake: fixes parsing for INCLUDE_METADATA in COPY INTO statement (#6150) @jcrobak
- [SNOWFLAKE] Adding support for extended constraint definitions (#6151) @babak-l1
- Snowflake: fixes parsing for PARSE_HEADER in FILE FORMAT statement (#6149) @jcrobak
- fix: avoid strip_newlines when encounter comments in inline segments (#6140) @Cynthia-Cheng
- More robust exception handling for dbt. (#6144) @alanmcruickshank
- postgres: Add
ENCRYPTED PASSWORD
option inCREATE USER
(#6143) @keraion - Fix support of INTERVAL in ClickHouse (#6112) @Pavel-Strybuk
- Add support for Snowflake Higher-Order Functions (#6136) @amardatar
- Method extraction and more robust typing in config. (#6135) @alanmcruickshank
- Add missing databricks and sparksql ALTER statements (#6102) @pahunter90
- fix: program_counter move in JinjaTracer (#6121) (#6123) @Cynthia-Cheng
- CI: allow hidden file upload for coverage files (#6139) @keraion
- Fix: added DOWNSTREAM keyword for TARGET_LAG on dynamic tables in Snowflake. (#6131) @mvastarelli
- Trino Dialect: update ARRAY type handling (#6127) @kirkhansen
- Split apart config module (#6128) @alanmcruickshank
- Add option for allowing only a subset of rules for noqa (#6115) @keraion
- TSQL: Allow for empty catch block in try-catch (#6116) @keraion
- Change evaluation order of literals before column refs in bracketed, delimited expressions (#6117) @keraion
- Fix spacing rules for functions (#5809) @WittierDinosaur
- SQLite: Add conflict_clause to unique table constraint (#6106) @WittierDinosaur
- SQLite: Support Raise Function (#6108) @WittierDinosaur
- SQLite: Create Trigger WHEN optionally bracketed (#6107) @WittierDinosaur
- Snowflake: Added
INTERVAL
s to Frame Clause (#6105) @keraion - Postgres: Add
IS UNKNOWN
(#6094) @keraion - RF02: Handle subquery column qualification (#6091) @keraion
- tsql: Allow leading dots in table references (#6093) @keraion
[3.1.1] - 2024-08-20
Highlights
This release brings a bumper lot of bugfixes, dialect improvements and other minor improvements across the board. Most notably:
- A rework of the structure of the docs. NOTE: This does change the url of some docs pages, but to prevent future moves, we've also provided permalinks to most important pages and rules. See the
conf.py
file in thedocs
folder for a full list of permalinks. - Solving rule conflicts between LT02 & LT02.
- Bugfixes to AM07, CV11, ST03, ST05 & RF03,
- Removes some redundant dependencies in for the dbt templater (which haven't been required for some time, but have been included in the install dependencies). Specifically:
markupsafe
,ruamel.yaml
,pydantic
&rich
. - And too many dialect improvements to summarise!
We've also seen fifteen new contributors to the project! Thanks to all of them for taking the time to contribute. πππππ
Whatβs Changed
- dbt Templater: Increase
dbt deps
test fixture timeout (#6088) @keraion - SparkSQL + Databricks: Add support for raw string literals (#6089) @D-to-the-K
- fixes #4855 - Add DECLARE statement in snowflake dialect (#6059) @YungChunLu
- Adding CTE to mysql views (#6077) @gone
- Rationalise config discovery routines. (#6080) @alanmcruickshank
- fix(dialect-trino): Trino ROW datatype definition in queries (#6085) @bonisb
- Databricks: Add support for GROUP BY ALL (#6082) @D-to-the-K
- fix(clickhouse): add support for tuple() and ENGINE MergeTree (#6079) @ogirardot
- Add perma-links for rules (#6066) @alanmcruickshank
- fix(clickhouse): add support for rename statement (#6073) @ogirardot
- fix(clickhouse): add support for INTO OUTFILE and supported FORMATs (#6065) @ogirardot
- LT04: Fix indentation conflict with LT02 (#6068) @keraion
- pre-commit: Disable progress bar (#6069) @keraion
- feat(clickhouse): add support for decimal(x,y), decimal32(x) and match (#6063) @ogirardot
- Big docs refactor. (#6052) @alanmcruickshank
- ST05: Handle set statement's subsequent queries (#6062) @keraion
- fix(clickhouse): add support for limit by and bracketed format (#6061) @ogirardot
- fix(clickhouse): add support for DateTime64(precision, tz) and Tuples() (#6060) @ogirardot
- Copy statement postgres v9 compatibility support (#5181) @Fullcure3
- Run dbt tests in py312 by default (#5861) @alanmcruickshank
- Extract path discovery routines from config and linter. (#6057) @alanmcruickshank
- MySQL: Added SET TRANSACTION parsing (#5781) @Xemptuous
- Support declare or replace variable statement for Databricks dialect (#6054) @urosstan-db
- Exclude Macros from a path provided (#6031) @culpgrant
- ST03: Detect CTE usage in nested
WITH
clauses (#6017) @keraion - TRINO: support FILTER after WITHIN GROUP agg expression (#6056) @rileymcdowell
- Fix/snowflake double quotes (#5727) @Starstruckk
- bq_table_function : Added functionality to parse table functionsin BigQuery (#5480) @moh-lch
- Fix Athena Partitioned By format for iceberg tables (#5399) @jverhoeks
- fix: redshift dialect, EXTENSION added (#6025) @rafalbog
- Fix ignored inline rule overrides (#5697) (#6010) @alesbukovsky
- Update the docs on RF03 (#6051) @alanmcruickshank
- RF03: Fixed some subquery reference scenarios (#6046) @keraion
- CV11: Remove rogue print statement (#6047) @keraion
- Snowflake: fixes parsing for AGGREGATE in CREATE FUNCTION statement (#6049) @hawle
- Snowflake:adds optional IF NOT EXISTS to ADD COLUMN (#6050) @hawle
- Replace types-pkg-resources with types-setuptools (#6039) @keraion
- Remove old deps for dbt templater (#6028) @alanmcruickshank
- Added GENERATED AS IDENTITY support to DataBricks dialect (#6004) @nicolb2305
- Add support for Clickhouse ORDER BY WITH FILL (#6018) @snikch
- Parse API example (#6021) @alanmcruickshank
- Add additional dateparts from DATEPART() (#6012) @yorickbouma
- MariaDB: Add support for GROUP BY ... ASC/DESC and GROUP BY ... WITH ROLLUP (#6009) @pprkut
- AM07: Handle set expressions with bracketed selects (#6005) @keraion
- MariaDB: Add support for DELETE/INSERT/REPLACE ... RETURNING (#6008) @pprkut
- MariaDB: Add mariadb specific syntax for generated columns (#6007) @pprkut
- Snowflake: fixes drop column if exists parsing rules (#5999) @hawle
- Fix TSQL Post Table Expr intrepreted as function (#6001) @ulixius9
New Contributors
- @yorickbouma made their first contribution in #6012
- @snikch made their first contribution in #6018
- @nicolb2305 made their first contribution in #6004
- @alesbukovsky made their first contribution in #6010
- @rafalbog made their first contribution in #6025
- @jverhoeks made their first contribution in #5399
- @moh-lch made their first contribution in #5480
- @Starstruckk made their first contribution in #5727
- @culpgrant made their first contribution in #6031
- @urosstan-db made their first contribution in #6054
- @ogirardot made their first contribution in #6060
- @D-to-the-K made their first contribution in #6082
- @bonisb made their first contribution in #6085
- @gone made their first contribution in #6077
- @YungChunLu made their first contribution in #6059