8000 AM07 throws an error whe using `dbt_utils.union_relations` with `include` or `exclude` arguments · Issue #5401 · sqlfluff/sqlfluff · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

AM07 throws an error whe using dbt_utils.union_relations with include or exclude arguments #5401

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
2 of 3 tasks
danisola opened this issue Nov 13, 2023 · 3 comments · Fixed by #6005
Closed
2 of 3 tasks

AM07 throws an error whe using dbt_utils.union_relations with include or exclude arguments #5401

danisola opened this issue Nov 13, 2023 · 3 comments · Fixed by #6005
Labels
bug Something isn't working

Comments

@danisola
Copy link

Search before asking

  • I searched the issues and found no similar issues.

What Happened

I'm linting a file with dbt_utils.union_relations with the include or exclude arguments.

Expected Behaviour

I would expect sqlfluff to execute normally and report on linting errors.

Observed Behaviour

Throws IndexError: list index out of range exceptions:

sqlfluff lint path/to/model/fct_automated_campaign_stats.sql
=== [dbt templater] Sorting Nodes...
=== [dbt templater] Compiling dbt project...
=== [dbt templater] Project Compiled.
CRITICAL   [AM07] Applying rule AM07 to 'models/analytics/marts/marketing/networks/fct_automated_campaign_stats.sql' threw an Exception: list index out of range                                                                                                                                  
Traceback (most recent call last):
  File "/Users/danilagares/workspace/jarvis/.venv/lib/python3.11/site-packages/sqlfluff/core/rules/base.py", line 855, in crawl
    res = self._eval(context=context)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/danilagares/workspace/jarvis/.venv/lib/python3.11/site-packages/sqlfluff/rules/ambiguous/AM07.py", line 216, in _eval
    set_segment_select_sizes, resolve_wildcard = self._get_select_target_counts(
                                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/danilagares/workspace/jarvis/.venv/lib/python3.11/site-packages/sqlfluff/rules/ambiguous/AM07.py", line 195, in _get_select_target_counts
    cnt, res = self.__resolve_selectable(selectable, query)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/danilagares/workspace/jarvis/.venv/lib/python3.11/site-packages/sqlfluff/rules/ambiguous/AM07.py", line 166, in __resolve_selectable
    _cols, _resolved = self.__resolve_selectable_wildcard(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/danilagares/workspace/jarvis/.venv/lib/python3.11/site-packages/sqlfluff/rules/ambiguous/AM07.py", line 134, in __resolve_selectable_wildcard
    _cols, _resolved = self.__resolve_wild_query(cte)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/danilagares/workspace/jarvis/.venv/lib/python3.11/site-packages/sqlfluff/rules/ambiguous/AM07.py", line 87, in __resolve_wild_query
    return self.__resolve_selectable(query.selectables[0], query)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/danilagares/workspace/jarvis/.venv/lib/python3.11/site-packages/sqlfluff/rules/ambiguous/AM07.py", line 166, in __resolve_selectable
    _cols, _resolved = self.__resolve_selectable_wildcard(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/danilagares/workspace/jarvis/.venv/lib/python3.11/site-packages/sqlfluff/rules/ambiguous/AM07.py", line 134, in __resolve_selectable_wildcard
    _cols, _resolved = self.__resolve_wild_query(cte)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/danilagares/workspace/jarvis/.venv/lib/python3.11/site-packages/sqlfluff/rules/ambiguous/AM07.py", line 87, in __resolve_wild_query
    return self.__resolve_selectable(query.selectables[0], query)
                                     ~~~~~~~~~~~~~~~~~^^^
IndexError: list index out of range
All Finished 📜 🎉!

How to reproduce

Here's an model example that shows this behaviour:

WITH sources AS (
{{
  dbt_utils.union_relations(
    relations=[
      ref('fct_apple_search_campaign_stats'),
      ref('fct_facebook_campaign_stats'),
      ref('fct_gdv360_campaign_stats'),
      ref('fct_google_campaign_stats'),
      ref('fct_outbrain_campaign_stats'),
      ref('fct_taboola_campaign_stats'),
      ref('fct_tiktok_ads_campaign_stats'),
    ],
    include=[
      'unique_id',
      'division',
      'source_mkt_channel',
      'source_network',
      'account_id',
      'account_name',
      'date',
      'campaign_id',
      'campaign_name',
      'campaign_type',
      'impressions',
      'clicks',
      'spend'
    ]
  )
}}
),

final AS (
  SELECT
    *,
    {{ get_vat('date', 'division') }} AS vat
  FROM sources
)

SELECT * FROM final

Dialect

redshift

Version

2.3.4

Configuration

[sqlfluff]
dialect = redshift
exclude_rules = L016,L031,L034,L039,L054,L059,L067
templater = dbt
large_file_skip_byte_limit = 30000

[sqlfluff:indentation]
tab_space_size = 2

[sqlfluff:rules:capitalisation.keywords]
capitalisation_policy = upper

[sqlfluff:rules:capitalisation.identifiers]
capitalisation_policy = upper

[sqlfluff:rules:capitalisation.functions]
capitalisation_policy = upper

[sqlfluff:rules:capitalisation.literals]
capitalisation_policy = upper

[sqlfluff:rules:capitalisation.types]
capitalisation_policy = upper

[sqlfluff:rules:references.special_chars] # Space needed for SFMC exports
allow_space_in_identifier = True

[sqlfluff:templater:jinja]
load_macros_from_path=macros

Are you willing to work on and submit a PR to address the issue?

  • Yes I am willing to submit a PR!

Code of Conduct
@danisola danisola added the bug Something isn't working label Nov 13, 2023
@danisola danisola changed the title AM07 throws an error whe using dbt_utils.union_relations with include or exclude arguments AM07 throws an error whe using dbt_utils.union_relations with include or exclude arguments Nov 13, 2023
@itsjliketheletter
Copy link
itsjliketheletter commented Feb 9, 2024 •
edited
Loading

Yep, have also had the same issue.

CODE:

with
unified_partitions as (
{{
dbt_utils.union_relations(
relations=[
source('EXAMPLE', 'Example_2019'),
source('EXAMPLE', 'Example_2020'),
source('EXAMPLE', 'Example_2021'),
source('EXAMPLE', 'Example_2022'),
source('EXAMPLE', 'Example_2023')
],
where = 'DWIsCurrent = 1',
source_column_name = none
)
}}
)

select * from unified_partitions

GIVES:

CRITICAL [AM07] Applying rule AM07 to 'models/STG/example/example.sql' threw an Exception: list index out of range
Traceback (most recent call last):
File "/usr/local/python/3.11.7/lib/python3.11/site-packages/sqlfluff/core/rules/base.py", line 507, in crawl
res = self._eval(context=context)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python/3.11.7/lib/python3.11/site-packages/sqlfluff/rules/ambiguous/AM07.py", line 216, in _eval
set_segment_select_sizes, resolve_wildcard = self._get_select_target_counts(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python/3.11.7/lib/python3.11/site-packages/sqlfluff/rules/ambiguous/AM07.py", line 195, in _get_select_target_counts
cnt, res = self.__resolve_selectable(selectable, query)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python/3.11.7/lib/python3.11/site-packages/sqlfluff/rules/ambiguous/AM07.py", line 166, in __resolve_selectable
_cols, _resolved = self.__resolve_selectable_wildcard(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python/3.11.7/lib/python3.11/site-packages/sqlfluff/rules/ambiguous/AM07.py", line 134, in __resolve_selectable_wildcard
_cols, _resolved = self.__resolve_wild_query(cte)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python/3.11.7/lib/python3.11/site-packages/sqlfluff/rules/ambiguous/AM07.py", line 87, in __resolve_wild_query
return self.__resolve_selectable(query.selectables[0], query)
~~~~~~~~~~~~~~~~~^^^
IndexError: list index out of range

Ill see what I can do about a PR but I'm very new to Open Source stuff 😅

@esenilsson
Copy link

I have the same issue with

  • Snowflake
  • dbt_utils.union_relations
  • include

@sziolko
Copy link
sziolko commented Jun 29, 2024

In my case with a similar issue changing to a pattern of:

with foo as (

    select *
    from
    {{ dbt_utils.union_relations(
...

resolved the error

@Timvde Timvde mentioned this issue Jun 29, 2024
Closed
3 tasks
@keraion keraion mentioned this issue Jul 12, 2024
Merged
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

AM07: Handle set expressions with bracketed selects keraion/sqlfluff
4 participants
@esenilsson @sziolko @danisola @itsjliketheletter
0