8000 "Cannot write" after altering table to add a partition on S3 root buckets · Issue #42 · duckdb/ducklake · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

"Cannot write" after altering table to add a partition on S3 root buckets #42

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
kiwialec opened this issue May 27, 2025 · 8 comments
Closed

"Cannot write" after altering table to add a partition on S3 root buckets #42

kiwialec opened this issue May 27, 2025 · 8 comments

Comments

@kiwialec
Copy link

I'm experiencing an error / bug when using partitioning. This is only mentioned in passing in the docs, so appreciate it might not be functional yet:

ATTACH 'ducklake:metadata2.ducklake' AS my_ducklake (DATA_PATH 's3://some-bucket/');
USE my_ducklake;
CREATE TABLE tester (
foo integer,
bar varchar
);

Inserting and querying at this point works, but after adding a partition

ALTER TABLE tester  SET PARTITIONED BY (foo);

However further inserts fail:

INSERT INTO tester (foo) VALUES (2),(3);

Results in: IO Error: Cannot write to "s3://some-bucket/" - it exists and is a file, not a directory!
@Mytherin
Copy link
Contributor

Thanks for the report!

Can you try running LOAD httpfs explicitly prior to the ATTACH to see if that solves the issue?

@kiwialec
Copy link
Author

The explicit LOAD httpfs; doesn't help - I get the same error.

@Mytherin
Copy link
Contributor

Interesting - does writing in the same configuration work without using partitioning?

@kiwialec
Copy link
Author

Yes - I have no problem using it without partitioning:

D INSTALL ducklake; LOAD httpfs;
D CREATE OR REPLACE SECRET secret (
      TYPE s3,
      PROVIDER credential_chain
  );
┌─────────┐
│ Success │
│ boolean │
├─────────┤
│ true    │
└─────────┘
D ATTACH 'ducklake:metadata.ducklake' AS my_ducklake (DATA_PATH 's3://marcode-test-ducklake/');
D USE my_ducklake;
D CREATE TABLE tester (
  foo integer,
  bar varchar
  );
D INSERT INTO tester (foo) VALUES (2),(3);
D INSERT INTO tester (foo) VALUES (2),(3);
D 
D select * from tester;
┌───────┬─────────┐
│  foo  │   bar   │
│ int32 │ varchar │
├───────┼─────────┤
│     2 │ NULL    │
│     3 │ NULL    │
│     2 │ NULL    │
│     3 │ NULL    │
└───────┴─────────┘
D ALTER TABLE tester  SET PARTITIONED BY (foo);
D select * from tester;
┌───────┬─────────┐
│  foo  │   bar   │
│ int32 │ varchar │
├───────┼─────────┤
│     2 │ NULL    │
│     3 │ NULL    │
│     2 │ NULL    │
│     3 │ NULL    │
└───────┴─────────┘
D INSERT INTO tester (foo) VALUES (2),(3);
IO Error:
Cannot write to "s3://marcode-test-ducklake/" - it exists and is a file, not a directory!

@Tishj
Copy link
Tishj commented May 27, 2025 •
edited
Loading

ducklake/src/storage/ducklake_insert.cpp

Line 325 in d53f5d7

result.file_path = data_path;

data_path is directly options.data_path stored in the DuckLakeCatalog, so this error makes sense, as this path points to the folder

@Mytherin
Copy link
Contributor

Thanks - I managed to reproduce. The issue seems to happen when using a root s3 bucket as a directory (e.g. s3://bucket/). As a work-around you can use a path within a bucket (e.g. s3://bucket/files/).

@Mytherin
Copy link
Contributor

I've pushed a fix in DuckDB here - duckdb/duckdb#17689

@Mytherin Mytherin changed the title "Cannot write" after altering table to add a partition "Cannot write" after altering table to add a partition on S3 root buckets May 28, 2025
@Mytherin
Copy link
Contributor

This is now fixed upstream, it should land in DuckDB v1.3.1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@Mytherin @Tishj @kiwialec
0