8000 Make JSON Lines sink support ".tar.gz" formatting. · Issue #505 · biolink/kgx · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Make JSON Lines sink support ".tar.gz" formatting. #505

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
DnlRKorn opened this issue Apr 8, 2025 · 0 comments
Open

Make JSON Lines sink support ".tar.gz" formatting. #505

DnlRKorn opened this issue Apr 8, 2025 · 0 comments
Labels
enhancement New feature or request

Comments

@DnlRKorn
Copy link
DnlRKorn commented Apr 8, 2025

Is your feature request related to a problem? Please describe.
I would like to be able to run a KGX transform where JSON Lines files are exported as a singular tar.gz file.
At this moment if run monarch-ingest and cat-merge to get some TSVs and then run a KGX transform command like

poetry run kgx transform --input-format tsv --input-compression tar.gz --output output/mokg-jsonl --output-format jsonl --output-compression tar.gz output/mokg.tar.gz
The "output-compression" parameter will end up ignored and the files will be outputted from this command will be two files; output/mokg-jsonl_nodes.jsonl.tar.gz and output/mokg-json_edges.jsonl.tar.gz.

Describe the solution you'd like
Allow the JSONL writer to be able to handle tar.gz as an output compression.

Describe alternatives you've considered
Make JSONL writer fail when passing in invalid compression parameter.

Additional context
Code pointing to TSV Sink Compression code right now.

kgx/kgx/sink/tsv_sink.py

Lines 138 to 149 in f5c8ab9

if self.mode:
archive_basename = f"{self.basename}.{archive_format[self.mode]}"
archive_name = os.path.join(
self.dirname if self.dirname else "", archive_basename
)
with tarfile.open(name=archive_name, mode=self.mode) as tar:
tar.add(self.nodes_file_name, arcname=self.nodes_file_basename)
tar.add(self.edges_file_name, arcname=self.edges_file_basename)
if os.path.isfile(self.nodes_file_name):
os.remove(self.nodes_file_name)
if os.path.isfile(self.edges_file_name):
os.remove(self.edges_file_name)

Code pointing to JSON Lines Sink Compression code right now.

kgx/kgx/sink/jsonl_sink.py

Lines 49 to 55 in f5c8ab9

if compression == "gz":
nodes_filename += f".{compression}"
edges_filename += f".{compression}"
NFH = gzip.open(nodes_filename, "wb")
self.NFH = jsonlines.Writer(NFH)
EFH = gzip.open(edges_filename, "wb")
self.EFH = jsonlines.Writer(EFH)

Notice that JSONL doesn't support tar.gz compression format.

kgx/kgx/sink/jsonl_sink.py

Lines 23 to 24 in f5c8ab9

compression: Optional[str]
The compression type (``gz``)

@DnlRKorn DnlRKorn added the enhancement New feature or request label Apr 8, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@DnlRKorn
0