8000 Add overwrite=True to cheta sync index table write statement by taldcroft · Pull Request #248 · sot/cheta · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Add overwrite=True to cheta sync index table write statement #248

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Apr 4, 2023
Merged

Add overwrite=True to cheta sync index table write statement #248

merged 1 commit into from
Apr 4, 2023

Conversation

taldcroft
Copy link
Member
@taldcroft taldcroft commented Mar 29, 2023 •
edited by jeanconn
Loading

Description

Daily cheta cron processing failed when updating the cheta sync archive. This is due to a missing overwrite=True arg in the astropy Table write() method. This was not seen in testing because the test only does one "update" into a fresh directory. This could be improved...

2023-03-29 04:49:05,151 Writing 1 row(s) to index file /proj/sot/ska3/flight/data/eng_archive/sync/acis2eng/index.ecsv
Traceback (most recent call last):
  File "/proj/sot/ska3/flight/bin/cheta_update_server_sync", line 33, in <module>
    sys.exit(load_entry_point('cheta==4.58.0', 'console_scripts', 'cheta_update_server_sync')())
  File "/proj/sot/ska3/flight/lib/python3.10/site-packages/cheta/update_server_sync.py", line 135, in main
    update_sync_repo(opt, logger, content)
  File "/proj/sot/ska3/flight/lib/python3.10/site-packages/cheta/update_server_sync.py", line 187, in update_sync_repo
    index_tbl = update_index_file(index_file, opt, logger)
  File "/proj/sot/ska3/flight/lib/python3.10/site-packages/cheta/update_server_sync.py", line 335, in update_index_file
    index_tbl.write(index_file, format="ascii.ecsv")
  File "/proj/sot/ska3/flight/lib/python3.10/site-packages/astropy/table/connect.py", line 130, in __call__
    self.registry.write(instance, *args, **kwargs)
  File "/proj/sot/ska3/flight/lib/python3.10/site-packages/astropy/io/registry/core.py", line 385, in write
    return writer(data, *args, **kwargs)
  File "/proj/sot/ska3/flight/lib/python3.10/site-packages/astropy/io/ascii/connect.py", line 28, in io_write
    return write(table, filename, **kwargs)
  File "/proj/sot/ska3/flight/lib/python3.10/site-packages/astropy/io/ascii/ui.py", line 940, in write
    raise OSError(NOT_OVERWRITING_MSG.format(output))
OSError: File /proj/sot/ska3/flight/data/eng_archive/sync/acis2eng/index.ecsv already exists. If you mean to replace it then use the argument "overwrite=True".

Interface impacts

None

Testing

Unit tests

  • Mac

Independent check of unit tests by Jean

  • Linux

Functional tests

On HEAD

ska3-kady$ cheta_update_server_sync  --sync-root ${SKA}/data/eng_archive --content=acis2eng
2023-03-29 06:15:20,234 Writing 1 row(s) to index file /proj/sot/ska3/flight/data/eng_archive/sync/acis2eng/index.ecsv
2023-03-29 06:15:21,881 Writing /proj/sot/ska3/flight/data/eng_archive/sync/acis2eng/2023-03-28T0245z/full.pkl.gz with 2603 rows of data and 106 msids
2023-03-29 06:15:23,395 Writing /proj/sot/ska3/flight/data/eng_archive/sync/acis2eng/2023-03-28T0245z/5min.pkl.gz with 261 rows of data and 101 msids
2023-03-29 06:15:25,958 Writing /proj/sot/ska3/flight/data/eng_archive/sync/acis2eng/2023-03-28T0245z/daily.pkl.gz with 0 rows of data and 101 msids
2023-03-29 06:15:25,963 Removing sync directory /proj/sot/ska3/flight/data/eng_archive/sync/acis2eng/2023-01-29T0334z
2023-03-29 06:15:25,981 Writing 60 row(s) to index file /proj/sot/ska3/flight/data/eng_archive/sync/acis2eng/index.ecsv
2023-03-29 06:15:26,021 Writing contents pickle /proj/sot/ska3/flight/data/eng_archive/sync/msid_contents.pkl.gz

On laptop

(ska3) ➜  ~ cheta_sync --content acis2eng
2023-03-29 06:17:17,538 Running cheta_update_client_archive version 4.58.0
2023-03-29 06:17:17,539   /Users/aldcroft/miniconda3/envs/ska3/lib/python3.10/site-packages/cheta/update_client_archive.py
2023-03-29 06:17:17,539 
2023-03-29 06:17:17,539 Updating client archive at /Users/aldcroft/ska/data/eng_archive
2023-03-29 06:17:17,801 Reading index file https://icxc.cfa.harvard.edu/aspect/cheta/sync/acis2eng/index.ecsv
2023-03-29 06:17:17,930 
2023-03-29 06:17:17,931 Processing full data for acis2eng
2023-03-29 06:17:18,102 Reading update date file https://icxc.cfa.harvard.edu/aspect/cheta/sync/acis2eng/2023-03-28T0245z/full.pkl.gz
2023-03-29 06:17:18,108   elapsed time: 0.006 sec
2023-03-29 06:17:18,110 Applying updates to 105 h5 files
2023-03-29 06:17:18,840   elapsed time: 0.730 sec
2023-03-29 06:17:18,841 Updating /Users/aldcroft/ska/data/eng_archive/data/acis2eng/archfiles.db3
2023-03-29 06:17:18,854   elapsed time: 0.013 sec
2023-03-29 06:17:18,855 
2023-03-29 06:17:18,855 Processing 5min data for acis2eng
2023-03-29 06:17:19,024 Reading update date file https://icxc.cfa.harvard.edu/aspect/cheta/sync/acis2eng/2023-03-28T0245z/5min.pkl.gz
2023-03-29 06:17:19,028   elapsed time: 0.004 sec
2023-03-29 06:17:19,031 Applying updates to 101 h5 files
2023-03-29 06:17:19,925   elapsed time: 0.894 sec
2023-03-29 06:17:19,926 
2023-03-29 06:17:19,926 Processing daily data for acis2eng
2023-03-29 06:17:20,045 Reading update date file https://icxc.cfa.harvard.edu/aspect/cheta/sync/acis2eng/2023-03-28T0245z/daily.pkl.gz
2023-03-29 06:17:20,045   elapsed time: 0.000 sec

@taldcroft taldcroft requested review from javierggt and jeanconn April 3, 2023 20:28
jeanconn
jeanconn approved these changes Apr 4, 2023
View reviewed changes
Copy link
Contributor
@jeanconn jeanconn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems fine. What is the thought on the priority of adding a test to do "updates" that would have caught this?

@taldcroft
Copy link
Member Author

What is the thought on the priority of adding a test to do "updates" that would have caught this?

There is some merit in having the test do a couple of updates to run through code that works on an existing sync archive. Maybe if I'm in that code again and remember then it'll happen but otherwise I'm not planning to prioritize it now.

@taldcroft taldcroft merged commit dc46a5e into master Apr 4, 2023
@taldcroft taldcroft deleted the fix-overwrite-index branch April 4, 2023 16:25
This was referenced Jun 1, 2023
Closed
Merged
@javierggt javierggt mentioned this pull request Jul 12, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jeanconn jeanconn jeanconn approved these changes

@javierggt javierggt Awaiting requested review from javierggt

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@taldcroft @jeanconn
0