8000 Add test of creating cmds archive and retry opening cmds.h5 by taldcroft · Pull Request #197 · sot/kadi · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Add test of creating cmds archive and retry opening cmds.h5 #197

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 3 commits into from
Dec 12, 2020
Merged

Add test of creating cmds archive and retry opening cmds.h5 #197

merged 3 commits into from
Dec 12, 2020

Conversation

taldcroft
Copy link
Member
@taldcroft taldcroft commented Dec 6, 2020 •
edited
Loading

Description

The driver here is to use ska_helpers.retry to avoid the 'Resource temporarily unavailable' error seen occasionally when updating the cmds archive. See also #196 for the "client-side" analog.

In order to test that this was working, I ported the regression test from ska_testr/packages/kadi/test_regress.sh into a unit test. In theory that ska_testr test can be removed (but it isn't hurting anything and runs quickly).

Testing

  • Passes unit tests on MacOS including new unit test (tested in ska3-dev with masters)
  • Functional testing

Functional testing

I forced the open file function to fail twice to see it fail and then subsequently succeed. This was done by calling this stub tables_open_file function instead of tables.open_file.

fail_count = 0
def tables_open_file(*args, **kwargs):
    global fail_count
    if fail_count < 2:
        fail_count += 1
        raise IOError()
    else:
        return tables.open_file(*args, **kwargs)

The output was:

(ska3-shiny) ➜  kadi git:(update-cmds) ✗ pytest -k test_commands -v -s
============================================================== test session starts ==============================================================
platform darwin -- Python 3.8.3, pytest-5.4.3, py-1.9.0, pluggy-0.13.1 -- /Users/aldcroft/miniconda3/envs/ska3-shiny/bin/python
cachedir: .pytest_cache
rootdir: /Users/aldcroft/git/kadi, inifile: pytest.ini
collected 60 items / 51 deselected / 9 selected                                                                                                 

kadi/cmds/tests/test_commands.py::test_find PASSED
kadi/cmds/tests/test_commands.py::test_filter PASSED
kadi/commands/tests/test_commands.py::test_find PASSED
kadi/commands/tests/test_commands.py::test_get_cmds PASSED
kadi/commands/tests/test_commands.py::test_get_cmds_zero_length_result PASSED
kadi/commands/tests/test_commands.py::test_get_cmds_inclusive_stop PASSED
kadi/commands/tests/test_commands.py::test_cmds_as_list_of_dict PASSED
kadi/commands/tests/test_commands.py::test_get_cmds_from_backstop_and_add_cmds PASSED
kadi/commands/tests/test_commands.py::test_commands_regress 2020-12-06 06:58:11,509 ******************************************
2020-12-06 06:58:11,509 Running: /Users/aldcroft/git/kadi/kadi/update_cmds.py
2020-12-06 06:58:11,510 Version: 5.4.1.dev2+g8135838.d20201206
2020-12-06 06:58:11,510 Time: Sun Dec  6 06:58:11 2020
2020-12-06 06:58:11,510 User: root
2020-12-06 06:58:11,510 Machine: sp4-mpt.occ.harvard.edu
2020-12-06 06:58:11,510 Processing args:
2020-12-06 06:58:11,510 {'data_root': '/private/var/folders/zn/910d7qgd1ydd4bvww62b6b9r0000gp/T/pytest-of-aldcroft/pytest-160/test_commands_regress0',
2020-12-06 06:58:11,510  'log_level': 10,
2020-12-06 06:58:11,510  'mp_dir': None,
2020-12-06 06:58:11,510  'start': '2020:159:00:00:00.000',
2020-12-06 06:58:11,510  'stop': '2020:189:00:00:00.000'}
2020-12-06 06:58:11,510 ******************************************
2020-12-06 06:58:11,510 No pars_dict file /private/var/folders/zn/910d7qgd1ydd4bvww62b6b9r0000gp/T/pytest-of-aldcroft/pytest-160/test_commands_regress0/cmds.pkl found, starting from empty dict
2020-12-06 06:58:11,576 Using mission planning files at /Users/aldcroft/ska/data/mpcrit1/mplogs
2020-12-06 06:58:11,583 Found 0 non-load commands between 2020:157:10:40:59.137 : 2020:189:00:00:00.000
2020-12-06 06:58:11,584 Found 28 timelines included within 2020:159:00:00:00.000 to 2020:189:00:00:00.000
2020-12-06 06:58:11,944 Read 1116 commands from /Users/aldcroft/ska/data/mpcrit1/mplogs/2020/JUN0120/oflsa/CR153_0501.backstop
2020-12-06 06:58:11,947   Got 263 backstop commands for timeline_id=426104305 and SCS=129
2020-12-06 06:58:12,010   Got 155 backstop commands for timeline_id=426104306 and SCS=132
2020-12-06 06:58:12,376 Read 1568 commands from /Users/aldcroft/ska/data/mpcrit1/mplogs/2020/JUN0820/oflsa/CR160_0004.backstop
2020-12-06 06:58:12,380   Got 228 backstop commands for timeline_id=426104307 and SCS=130
2020-12-06 06:58:12,446   Got 150 backstop commands for timeline_id=426104308 and SCS=133
2020-12-06 06:58:12,513   Got 268 backstop commands for timeline_id=426104309 and SCS=128
2020-12-06 06:58:12,578   Got 145 backstop commands for timeline_id=426104310 and SCS=131
2020-12-06 06:58:12,645   Got 306 backstop commands for timeline_id=426104311 and SCS=129
2020-12-06 06:58:12,710   Got 407 backstop commands for timeline_id=426104312 and SCS=132
2020-12-06 06:58:13,076 Read 1359 commands from /Users/aldcroft/ska/data/mpcrit1/mplogs/2020/JUN1520/oflsa/CR166_1003.backstop
2020-12-06 06:58:13,079   Got 273 backstop commands for timeline_id=426104313 and SCS=130
2020-12-06 06:58:13,144   Got 133 backstop commands for timeline_id=426104314 and SCS=133
2020-12-06 06:58:13,207   Got 258 backstop commands for timeline_id=426104315 and SCS=128
2020-12-06 06:58:13,271   Got 144 backstop commands for timeline_id=426104316 and SCS=131
2020-12-06 06:58:13,334   Got 244 backstop commands for timeline_id=426104317 and SCS=129
2020-12-06 06:58:13,398   Got 143 backstop commands for timeline_id=426104318 and SCS=132
2020-12-06 06:58:13,461   Got 54 backstop commands for timeline_id=426104319 and SCS=130
2020-12-06 06:58:13,525   Got 35 backstop commands for timeline_id=426104320 and SCS=133
2020-12-06 06:58:13,881 Read 1196 commands from /Users/aldcroft/ska/data/mpcrit1/mplogs/2020/JUN2220/oflsa/CR174_0901.backstop
2020-12-06 06:58:13,885   Got 419 backstop commands for timeline_id=426104321 and SCS=128
2020-12-06 06:58:13,949   Got 281 backstop commands for timeline_id=426104322 and SCS=131
2020-12-06 06:58:14,014   Got 258 backstop commands for timeline_id=426104323 and SCS=129
2020-12-06 06:58:14,212   Got 175 backstop commands for timeline_id=426104324 and SCS=132
2020-12-06 06:58:14,568 Read 1406 commands from /Users/aldcroft/ska/data/mpcrit1/mplogs/2020/JUN2920/oflsa/CR181_0402.backstop
2020-12-06 06:58:14,572   Got 296 backstop commands for timeline_id=426104325 and SCS=130
2020-12-06 06:58:14,638   Got 172 backstop commands for timeline_id=426104326 and SCS=133
2020-12-06 06:58:14,702   Got 370 backstop commands for timeline_id=426104327 and SCS=128
2020-12-06 06:58:14,768   Got 223 backstop commands for timeline_id=426104328 and SCS=131
2020-12-06 06:58:14,831   Got 178 backstop commands for timeline_id=426104329 and SCS=129
2020-12-06 06:58:14,894   Got 109 backstop commands for timeline_id=426104330 and SCS=132
2020-12-06 06:58:15,263 Read 1908 commands from /Users/aldcroft/ska/data/mpcrit1/mplogs/2020/JUL0620/oflsb/CR187_1502.backstop
2020-12-06 06:58:15,269   Got 377 backstop commands for timeline_id=426104331 and SCS=130
2020-12-06 06:58:15,336   Got 242 backstop commands for timeline_id=426104332 and SCS=133
2020-12-06 06:58:15,361 Read total of 384 orbit commands
2020-12-06 06:58:15,365 Read total of 6690 commands (0 non-load commands)
WARNING: tables_open_file(/private/var/folders/zn/910d7qgd1ydd4bvww62b6b9r0000gp/T/pytest-of-aldcroft/pytest-160/test_commands_regress0/cmds.h5, mode=a) exception: , retrying in 1 seconds...
WARNING: tables_open_file(/private/var/folders/zn/910d7qgd1ydd4bvww62b6b9r0000gp/T/pytest-of-aldcroft/pytest-160/test_commands_regress0/cmds.h5, mode=a) exception: , retrying in 4 seconds...
2020-12-06 06:58:16,898 Created h5 cmds table /private/var/folders/zn/910d7qgd1ydd4bvww62b6b9r0000gp/T/pytest-of-aldcroft/pytest-160/test_commands_regress0/cmds.h5
2020-12-06 06:58:16,899 Upated HDF5 cmds table /private/var/folders/zn/910d7qgd1ydd4bvww62b6b9r0000gp/T/pytest-of-aldcroft/pytest-160/test_commands_regress0/cmds.h5
2020-12-06 06:58:16,900 Wrote 608 pars_dict values (608 new) to /private/var/folders/zn/910d7qgd1ydd4bvww62b6b9r0000gp/T/pytest-of-aldcroft/pytest-160/test_commands_regress0/cmds.pkl
PASSED

======================================================= 9 passed, 51 deselected in 9.90s ========================================================

@taldcroft taldcroft merged commit ced97b4 into master Dec 12, 2020
@taldcroft taldcroft deleted the 8BCD update-cmds branch December 12, 2020 19:17
This was referenced Jan 5, 2021
Closed
Merged
@javierggt javierggt mentioned this pull request Mar 2, 2021
Merged
@javierggt javierggt mentioned this pull request Apr 21, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@javierggt javierggt Awaiting requested review from javierggt

@jeanconn jeanconn Awaiting requested review from jeanconn

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@taldcroft
0