Use Python to make text report from html #441

jeanconn · 2024-04-30T18:13:25Z

Description

Use Python to make text report from html

This is intended to support our arm64 build, as I have not quite gotten Perl HTML::TableExtract to build successfully on arm64 and perhaps we just don't need it.

Interface impacts

This changes newlines in the text file output of starcheck.

I think the only thing parsing the text file is the starcheck_parser code, which is functionally checked as part of this PR. Load review also uses diffs of the text file, but this PR does not introduce diffs between the vehicle and full loads.

Testing

Unit tests

Mac

(ska3-flight-2024.1rc4) flame:starcheck jean$ pytest
================================================ test session starts ================================================
platform darwin -- Python 3.11.8, pytest-7.4.4, pluggy-1.4.0
rootdir: /Users/jean/git
configfile: pytest.ini
plugins: timeout-2.2.0, anyio-4.3.0
collected 14 items                                                                                                  

starcheck/tests/test_state_checks.py .............                                                            [ 92%]
starcheck/tests/test_utils.py .                                                                               [100%]

================================================= warnings summary ==================================================
../../miniconda3/envs/ska3-flight-2024.1rc4/lib/python3.11/site-packages/tables/node.py:251
../../miniconda3/envs/ska3-flight-2024.1rc4/lib/python3.11/site-packages/tables/node.py:251
starcheck/starcheck/tests/test_state_checks.py::test_get_obs_man_angle[646940289.224-143.4]
  /Users/jean/miniconda3/envs/ska3-flight-2024.1rc4/lib/python3.11/site-packages/tables/node.py:251: DeprecationWarning: `alltrue` is deprecated as of NumPy 1.25.0, and will be removed in NumPy 2.0. Please use `all` instead.
    self._v_objectid = self._g_open()

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
========================================== 14 passed, 3 warnings in 7.76s ===========================================
(ska3-flight-2024.1rc4) flame:starcheck jean$ git rev-parse HEAD
a9edae5f73b558b0df4230cb5bf33edabc860786

Independent check of unit tests by [REVIEWER NAME]

[PLATFORM]:

Functional tests

jeanconn-fido> pwd
/proj/sot/ska/www/ASPECT_ICXC/test_review_outputs/starcheck/starcheck-pr441
jeanconn-fido> /proj/sot/ska/bin/diff2html MAY1324B_master.txt MAY1324B_master_vehicle.txt > MAY1324B_masters_diff.html
jeanconn-fido> /proj/sot/ska/bin/diff2html MAY1324B_test.txt MAY1324B_test_vehicle.txt > MAY1324B_test_diff.html

The full load to vehicle output diffs in the test outputs are consistent with the same diffs in the 'master' branch diffs:

https://icxc.cfa.harvard.edu/aspect/test_review_outputs/starcheck/starcheck-pr441/MAY1324B_masters_diff.html

https://icxc.cfa.harvard.edu/aspect/test_review_outputs/starcheck/starcheck-pr441/MAY1324B_test_diff.html

The text files as parsed by starcheck.parser show that the new output appears to work better with the parser in that the extra spacing by chance fixes a bug that the maneuvers of the maneuver-only observations were not getting into the dictionary of the parser in the output from master. There's no real impact there as the starcheck parser output is intended to work to create and update a database of star catalogs (and the maneuver-only observations don't have those).

In [1]: import starcheck.parser

In [2]: test = starcheck.parser.read_starcheck('MAY1324B_test.txt')

In [3]: master = starcheck.parser.read_starcheck('MAY1324B_master.txt')

In [4]: from itertools import count

In [6]: for i, t, m in zip(count(), test, master):
   ...:     if t != m:
   ...:         print(i)
   ...: 
12
19
27

In [8]: test[12]
Out[8]: 
{'obsid': 43447,
 'obs': {'obsid': 43447,
  'point_ra': 331.015146,
  'point_dec': 0.739498,
  'point_roll': 108.991505},
 'manvrs': [{'mp_targquat_time': '2024:136:02:52:04.357',
   'mp_targquat_vcdu_cnt': 1737502,
   'target_Q1': 0.78721495,
   'target_Q2': -0.20734763,
   'target_Q3': -0.14024789,
   'target_Q4': 0.56358682,
   'angle_deg': 0.01,
   'duration_sec': 240,
   'end_date': '2024:136:02:56:10',
   'instance': 0}],
 'catalog': {},
 'warnings': [],
 'pred_ccd_temp': None}

In [9]: master[12]
Out[9]: 
{'obsid': 43447,
 'obs': {'obsid': 43447,
  'point_ra': 331.015146,
  'point_dec': 0.739498,
  'point_roll': 108.991505},
 'manvrs': {},
 'catalog': {},
 'warnings': [],
 'pred_ccd_temp': None}

I confirmed that the text block (from the output of master) is consistent with the maneuver information printed for 43447 and parsed.

OBSID: 43447  
RA, Dec, Roll (deg):   331.015146     0.739498   108.991505
BACKSTOP GUIDE_SUMM OR MANVR DOT TLR 

MP_TARGQUAT at 2024:136:02:52:04.357 (VCDU count = 1737502)
  Q1,Q2,Q3,Q4: 0.78721495  -0.20734763  -0.14024789  0.56358682
  MANVR: Angle=   0.01 deg  Duration= 240 sec  End= 2024:136:02:56:10

In [10]: test[19]
Out[10]: 
{'obsid': 43445,
 'obs': {'obsid': 43445,
  'point_ra': 194.184776,
  'point_dec': -57.404593,
  'point_roll': 306.588572},
 'manvrs': [{'mp_targquat_time': '2024:137:05:55:55.852',
   'mp_targquat_vcdu_cnt': 2117722,
   'target_Q1': 0.37707481,
   'target_Q2': 0.44414365,
   'target_Q3': -0.75091653,
   'target_Q4': 0.31092662,
   'angle_deg': 54.89,
   'duration_sec': 1392,
   'end_date': '2024:137:06:19:13',
   'instance': 0}],
 'catalog': {},
 'warnings': [],
 'pred_ccd_temp': None}

In [11]: master[19]
Out[11]: 
{'obsid': 43445,
 'obs': {'obsid': 43445,
  'point_ra': 194.184776,
  'point_dec': -57.404593,
  'point_roll': 306.588572},
 'manvrs': {},
 'catalog': {},
 'warnings': [],
 'pred_ccd_temp': None}

OBSID: 43445  
RA, Dec, Roll (deg):   194.184776   -57.404593   306.588572
BACKSTOP GUIDE_SUMM OR MANVR DOT TLR 

MP_TARGQUAT at 2024:137:05:55:55.852 (VCDU count = 2117722)
  Q1,Q2,Q3,Q4: 0.37707481  0.44414365  -0.75091653  0.31092662
  MANVR: Angle=  54.89 deg  Duration= 1392 sec  End= 2024:137:06:19:13

In [12]: test[27]
Out[12]: 
{'obsid': 43441,
 'obs': {'obsid': 43441,
  'point_ra': 331.015146,
  'point_dec': 0.739498,
  'point_roll': 109.521944},
 'manvrs': [{'mp_targquat_time': '2024:138:18:21:04.357',
   'mp_targquat_vcdu_cnt': 2629365,
   'target_Q1': 0.78981533,
   'target_Q2': -0.20799461,
   'target_Q3': -0.13928659,
   'target_Q4': 0.55993681,
   'angle_deg': 0.01,
   'duration_sec': 240,
   'end_date': '2024:138:18:25:10',
   'instance': 0}],
 'catalog': {},
 'warnings': [],
 'pred_ccd_temp': None}

In [13]: master[27]
Out[13]: 
{'obsid': 43441,
 'obs': {'obsid': 43441,
  'point_ra': 331.015146,
  'point_dec': 0.739498,
  'point_roll': 109.521944},
 'manvrs': {},
 'catalog': {},
 'warnings': [],
 'pred_ccd_temp': None}

OBSID: 43441  
RA, Dec, Roll (deg):   331.015146     0.739498   109.521944
BACKSTOP GUIDE_SUMM OR MANVR DOT TLR 

MP_TARGQUAT at 2024:138:18:21:04.357 (VCDU count = 2629365)
  Q1,Q2,Q3,Q4: 0.78981533  -0.20799461  -0.13928659  0.55993681
  MANVR: Angle=   0.01 deg  Duration= 240 sec  End= 2024:138:18:25:10

And everything else is identical.

taldcroft

About the diffs, the description says "minor diffs" but I did the diff between master and test and found only diffs in the number of newlines. So you should update the description to make that much stronger statement.

About the suggested code change, you don't need to re-do all the testing but just run once and confirm that the test text output does not change.

starcheck/utils.py

Use Tom's list comprehension for the output. Co-authored-by: Tom Aldcroft <taldcroft@gmail.com>

taldcroft · 2024-05-13T13:52:53Z

This just needs documentation that the output for MAY1324B is unchanged from the last commit.

jeanconn · 2024-05-13T13:54:32Z

Right, I just actually need to do that first 😄

jeanconn · 2024-05-13T16:17:34Z

I reran the output at the previous "last commit" (because the agasc supplement has changed) and confirmed just expected diffs.

ska3-jeanconn-fido> diff MAY1324B_test.txt MAY1324B_rtest.txt
1,2c1,2
<  ------------  Starcheck 14.8.2.dev1+ga9edae5    -----------------
<  Run on Mon May 13 12:05:54 EDT 2024 by jeanconn from fido.cfa.harvard.edu
---
>  ------------  Starcheck 14.8.2.dev4+g21ef617    -----------------
>  Run on Mon May 13 12:15:22 EDT 2024 by jeanconn from fido.cfa.harvard.edu

Use Python to make text from html

a9edae5

jeanconn marked this pull request as ready for review May 10, 2024 14:35

jeanconn requested review from taldcroft and javierggt May 10, 2024 14:35

taldcroft approved these changes May 11, 2024

View reviewed changes

starcheck/utils.py Outdated Show resolved Hide resolved

jeanconn and others added 2 commits May 13, 2024 09:48

Update starcheck/utils.py

3066b6e

Use Tom's list comprehension for the output. Co-authored-by: Tom Aldcroft <taldcroft@gmail.com>

Fix indent

48906c8

Fix wrong name in join

21ef617

jeanconn merged commit 144485d into master May 13, 2024

jeanconn deleted the html branch May 13, 2024 16:18

javierggt mentioned this pull request May 13, 2024

Update sot/starcheck to 14.9.0 sot/skare3#1348

Closed

jeanconn mentioned this pull request May 20, 2024

2024.5 sot/skare3#1350

Merged

javierggt mentioned this pull request Jun 26, 2024

2024.6 sot/skare3#1363

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use Python to make text report from html #441

Use Python to make text report from html #441

Use Python to make text report from html #441

Use Python to make text report from html #441

Conversation

Description

Interface impacts

Testing

Unit tests

Functional tests

Choose a reason for hiding this comment