10000 Configure kadi logger globally + other refactoring by taldcroft · Pull Request #400 · sot/starcheck · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content
8000

Configure kadi logger globally + other refactoring #400

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Nov 4, 2022

Conversation

taldcroft
Copy link
Member
@taldcroft taldcroft commented Oct 16, 2022

Description

Fixes the issue noted on slack (#missionplanning "SOT MP run of starcheck did not generate the expected output files for the OCT1722B products"), that get_dither_kadi_state would run before KADI_SCENARIO=flight was set, causing HEAD starcheck to still have a network dependence for SOT MP's run of the tool.

In the course of implementing the fix, some refactoring and rework of existing code was useful.

Interface impacts

Testing

Unit tests

  • No unit tests

Functional tests

Disabled occweb access and confirmed that master fails to use 'flight' scenario:

Traceback (most recent call last):
  File "/proj/sot/ska/jeanproj/git/starcheck/starcheck/utils.py", line 26, in wrapper
    return func(*args, **kwargs)
  File "/proj/sot/ska/jeanproj/git/starcheck/starcheck/utils.py", line 100, in get_dither_kadi_state
    state = states.get_continuity(date, cols)
  File "/proj/sot/ska3/flight/lib/python3.8/site-packages/kadi/commands/states.py", line 1697, in get_continuity
    cmds = commands.get_cmds(start, stop, scenario=scenario)
  File "/proj/sot/ska3/flight/lib/python3.8/site-packages/kadi/commands/commands.py", line 54, in get_cmds
    cmds = get_cmds_(start=start, stop=stop, inclusive_stop=inclusive_stop, **kwargs)
  File "/proj/sot/ska3/flight/lib/python3.8/site-packages/kadi/commands/commands_v2.py", line 216, in get_cmds
    cmds_recent = update_archive_and_get_cmds_recent(
  File "/proj/sot/ska3/flight/lib/python3.8/site-packages/kadi/commands/commands_v2.py", line 312, in update_archive_and_get_cmds_recent
    loads = update_loads(scenario, cmd_events=cmd_events, lookback=lookback, stop=stop)
  File "/proj/sot/ska3/flight/lib/python3.8/site-packages/kadi/commands/commands_v2.py", line 911, in update_loads
    contents = occweb.get_occweb_dir(dir_year_month)
  File "/proj/sot/ska3/flight/lib/python3.8/site-packages/kadi/occweb.py", line 322, in get_occweb_dir
    html = get_occweb_page(path, timeout=timeout, cache=cache)
  File "/proj/sot/ska3/flight/lib/python3.8/site-packages/kadi/occweb.py", line 290, in get_occweb_page
    req.raise_for_status()  # raise exception if not 200
  File "/proj/sot/ska3/flight/lib/python3.8/site-packages/requests/models.py", line 960, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://occweb.cfa.harvard.edu/occweb/FOT/mission_planning/PRODUCTS/APPR_LOADS/2022/SEP
HTTPError: 401 Client Error: Unauthorized for url: https://occweb.cfa.harvard.edu/occweb/FOT/mission_planning/PRODUCTS/APPR_LOADS/2022/SEP at line 26
 at ./starcheck/src/starcheck.pl line 37.
	main::__ANON__("HTTPError: 401 Client Error: Unauthorized for url: https://oc"...) called at (eval 184) line 3
	main::get_dither_kadi_state("__main__", "get_dither_kadi_state", "2022:289:13:17:00.000") called at ./starcheck/src/starcheck.pl line 274

Confirmed that this PR (with the same disabled .netrc etc) works and that KADI_SCENARIO is set to flight before/for all kadi command queries.

ska3-jeanconn-fido> ./sandbox_starcheck -dir /data/mpcrit1/mplogs/2022/OCT1722/oflsb/ -out ob_test
Using backstop file /data/mpcrit1/mplogs/2022/OCT1722/oflsb//CR289_1303.backstop
Using guide summary file /data/mpcrit1/mplogs/2022/OCT1722/oflsb//mps/mgOCT1722B.sum
Using OR file /data/mpcrit1/mplogs/2022/OCT1722/oflsb//mps/or/OCT1722_B.or
Using maneuver file /data/mpcrit1/mplogs/2022/OCT1722/oflsb//mps/mmOCT1722B.sum
Using DOT file /data/mpcrit1/mplogs/2022/OCT1722/oflsb//mps/mdOCT1722B.dot
Using mech check file /data/mpcrit1/mplogs/2022/OCT1722/oflsb//output/TEST_mechcheck.txt
Using fidsel file /data/mpcrit1/mplogs/2022/OCT1722/oflsb//History/FIDSEL.txt
Using dither file /data/mpcrit1/mplogs/2022/OCT1722/oflsb//History/DITHER.txt
Using radmon file /data/mpcrit1/mplogs/2022/OCT1722/oflsb//History/RADMON.txt
Using simtrans file /data/mpcrit1/mplogs/2022/OCT1722/oflsb//History/SIMTRANS.txt
Using simfocus file /data/mpcrit1/mplogs/2022/OCT1722/oflsb//History/SIMFOCUS.txt
Using attitude file /data/mpcrit1/mplogs/2022/OCT1722/oflsb//History/ATTITUDE.txt
Using characteristics file /data/mpcrit1/mplogs/2022/OCT1722/oflsb//mps/ode/characteristics/CHARACTERIS_18JAN21
Using aimpoint file /data/mpcrit1/mplogs/2022/OCT1722/oflsb//output/OCT1722B_dynamical_offsets.txt
Using config file /proj/sot/ska/jeanproj/git/starcheck/starcheck/data/characteristics.yaml
Using odb file /proj/sot/ska/jeanproj/git/starcheck/starcheck/data/fid_CHARACTERISTICS
Using agasc_file file /proj/sot/ska3/flight/data/agasc/proseco_agasc_1p7.h5
Using manerr file /data/mpcrit1/mplogs/2022/OCT1722/oflsb//output/OCT1722B_ManErr.txt
Using processing summary file /data/mpcrit1/mplogs/2022/OCT1722/oflsb//mps/msOCT1722B.sum
Using TLR file /data/mpcrit1/mplogs/2022/OCT1722/oflsb//CR289_1303.tlr
Using banned_agasc file /proj/sot/ska/jeanproj/git/starcheck/starcheck/data/agasc.bad
Using bad_pixel file /proj/sot/ska/jeanproj/git/starcheck/starcheck/data/ACABadPixels
Using acq_star_rdb file /proj/sot/ska/jeanproj/git/starcheck/starcheck/data/bad_acq_stars.rdb
Using gui_star_rdb file /proj/sot/ska/jeanproj/git/starcheck/starcheck/data/bad_gui_stars.rdb
Getting commands from archive only
Loaded /proj/sot/ska3/flight/data/kadi/cmds2.h5 with 1431351 commands
Loaded /proj/sot/ska3/flight/data/kadi/cmds2.pkl with 147968 pars
Read 258 ACA bad pixels from /proj/sot/ska/jeanproj/git/starcheck/starcheck/data/ACABadPixels
Read 46 bad AGASC IDs from /proj/sot/ska/jeanproj/git/starcheck/starcheck/data/agasc.bad
#####################################################################
# calc_ccd_temps run at Tue Oct 18 10:36:10 2022 by jeanconn
# Continuity run_start_time = 2022:291:14:36:09.564
# calc_ccd_temps version = 13.16.1.dev3+g75cba91
# chandra_models version = 3.43
# kadi version = 7.0.2
#####################################################################

Using backstop file /data/mpcrit1/mplogs/2022/OCT1722/oflsb/CR289_1303.backstop
Found 2274 backstop commands between 2022:289:13:17:00.000 and 2022:297:11:45:00.000
RLTT = 2022:289:13:20:00.000
sched_stop = 2022:297:11:45:00.000
Fetching telemetry between 2022:288:13:17:00.000 and 2022:289:13:17:00.000
Getting commands from archive only
Getting commands from archive only
Calculating ACA thermal model
Propagation initial time and ACA: 2022:289:13:02:46.816 -8.76
Making temperature check plots
Writing plot file ob_test/ccd_temperature.png
Checking star catalog for obsid 45165
Checking star catalog for obsid 45164
Checking star catalog for obsid 45162
Checking star catalog for obsid 45161
Checking star catalog for obsid 45160
Checking star catalog for obsid 45159
Checking star catalog for obsid 26641
Checking star catalog for obsid 27505
Checking star catalog for obsid 25244
Checking star catalog for obsid 27517
Checking star catalog for obsid 27512
Checking star catalog for obsid 27514
Checking star catalog for obsid 26031
Checking star catalog for obsid 27271
Checking star catalog for obsid 27305
Checking star catalog for obsid 27306
Checking star catalog for obsid 27307
Checking star catalog for obsid 27308
Checking star catalog for obsid 27309
Checking star catalog for obsid 27310
Checking star catalog for obsid 27311
Checking star catalog for obsid 45158
Checking star catalog for obsid 45157
Checking star catalog for obsid 45156
Checking star catalog for obsid 45154
Checking star catalog for obsid 45153
Checking star catalog for obsid 45152
Checking star catalog for obsid 45151
Checking star catalog for obsid 25805
Checking star catalog for obsid 25194
Checking star catalog for obsid 27515
Checking star catalog for obsid 27511
Checking star catalog for obsid 25806
Checking star catalog for obsid 27506
Checking star catalog for obsid 25294
Checking star catalog for obsid 25310
Checking star catalog for obsid 27509
Checking star catalog for obsid 27272
Checking star catalog for obsid 27273
Checking star catalog for obsid 27312
Checking star catalog for obsid 27313
Checking star catalog for obsid 27314
Checking star catalog for obsid 27315
Checking star catalog for obsid 27316
Checking star catalog for obsid 27337
Checking star catalog for obsid 27338
Checking star catalog for obsid 27339
Checking star catalog for obsid 45150
Checking star catalog for obsid 45149
Checking star catalog for obsid 45148
Checking star catalog for obsid 45146
Checking star catalog for obsid 45145
Checking star catalog for obsid 45144
Checking star catalog for obsid 45143
Checking star catalog for obsid 25111
Checking star catalog for obsid 27510
Checking star catalog for obsid 23863
Checking star catalog for obsid 27516
Checking star catalog for obsid 27508
Checking star catalog for obsid 27513
Checking star catalog for obsid 27507
Checking star catalog for obsid 27482
Checking star catalog for obsid 25550
Checking star catalog for obsid 27317
Checking star catalog for obsid 27318
Checking star catalog for obsid 27319
Checking star catalog for obsid 27320
Checking star catalog for obsid 27321
Checking star catalog for obsid 24407
Checking star catalog for obsid 45142
Getting commands from archive only
Wrote HTML report to ob_test.html
Wrote text report to ob_test.txt

Testing of 403a1d6

By default the output does not include any kadi logging output. Confirmed this with:

./sandbox_starcheck -dir ~/ska/data/mpcrit1/mplogs/2022/OCT1722/oflsb/ -out ob_test 

Also ran with -verbose=0 and got the same results. Then ran with -verbose=2 and got kadi DEBUG output:

...
Using acq_star_rdb file /Users/aldcroft/git/starcheck/starcheck/data/bad_acq_stars.rdb
Using gui_star_rdb file /Users/aldcroft/git/starcheck/starcheck/data/bad_gui_stars.rdb
Getting cmd_events from https://docs.google.com/spreadsheets/d/19d6XqBhWoFjC-z1lS1nM6wLE_zjr4GYB1lOvrEGCbKQ/export?format=csv
Writing 53 cmd_events to /Users/aldcroft/.kadi/cmd_events.csv
Getting OCCweb FOT/mission_planning/PRODUCTS/APPR_LOADS/2022/OCT with cache=False
Already have /Users/aldcroft/.kadi/loads/OCT1022A.pkl.gz
Already have /Users/aldcroft/.kadi/loads/OCT1722B.pkl.gz
Already have /Users/aldcroft/.kadi/loads/OCT2122A.pkl.gz
Already have /Users/aldcroft/.kadi/loads/OCT2422A.pkl.gz
Already have /Users/aldcroft/.kadi/loads/OCT2422B.pkl.gz
Already have /Users/aldcroft/.kadi/loads/OCT2722A.pkl.gz
Already have /Users/aldcroft/.kadi/loads/OCT3122A.pkl.gz
Getting OCCweb FOT/mission_planning/PRODUCTS/APPR_LOADS/2022/NOV with cache=False
No OCCweb directory for FOT/mission_planning/PRODUCTS/APPR_LOADS/2022/NOV
Including loads OCT1022A, OCT1722B, OCT2122A, OCT2422A, OCT2422B, OCT2722A, OCT3122A
Load OCT1022A has 2143 commands with RLTT=2022:283:02:40:34.206
Load OCT1722B has 2274 commands with RLTT=2022:289:13:20:00.000
... and so on

Previous behavior reported by @jeanconn, NO LONGER accurate

The new STDERR output has "Getting commands from archive only" whenever a call was made to kadi commands. Seems fine. The first call also shows when the commands were loaded (happens once)

Loaded /proj/sot/ska3/flight/data/kadi/cmds2.h5 with 1431351 commands
Loaded /proj/sot/ska3/flight/data/kadi/cmds2.pkl with 147968 pars

This is fine as well.

@jeanconn
Copy link
Contributor

What was the hidden dependence? Or, equivalently, how does this fix it?

@taldcroft
Copy link
Member Author

The call to kadi get_cmds to set the dither history. So not exactly "hidden", but we both missed this in my patch that just addressed the call in calc_ccd_temps.

@taldcroft
Copy link
Member Author

And the fix is to set the environment variable early in the perl script before anything else happens.

@jeanconn
Copy link
Contributor

Makes sense. Thanks!

@taldcroft taldcroft changed the title WIP: Configure kadi logger globally + other refactoring Configure kadi logger globally + other refactoring Oct 17, 2022
@taldcroft
Copy link
Member Author

@jeanconn - just a reminder here that you should take over the functional testing of this, which I can then review.

@jeanconn jeanconn self-assigned this Oct 17, 2022
logger = logging.getLogger(name)
logger.setLevel(loglevel)

# Remove existing handlers if calc_ccd_temps is called multiple times
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is no longer in calc_ccd_temps so I assume this comment should go. Not 100% if the code is right for it as well but...

logger.addHandler(console)

filehandler = logging.FileHandler(
filename=os.path.join(outdir, 'run.dat'), mode='w')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And now this happens before outdir is made, so should perhaps make outdir here as needed.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I updated to just make the output directory here for now.

@jeanconn
Copy link
Contributor

@taldcroft I don't think I can request a review on it from you, but is that what you had in mind for functional testing or would you like some other variations?

@jeanconn
Copy link
Contributor

@taldcroft figured I'd check once more about this now that you are back.

@javierggt javierggt mentioned this pull request Nov 2, 2022
29 tasks
@taldcroft
Copy link
Member Author

@jeanconn - I have reviewed your functional testing and it looks fine. I pushed another commit so I'll re-request a final review from you. I'll update the functional test description momentarily.

@taldcroft taldcroft requested a review from jeanconn November 3, 2022 16:34
@jeanconn jeanconn merged commit 8fed863 into master Nov 4, 2022
@jeanconn jeanconn deleted the fix-kadi-scenario-etc branch November 4, 2022 13:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants
0