8000 added magnitude estimation (formerly known as mag-stats) by javierggt · Pull Request #48 · sot/agasc · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

added magnitude estimation (formerly known as mag-stats) #48

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 12 commits into from
Nov 12, 2020

Conversation

javierggt
Copy link
Contributor
@javierggt javierggt commented Oct 20, 2020

Description

This is a modified version of what is in the mica mag-stats branch. I renamed most modules and functions.

Testing

  • Passes unit tests on MacOS, linux, Windows (at least one required)
  • Functional testing

Fixes #

Copy link
Member
@taldcroft taldcroft left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quick first comments, I just realized this is out there and we want to get it into shiny.

agasc/agasc.py Outdated
return t[0]


def get_stars(ids, agasc_file=None, dates=None, fix_color1=True):
def get_stars(ids, agasc_file=None, dates=None, fix_color1=True, supplement_file=None):
Copy link
Member

Choose a 8000 reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not needed now.

STARS_OBS_NP = None


def _load_startcat_commands(tstop=None):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typo in the name?

@taldcroft
Copy link
Member

@javierggt - I'm working on a commit to implement my suggestions, so don't worry about this.

@taldcroft
Copy link
Member

The current dtype of the entries in the mags table in the supplement is:

      dtype=[('agasc_id', '<i8'), ('color', '<f4'), ('mag_aca', '<f8'), ('mag_aca_err', '<f8'), ('last_obs_time', '<f8')]

I would suggest instead:

      dtype=[('agasc_id', '<i4'),  ('mag_aca', '<f4'), ('mag_aca_err', '<f4'), ('last_obs_time', '<f8')])

@taldcroft
Copy link
Member

@javierggt - I committed a new approach for doing AGASC queries that also includes tests. I also just realized that having the use_mag_est parameter will be helpful in proseco unit testing because we can force that to be False for testing and have reproducible results (always a good thing).

@javierggt
Copy link
Contributor Author

I'm good with that. I will start using this.

@taldcroft
Copy link
Member

The existing file /proj/sot/ska/www/ASPECT_ICXC/tmp/jgonzalez/mag_stats/agasc_supplement.h5 does not have the original bad table leaf.

@javierggt javierggt force-pushed the magnitude-supplement branch from 0022822 to b543863 Compare November 6, 2020 21:01
@taldcroft taldcroft requested a review from jeanconn November 9, 2020 15:35
:return: dict
dictionary with stats
"""
min_mag_obs_err = 0.3
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe this is the bit that needs to be 0.03?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed.

@jeanconn jeanconn mentioned this pull request Nov 10, 2020
3 tasks
Copy link
Contributor
@jeanconn jeanconn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With a review of the testing and light review of the agasc/agasc.py changes, this seems fine to me. I do wonder:

  1. would we be better off with downstream fixes instead of the old and new hacks on COLOR1?
  2. do we still want the color1=1.5 acquisition grid model for a star that (before this code changed its value to 1.49) had COLOR1=1.5? I wasn't sure how much of that model selection was based on larger mag error and how much was based on redder stars (not that we can tease those apart completely) I'm also not sure how different the models in there are at this point.

@taldcroft
Copy link
Member

No we do not want the color1=1.5 model for stars that have an estimated mag. The ONLY reason for the distinct model is that the mag_errs for color1=1.5 stars follow a completely different distribution. Once we know the mag of a star then it is no longer in that weird distribution, instead it follows a gaussian distribution (more-or-less) with MAG_ACA_ERR. The red-ness of a star is not relevant once you have a mag.

@taldcroft
Copy link
Member

would we be better off with downstream fixes instead of the old and new hacks on COLOR1?

What do you mean by downstream fixes?

@jeanconn
Copy link
Contributor

By downstream fixes, I meant changing the handling in proseco sparkles starcheck chandra_aca to be more explicit instead of implicit... basically to remove anything that is using COLOR1=1.5 or COLOR1=0.7 checks. But I suppose with the current agasc and supplement, we'd be left with a bit of a mess of logic around RSV1, COLOR2, MAG_CATID, unless we added a new column that somewhat defined "does this need extra mag error padding or special handing in the acq model".

I understand that this hack in the agasc code to rewrite COLOR1 works fine with code in the wild like:

https://github.com/sot/proseco/blob/7ad393513387cbe8e08396793ecc5c144f6ace12/proseco/core.py#L1130

just also wondering if that code in proseco really belongs in the agasc module.

Maybe my point in the end is that this agasc change should also kick off an audit/sanity check of how we're using COLOR1 in our code.

@javierggt
Copy link
Contributor Author

I guess downstream would mean changes in proseco.

I added the 'color' column to the supplement, which in principle allows you to specify color in the supplement as well, but I can remove that today since it is not used.

@taldcroft
Copy link
Member

I added the 'color' column to the supplement, which in principle allows you to specify color in the supplement as well, but I can remove that today since it is not used.

Yes, color does not belong in the supplement since we do not measure it with the ACA.

@javierggt
Copy link
Contributor Author

removed it.

@taldcroft
Copy link
Member

Indeed we would need a new column which basically says that a star behaves like a color=0.7 or color=1.5 star. Given all the code out there (and not just in a few core packages, but probably dozens of little scripts and whatnot) that uses this existing convention, I don't see a big advantage when the end result will be exactly the same. Doing the upstream single-source fix in the agasc package and sticking with the legacy encoding that is embedded in the color seems cleanest to me.

I agree we might want to migrate that fix in proseco up to agasc.

@jeanconn
Copy link
Contributor

Sounds good. Thanks for having the conversation about it.

@taldcroft taldcroft merged commit c1a011b into master Nov 12, 2020
@taldcroft taldcroft deleted the magnitude-supplement branch November 12, 2020 16:05

- ``MAG_ACA`` and ``MAG_ACA_ERR`` are set according to the supplement.
- ``MAG_CATID`` (mag catalog ID) is set to ``MAG_CATID_SUPPLEMENT`` (128).
- If COLOR1 is 0.7 or 1.5 then it is changed to 0.69 or 1.49 respectively.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure if we might want a clarifying comment indicating "if the COLOR1 hasn't already been updated based on APASS" or some such.

@javierggt javierggt mentioned this pull request Dec 7, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants
0