List view
It'd be beneficial to create some basica support for a way to describe on a high level the "workflow" or steps ASGS would run as the forecast loop. I am envisioning something similar to a docker compose file (basically YAML) that is aware of the local ASGS environment (ref: https://docs.docker.com/compose/compose-file/). Might also revisit "securerun".
No due dateStuff for making ASGS more portable and empowering users more
No due date•9/10 issues closed``` asgs_main.sh: # TODO: This function should autodetect the hotstart file format, asgs_main.sh:# TODO: Refactor this code so that it runs one serial script that does asgs_main.sh: # and adcprep it. TODO: When real time flux data become available, asgs_main.sh: # TODO: Autodetect the format of the hotstart files to read (the asgs_main.sh: # jgfdebug: TODO: FIXME: Hardcoded the time varying weirs input file asgs_main.sh:# TODO: This should be refactored and streamlined as described in the TODO cloud/general/asgs-brew.pl:# TODO - add in preconditions, need per step validation of required options/params cloud/general/asgsh_profiles/asgsh_profile-ar: JOBENV=( gmt.sh gdal.sh imagemagick.sh ) #TODO doc/paper/asgs.tex:The ADCIRC model offers eleven (TODO: IS THIS THE NUMBER) key doc/paper/asgs.tex:Has been used for many years (TODO: how many) for studies in doc/paper/fleming_asgs_ecm13.tex:The ADCIRC model offers eleven (TODO: IS THIS THE NUMBER) key doc/paper/fleming_asgs_ecm13.tex:Has been used for many years (TODO: how many) for studies in get_flux.pl: # TODO: error message so the ASGS can retry? get_nam.pl:# TODO : eventually get this parameter setting from scenario.json instead of run.properties input/awip_lambert_interp.F:C + TODO: Eliminate floating point comparison tests in main program, input/data_assimilation/loadProcessedStations.m:% TODO: Remove reference stations that don't have enough observations. input/data_assimilation/offsetSurfaceGen.m:% jgf: TODO: need to document the data are in these mat files input/lambertInterpRamp.f:C + TODO: Eliminate floating point comparison tests in main program, output/POSTPROC_KMZGIS/POST_SCRIPT_Corps.sh:# platform. TODO: This program contains several obvious bugs and shortcomings output/adc2vtk.pl: #TODO: remove this file from the list so the script can continue output/adcirc2netcdf.f90: ! TODO : finish implementing gridded data output/adcircdata.f90:rundes = 'rundes' !TODO: make adcirc write this value to netcdf output files output/adcircdata.f90:runid = 'runid' !TODO: make adcirc write this value to netcdf output files output/adcmesh.f90:! a netcdf file. See TODO comments at the end of the subroutine. output/adcmesh.f90:! TODO: The NetCDF files written by ADCIRC don't actually contain the output/adcmesh_boundaryfinder.f90:! a netcdf file. See TODO comments at the end of the subroutine. output/adcmesh_boundaryfinder.f90:! TODO: The NetCDF files written by ADCIRC don't actually contain the output/asgsio.f90:rundes = 'rundes' !TODO: make adcirc write this value to all netcdf output files output/asgsio.f90:runid = 'runid' !TODO: make adcirc write this value to all netcdf output files output/checkAdcircMesh.f90:! TODO: This does not include boundary conditions. output/checkAdcircMesh.f90: ! TODO: handle elevation specified boundaries output/checkAdcircMesh_boundaryfinder.f90:! TODO: This does not include boundary conditions. output/checkAdcircMesh_boundaryfinder.f90: ! TODO: handle elevation specified boundaries output/diffAdcircNetCDF.f90:! TODO: Implement the finding of relative differences, using the gold output/generateXDMF.f90: ! TODO:load bathy depth to interpolate that too output/generateXDMF.f90: p(ip)%z = 0.d0 ! TODO: could provide options for setting this output/generateXDMF.f90:! TODO: this assumes the same number of datasets in each file and that output/generateXDMF.f90:! TODO: This assumes that if the netcdf contains a multicomponent quantity output/generateXDMF.f90: ! TODO: Fix this hack for 2D velocity output/gomex_renci_post.sh:# TODO: Add in-situ post processing and graphics generation if needed, and output/modMeshData.f90:! TODO: This only handles water level offset files but this could output/modMeshData.f90:! TODO: netCDF support output/pullStationTimeSeries.f90:! TODO: fix the maureparticle interface so it determines the number of datasets, time increment, and start time automatically output/test/irene_post.sh:# TODO: Fix availability/acquisition of gold data ... for now, I've output/test/qa_post.sh:# TODO: Fix availability/acquisition of gold data ... for now, I've output/tristreamline.py:# TODO: option for input file name output/viz_driver.sh:# TODO: for parallel execution? output/workflow.sh:# TODO: compute duration(s) of exposure to water of a certain height output/workflow.sh:# TODO: lookup vdatum conversions qscript.template:# job properties (TODO: Add json propertes) qscript.template-test:# job properties (TODO: Add json propertes) set_flux.pl: # TODO: implement stage discharge set_flux.pl: # TODO: only one stage discharge curve at the moment tides/CMakeLists.txt:# TODO: Fix and add ec2001v2d_tide_interp tides/tide_fac.f:C TODO: IMPLICIT NONE util/input/nodalattr/extractNodalAttributesProperties.sh: # TODO: This does not handle the case where the datum offset is variable util/levee_tools_etc/STATIONS2KML.f: open(61,file=trim(adjustl(bathyTopoFile)),action='read') ! TODO: make this more robust ```
No due dateThere are many `#FIXME` commends in the code, this milestone is to capture them ``` git grep -n FIXME | grep -v config ``` Active (issues created): ``` asgs_main.sh:341: # jgfdebug: TODO: FIXME: Hardcoded the time varying weirs input file asgs_main.sh:568: # FIXME: how to handle this situation? asgs_main.sh:723: # FIXME: there is a hack in qscript.pl to change this to 20 for the priority queue on asgs_main.sh:1618: # FIXME: the following are legacy properties from 2014stable asgs_main.sh:1649: # FIXME: the following are legacy properties from 2014stable asgs_main.sh:2075: # FIXME: this does not download the SWAN hotstart file from the URL; asgs_main.sh:2078: # FIXME: this also assumes that the hotstart format for this instance asgs_main.sh:2726: # FIXME: writeProperties? ``` Waiting (issues not created yet): ``` archive/enstorm_pedir_removal.sh:75: # FIXME: the name of the asgs_swan.prt file is provided in swaninit but it archive/enstorm_pedir_removal.sh:141:# FIXME : this needs to be cleaned up so that the subdomain hotstart cloud/Dockerfiles/Dockerfile.dev:73:# # get from dockerhub (FIXME: not yet) cloud/Dockerfiles/Dockerfile.dev:74:# FIXME (not yet): docker pull asgsdockerhub/master cloud/Dockerfiles/Dockerfile.dev:122:# FIXME : Eventually we need to use environment variables to pass control_file_gen.pl:430: # FIXME: static offset start and end hardcoded to coldstart and 15.0 days control_file_gen.pl:610:# FIXME: if the stormname property exists but is null or empty, it should be get_nam.pl:63:# FIXME : $enstorm needs to be eliminated in favor of $stage which can be "nowcast" or "forecast" get_nam.pl:151:# FIXME : this is repeated code with forecastselection from above that should be turned into a sub get_nam.pl:562: # FIXME : there has to be a better way get_nam.pl:779: # FIXME: undo this hardcode for downloading files with curl using input/machines/hatteras/hatteras.adcprep.template.slurm:55:# FIXME: insert code to detect failures in adcprep that still allow it to metadata.pl:41:my $keys = "null"; # keys for values to read FIXME: only works if a single key is provided monitoring/time_properties.py:30:# FIXME: add ability to search for runid output/accumulateMinMax.sh:82:# FIXME: this makes an assumption that previous advisory number is one output/asgsio.f90:334: f%dataFileCategory = MESH !FIXME: or fort.88? output/asgsio.f90:1058: !FIXME: finish code for parsing OWI data output/cpra_postproc/createPPT.sh:58: stormname=NAM #FIXME: make this more general output/cpra_slide_deck_post.sh:102: STORMNAME=NAM #FIXME: make this more general output/diffAdcircNetCDF.f90:88:! jgf20150723: UNFORTUNATELY ASSUMING WE HAVE TWO MAXELE.63.NC FILES !FIXME output/diffAdcircNetCDF.f90:140:! !FIXME! should change the name of the variable to reflect the output/makefile:203:# FIXME: make FigureGen is not working yet output/nco_ncrcat.pl:153: # FIXME: we're assuming all files have the same cold start date output/nco_ncrcat.pl:178: # FIXME: this will probably result in concatenating too much data, output/opendap_post.sh:62:IFS=' ' read -r -a SERVERS <<< "$serverList" # FIXME: contains "(" and ")" (don't use eval) output/opendap_post.sh:69:IFS=' ' read -r -a FILES <<< "$fileList" # FIXME: contains "(" and ")" (don't use eval) output/opendap_post.sh:102:timeoutRetryLimit=${timeoutRetryLimit:-5} # FIXME: hardcoded to 5; make this more granular output/opendap_post.sh:115: # FIXME: enable Operator to override TDS parameter settings from platforms.sh output/opendap_post.sh:142: basin="al" # FIXME: write/read a property instead of hardcoding the atlantic basin output/opendap_post.sh:262: mkdirRetryLimit=10 # FIXME: hardcoded for now output/opendap_post.sh:534: # jgf20160322: FIXME: post to opendap even if there was an error so we can output/pullStationTimeSeries.f90:504: ! FIXME: this only measures the first instance of nonzero particle count output/stationProcessor.f90:23:! FIXME: This code assumes that the station files are provided in output/stationProcessor.f90:388:! FIXME: this may not do the right thing for multicomponent (i.e., vector) quantities output/storm_history.sh:82:# FIXME: this makes an assumption that previous advisory number is one platforms.sh:570: # FIXME: create something more reliable/repeatable qscript.pl:129: # FIXME: this is a hack to handle an idiosyncracy in ppn on queenbee and supermic storm_track_gen.pl:436:# FIXME: if the stormname property exists but is null or empty, it should be ```
No due date•2/7 issues closed- No due date•5/5 issues closed
The ability for operators on different HPC systems or instances of running ASGS is really interesting, but requires some investigation. This milestone is to capture this research, code updates, and documentation meant to make this happen.
No due date•1/1 issues closedthings we need to do to finish preparing for the 2020 Operational Window
No due date•18/19 issues closedChanges here are meant to facilitate readiness drils, onboarding, testing, and integration testing
No due date•5/5 issues closedtasks related to integration with the CERA Heartbeat service
No due date•2/2 issues closedDriven by utilizing AWS' Simple Email Service (via SMTP credentials), ultimately this solution will be a mixture of developed tools and procedures for onboarding both Operators and Resource Providers.
No due date•5/5 issues closedDecide on: - source type (kind of mark up) - overall structure/areas -create documentation to cover current user classes (admin, operator, resource provider, etc), workflows, and installation/porting/integration guides
No due date•3/6 issues closedmake available via asgsh's 'verify' command and using a driver script (non-interactive) the purpose of this would be to provide absolute compliance that an ASGS environment is set up correct; the linter would inspect scripts statically (without executing them) for common errors and coding standards
No due date- No due date•28/28 issues closed
Clients and stakeholders at CPRA have identified Operational priorities for ASGS based on lessons learned from the 2018 hurricane season. This milestone covers the issues and features associated with these priorities.
Overdue by 5 year(s)•Due by June 1, 2020•4/4 issues closedADCIRC v54 will have the ability to accept a new nodal attribute for vertical datum specification (i.e., the node-by-node difference between the vertical datum of the mesh and the geoid). This capability must be integrated into the ASGS, allowing the ASGS to understand what datum the results are relative to, and to be able to use that information during post processing to provide data to decision makers in the vertical datum they expect.
Overdue by 5 year(s)•Due by August 1, 2019The ASGS Operator Corps has always functioned in a loose, independent, distributed fashion. In order to improve reliability while reducing the on-call burden associated with each Operator, improved team coordination and training will be implemented to synchronize the ASGS Operators.
Overdue by 5 year(s)•Due by August 1, 2019•1/2 issues closedThe developer, operator, and collaborator (i.e., interface with downstream post processing technologies, including documentation of metadata scheme) must be brought up to date to encourage and facilitate collaboration. CERA documentation tasks are also part of this milestone.
Overdue by 5 year(s)•Due by August 1, 2019•1/1 issues closedThe run.properties file was produced in the past purely for use in providing information to downstream data consumers like CERA and StormSurgeViz. This file has steadily increased in importance, and has become the preferred solution to long lists of command line arguments for various subsystems. This milestone marks the transition to an operating paradigm where the ASGS starts setting up a run by creating the associated run.properties files. Then the various subsystems (for example, the fort.15 file creation script, the post processing, etc) all read the run.properties files to figure out what to do for that job. This milestone must also include reasonable defaults for each piece of metadata, the inclusion of all the metadata into the netcdf output files, and official documentation of all metadata for use by downstream data consumers.
Overdue by 5 year(s)•Due by August 1, 2019•4/5 issues closedThe ASGS will collect real time wind, wave, and water level data, be able to compare with model outputs, and compute model skill indices.
Overdue by 6 year(s)•Due by July 1, 2019•0/2 issues closedThe ASGS has to guess when it is computing estimated wall clock times because performance characteristics differ based on hardware platform, number of cores, number of dedicated writer cores (if any), use of SWAN coupling, overall load on the hpc platform due to other users, and other factors. When this milestone is complete, the ASGS will be able to collect performance information across all platforms, plot it via gnuplot or similar, report it via status monitoring, and use it in its algorithm for estimating wall clock time.
Overdue by 6 year(s)•Due by June 1, 2019•1/5 issues closedThe capability to have automatically varying river forecast updates in ADCIRC is currently under developments. In order to take advantage of this capability, the ASGS must be able to download the latest river forecast water levels and plug those into the boundary conditions for ADCIRC.
Overdue by 6 year(s)•Due by July 1, 2019•1/2 issues closedIn the past, the ASGS required a partially templatized fort.15 file with parameters that to be filled in being surrounded with percent signs. This system works well but inherently limits which parameters can be modified in an automated way by the ASGS. It also required a manual process of creating a new fort.15 template file for each new mesh that needed to be integrated, which introduced delays and typos. This milestone will result in the ability to use a universal fort.15 template where every parameter can be modified dynamically by ASGS, and will require the setting of reasonable model defaults for every parameter.
Overdue by 5 year(s)•Due by June 1, 2020•2/2 issues closedWater level data assimilation workflow integrated across all platforms. This includes the conversion of automated water level offset scripting from matlab to python (to enable and simplify cross platform execution). It also includes enabling an Operator to specify static offset surfaces and to change the offset surface that is being applied.
Overdue by 6 year(s)•Due by May 1, 2019In order to support a wider range of nowcast/forecast behavior, including but not limited to the integration of water level data assimilation, the ASGS must support an ensemble of nowcasts for each nowcast cycle. For water level data assimilation, this would include two nowcasts: one without data assimilation and one with data assimilation. It could also include a nowcast with and without far field winds, or with multiple types of far field winds, with a subsequent forecast only using one of the nowcast types (the Operator would need to have the capability to specify which nowcast each forecast should start from).
Overdue by 6 year(s)•Due by April 1, 2019•1/2 issues closedADCIRC has the ability to combine OWI and GAHM to produce a blended wind field. This milestone marks the achievement of support for that capability inside ASGS.
Overdue by 6 year(s)•Due by March 1, 2019•9/12 issues closed