Tags · openmobilityfoundation/mds-core

This repository was archived by the owner on Oct 29, 2021. It is now read-only.
0.1

Catch-up from lacuna-tech/mds-core (#249)


*Autogenerated Release notes, commits, authors from Squash:*

* [mds-metrics] Fix binning bug (#70)

* Fix weird array construction bug in metrics API binning method

* Remove async modifier

* Cleanup and add state_snapshot test

* Cleanup

* Bump root version

* Publish

 - @container-images/env-inject@0.1.21
 - @container-images/mds-agency@0.1.22
 - @container-images/mds-audit@0.1.23
 - @container-images/mds-compliance@0.1.22
 - @container-images/mds-daily@0.1.22
 - @container-images/mds-metrics-sheet@0.1.23
 - @container-images/mds-metrics@0.0.4
 - @container-images/mds-native@0.0.20
 - @container-images/mds-policy-author@0.1.22
 - @container-images/mds-policy@0.1.22
 - @container-images/mds-provider@0.1.22
 - @mds-core/mds-agency@0.0.24
 - @mds-core/mds-api-authorizer@0.1.22
 - @mds-core/mds-api-helpers@0.1.22
 - @mds-core/mds-api-server@0.1.22
 - @mds-core/mds-audit@0.1.34
 - @mds-core/mds-cache@0.1.22
 - @mds-core/mds-compliance@0.1.23
 - @mds-core/mds-daily@0.0.23
 - @mds-core/mds-db@0.1.22
 - @mds-core/mds-logger@0.1.20
 - @mds-core/mds-metrics-sheet@0.0.23
 - @mds-core/mds-metrics@0.0.4
 - @mds-core/mds-native@0.0.20
 - @mds-core/mds-policy-author@0.0.19
 - @mds-core/mds-policy@0.0.23
 - @mds-core/mds-provider@1.0.22
 - @mds-core/mds-providers@0.1.22
 - @mds-core/mds-stream@0.1.22
 - @mds-core/mds-test-data@0.1.22
 - @mds-core/mds-types@0.1.19
 - @mds-core/mds-utils@0.1.22

* Add some more logs for the Metrics API prototype (#73)

* Bump root package version

* Publish

 - @container-images/env-inject@0.1.22
 - @container-images/mds-agency@0.1.23
 - @container-images/mds-audit@0.1.24
 - @container-images/mds-compliance@0.1.23
 - @container-images/mds-daily@0.1.23
 - @container-images/mds-metrics-sheet@0.1.24
 - @container-images/mds-metrics@0.0.5
 - @container-images/mds-native@0.0.21
 - @container-images/mds-policy-author@0.1.23
 - @container-images/mds-policy@0.1.23
 - @container-images/mds-provider@0.1.23
 - @mds-core/mds-agency@0.0.25
 - @mds-core/mds-api-authorizer@0.1.23
 - @mds-core/mds-api-helpers@0.1.23
 - @mds-core/mds-api-server@0.1.23
 - @mds-core/mds-audit@0.1.35
 - @mds-core/mds-cache@0.1.23
 - @mds-core/mds-compliance@0.1.24
 - @mds-core/mds-daily@0.0.24
 - @mds-core/mds-db@0.1.23
 - @mds-core/mds-logger@0.1.21
 - @mds-core/mds-metrics-sheet@0.0.24
 - @mds-core/mds-metrics@0.0.5
 - @mds-core/mds-native@0.0.21
 - @mds-core/mds-policy-author@0.0.20
 - @mds-core/mds-policy@0.0.24
 - @mds-core/mds-provider@1.0.23
 - @mds-core/mds-providers@0.1.23
 - @mds-core/mds-stream@0.1.23
 - @mds-core/mds-test-data@0.1.23
 - @mds-core/mds-types@0.1.20
 - @mds-core/mds-utils@0.1.23

* [mds-db] Fix `editGeography` (#74)

* Fix `editGeography`.

* ts assist

* Fixed `editGeography` with TS assist.

* Bump root version

* Publish

 - @container-images/env-inject@0.1.23
 - @container-images/mds-agency@0.1.24
 - @container-images/mds-audit@0.1.25
 - @container-images/mds-compliance@0.1.24
 - @container-images/mds-daily@0.1.24
 - @container-images/mds-metrics-sheet@0.1.25
 - @container-images/mds-metrics@0.0.6
 - @container-images/mds-native@0.0.22
 - @container-images/mds-policy-author@0.1.24
 - @container-images/mds-policy@0.1.24
 - @container-images/mds-provider@0.1.24
 - @mds-core/mds-agency@0.0.26
 - @mds-core/mds-api-authorizer@0.1.24
 - @mds-core/mds-api-helpers@0.1.24
 - @mds-core/mds-api-server@0.1.24
 - @mds-core/mds-audit@0.1.36
 - @mds-core/mds-cache@0.1.24
 - @mds-core/mds-compliance@0.1.25
 - @mds-core/mds-daily@0.0.25
 - @mds-core/mds-db@0.1.24
 - @mds-core/mds-logger@0.1.22
 - @mds-core/mds-metrics-sheet@0.0.25
 - @mds-core/mds-metrics@0.0.6
 - @mds-core/mds-native@0.0.22
 - @mds-core/mds-policy-author@0.0.21
 - @mds-core/mds-policy@0.0.25
 - @mds-core/mds-provider@1.0.24
 - @mds-core/mds-providers@0.1.24
 - @mds-core/mds-stream@0.1.24
 - @mds-core/mds-test-data@0.1.24
 - @mds-core/mds-types@0.1.21
 - @mds-core/mds-utils@0.1.24

* fix error message (#77)

* mdscl/istio: install (un)install via helm straight up (#75)

* mdscl/istio: install (un)install via helm straight up

* mdsctl/istio: uninstallIstio cleanup

* [mds-audit] Support attaching media to audits (#76)

* Add new tables for audit attachments

* Integrate with audit attachments db tables

* New endpoints to add and delete attachments; return attachments with audits

* Lint

* Minor cleanups to s3 upload key and base url

* Move S3 config to env variables

* Let caller catch writeAttachment exception; type fixes

* Lint

* [mds-audit] Include initial provider state with audits returned via API (#42)

* Add MDS vehicle status and location to get audit response

* Add test for provider info in audit response

* Capitalize DESC keyword

* Add provider_event_time to Get Audit response

* Remove references to dropped columns from schema

* Use enums

* Fix import

* Make ORDER BY configurable in readEventsWithTelemetry

* Fix errors when running `yarn image`

* Use optional chaining from TS 3.7

* Bump root version

* Publish

 - @container-images/env-inject@0.1.24
 - @container-images/mds-agency@0.1.25
 - @container-images/mds-audit@0.1.26
 - @container-images/mds-compliance@0.1.25
 - @container-images/mds-daily@0.1.25
 - @container-images/mds-metrics-sheet@0.1.26
 - @container-images/mds-metrics@0.0.7
 - @container-images/mds-native@0.0.23
 - @container-images/mds-policy-author@0.1.25
 - @container-images/mds-policy@0.1.25
 - @container-images/mds-provider@0.1.25
 - @mds-core/mds-agency@0.0.27
 - @mds-core/mds-api-authorizer@0.1.25
 - @mds-core/mds-api-helpers@0.1.25
 - @mds-core/mds-api-server@0.1.25
 - @mds-core/mds-audit@0.1.37
 - @mds-core/mds-cache@0.1.25
 - @mds-core/mds-compliance@0.1.26
 - @mds-core/mds-daily@0.0.26
 - @mds-core/mds-db@0.1.25
 - @mds-core/mds-logger@0.1.23
 - @mds-core/mds-metrics-sheet@0.0.26
 - @mds-core/mds-metrics@0.0.7
 - @mds-core/mds-native@0.0.23
 - @mds-core/mds-policy-author@0.0.22
 - @mds-core/mds-policy@0.0.26
 - @mds-core/mds-provider@1.0.25
 - @mds-core/mds-providers@0.1.25
 - @mds-core/mds-stream@0.1.25
 - @mds-core/mds-test-data@0.1.25
 - @mds-core/mds-types@0.1.22
 - @mds-core/mds-utils@0.1.25

* Backend method for finding policies that use particular geographies. (#49)

* [mds-logger] Overloading log with log level parameter to support ternary conditions & fluent syntax (#71)

* Overloading log with log level parameter to support ternary conditions & fluent syntax

* remove psql-client (#84)

* need to have the domain inserted for both TLS and non-TLS (#83)

* allow additional hosts for egress if needed (#82)

* Add more granular readAllEvents performance logging (#80)

* [mds-schema-validators] Pull all JSON Schema-related definitions and code into a new package. (#78)

* Consolidating validator/schema-related code in mds-policy to mds-utils

* Consolidating validator/schema-related code in mds-compliance to mds-utils

* Consolidating validator/schema-related code in mds-compliance to mds-utils

* extracting JSON Schema definitions and code into mds-json-schema

* fixing build errors caused by moving validation to new package

* renaming mds-json-schema -> mds-schema-validators and cleaning up build issues

* moving ValidationError back to mds-utils

* removing hapi/joi from mds-utils

* updating tsconfigs

* Referencing RULE_TYPES when constructing schemas

* Revert "Referencing RULE_TYPES when constructing schemas"

This reverts commit 87def6e65910b44b3e632f6b3525bf993804882b.

* Referencing RULE_TYPES when constructing schemas

* Configuration Service/API (#91)

* Config service client and tests

* Config API and tests

* Config API container image

* Add debug configurations for config api/service

* create a configmap from files in helm/mds/mds-config, mount it to the mds-config pod under that path /mds-config

* Allow settings to use .json or .json5 extensions

* [mds-compliance] Compliance engine sorts vehicles by earliest telemetry data, so most … (#89)

* Compliance engine sorts vehicles by earliest telemetry data, so most recent events will be the ones tagged as violators.

* Updating sort function.

* Removing unnecessary spread operator.

* Fix helm configmap glob (#95)

* [mds-cache, mds-db] Instrument some cache and db operations with timing logs (#97)

* Instrument some cache and db operations with timing logs

* Add provider to getVehicles timing log

* Update dev/tooling dependencies. (#93)

* Upgrade to latest package dependency versions

* Clean up test scripts using config files

* Bump some coverage levels

* [mds-audit] Relax lookup VIN response requirements (#96)

* Relax lookup VIN response requirements

* Add logs for missing info; return telemetry when possible

* Bump root version

* Publish

 - @container-images/env-inject@0.1.25
 - @container-images/mds-agency@0.1.26
 - @container-images/mds-audit@0.1.27
 - @container-images/mds-compliance@0.1.26
 - @container-images/mds-config@0.0.2
 - @container-images/mds-daily@0.1.26
 - @container-images/mds-metrics-sheet@0.1.27
 - @container-images/mds-metrics@0.0.8
 - @container-images/mds-native@0.0.24
 - @container-images/mds-policy-author@0.1.26
 - @container-images/mds-policy@0.1.26
 - @container-images/mds-provider@0.1.26
 - @mds-core/mds-agency@0.0.28
 - @mds-core/mds-api-authorizer@0.1.26
 - @mds-core/mds-api-helpers@0.1.26
 - @mds-core/mds-api-server@0.1.26
 - @mds-core/mds-audit@0.1.38
 - @mds-core/mds-cache@0.1.26
 - @mds-core/mds-compliance@0.1.27
 - @mds-core/mds-config-service@0.0.2
 - @mds-core/mds-config@0.0.2
 - @mds-core/mds-daily@0.0.27
 - @mds-core/mds-db@0.1.26
 - @mds-core/mds-logger@0.1.24
 - @mds-core/mds-metrics-sheet@0.0.27
 - @mds-core/mds-metrics@0.0.8
 - @mds-core/mds-native@0.0.24
 - @mds-core/mds-policy-author@0.0.23
 - @mds-core/mds-policy@0.0.27
 - @mds-core/mds-provider@1.0.26
 - @mds-core/mds-providers@0.1.26
 - @mds-core/mds-schema-validators@0.1.2
 - @mds-core/mds-stream@0.1.26
 - @mds-core/mds-test-data@0.1.26
 - @mds-core/mds-types@0.1.23
 - @mds-core/mds-utils@0.1.26

* Fix debugging for unit tests (#105)

* mdsctl: prompt-to-proceed if not using docker-desktop (#85)

* mdsctl: prompt-to-proceed if not using docker-desktop

* mdsctl: bit of cleanup

* Add moped to VEHICLE_TYPES enum (#106)

* Support merging multiple settings properties (#108)

* Support merging multiple settings files

* Get settings property names from query string

* Support existing single settings property API

* [mds-agency] [mds-db] Initial Stop Prototype (#47)

* Add Stop datatype, and r/w endpoints to Agency

* Add proper validation when POSTing stops

* Add some basic error handling

* Start adding tests...

* Clean up bad db types

* Test cleanup

* More cleanup

* More cleanup pt2

* More cleanup

* Add read back test

* More tests

* Last of the tests?

* Some formatting cleanup

* Remove unused import

* Make geography_id optional for stops, add lat/lng requirement

* Cleanup

* Review nits

* Clean up schema, use SqlVals()

* Remove markdown plugin

* Update carshare -> car, and add moped type to vehicleTypesCountMapSchema definition

* Break build... this is intentional, I promise

* Revert "Break build... this is intentional, I promise"

This reverts commit d8ba98963e3d2c523aa8212e9715f3adf76bd057.

* Fixture updates

* Add recorded column

* Cleanup

* Add option to skip bbox check when fetching vehicles by bbox (#113)

* [mds-audit] Speed up VIN lookup by parallelizing cache reads (#110)

* Speed up VIN lookup by parallelizing cache reads

* Fix build error (#114)

* Update VS Code ESLint settings (#117)

* [mds-db] Refactor makeReadOnlyQuery to permit use of SqlVals() (#116)

* Refactor makeReadOnlyQuery to permit use of SqlVals()

* Fix minor oversight

* [mds-agency] [mds-web-sockets] Basic outbound websocket implementation for MDS-Agency (#99)

* First pass at WS

* Add tsconfig

* Add basic websocket emission to mds-agency

* Emission cleanup

* Add tsconfig

* Cleanup

* Dependency cleanup

* More dependency cleanup

* WIP

* More WIP

* WIP

* WIP

* Manual event triggering works, wheee

* Remove agency socket interaction, to be replaced with reading from KNE

* Added basic auth handling, need to add logic for token expiration. Could also just add a 30s heartbeat requirement.

* Validate auth

* More auth cleanup

* Header validation cleanup

* Generate random events and telemetry to send down websocket

* Nits

* Add outbound ws client

* Message emission working

* Purge console.log

* Clean up authorizers

* Cleanup

* Bump deps

* Bump deps

* Cleanup

* Test coverage

* Cleanup

* Moar cleanup

* So much cleanup

* Bap

* nits

* Moar nits

* Refactor async in forEach to use Promise.all()

* Rename nit

* Export cleanup

* Exception handling cleanup

* Add missing index.ts file

* Remove stale clients

* Make sure to kill the connections too

* Exec function

* Minor authorizer refactor (#111)

* Force exit for agency tests to close open socket

* Moar exit forces

* Import alignment

* Cleanup

* Changes for internal routing and basic helm config

* Remove test code

* Cleanup

* [mds-agency] Fix import of package with no default export (#120)

* [mds-web-sockets] Check for admin:all scope when authenticating clients (#121)

* Add admin-all scope check when authenticating ws clients

* Nullish coalesce all the things

* Remove unnecessary check

* Fix image name for mds-web-sockets (#122)

* .Values.api.env will populate all APIs' environments (#92)

* [mds-web-sockets] Don't always start WebSocket server (#123)

* Don't always start WebSocket server

* Add missing launch file

* use branch name for docker image tag suffix (#124)

* use literal branch name as docker suffix

* portable sed

* [mds-web-sockets] Fix PING message handling

* [mds-processors] Refined Processor Implementation (#104)

* Cleanup

* update requirement versions

* revert to original dir copy

* cleanup

* Beginning of cleanups

* more cleanup/ removed streaming logic scoped for Q1

* trip-proc cleanup

* type fixes

* ts refactor

* helm reconcile

* mds-logger

* metric cleanup

* helm fixes

* breaking db methods into seperate file

* cache/db refactor

* Clean up geo.ts

* Clean up annotation.ts

* Clean up proc-event.ts

* Clean up proc.ts

* Clean up misc

* Clean up metrics.ts

* Update MetricsTableRow type

* Resolve/suppress all linter warnings, and resolve compilation errors.

* Add eslint-reasons to most directive disables

* Refactor geo.ts files into utils

* Add stricter typing

* Cleanup

* Fix dependencies/versions

* cache/db fixes

* proc fixes

* mdsctl: cleanup (comment out nats)

* Cleanup

* Optional chaining

* Add ParseError

* upgrade natss

* mdsctl: update

* mdsctl: update bin/mdsctl add knative(broker,channel)

* fix typo

* mdsclt: ko hack, mds pause for natss

* mdsctl: s/pause/condition/g

* [WIP] [mds-metrics] Add dump all metrics rows API endpoint (#94)

* WIP

codify types

add initial request handler test

add query construction tests

named args obj (no more defaults)

Add explanatory comments

more comments

* s/timestamp/start_time

* rename some things + cleanup

* update test

* fix incorrect import

* fix type signature

* mdsctl: kne w/o namespace labeling

* Remove default param val

* - Add id column to reports tables
- Restore sorting of code

* Do not initialize the db at startup!

* Reduce unnecessary switch cases

* Fix lint error that's been breaking build

* Upgrade to cloudevents 1.0 with typings

* added presets to mdsctl

* update proc-event

* mdsctl uncomment natss

* Remove deprecated Redis stream

* Remove package-lock file

* Generic server for handling knative cloudevents

* Make handler async

* Unit test placeholder

* [mds-metrics] Adds geography_id and provider_id query params to /metr… (#98)

* [mds-metrics] Adds geography_id and provider_id query params to /metrics/all

Usage: GET /metrics/all?geography_id={}&provider_id={}

Both fields are optional.
Each id query param must be a valid UUID or you'll get a 500.

TODO test w/ cURL or Insomnia @evanxadkins

This is also going into PR #88

Closes [MDSAAS-487](https://lacunadotai.atlassian.net/browse/MDSAAS-487)

* fix up lint and add comment

* remove geography_id query param

* initial stab at vehicle_type querying

* vehicle_type query param support

* add english bin_size support

* switch to nifty time utils

* crazy time validation logic

* add note

* fix mutable moment bug

* add note

* clean up request handler

* clean up notes

* add tsv format query param

* add note

* add some tsv export tests

* fix failing test

* tests should be more stable now

* fix import

* purge link

* remove flaky test (again)

* lint

* type enforcement

* eslint errors

* Regenerate lock file

* Switch trip/provider handlers to event server

* Clean up a couple test commands/timeouts

* Add better parsing logs

* trip-proc update

* promise handling

* prov-proc update

* metrics update

* Handle parse errors from cron source events

* Improve test coverage

* Minor cleanup

* Feature/jwtodd events trips provider processor (#101)

* nats: internal deployment of processors

* cjs: remove hardwired minute interval

* helm: fix cronjobsource schedule

* processors: add cronjob template

* interval: 1m

* event sync

* nats: processor (internal-apis; no need to port-forward), simulator cli, natss broker/channel

* helm/mds: template cleanup(broker, processor-cronjob)

* mdsctl: tad better ko ref

* helm/mds cleanup

* mdsctl: make natss a pre-condition of mds

* mdsctl: cleanup

* bug fixes

* Package update

* [mds-metrics] Add stub endpoint (#103)

* add stub endpoint

* fix async/string

* simplify JSON tsv processing

* re-enable/fix up tsv export in main api

* WIP

* cache update

* cache update

* Update test scripts for new packages

* Update versions of mds-core dependencies

* Add missing dependencies, fix build/test errors

* package updates

* minor syntax changes

* mdsctl: sed of osx; todo: fix for all

* mdsctl: better (osx,windows) natss install

* multimodal changes

* mdsctl: support deleting 2+ images

* mdsctl: build image cleanup, simulator build

* event_count metric addition

* [mds-db] [mds-metrics] Add SqlVals() to processors db queries (#107)

* refactor to use SqlVals()

* revert unrelated changes

* helm/mds: +mds-config, for t in helm/mds/templates/* s/key/name/g, move natss-channel into deploy namespace (eg mds); verified: simulator, logs mds-(event,provider,trip)-processor

* Add unit tests to mds-event-processor (#112)

* Revert "[mds-db] [mds-metrics] Add SqlVals() to processors db queries (#107)"

This reverts commit 0cee743787328d6ffa4bbca963370a3030218f4d.

* fix weird lint error

* initial test

* crlf yikes

* moar tests

* lower codecov

* more unit tests

* small helm changes

* sqlVals

* small error handling changes

* Revert "sqlVals"

This reverts commit f2299b25606045e219fe0a1c374ae8140afbd89b.

* vscode settings

* kne/natss:0.11.0, multi-tenant nats/redis msg routing (#115)

* mdsctl: delete 2+ images

* filter prep

* natss: conditionally prefix the event:type with the tenantId else use 'mds' as the default

* rm tenantId dbug

* events/multi-tenancy: patch db statments to vary event-type prefix, attempted to push the tenentId down to utils but bad things happened to my system; next: test concurrent tenants, verify db statements

* kne: move deploy process to helm (from kubectl apply -f [url])

* nats: helm'ify

* nats/helm: pin to gcr

* nats: prioritize (un)install natss via helm

* redis: multi-tenant

* multi-tenant: cleanup

* mdsctl: not to sync kne-contrib from a release

* redis/multi-tenant: clean up comment

* redis/multi-tenant: clean up argument quoting

* mdsctl: helm kne-eventing-init

* nats/multi-tenant: filter out tenant-routing prefix prior to forwarding the message to the handler

* natss event processor: clean-up

* helm/kne-contrib: r0.11.0 natss-channel.yaml

* mdsctl: fixme cleanup

* tenantid: better ternary

* Clean up cloud event parsing.

* Suppress deprecation warnings

* Feature/maxj/sqlvals (#119)

* [mds-audit] Speed up VIN lookup by parallelizing cache reads (#110)

* Speed up VIN lookup by parallelizing cache reads

* Fix build error (#114)

* Update VS Code ESLint settings (#117)

* [mds-db] Refactor makeReadOnlyQuery to permit use of SqlVals() (#116)

* Refactor makeReadOnlyQuery to permit use of SqlVals()

* Fix minor oversight

* re-enable sqlvals

* [mds-agency] [mds-web-sockets] Basic outbound websocket implementation for MDS-Agency (#99)

* First pass at WS

* Add tsconfig

* Add basic websocket emission to mds-agency

* Emission cleanup

* Add tsconfig

* Cleanup

* Dependency cleanup

* More dependency cleanup

* WIP

* More WIP

* WIP

* WIP

* Manual event triggering works, wheee

* Remove agency socket interaction, to be replaced with reading from KNE

* Added basic auth handling, need to add logic for token expiration. Could also just add a 30s heartbeat requirement.

* Validate auth

* More auth cleanup

* Header validation cleanup

* Generate random events and telemetry to send down websocket

* Nits

* Add outbound ws client

* Message emission working

* Purge console.log

* Clean up authorizers

* Cleanup

* Bump deps

* Bump deps

* Cleanup

* Test coverage

* Cleanup

* Moar cleanup

* So much cleanup

* Bap

* nits

* Moar nits

* Refactor async in forEach to use Promise.all()

* Rename nit

* Export cleanup

* Exception handling cleanup

* Add missing index.ts file

* Remove stale clients

* Make sure to kill the connections too

* Exec function

* Minor authorizer refactor (#111)

* Force exit for agency tests to close open socket

* Moar exit forces

* Import alignment

* Cleanup

* Changes for internal routing and basic helm config

* Remove test code

* Cleanup

* [mds-agency] Fix import of package with no default export (#120)

* remove quotes around vals.add()

* follow existing pattern for IN

* clean up unused imports

* more cleanup

* more lint?

* helm/mds: fix values(apis,processors)

* Fix Cloud Event emitter cache

* Remove deprecated test

* Restore cloud event test

* minor proc package fix

* Feature/maxj/additional query params (#118)

* specify multiple provider_id query values

* clean up types, normalize logic to arrays

* add multiple bin processing support

* clean up bin_size handling

* add some additional handling code

* %s/let/return/g

* clean up types/naming

* clean up default arg

* fix test

* mdsctl: fully helm'ify the mds install (by labelling the namespace; noting the cost is an extra 'default' in-memory broker)

* CE prefix metric query removal

* Dependency updates

* added README

* Small cleanups

* Fix helm template

* Fix project reference

* Add unit test to fix coverage

* add recorded column to reporting tables

* Update README.md

* added now util function

* Update README.md

* run triggers on the top of the hour

* helm/kne: remove in-memory-controller; tested with `mdsctl -p processors install:mds,simulator` and observing events propogated to logs

* helm/mds: add `postgresql.password` empty-by-default field with clarifying comment

* helm/knative: remove imc k8s deploy

* [mds-provider-processor] A few unit tests for mds-provider-processor. (#128)

* A few unit tests for mds-provider-processor.

* Fixed test threshold.

* this is not right but it makes tests pass

* Clean up project references

* Adding cache.initialize for consistency.

* unit tests for metric utils

* formatting

* cleanup

* Restore configmap functionality

* type reduction

* hour cron interval

* Mount config volume in all pods (#130)

Co-authored-by: Neil Goldader <avatarneil@gmail.com>
Co-authored-by: Michael Durling <michael@ellis-and-associates.com>
Co-authored-by: James Todd <99516+jwtodd@users.noreply.github.com>
Co-authored-by: Max Johansen <macsj200@gmail.com>
Co-authored-by: Alex Gottschalk <389982+invertigo@users.noreply.github.com>
Co-authored-by: janedotx <jane.x.huang@gmail.com>

* [mds-web-sockets] Add proper env-injection to mds-web-sockets (#133)

* mdsctl: remove dashboard, get cli:postgresql and cli:redis working again (#126)

* mdsclt: rm dashhboard, fix cli

* mdsctl: get cli:[postgresql|redis] working again

* mdsctl: add helm@2 to PATH given it is installed in a non-default/opt location; note: dep on #104, will work as-is if pushed prior

* helm/mds: match #104 re $.Values.apis.env parse error

* mdsctl: helm init cleanup, don't display usage message when an error is present

Co-authored-by: Neil Goldader <avatarneil@gmail.com>

* [mds-agency] Don't retry then send a 500 on a failed cache/stream/socket write (#134)

* Don't retry then send a 500 on a failed cache/stream/socket write

* Use recorded_event rather than event

* Refactor telemetry writing

* fix crash if values not present (#125)

* [mds-audit] Return last active device registered if db has duplicates for provider + vin (#127)

* Return last device registered if db has duplic
8000
ates for provider + vin

* Fix lint

* Prefer an active device when de-duping

* Load configuration data via service (#132)

* Load configuration data via service

* Rename a few things...

* Fix tests

Co-authored-by: Neil Goldader <avatarneil@gmail.com>

* [mds-agency] [mds-utils] Extract normalizeToArray() and leverage in Daily  (#136)

* refactor normalizeToArray()

* use new helper

* [mds-*] Append commit hash to image tag if on non-master branch (#139)

* Append commit hash to image tag if on non-master branch

* Review nits

* replace heisentest with unit tests (#138)

Co-authored-by: Neil Goldader <avatarneil@gmail.com>

* Update package dependencies (#140)

* [mds-web-sockets] Initial web-socket test to get the framework in place. (#137)

* Initial web-socket test to get the framework in place.

* Minor refactor

* mdsctl: number of usability improvements (#135)

* mdsctl: seed PATH for non-default install bins; confirm remote-deploy only once

* mdsctl: tiller aws

* mdsctl: cleanup

* mdsctl: better remoting, add ability to cycle a pod, lessen remote warning check

* mdsctl: add cycle default, all services and procesors

* mdsctl: cleanup neg responses

* mdsctl: nit

* mdsctl: deploy in-cluster pgcl/redis-cli; note: can remove pcli/redis from brew

* mdsctl: rm brew(pgcli, redis)

* mdsctl: rm dead preset hook

* mdsctl: fix bugs(uninstallHelm, don't require MDS_SIMULATOR_REPOSISORY if MDS_SIMULATOR is set to a viable install)

* mdsctl: move some non-obvious configuration switches to comment-notes

* helm/natss: provide ability to override default resources(requests.cpu, limits.memory)

* readme: update mdsctl invocation

* mdsctl: +grafana; note: not configured correctly just yet

* mdsctl: +(un)install(dnslookup,curl,grafana)

* mdsctl: support kn(eventing,monitoring,serving)

* mdsctl: helm-delete by namespace

* mdsctl: touch ~/.bashrc if it doesn't exist, better bash command-to-brew-recipe mapping support

* helm/natss: set default memory to 512M, override with 32M

* mdsctl: install:knative(monintoring); will be continued in another branch

* natss: default memory=512M, fix mdsctl dns-lookup ns bug

* mdsctl: processors(+websockets)

* [mds-agency] [mds-web-sockets] Improve agency/telemetry error handling, and add basic health check to WebSocket client (#141)

* First pass at cleaning up telemetry error handling

* Patch the client to properly handle not being able to connect to web-sockets

* More attemps at fixing client code...

* Moar cleanup

* More cleanup

* add unified state representation (#144)

Co-authored-by: Neil Goldader <avatarneil@gmail.com>

* [mds-metrics] Fixes metrics scope processing (#146)

* adds metrics:read and metrics:read:provider scoping

* fix error msg

* add note

* push provider query logic to middleware

* simplify logic

* fix import

* lint

* fix unit tests

* coverage

* alphabet

* add state transition diagram (#148)

* Support async access token scope validators (#147)

* Support async access token scope validators

* Dependency updates

* ESLint/Prettier

* remove self loop (#149)

* [mds-utils] Remove unused state transition function (#145)

* remove unused function

* lint

* add expected transition values

* Make redis multi calls async (#151)

* [mds-*] Upgrade to Node.js 12 (v12.14.1) runtime (#154)

* Upgrade to Node.js 12 (v12.14.1) runtime

* Upgrade node types

* [mds-metrics] Add unauthorized error if query params attempt to exceed scope (#156)

* add unauthorized error if query params attempt to exceed scope

* reorder scope checking

* Feature/eadks proc cleanup (#150)

* cleanup spike

* telemetryMap structure change

* optimize registered count metric

* fixed test case

* null vehicle_type cache read bug fix

* service area test cases

* test case type enforcement

* cleanup

* query fix

* reduce logic

* removed unused import

* added error msg

* removed unused import

* syntax edits

* migration script and sql logs

* Remove provider API from City Services cluster (#159)

* Remove provider API from City Services cluster

* Remove provider API scopes

* fix heisenbug by moving tests to end (#161)

Co-authored-by: Neil Goldader <avatarneil@gmail.com>

* Extend daily timeout (#162)

* increase timeout

* timeout env var

* reorder number cast

* make sure we have AWS env vars before we make the call asking for AWS creds (#164)

* two changes to fix heisenbug pt 2 in policy-author (#165)

* [mds-cache][mds-processors] Cache optimization for event and trip processors (#160)

* cleanup spike

* telemetryMap structure change

* optimize registered count metric

* fixed test case

* null vehicle_type cache read bug fix

* service area test cases

* test case type enforcement

* cleanup

* query fix

* restructure spike

* hscan addition

* proc cache changes

* hscan reformat

* hscan bug fix

* reduced cache maintanence logic

* removed unused import

* edit to cache method names

* merge cleanup

* merge cleanup

* more merge cleanup

* comment syntax

* map edits

* reduced logic

* typdef fixme comment

* revert values.yaml

Co-authored-by: Mark Maxham <max@ellis-and-associates.com>
Co-authored-by: Neil Goldader <avatarneil@gmail.com>

* forward parsing error to client (#166)

* Don't mutate datetime (#167)

* no mutation

* refactor modules + add test

* Add moment-based timestamps

* Export getCurrentDate()

Co-authored-by: Neil Goldader <avatarneil@gmail.com>

* Fix clone error on Windows by removing unit test for invalid filename (#168)

* [mds-stream] [mds-event-processor] NATS implementation for mds-event-processor. (#143)

* First pass at nats implementation for mds-event-processor

* Some cleanup

* Works in-cluster with manual IP for NATS.connect args

* Actually await things

* nats: mdsctl(nats), disable kne/natss

* Remove trigger.yaml

* Check env.NATS instead of env.SINK

* mdsctl: natss

* Switch from vanilla nats to nats-streaming. This enables us to sychronously handle subscriptions.

* stan ftw

* add STAN_CLUSTER_ID env var

* WIP

* WIP

* WIP

* WIP

* Re-add event-server wrapper for mds-event-processor

* Revert some prior changes, structure events with CE

* Add credentials support

* stan.creds

* Add STAN_CREDS env var

* Some changes...

* stan.creds mount

* cleanup

* Stop using mds-event-server because it causes problems, working end-to-end in cluster

* Patch

* infer image:tags from remote registry

* +gcloud

* Re-add getRawData()

* Re-add mds-event-server, switch to using nats-streaming 0.3.0 client

* Clean up dependencies

* Update dependencies

* Ack failed messages

* helm(nats,stan)

* stan/cleanup

* +nats-box

* +nats-box, pin cluster-id

* stan cluster id

* add [nats|stan]x which are single-pod no-auth nats-io/k8s deploys

* helm'ify (nats,stan)x

* Cluster handled pod routing

* +stan=statefulset

* +stans:0.16.2

* u/d

* Event-server refactoring to improve readability

* Add getNats() to get a persistent nats client and save it if it falls over

* Add re-connect logic

* Turns out the nats client will attempt to reconnect for us, groovy.

* Enable reconnects

* Misc cleanup

* disable istio-injection in nats deployments

* prune kne

* cleanup

* stan: disable istio

* mdsctl: patch nats helm so that it's operations are distinct from init,cluster

* (un)install:stan fix

* cleanup natsbox conf

* install {tools}/helm:2.14.1 if not preent in the env

* bump stan disk allocation

* allow for more-more-then-one (natsbox, curl, dns) deploys

* add ServiceEntry egress for nats, and DestinationRule to disable mTLS for the destination namespace

* switch default nats namespace to "nats" from "default", use correct nats endpoint for stan

* mdsctl: +(nats,stan)-local, rm(kne*); next: conf mds->nats.nats

* mds: add env NATS=nats.nats, set STAN_CLUSTER=stan; would like to remove env STAN, STAN_CREDENTIALS, SINK

* patch

* helm/mds/.../egress.yaml: fix typo

* patch: get image versions

* +kubectx, helm/mds/deployment conf nats ns

* cold-start install patch

* cleanup

* stan: set max-message,max-bytes=0 (unlimited)

* mdsctl: keep-cache support for yarn

* trip-proc cron job

* Cron restart policy fix

* cronjobs in the house

* sync mds-trip

* cleanup

* mds/templates: image w/ & w/o registry

* cleanup env

* Remove CE_NAME and STAN in favor of NATS

* Update STAN_CLUSTER to nats-streaming

* Switch to using env.NATS for streaming

* More switching to env.NATS

* provider-proc cron edits

* Await parseTripsEvents

* u/d readme(mdsctl)

* u/d mdsctl

* u/d mdsctl

* mdsctl: support provider=aws:imageTag (get latest tag by image name)

* mdsctl: u/d readme

* Optional processor fix

* mount mds-config configmap

* Remove db.startup, cache.startup, getConfig. They're not working properly at the moment

* Remove db.startup from mds-provider-processor

* +surveyor, rename stan-statefulset to nats-streaming-server

* helm/mds: disable (trip, provider) processors, as they are invokved via cron

* surveyor: rm creds

* helm/natsbox: rename to nats-box

* Force exit after function is done

* use namespace name as default TENANT_ID

* Fix bad variable injection

* allow empty tenantId for legacy installs (ladot), use namespace if key is missing from .Values

* helm/mds: clean-up proto dead-code

* not using knative anymore

* consistent naming

* cleanup resource names

* differentiate nats/stan config-map names

* s/example-nats/nats/

* switch cron to hourly, turn off migrations for agency

* Update mdsctl

* Update README.md

* Update mdsctl

* Update mdsctl

* remove dependency on ~/.bashrc

* mdsctl: remove persisting conf in ~/.bashrc

* minor refactor - put event-processor into APIs as a service deployment and put trip- and provider- processors into cronjobs.  retire old metrics-sheet specific cronjob

* pull out nats-surveyor to be applied to a subsequent branch

* Drop nullish coalescence operator in favor of logical OR for TENANT_ID

* Update TENANT_REGEXP

* update template to match yaml, remove spurious env vars

* helm/nats: patch stan conf (store typo)

* Switch processors -> cronjobs

* mdsctl: build/yarn-clean

* helm: prune nats-local

* Typo

* Env var defaults cleanup

* u/d readme re k8s conf

* Add comment detailing writeCloudEvent

Co-authored-by: James Todd <99516+jwtodd@users.noreply.github.com>
Co-authored-by: Alex Gottschalk <389982+invertigo@users.noreply.github.com>
Co-authored-by: Evan Adkins <53316916+evanxadkins@users.noreply.github.com>

* Update README.md

readme/docker-desktop-config cleanup

* Assume cache as source-of-truth for  output to get time since last event due to performance problems in production (#169)

* istio-disable cronjobs (#170)

* [mds-*] Add Okteto Configuration for in-cluster debugging (#171)

* First-pass at okteto integration

* Add per-service okteto config

* Move config back to top level

* Update docs, fix missing name

* update docs

* Oops

* Update docs

* Update docs moar

* Docs updates

* Bump okteto resources massively to avoid crashes at debug init (#172)

* Make all tests async (#173)

* change envoy idle timeout to offset from nodejs idle timeout.  reduces incidents of a race condition where envoy attempts to connect with a timed-out node, resulting in a 503 UC

* [mds-db] Refactor en-mass import/exports in mds-db to be more readable (#177)

* Update Config API to support partial configs (#178)

* Refactor config service to improve error handling and test coverage

* Add partial config support

* Unit tests for partial config

* Change error (404) response body

* Upgrade dependencies; fix typescript/lint errors (#179)

* Export interfaces (#181)

* move annotation for preventing istio sidecar injection to correct met… (#180)

* move annotation for preventing istio sidecar injection to correct metadata location

* move annotation for preventing istio sidecar injection to correct metadata location

* remove dupe

* too many specs and templates

* [mds-web-sockets] First pass at some internal JWT validation. (#157)

* Promisify all the things

* Cleanup

* Working with Bearer tokens

* WIP

* Working tests whoop

* Fix error message

* Increase default timeout values to reduce spurious 503 errors from Istio (#184)

* Update Agency, Policy, and the root README docs (#182)

* OMF-26 API Version Middleware (#189)

* API Version Middleware

* Minor corrections

* Swallow STAN fatal errors. Previously, these would cause a node crash dump. (#190)

* Convert empty event.event_type_reason to null (#195)

* [mds-cache] Clean up promise logic in hreads (#153)

* Await batch hgetallasync rather than awaiting each invocation

* Remove Promise junk

* Remove janky promises, update typedef

* Promise.all all of the things

* Remove inner Promise.all

* Linterz

* yarn upgrade-interactive --latest (#193)

* Fix redis multi definition (#196)

* Jurisdiction API (GET, POST endpoints) (#187)

* getOne/getAll jurisdiction service methods

* create Jurisdiction service method

* initialize/shutdown jurisdiction service

* Initial GET /jurisdictions endpoint

* Drop schema after unit tests

* Clean up return statement

* Add container image for MDS Jurisdiction API

* API enpoints for create / find jurisdiction

* Add validation on create

* Update copyright notices

* Use port 4011 for local jurisdiction API

* Return 404 when jurisdiction not found

* Add some error typing to jurisdiction service

* Remove unused type

* Scopes for writing jurisdictions + api tests

* Add scopes for jurisdiction read

* Add jurisdiction claim filtering

* Stop tests from failing in the presence of extra db tables

* mdsctl installs mochawesome; commiting

* Rename a scope

* Add copyright notices

* Add versioning middleware to Jurisdiction API

* Updated lock file

* Merge origin/develop and update dependencies

* mdsctl yarn.lock updates

* Add some DB connection logging

* Use R/W connection for running TypeORM migrations (#197)

* Clean up Versioned API types; Reorganize Jurisdiction/Native APIs (#198)

* Refactor Jurisdiction API

* Clean up Native API types

* Rename some private interfaces/types

* Use array destructuring

* Reorganize Native API middleware/handlers

* Rename utils files

* Catch db connection errors separately (#199)

* [mds-provider-processor] Provider filter for device cache  (#174)

* provider-device cache call

* cron schedule revert

* updated tests

* test case bug

* cleanup

* nullish coalescing

* registered device map refactor

* registered device map refactor

* cleanup

* [mds-web-sockets] Add structured response for authentication success and failure (#204)

* Add structured response for authentication success and failure

* Fix test

* [mds-agency] [mds-utils] Add better object validation to agency. (#200)

* Add device object validation, begin working on building out event validators

* Add event and telemetry validation to agency request handlers

* Remove comment

* Refactor to match other usage better, add allowUnknown validator option

* More cleanup

* Cleanup

* Endpoints to update/delete a jurisdiction (#201)

* Update a single jurisdiction

* Add unit tests

* Sort versions by timestamp

* Support "deleting" jurisdictions (with tests)

* Separate out ORM operations into their own module.

* Fix import statements

* Tidy up some error messaging

* Add a comment

* carve out exception to jwt auth for websockets api (auth is done internally due to different headers for ws protocol) (#192)

* [mds-*] Merge in OMF/develop (#206)

* Kubernetes Contribution (#233)

* mdsctl/preset: onward to gh

* mdsctl: landed in gh

* Add additional test + simplify sum logic

* generalize over summing any columns

* fix test index type

* Split up mds-db into more manageable files.

* mdsctl: open:dashboard patch

* Increase test timeout to support k8s DB/Redis (#4)

* mdsctl: uinstall:simulator

* mdsctl: add preset:no-events

* mdsctl: -p:no-events

* mdsctl: install:simulator python pathing

* review feedback, boiling all of index.ts down

* k8s/util-namespace: bring forward from cola

* Add last provider event to audit_events table on every audit event

* Add endpoint to get vehicle by provider and vin

* mdsctl: support running from path

* Add geos summaries endpoint.

* mdsctl: support running from any directory, add --quiet option to be 'less verbose' (can likely do more)

* Consolidate provider event flattening, fix function-called-before-definition lint error

* review feedback

* mdsctl: provide a means to disable (postgresql, redis) persistence

* mdsctl: conditionally update ~/[rc] to add mdsctl to PATH env

* mdsctl: conditionally add mdsctl to PATH

* mdsctl: fix install:logging namespace

* Bump versions

* mdsctl: override fluentbit

* mdsctl: logging local

* mdsctl: don't istio sidecar logging

* test fails fuck

* mdsctl: push logging to util namespace

* Fixed Geography type.

* Using fancier types.

* improving tests

* review feedback

* mdsctl: leverage getopts/optspec

* mdsctl: cleanup

* Resolve Geography related build errors.

* Implement count minimums.

* Break readGeographies into separate methods for reading Geography and GeographySummary objects.

* Rename readGeographiesSummary -> readGeographiesSummaries, and replace duplicate logic with a call to readGeographies then simply strip out the geography_json

* Minor tweaks due to vscode doing some poor automagical renaming

* Better parameter destructuring

* Rename readGeographiesSummaries -> readGeographySummaries

* Resolve lint error, oops

* Add additional logging info to getProviderMetrics()

* Bump version

* mdsctl: tab completion

* [mds-agency] s/car/carshare/g MDSAAS-277

Slack discussion https://lacuna-tech.slack.com/archives/CHXQXQK5M/p1571325224046400

* update ref to enum key

* add a small test

* mdsctl: remove python env dep

* mdsctl: s/presets/preset/g

* Resolve bad import

* finitio

* MDSAAS-275 Remove AWS Lambda handlers, etc. (#25)

* MDSAAS-275 Remove AWS Lambda handlers, etc.

* Remove lambda deployment script runner

* Run ESLint concurrently before unit tests

* Remove AWS reference from comment

* Remove explicit typing

* Bump version

* Publish

 - @container-images/env-inject@0.1.13
 - @container-images/mds-agency@0.1.14
 - @container-images/mds-audit@0.1.15
 - @container-images/mds-compliance@0.1.14
 - @container-images/mds-daily@0.1.14
 - @container-images/mds-metrics-sheet@0.1.15
 - @container-images/mds-native@0.0.12
 - @container-images/mds-policy-author@0.1.14
 - @container-images/mds-policy@0.1.14
 - @container-images/mds-provider@0.1.14
 - @mds-core/mds-agency@0.0.16
 - @mds-core/mds-api-authorizer@0.1.14
 - @mds-core/mds-api-helpers@0.1.14
 - @mds-core/mds-api-server@0.1.14
 - @mds-core/mds-audit@0.1.26
 - @mds-core/mds-cache@0.1.14
 - @mds-core/mds-compliance@0.1.15
 - @mds-core/mds-daily@0.0.15
 - @mds-core/mds-db@0.1.14
 - @mds-core/mds-logger@0.1.12
 - @mds-core/mds-metrics-sheet@0.0.15
 - @mds-core/mds-native@0.0.12
 - @mds-core/mds-policy-author@0.0.11
 - @mds-core/mds-policy@0.0.15
 - @mds-core/mds-provider@1.0.14
 - @mds-core/mds-providers@0.1.14
 - @mds-core/mds-stream@0.1.14
 - @mds-core/mds-test-data@0.1.14
 - @mds-core/mds-types@0.1.11
 - @mds-core/mds-utils@0.1.14

* comments

* Rename previous_geographies column and update deleteGeography so it handles all cases where read_only is not true.

* Fix bad null checking. Closes MDSAAS-289

* MDSAAS-202 Upgrade to Node.js 10.16.3 LTS runtime (#12)

* MDSAAS-202 Upgrade to Node.js 10.16.3 LTS

* Suppress node deprecation warnings

* review feedback

* bump root version

* Publish

 - @container-images/env-inject@0.1.14
 - @container-images/mds-agency@0.1.15
 - @container-images/mds-audit@0.1.16
 - @container-images/mds-compliance@0.1.15
 - @container-images/mds-daily@0.1.15
 - @container-images/mds-metrics-sheet@0.1.16
 - @container-images/mds-native@0.0.13
 - @container-images/mds-policy-author@0.1.15
 - @container-images/mds-policy@0.1.15
 - @container-images/mds-provider@0.1.15
 - @mds-core/mds-agency@0.0.17
 - @mds-core/mds-api-authorizer@0.1.15
 - @mds-core/mds-api-helpers@0.1.15
 - @mds-core/mds-api-server@0.1.15
 - @mds-core/mds-audit@0.1.27
 - @mds-core/mds-cache@0.1.15
 - @mds-core/mds-compliance@0.1.16
 - @mds-core/mds-daily@0.0.16
 - @mds-core/mds-db@0.1.15
 - @mds-core/mds-logger@0.1.13
 - @mds-core/mds-metrics-sheet@0.0.16
 - @mds-core/mds-native@0.0.13
 - @mds-core/mds-policy-author@0.0.12
 - @mds-core/mds-policy@0.0.16
 - @mds-core/mds-provider@1.0.15
 - @mds-core/mds-providers@0.1.15
 - @mds-core/mds-stream@0.1.15
 - @mds-core/mds-test-data@0.1.15
 - @mds-core/mds-types@0.1.12
 - @mds-core/mds-utils@0.1.15

* match path prefix, optionally followed by a slash, optionally followed by [-a-z] character set

* Populate publish_date on geographies.

* review feedback

* incrase timeout

* Fix bad geographies migration

* Minor fix

* Bump main version

* Publish

 - @container-images/env-inject@0.1.15
 - @container-images/mds-agency@0.1.16
 - @container-images/mds-audit@0.1.17
 - @container-images/mds-compliance@0.1.16
 - @container-images/mds-daily@0.1.16
 - @container-images/mds-metrics-sheet@0.1.17
 - @container-images/mds-native@0.0.14
 - @container-images/mds-policy-author@0.1.16
 - @container-images/mds-policy@0.1.16
 - @container-images/mds-provider@0.1.16
 - @mds-core/mds-agency@0.0.18
 - @mds-core/mds-api-authorizer@0.1.16
 - @mds-core/mds-api-helpers@0.1.16
 - @mds-core/mds-api-server@0.1.16
 - @mds-core/mds-audit@0.1.28
 - @mds-core/mds-cache@0.1.16
 - @mds-core/mds-compliance@0.1.17
 - @mds-core/mds-daily@0.0.17
 - @mds-core/mds-db@0.1.16
 - @mds-core/mds-logger@0.1.14
 - @mds-core/mds-metrics-sheet@0.0.17
 - @mds-core/mds-native@0.0.14
 - @mds-core/mds-policy-author@0.0.13
 - @mds-core/mds-policy@0.0.17
 - @mds-core/mds-provider@1.0.16
 - @mds-core/mds-providers@0.1.16
 - @mds-core/mds-stream@0.1.16
 - @mds-core/mds-test-data@0.1.16
 - @mds-core/mds-types@0.1.13
 - @mds-core/mds-utils@0.1.16

* better regex - match the pathPrefix followed by either EoL or a slash and zero or more characters

* route by FQDN in ingress

* update values for ingress routing by domain

* ALTER TABLE RENAME COLUMN ... TO ... (#36)

* remove letsencrypt option - never worked reliably and not the ingress is not compatible with non-LE TLS (not possible to switch back and forth)

* remove letsencrypt option - never worked reliably and not the ingress is not compatible with non-LE TLS (not possible to switch back and forth)

* standardize API TCP ports to 4000, remove some never-used selectors for enabling/disabling DB and Cache env vars, make Eventing selector global instead of per-api

* bump package version

* Publish

 - @container-images/env-inject@0.1.16
 - @container-images/mds-agency@0.1.17
 - @container-images/mds-audit@0.1.18
 - @container-images/mds-compliance@0.1.17
 - @container-images/mds-daily@0.1.17
 - @container-images/mds-metrics-sheet@0.1.18
 - @container-images/mds-native@0.0.15
 - @container-images/mds-policy-author@0.1.17
 - @container-images/mds-policy@0.1.17
 - @container-images/mds-provider@0.1.17
 - @mds-core/mds-agency@0.0.19
 - @mds-core/mds-api-authorizer@0.1.17
 - @mds-core/mds-api-helpers@0.1.17
 - @mds-core/mds-api-server@0.1.17
 - @mds-core/mds-audit@0.1.29
 - @mds-core/mds-cache@0.1.17
 - @mds-core/mds-compliance@0.1.18
 - @mds-core/mds-daily@0.0.18
 - @mds-core/mds-db@0.1.17
 - @mds-core/mds-logger@0.1.15
 - @mds-core/mds-metrics-sheet@0.0.18
 - @mds-core/mds-native@0.0.15
 - @mds-core/mds-policy-author@0.0.14
 - @mds-core/mds-policy@0.0.18
 - @mds-core/mds-provider@1.0.17
 - @mds-core/mds-providers@0.1.17
 - @mds-core/mds-stream@0.1.17
 - @mds-core/mds-test-data@0.1.17
 - @mds-core/mds-types@0.1.14
 - @mds-core/mds-utils@0.1.17

* Upgrade dependencies (#29)

* Upgrade dependencies to latest versions

* More upgrades

* hard-code container port

* initial commit, docker-compose with an nginx gateway proxy

* default ports

* move out of subdir

* simplify PG_HOST_READER - can leave unset and the mds-db code will fall back to PG_HOST, make PG_PORT optional

* add optional slack integration vars, common secret for APIs

* changed secret name

* Add standard API request logging (#43)

* Add standard request logging

* port must be string not int

* fix slack token and secret

* fix value for ingress domain FQDN

* Remove provider event processor (#41)

* Remove provider event processor

* Drop deprecated provider tables

* Enable SQL logging during migrations

* Only drop tables that are/were part of the schema

* need to quote domains in case of a wildcard globa ("*")

* Remove read_only column from geographies.

* Remove unused provider-stats cache methods

* bump version

* hard-code port number

* correct secret name

* template docker-compose with latest build versions

* Add perf logging to MDS Daily db/cache calls

* fluentbit (mds/requirements rm, util/values pin to es host) (#50)

* narrow fluentbit refs

* fluentbit: rm from mds/requirements, pin to elasticsearch-master in util/values

* helm/(mds,util): rm fluent from mds

* pull changes from fix/michael/remove-provider-api

* Add perf logging to MDS Daily db/cache calls

* Resolve linter errors

* Bump root package version

* Publish

 - @container-images/env-inject@0.1.17
 - @container-images/mds-agency@0.1.18
 - @container-images/mds-audit@0.1.19
 - @container-images/mds-compliance@0.1.18
 - @container-images/mds-daily@0.1.18
 - @container-images/mds-metrics-sheet@0.1.19
 - @container-images/mds-native@0.0.16
 - @container-images/mds-policy-author@0.1.18
 - @container-images/mds-policy@0.1.18
 - @container-images/mds-provider@0.1.18
 - @mds-core/mds-agency@0.0.20
 - @mds-core/mds-api-authorizer@0.1.18
 - @mds-core/mds-api-helpers@0.1.18
 - @mds-core/mds-api-server@0.1.18
 - @mds-core/mds-audit@0.1.30
 - @mds-core/mds-cache@0.1.18
 - @mds-core/mds-compliance@0.1.19
 - @mds-core/mds-daily@0.0.19
 - @mds-core/mds-db@0.1.18
 - @mds-core/mds-logger@0.1.16
 - @mds-core/mds-metrics-sheet@0.0.19
 - @mds-core/mds-native@0.0.16
 - @mds-core/mds-policy-author@0.0.15
 - @mds-core/mds-policy@0.0.19
 - @mds-core/mds-provider@1.0.18
 - @mds-core/mds-providers@0.1.18
 - @mds-core/mds-stream@0.1.18
 - @mds-core/mds-test-data@0.1.18
 - @mds-core/mds-types@0.1.15
 - @mds-core/mds-utils@0.1.18

* Purge mds-provider implementation

* Update LICENSE

* Remove mds-cache weird duplicate functions

* Clean up mds-audit weirdness

* Clean up mds-db weirdness

* Clean up mds-types weirdness

* Clean up helm template weirdness

Co-authored-by: Hunter Owens <owens.hunter@gmail.com>

* Fix yarn.lock

* [mds-policy-author][mds-geography-author] Extract geography author service from policy author. (#203)

* Extract geography author service from policy author.

* review feedback

* review feedback

* review feedback

* RF

* RF

* satisfy linter

* SqlVals and vals_list used to sanitize db input.

* rf

* helm unittest (#209)

* mdsctl: helm unittest service

* heml uuniest: auth

* helm unittest: auth disable

* helm unittest: autoscaler

* helm unittest: +configmap

* helm unittest: cronjob

* helm unittest: egress

* helm unittest: ingress

* helm unittest: ingress domain

* helm unittest: deployment

* helm unittest: deployment secret

* helm unittest: deployment overrides

* helm unittest: better suite labels

* helm force upgrade

* mdsctl: ensure resolvable k8s endpoint (thx jane)

* Reorder writing an event and related telemetry to avoid race condition where event is written but telemetry has not been written yet (#205)

* [mds-agency] [mds-web-sockets] Add NATS-Streaming consumption for mds-web-sockets (#208)

* Add mds-web-sockets consumption of NATS stream rather than agency having to persist a direct connection with the web-socket server

* Clean up okteto.yml

* Handle STAN errors better

* eslint-ignore

* Istanbul, not eslint

* [mds-policy] [mds-geography] Extract read-only geography service from policy service (#211)

* Extract read-only geo points from Policy.

* making mds-policy tests pass

* Tests for mds-geography.

* pacifying linter

* remove todo comment

* removing TODO comment

* Feature/alex/monitoring alerting (#213)

* initial commit, canned alerts for prometheus

* whitespace, add additional alerts

* clean up summaries and descriptions

* clean up slack notifications

* better tracking of container restarts (count over time instead of absolute), better labeling of statefulset failures

* use rate over last five minutes

* bump version

* update lockfile, try removing auth from grafana (thanks james) (currently broken)

* add prometheus datasource, some useful dashboards.  remove login since access requires authenticated kubectl

* update

* ignore downloaded charts

* more meaningful restart rate

* tuning

* add alarms for pod CPU and memory going over limit

* switch disk usage alerts from static to predictive

* Upgrade package dependencies (#212)

* Upgrade package dependencies

* Add type annotation to fix build error

* This test was destined to fail starting yesterday (#215)

* [mds-processors][mds-metrics] removal for OMF release (#210)

* removal spike

* db/cache util and type removal

* turf deps

* purge metrics + more processors

* remove cron from mdsctl

Co-authored-by: Max Johansen <macsj200@gmail.com>

* [mds-policy-author] Minor changes to policy to bring it back in line with the spec. (#217)

* Fixing parameters for policy author.

* Adding User Rule type to Policy.

* [mds-stream] Add outbound kafka stream support (#216)

* Add outbound kafka stream support

* Cleanup

* Review cleanup pt1

* Refactoring and cleanup

* Call KafkaStream.initialize() from top-level initialize method

* Call stream.initialize() at the top of mds-agency/request-handlers

* Update README.md

* Remove stream.initialize() from agency tests

* Remove stream.initialize() from compliance tests

* Writer refactor, and reader implementation

* Fix unhappy exports

* Remove callback fallback :P

* clean up stream init

* Run with new Prettier 2.0 defaults (#221)

* Run with new prettier defaults

* Revert trailingComma to previous default

* Revert arrowParens to previous default

* Removing Slack client and pushover notifications (#226)

* Removing Slack client and pushover notifications

* Implement new method to redact sensitive text

* Kafka Stream Processor (#224)

* Beginnings of initial refactor

* More cleanup of kafkajs, removal of node-rdkafka

* Rename and refactor

* Use Nullable<T>

* Remove temporary type declaration

* Remove file

* Make producers topic specific

* Basic functional streap labeler

* Remove EventServer

* Remove mds-event-server

* Refactor stream processing

* Fix module export issues

* Add container image for vehicle event processor

* Fix package name

* Add helm deployment for event-processor

* await NATS write instead of returning

* Remove reliance on deprecated service_area_id

* Incorporate review feedback

* Merge remote-tracking branch 'origin/develop' into feature/neil/begin-kafka-refactor-to-kafkajs

Co-authored-by: Neil Goldader <neilgoldader@gmail.com>

* [mds-utils] Update relative time parsing to snap to the nearest hour when 'now' is provided as a query param (#223)

* Update relative time parsing to snap to the nearest hour when  is provided as a query param

* Undo old prettier weirdness

* Add caching of device labels (#229)

* Add caching of device labels

* Rename some consts

* Add control methods to stream processor (#232)

* Add control methods to stream processor

* Minor package refactoring

* Add copyright notices

* Remove some spammy logs from mds-cache

* [mds-stream-processor] Add geo labeling for mds-stream-processor (#231)

* Add geo labeling for mds-stream-processor

* Add tests

* Add Sinon devDependency

* Add LatencyLabeler tests

* Filter based on point in shape, then map to geography_id

* Remove now unused import

* Only compute for published geographies

* [ladot-service-areas] [mds-agency] [mds-daily] Purge service-areas (#219)

* Purge service-areas

* Rename mds-test-data/ladot-service-areas to mds-test-data/test-areas

* Get rid of require statements

* Remove mds-metrics-sheet vehicle counts (#228)

* Remove mds-metrics-sheet vehicle counts

* Remove vehicle-counts.ts

* Remove duplicate Nullable<T> definition

* Fix mds-metrics-sheet container image build (#233)

* Reorganize files

* Remove extraneous vehicle-counts references

Co-authored-by: Neil Goldader <neilgoldader@gmail.com>

* Fix event type for vehicle reads in case of a cache miss (#225)

* Fix Project References (#234)

* Agency should not be writing tel
8000
emetry to socket

* Fix typescript project references

* Make sinon a dev dependency

* Upgrade dependencies

* [mds-native] Removing mds-native. (#214)

* Removing mds-native.

* Removing helm tests.

* Removing reference in nginx.

* [mds-db] [mds-policy-author][mds-compliance] Fix/final separation of policy from geo (#218)

* Publishing a policy no longer publishes a geography. Instead it throws an error.

* minor fixes

* Removing container-images directory for mds-native. (#236)

* Remove leftover artifacts of mds-metrics (#238)

* [mds-stream] [mds-stream-processor] TENANT_ID prefix for Kafka Streams (#235)

* Add TENANT_ID prefix to kafka to support multi-tenancy on one Kafka cluster

* Yarn.lock changes I guess?

* Add a couple source/sink examples (#237)

* Re-add stream init that was removed, thanx git (#239)

* Add FileSource and FileSync connectors for mds-stream-processors (#240)

* [mds-stream-processor] First pass at vehicle telemetry processor (#241)

* First pass at vehicle telemetry processor

* Add getEnvVar method to mds-utils

* Refactor getEnvVar

* Refactor TelemetryFlattener to TelemetryLabeler, add OptionalTelemetryLabeler

* Rename FlattenedTelemetry, update NullableProperties type

* [helm] Update helm for telemetry processor (#245)

* Update helm for telemetry processor

* do not create auth policies or ingress virtualservices for APIs with no pathPrefix (ie internal-only services)

Co-authored-by: Alex Gottschalk <alex.gottschalk@gmail.com>

* Initial Metrics Service (#244)

* Add ORM model for metrics

* Update dependencies

* Create some service helper methods

* Implement metrics filtering on read

* Allow specifying one of more filter values

* Refactor metric query tests to minimize redundancy

* Refactor utility function and add missing licenses

* Remove OptionalTelemetryLabeler

* Code review revisions

* was missing geography api (#242)

* Split metrics-service into server/client modules (#246)

* [mds-stream] Switch from NATS Streaming -> Vanilla NATS (#243)

* First pass

* Minor fix that will be replaced once I merge in

* Retry forever

* waitOnFirstConnect fix

* Fix variable names

* Add typing for NATS messages

* NATS env var refactor

* nats-streaming folder -> nats folder

* Thanks F2

* Path fix

* Enable ES2019 features in node:12.14.1-alpine (#247)

* Refactor ORM Connection Configuration (#249)

* Fix coverage excludes

* Support named connections

* Curry the ORM connection factory functions

* Refactor Jurisdiction Service (#250)

* ServiceProvider interface

* CreateRepository function

* CreateRepositoryMethod function

* Move jurisdiction types/utils to service package (#251)

* Move jurisdiction types/utils to service package

* Use helper method for filter

* Move schema to separate file

* Export default connection for TypeORM CLI (#252)

* Export default connection for TypeORM CLI

* More robust destructuring

* Fix import statement

* Clean up error handling

* Improve test coverage

* Use recommended typescript config for istanbul (#255)

* Use recommended typescript config for istanbul

* Clean up some config files

* Create types for mapping between entity/domain models (#257)

* Create types for mapping between entity/domain

* Fix test failure

* Fluent model mapper syntax

* Use object schema for jurisdiction validation

* Fix imports and make mapper syntax consistent (#258)

* [mds-stream-processor] First pass at pre-computed surrounding bbox (#253)

* First pass at pre-computed surrounding bbox

* Add some tests

* Remove console.log

* Typecasting is bads

* Add geography caching test

* Remove unused import

* Better destructuringz

Co-authored-by: Mark Maxham <max@ellis-and-associates.com>

* Update prettier config to enforce single quotes only in js/ts files, since we may want to have double quotes in yaml files. (#256)

* Update prettier config to enforce single quotes only in js/ts files, since we may want to have double quotes in yaml files.

* Making file formatting changes restricting to js and ts files

* [mds-config][mds-config-service] Remove mds-config and mds-config-service  (#254)

* Removing mds-config and mds-config-service, as these will not be part of the OMF contribution.

* Removing mds-config and mds-config-service as part of prepping for the OMF contribution.

* Reverting single quotes in yaml files.

* Reverting more single quotes.

Co-authored-by: Mark Maxham <max@ellis-and-associates.com>

* Allow services to specify TypeORM CLI options. (#259)

Use typeorm executable in ./node_modules/.bin

* Serializable service errors (#261)

* Use service provider interface for unit tests

* Move metrics service

* Move jurisdiction service

* Serializable service error objects

* Abstract processing of repository errors

* RepositoryError.GetProperty helper

* Use static error factory and type guards

* Minor refactor of type guard syntax

* Resolution of type errors that resul…