This is a multi-part repository it contains code for:
- The freva-rest service defining rest endpoints that make up the freva server services
- The freva-client library that provide command line interfaces and python libraries for their rest service counterparts.
- The freva-data-loader-portal that implements rules of how to open different data sources and stream them via zarr.
-
Make sure you have Python 3.8+ installed.
-
Clone this repository:
git clone --recursive git@github.com:freva-org/freva-nextgen.git cd freva-nextgen
-
Install all components:
make installAlternatively you can create an encapsulated conda enviroment:
conda create -n freva-nextgen cfgrib conda run -n freva-nextgen make install
Various services, such as apache solr are needed to run the rest services system
in a development environment. Here we set up these services in a containers
using the docker-compose or podman-compose command, ensure
you have docker-compose or podman-compose installed on your system.
Then, run the following command:
USER=$(whoami) docker-compose -f dev-env/docker-compose.yaml up -d --remove-orphansif you use podman:
python -m pip install podman-compose
USER=$(whoami) podman-compose -f dev-env/docker-compose.yaml up -d --remove-orphansThis will start the required services and containers to create the development environment. You can now develop and test the project within this environment.
After the containers are up and running you will need to install the REST server and start it doing the following:
python run_server.py --config dev-env/api_config.toml --debug --dev -p 7777 -fThe --debug and --dev flag will make sure that any changes are loaded.
You can choose any port you like. Furthermore the --dev flag will pre
load any existing test data. If you don't like that simply do not pass the
--dev flag.
The following users are available for log on:
- uid:
<YOUR-USER>, password:secret - uid:
johndoe, password:johndoe123 - uid:
janedoe, password:janedoe123 - uid:
alicebrown, password:alicebrown123 - uid:
bobsmith, password:bobsmith123 - uid:
lisajones, password:lisajones123
<YOUR-USER> is the user name that started the compose command.
Unit tests, Example notebook tests, type annotations and code style tests are done with tox. To run all tests, linting in parallel simply execute the following command:
tox -p 3You can also run the each part alone, for example to only check the code style:
tox -e lintavailable options are lint, types, test.
Tox runs in a separate python environment to run the tests in the current environment use:
python -m pip install -e freva-rest[tests] freva-client freva-data-portal-worker
pytest -vv ./testsThis repository defines two docker images that can be used for production
freva-rest-api(restAPI)data-loader-worker(zarr streaming)
These images are automatically build via the GitHub CI pipeline. Yet for debugging they can be built using the following commands:
- First setup mulitarch build:
docker run --rm --privileged multiarch/qemu-user-static --reset -p yes && \
docker buildx create --name multiarch --use && \
docker buildx inspect --bootstrap- Build for the
data-loader-workerimage for the target platforms
docker buildx build --no-cache --platform linux/amd64,linux/arm64,linux/ppc64le\
-t ghcr.io/freva-org/freva-data-loader:latest \
-t ghcr.io/freva-org/freva-data-loader:<VERSION> \
--build-arg=VERSION=<VERSION> \
--build-arg=CMD=data-loader-worker .- Build for the
freva-rest-apiimage for the target platforms
docker buildx build --no-cache --platform linux/amd64,linux/arm64,linux/ppc64le\
-t ghcr.io/freva-org/freva-rest-api:latest \
-t ghcr.io/freva-org/freva-rest-api:<VERSION> \
--build-arg=VERSION=<VERSION> \
--build-arg=CMD=freva-rest-server .If you are using podman make sure buildah and qemu are set up for your OS.. You can then replace the
docker buildxcommands withbuildah.
Once the development is finished and you decide that it's time for a new release of the software use the following command to trigger a release procedure:
tox -e releaseThis will check the current version of the main branch and trigger
a GitHub continuous integration pipeline to create a new release. The procedure
performs a couple of checks, if theses checks fail please make sure to address
the issues.
This project is licensed under the MIT License - see the LICENSE file for details.