Docker compose files for starting a NuoDB database on your local host;
- and, optionally, initializing the database state from an existing NuoDB backup.
NOTE: docker compose is intentionally a very simple tool - so the commands are more wordy than would be ideal; some desirable automation is just not possible; and some benign error messages are occasionally emitted.
- Create a local NuoDB database in Docker on a developer's laptop;
- Create a local copy of an existing database for diagnostic purposes;
- see the
IMPORT_LOCALandIMPORT_REMOTEoptions;
- see the
- Create a local copy of an existing database for UAT or testing purposes;
- see the
IMPORT_LOCALandIMPORT_REMOTEoptions;
- see the
- Create a local database for testing versions, SQL, ...
- Create a simple multi-engine database on a single cloud node (VM);
- Create multiple local databases for testing different software versions of either NuoDB or an application.
distributed(default)- This is a database comprising a separate container for each database process:
- Separate AP (admin), TE, and SM containers - one for each NuoDB process;
- This is the same topology used when NuoDB is deployed in production and under Kubernetes, but may be overkill for a local machine - especially a laptop.
- This is a database comprising a separate container for each database process:
distributed + insights- This is a database comprising 3 separate containers for the database processes (as above), plus additional containers including
influxdbandgrafanato enable metrics publishing and display. - This is currently the only configuration that supports NuoDB
Insightsmonitoring.
- This is a database comprising 3 separate containers for the database processes (as above), plus additional containers including
monolith- This is a database in a single container (monolith). All 3 NuoDB processes are running inside the same container.
- This is not how NuoDB is deployed in production, but is an easily managed, resource optimized, option on a local machine.
- This is the simplest configuration for a developer to run a local DB for testing and debugging.
- This is a database in a single container (monolith). All 3 NuoDB processes are running inside the same container.
instadb- This is a database in a single container (as per
monolith) - but with dynamic port mapping.- This allows multiple
instadbdatabases to run on the same host simultaneously. - The downside is that because of the dynamically-mapped ports, the mapped public port cannot be predicted, and so external connections (such as non-containerized applications running on the same computer) will need to use the
direct=trueconnection property; - To ensure container name uniqueness, each
instadbinstance must be running in a differentprojectfrom any otherinstadbinstance.
- This allows multiple
- This is a database in a single container (as per
These Docker compose files will create:
- A new Docker network specifically for the project;
- A database in one of 4 possible configurations (see above).
A second TE can be added to a distributed database
- See the
scale-te2profile.
Note that all container names will have the project name embedded, which is the name of the directory (nuodb), but you can override the project name using the -p option to docker compose.
- This default is fine for all configurations except
instadbdatabases (you can use the default for your firstinstadbdatabase, but all subsequent databases must be named explicitly using-pto ensure unique container names).
-
Clone this repo.
-
cdto thenuodbdirectory. -
Copy the
env-defaultfile to.env(this step is NOT optional). -
If using NuoDB v6.0.2 or greater, acquire a NuoDB license file.
- If you or your organisation do not have a valid NuoDB license file, contact NuoDB support to request one: NuoDB.Support...at...3ds.com.
- Store your NuoDB license file somewhere on your local disk
-
Edit the
.envfile:ENGINE_MEM- Sets the memory cache size for each TE and SM.
SQL_ENGINE:- If you want to use a specific SQL engine, such as
scalar(the vector engine,vee, is the default). You will also need an image that supports that engine;
- If you want to use a specific SQL engine, such as
EXTERNAL_ADDRESS:- If you want to access the database from outside the
docker network- for example from an app running directly on the local host - then setEXTERNAL_ADDRESS;- either in the
.envfile, or by settingEXTERNAL_ADDRESSon thedocker compose upcommand-line (Linux/MacOS) or by first settingEXTERNAL_ADDRESSas an environment variable (Windows); - set to the address of the local host machine (Ex
192.168.0.123); - on some platforms, setting
EXTERNAL_ADDRESSto127.0.0.1also works;
- either in the
- If you want to access the database from outside the
LICENSE_PATH:- If you have a valid NuoDB license file - per step #4 above - then set
LICENSE_PATHto point to that file path.- Eg:
LICENSE_PATH=./nuodb.lic
- Eg:
- If you have a valid NuoDB license file - per step #4 above - then set
IMPORT_LOCAL,IMPORT_REMOTE,IMPORT_TIMEOUT,IMPORT_AUTH,IMPORT_LEVEL- If you want to import initial state from a database backup into the new database, set
IMPORT_LOCALand/orIMPORT_REMOTE(seeNotesbelow for details ofIMPORT_LOCALandIMPORT_REMOTE);- the
importoperation is only performed when the archive dir is empty - so the SM container can be stopped and restarted without being reinitialized each time. - if you have set
IMPORT_LOCALorIMPORT_REMOTEand it is a large archive that takes multiple minutes to import, you will need to setIMPORT_TIMEOUTto a value larger than the time taken to import - in order to stop the DB startup from timing out before the IMPORT has completed.
- the
- If you want to import initial state from a database backup into the new database, set
NOTE: In earlier versions of docker, the docker-compose command was the only form that worked with nuodb-compose.
However, with newer versions of docker both docker-compose and docker compose work. docker-compose is slated to be removed from the docker product.
-
A database is created using
docker compose ... up ...;- NOTE: the
docker compose ... downcommand will destroy the database and all its storage. More on this below:
- NOTE: the
-
A database can be
stoppedWITHOUT deleting its storage:docker compose ... stop ...;
-
A stopped database can be
restartedand will CONTINUE using its previous existing storage:docker compose ... start ...;- this is often needed after a host machine has been woken up after sleep/hibernation.
-
A database that was running, but has stopped for some environmental reason, can be
restartedwith a single command:docker compose ... restart- for example, if the host machine was hibernated, and then re-awakened, the database will typically need to be
restarted.
-
A database is
deletedWITH its storage using:docker compose ... down;
-
A client app connects to a database by configuring a port on the host network - set with the variable
EXTERNAL_ADDRESS- into its connection string/params;- example:
EXTERNAL_ADDRESS=192.167.0.123if the local host's network address is192.168.0.123 - example:
EXTERNAL_ADDRESS=localhostifdockercan resolvelocalhost.
- example:
-
Because
docker composedoes not scope the Docker networks it creates to a particular file or profile,docker compose ... downmay attempt to delete a network that is still in use by a different database. The error looks like the following, and can be ignored:⠿ Network nuodb_net Error 0.0s failed to remove network df0df85905b1702fea9c1a20a1142b9f4ff85f07844087b520f072c8a6af5e68: Error response from daemon: error while removing network: network nuodb_net id df0df85905b1702fea9c1a20a1142b9f4ff85f07844087b520f072c8a6af5e68 has active endpoints
WARNING: In the following do not confuse start and stop with up and down.
docker upis used to create the database container and start it running.- You can stop and restart the container using
docker stopanddocker start- no database data will be lost. docker downdestroys the container and your database will be lost.
NOTE: the distributed database is the default configuration.
createwith:docker compose up -dstop(temporarily) all containers with:docker compose stop- this will NOT delete the database storage;
restarta stopped database with:docker compose startstopandstarta database in limbo withdocker compose restartdelete- including storage - withdocker compose downconnectto adistributeddatabase using the value ofEXTERNAL_ADDRESSin the connection string.
- To scale out a
distributeddatabase with a second TE:docker compose --profile scale-te2 up -d
- to scale in a
te2on adistributeddatabase:docker compose --profile scale-te2 stop
- to delete a
distributeddatabase plus its scaled-outte2in a single command:docker compose --profile scale-te2 down
NOTE: the monolith topology must be specified explicitly by using -f monolith.yaml.
createwith:docker compose -f monolith.yaml up -dstop(temporarily) all containers with:docker compose -f monolith.yaml stop- this will NOT delete the database storage;
restarta stopped database with:docker compose -f monolith.yaml startstopandstarta database in limbo withdocker compose -f monolith.yaml restartdelete- including storage - withdocker compose -f monolith downconnectto amonolithdatabase using the value ofEXTERNAL_ADDRESSin the connection string;- example connection string:
jdbc:com.nuodb://192/.168.0.123/demo - example connection string:
jdbc:com.nuodb://localhost/demo
- example connection string:
NOTE: the instadb topology must be specified explicitly by using -f instadb.yaml.
createwith:docker compose -f instadb.yaml up -dstop(temporarily) all containers with:docker compose -f instadb.yaml stop- this will NOT delete the database storage;
restarta stopped database with:docker compose -f instadb.yaml startstopandstartan instadb database in limbo with:docker compose -f instadb.yaml restartdelete- including storage - withdocker compose -f instadb.yaml downconnectto aninstadbby settingdirect=truein the connection properties;- and setting the value of
EXTERNAL_ADDRESS:<mapped-port>in the connection string;- where
<mapped-port>is the public port mapped to port48006for that container.
- where
- and setting the value of
Port mapping example:
Note that the TE is always on container (internal) port 48006
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
458760270ee7 nuodb/nuodb:5.0.0.2 "docker-entrypoint.s…" 6 minutes ago Up 6 minutes 0.0.0.0:54587->8888/tcp, 0.0.0.0:54584->48004/tcp, 0.0.0.0:54585->48005/tcp, 0.0.0.0:54586->48006/tcp nuodb-instadb-1
$ docker compose port instadb 48006
0.0.0.0:54586In this example, to connect to the database (actually to its TE) use the value of EXTERNAL_ADDRESS:54586 in the connection string (along with direct=true).
- Example connection string:
jdbc:com.nuodb://192.168.0.123:54586/demo?direct=true - Example connection string:
jdbc:com.nuodb://localhost:54586/demo?direct=true
createan additionalinstadbwith:docker compose -p <new-project-name> -f instadb.yaml up -dstop(temporarily) all containers of a specificinstadbwith:docker compose -p <project-name> -f instadb.yaml stop- this will NOT delete the database storage;
restarta stopped database with:docker compose -p <project-name> -f intadb.yaml startstopandstartan instadb in limbo with:docker compose -p <project-name> -f instadb.yaml restartdelete- in 9108 cluding storage - withdocker compose -p <project-name> -f instadb.yaml down
The initial state of the database can be imported using IMPORT_LOCAL and/or IMPORT_REMOTE.
NOTE: You can specify environment variables on the command-line on Linux or MacOS, by setting them before the docker compose command.
- Example:
$ IMPORT_LOCAL=./mydb.bak.tgz STARTUP_TIMEOUT=300 docker compose up -d
Steps required:
-
Set
IMPORT_LOCALto a path on the local machine. The SM container will mount this path as a volume and import its contents into the archive dir prior to starting the SM process (presuming the archive is empty);The path that
IMPORT_LOCALpoints to can be one of:- a
tar.gzipfile of anuodb backup; - a directory containing a
nuodb backup.
Note: that a
nuodb backupcan come in one of two formats:- a nuodb
backup set- which is the result of ahotcopy --type fullbackup; - a nuodb
archive- which is the result of ahotcopy --type simple, or just a copy of an SM archive and journal copied while the SM is NOT running.
Note: a
backupsetcan only be imported from a directory. - a
-
Set
IMPORT_REMOTEto the URL of a remotefilehosted on a server - typically accessed throughhttp(s)or(s)ftp.- Example:
https://some.server.io/backup-4-3.tz - Example:
sftp://sftp.some.domain.com/backup-4-3.tz
Note: The SM container will download a
remotefile via the URL and extract it into the archive directory prior to starting the SM process. - Example:
-
If you set both
IMPORT_LOCALandIMPORT_REMOTE, thenIMPORT_REMOTEis treated as the remote source, andIMPORT_LOCALis treated as a locally cached copy. Hence the behaviour is as follows:- If
IMPORT_LOCALis a non-emptyfileordirectory, then it is used directly, andIMPORT_REMOTEis ignored. - If
IMPORT_LOCALis an emptyfilethenIMPORT_REMOTEis downloaded intoIMPORT_LOCAL, and theimportis then performed byextractingfromIMPORT_LOCALinto thearchive;- note this only works for a
tar.gzipfile of anarchive(see above).
- note this only works for a
- If
IMPORT_LOCALis an emptydirectorythenIMPORT_REMOTEis downloaded and extracted intoIMPORT_LOCAL, and theimportis then performed fromIMPORT_LOCALinto thearchive;- note this works for both forms of
nuodb backup(see above); - Note: importing from a
directorycan be significantly slower than importing (extracting directly) from atar.gzipfile.
- note this works for both forms of
Hence: To cause the initial download from
IMPORT_REMOTEto be cached inIMPORT_LOCAL,IMPORT_LOCALmust exist and be empty. To ensure this, you can do something like the following:-
$ rm -rf a/b/c -
$ touch a/b/cormkdir -p /a/b/cNow you can set
IMPORT_REMOTEas needed, and setIMPORT_LOCALtoa/b/c.
- If
-
If you get an error in the form:
WARNING: The NUODB_IMAGE variable is not set. Defaulting to a blank string. WARNING: The PEER_ADDRESS variable is not set. Defaulting to a blank string. WARNING: The IMPORT_SOURCE variable is not set. Defaulting to a blank string. ... ERROR: Couldn't find env file: /a/b/c/nuodb/.env- then you have most likely forgotten to copy the
env-defaultfile to.env; - or (less popular) not created your own
.envfile from scratch.
- then you have most likely forgotten to copy the
-
If you get one or more warnings of the form:
WARNING: The <xyz> variable is not set. Defaulting to a blank string.- but NO ERROR, then you probably need to update your
.envfile. - Do a
diffof.envvsenv-default, and look for new VARs defined in the latestenv-default.
- but NO ERROR, then you probably need to update your
-
If you get an error of the form:
ERROR: for nuodb_nuocd-sm_1 Cannot start service nuocd-sm: container ... is not runningcheck the logs of the
sm_1container to see why it has failed to start.- To check the logs of any container, run
docker logs <name-of-container> - Example:
docker logs nuodb_sm_1
- To check the logs of any container, run
-
If you get an error in the form:
IMPORT_REMOTE is not a valid URL: ... - import abortedthen you have not set
IMPORT_REMOTEto a valid URL.- URL is expected to be in the form:
<protocol>://<host>/<path> - Example:
sftp://myhost.com/my-archives/backup-xyz.tar.gzip
- URL is expected to be in the form:
-
If you get an error in the form:
This database has <n> archives with no running SM. No new SMs can start while the database is in this state.then you have somehow restarted the database with existing archives but too few running SMs.
- This could happen if an import has somehow failed after the initial import started, and you restart with
IMPORT_Xset. - This could also happen if an SM has shut down, and you try to restart it with
docker compose up, but have accidentally setIMPORT_X. (You cannot attempt to import the database state if there is existing state in some archive - even if the SM for that archive is not currently running.) - Follow the instructions following the error message to resolve the problem(s), and then continue stating with:
... docker compose up -d
- This could happen if an import has somehow failed after the initial import started, and you restart with
-
If an error causes only part of a
distributeddatabase to be deployed, you can start the remaining containers - after fixing the error - by simply running... docker compose up -dagain. Theupcommand only starts those containers that are not currently running.- When running
... docker compose upa subsequent time, you need to decide if you still need to setIMPORT_Xvariable(s):- you DON'T need to if the database state has already been successfully imported;
- you probably DO need to if you had them set for the original
docker compose upcommand, and theimporthas not yet succeeded.
- When running
-
If you get an error about being unable to delete a network because it has active end-points, you can normally safely ignore this.
⠿ Network nuodb_net Error 0.0s failed to remove network df0df85905b1702fea9c1a20a1142b9f4ff85f07844087b520f072c8a6af5e68: Error response from daemon: error while removing network: network nuodb_net id df0df85905b1702fea9c1a20a1142b9f4ff85f07844087b520f072c8a6af5e68 has active endpointsIf you think that the network should have been deleted, then run the
docker compose ... downcommand once more. If the end-points have now been successfully deleted, the network delete will now succeed.