Eternal DB is a Java embedded time series database/data storage engine with a data retention policy based on a disk space. As oldest records are removed when needed to reclaim disk space, this database allows to collect data eternally. It gives unmatched control on a disk resource usage. Moreover, data are easily accessible for any tools as they are kept in regular disk files on a given file system. It's a schema-less document database as records are just JSON documents.
Some possible use cases:
- collecting measurement data
- collecting metrics
- collecting market data
In above cases the most important thing for a running system is to keep some time window of latest data. Eternal DB allows to express this time window in terms of a disk space.
Please note, that this is not a classic database system like RDBMS. It doesn't support SQL, users, transactions, constraints, relations, triggers, stored procedures, etc. Maybe the best term is a "data storage engine".
As this is designed to
D8C4
be used under Linux, it requires Coreutils https://www.gnu.org/software/coreutils/ or a compatible software module
that supports running the command du -sk. Most likely it can work with BusyBox as well: https://www.busybox.net/downloads/BusyBox.html#du
(not verified yet).
NOTE: This project is a work-in-progress (WIP).
- It is an embedded database (data store) that is easy to add to a program written in Java.
- It has a disk space based data retention policy. This is meant to keep as much data as possible within a defined disk space limit.
- When reclaiming disk space, oldest data records are removed.
- Records are kept in a text format in files on a given filesystem and these files are easily readable by other programs.
- Records are considered documents, like in document databases. No fixed data structure is forced.
- A record always has a timestamp and time is considered an indexed attribute.
- It is meant to run in a Linux environment.
- It is thread-safe.
- It is fault-tolerant. A single error should not block you from reading all your data. As files are written only by appending a corrupted file may only be a result of not closing a database object before JVM exit. In this case some records may be lost, but a single data file may have only the last record corrupted. A database can continue using this file when started again.
- This is not meant to run as a standalone server enabling multiple clients to connect and use.
- Ability to configure other data retention policy is not required.
- Support for any other operating system is not required if such an operating system cannot provide Linux-compatible runtime environment.
- Strict schema validation for data is not a goal.
- Strict transactions support is not a goal.
- Data access control is not a goal. A user is responsible to configure desired access rights to a directory defined for data files.
- Valid use of a single database storage from many database instances is not a goal.
- Support for efficient searching of records based on any other criteria than time period is not a goal.
- Project is developed in Java 21.
- Records are kept formatted as single line JSON each.
- Each line in a data file is beginning with timestamp being number of millis from the Epoch (as given by
System.currentTimeMillis()) and formatted with radix of 32 (to use less disk space), followed by the tab character, followed by JSON one-liner. - Unix line endings are used in data files, as it is only 1 byte and it makes it easier to detect corrupted file.
- Data are persisted as a disk files in directory tree that is representing periods of time.
- Project is avoiding unnecessary dependencies. No Vavr, Project Reactor, Guava, etc.
- The database should be easy to use in Spring Boot based application.
- By design there is no protection against improper use like two Eternal-db instances (class
Database) writing independently using overlapping directories on disk. Just to keep it simple. - A single Eternal-db database folder can be shared by multiple independently used read-only Eternal-db instances (class
ReadOnlyDatabase).
Add dependency to eternal-db library (use the latest version):
<dependency>
<groupId>io.github.k_tomaszewski</groupId>
<artifactId>eternal-db</artifactId>
<version>1.0.0</version>
</dependency>As of now the library is published only in GiHub Packages repository, so you need to add additional Maven repository in your pom.xml like this:
<repositories>
<repository>
<id>github_k-tomaszewski_eternal-db</id>
<url>https://maven.pkg.github.com/k-tomaszewski/eternal-db</url>
</repository>
</repositories>It seems that GitHub Packages repository requires authentication even for public
artifacts, so you need to have a GitHub account and you need to set up your credentials
(GitHub username and access token) for Maven in the settings.xml file. Ref:
- https://maven.apache.org/guides/mini/guide-multiple-repositories.html
- https://maven.apache.org/guides/mini/guide-deployment-security-settings.html
- https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-apache-maven-registry#authenticating-with-a-personal-access-token
To use it you create a single object of class io.github.k_tomaszewski.eternaldb.Database.
Such object gives access to operations on a single data store.
Please note that the Database is a generic class and has a type parameter. This is used for type
checking only, so a compiler can detect when a wrong object is passed for writing method.
To create instance of the Database class you need to provide a DatabaseProperties object, which contains following
configuration properties:
- path to a directory where data files are going to be stored
- limit of disk space to use, given as integer number of megabytes (MB)
- object being strategy for serialization and deserialization, the library provides
JacksonSerializationinstance as a default - object being strategy for naming data files, the library provides
BasicFileNaminginstance as a default - timestamp supplier - object implementing
ToLongFunction<T>that returns timestamp, possibly based on a record being written. This is optional.
Example:
Database<MyRecord> db = new Database<>(new DatabaseProperties<>(Path.of("/home/db"), 100));You need one such object for a single data storage directory. Instances of Database class are
thread-safe. You must not use many instances of Database class that are configured to use
the same directory or subdirectories of each other.
A basic method for writing data: void write(T record, long recordMillis). To use it you
need to give record timestamp when writing as this is a time series database. Example:
db.write(myRecord, System.currentTimeMillis());If a database is configured with a timestamp supplier (an object implementing ToLongFunction<T> interface),
then another data writting method can be used: void write(T record). Calling write(x) is equivalent
of calling write(x, timestampSupplier.applyAsLong(x)). This can be useful when your domain model
already contains a timestamp attribute.
There is just one method for reading data: Stream<Timestamped<U>> read(Class<U> type, Long minMillis, Long maxMillis).
This is designed to read a set of records with timestamps matching a given range [minMillis, maxMillis].
Both ends of a time range are optional. Example of reading all records with timestamps starting
with given time:
List<MyRecord> entities = db.read(MyRecord.class, fromMillis, null)
.map(Timestamped::record).toList();Here the given type (MyRecord.class in the example above) is used for deserialization purpose.
The main data filter provided is a time-range, as described above. This can take following variants:
- no time-range filtering: minMillis = null and maxMillis = null
- open-ended time-range: minMillis = null or maxMillis = null
- time-range filtering: minMillis and maxMillis are both not null
Other means of filtering may be applied on a returned java.util.Stream using its filter method.
You can use java.util.Stream::count method for counting.
You can use ReadOnlyDatase class instead of Database class to get a read-only access to your database.
Actually Database class extends ReadOnlyDatase class by adding writing functionality.
After you have collected interesting data with Eternal DB you can decide to compress the whole database directory to a ZIP file. Later on you can get a read-only access to such ZIP-compressed database as follows:
ReadOnlyDatabase roDb = ReadOnlyDatabase.fromZip(Path.of("your/db.zip"));After all interactions with the database are done, for example at the end of your program, you should close the database object:
db.close();You don't need to close a database before the end of your program. It's a lightweight object and has rather small memory usage.
The library provides a default class for serlialization/deserialization strategy: io.github.k_tomaszewski.eternaldb.JacksonSerialization.
It uses its own instance of Jackson ObjectMapper (precisely: JsonMapper). One can add customization by
creating this object and passing Consumer<ObjectMapper> to the constructor. Example:
Consumer<ObjectMapper> customizer = objectMapper -> objectMapper.registerModule(new JavaTimeModule())
.disable(SerializationFeature.WRITE_DATES_AS_TIMESTAMPS)
.enable(SerializationFeature.WRITE_DATES_WITH_ZONE_ID)
.disable(DeserializationFeature.ADJUST_DATES_TO_CONTEXT_TIME_ZONE);
Database<MyRecord> db = new Database<>(new DatabaseProperties<>(Path.of("/home/db"), 100)
.setSerialization(new JacksonSerialization(customizer)));Contributions are welcome. If you want to contribute, just make a pull request. Please contact me before to discuss your idea: krzysztof.tomaszewski (at) gmail.com