NOTE: Issues tracker is disabled. You are welcome to contribute, pull requests accepted.
EJDB2 is an embeddable JSON database engine published under MIT license.
The Story of the IT-depression, birds and EJDB 2.0
- C11 API
- Single file database
- Online backups support
- 500K library size for Android
- iOS / Android / React Native / Flutter integration
- Simple but powerful query language (JQL) as well as support of the following standards:
- Support of collection joins
- Powered by IOWOW - The persistent key/value storage engine
- HTTP REST/Websockets endpoints powered by IWNET and BearSSL.
- JSON documents are stored in using fast and compact binn binary format
- Native language bindings
- Supported platforms
- JQL query language
- Indexes and performance
- Network API
- C API
- License
| Linux | macOS | iOS | Android | Windows | |
|---|---|---|---|---|---|
| C library | ✔️ | ✔️ | ✔️ | ✔️ | ✔️1 |
| NodeJS | ✔️ | ✔️ | ❌3 | ||
| Java | ✔️ | ✔️ | ✔️ | ✔️2 | |
| DartVM5 | ✔️ | ✔️2 | ❌3 | ||
| Flutter5 | ✔️ | ✔️ | |||
| React Native5 | ❌4 | ✔️ | |||
| Swift5 | ✔️ | ✔️ | ✔️ |
[5] Bindings are unmaintained Contributors needed.
[1] No HTTP/Websocket support #257
[2] Binaries are not distributed with dart pub. You can build it manually
[3] Can be build, but needed a linkage with windows node/dart libs.
[4] Porting in progress #273
- Go
- Rust
- .Net
- Haskell
- Pharo
- Lua
- EJDB 2.0 core engine is well tested and used in various heavily loaded deployments
- Tested on
Linux,macOSandFreeBSD. Has limited Windows support - Old EJDB 1.x version can be found in separate ejdb_1.x branch. We are not maintaining ejdb 1.x.
Are you using EJDB? Let me know!
EJDB2 code ported and tested on High Sierra / Mojave / Catalina
EJDB2 Swift binding for MacOS, iOS and Linux. Swift binding is outdated at now. Looking for contributors.
brew install ejdb
cmake v3.24 or higher required
git clone --recurse-submodules git@github.com:Softmotions/ejdb.git
mkdir build && cd build
cmake .. -DCMAKE_BUILD_TYPE=Release
make install
mkdir build && cd build
cmake .. -DCMAKE_BUILD_TYPE=Release -DPACKAGE_DEB=ON
make packagemkdir build && cd build
cmake .. -DCMAKE_BUILD_TYPE=Release -DPACKAGE_RPM=ON
make packageEJDB2 can be cross-compiled for windows
Note: HTTP/Websocket network API is disabled and not yet supported
Nodejs/Dart bindings not yet ported to Windows.
Cross-compilation Guide for Windows
IWSTART is an automatic CMake initial project generator for C projects based on iowow / iwnet / ejdb2 libs.
https://github.com/Softmotions/iwstart
EJDB query language (JQL) syntax inspired by ideas behind XPath and Unix shell pipes. It designed for easy querying and updating sets of JSON documents.
JQL parser created created by peg/leg — recursive-descent parser generators for C Here is the formal parser grammar: https://github.com/Softmotions/ejdb/blob/master/src/jql/jqp.leg
Notation used below is based on SQL syntax description:
| Rule | Description |
|---|---|
' ' |
String in single quotes denotes unquoted string literal as part of query. |
{ a | b } |
Curly brackets enclose two or more required alternative choices, separated by vertical bars. |
[ ] |
Square brackets indicate an optional element or clause. Multiple elements or clauses are separated by vertical bars. |
| |
Vertical bars separate two or more alternative syntax elements. |
... |
Ellipses indicate that the preceding element can be repeated. The repetition is unlimited unless otherwise indicated. |
( ) |
Parentheses are grouping symbols. |
| Unquoted word in lower case | Denotes semantic of some query part. For example: placeholder_name - name of any placeholder. |
QUERY = FILTERS [ '|' APPLY ] [ '|' PROJECTIONS ] [ '|' OPTS ];
STR = { quoted_string | unquoted_string };
JSONVAL = json_value;
PLACEHOLDER = { ':'placeholder_name | '?' }
FILTERS = FILTER [{ and | or } [ not ] FILTER];
FILTER = [@collection_name]/NODE[/NODE]...;
NODE = { '*' | '**' | NODE_EXPRESSION | STR };
NODE_EXPRESSION = '[' NODE_EXPR_LEFT OP NODE_EXPR_RIGHT ']'
[{ and | or } [ not ] NODE_EXPRESSION]...;
OP = [ '!' ] { '=' | '>=' | '<=' | '>' | '<' | ~ }
| [ '!' ] { 'eq' | 'gte' | 'lte' | 'gt' | 'lt' }
| [ not ] { 'in' | 'ni' | 're' };
NODE_EXPR_LEFT = { '*' | '**' | STR | NODE_KEY_EXPR };
NODE_KEY_EXPR = '[' '*' OP NODE_EXPR_RIGHT ']'
NODE_EXPR_RIGHT = JSONVAL | STR | PLACEHOLDER
APPLY = { 'apply' | 'upsert' } { PLACEHOLDER | json_object | json_array } | 'del'
OPTS = { 'skip' n | 'limit' n | 'count' | 'noidx' | 'inverse' | ORDERBY }...
ORDERBY = { 'asc' | 'desc' } PLACEHOLDER | json_path
PROJECTIONS = PROJECTION [ {'+' | '-'} PROJECTION ]
PROJECTION = 'all' | json_path
json_value: Any valid JSON value: object, array, string, bool, number.json_path: Simplified JSON pointer. Eg.:/foo/baror/foo/"bar with spaces"/*in context ofNODE: Any JSON object key name at particular nesting level.**in context ofNODE: Any JSON object key name at arbitrary nesting level.*in context ofNODE_EXPR_LEFT: Key name at specific level.**in context ofNODE_EXPR_LEFT: Nested array value of array element under specific key.
Lets play with some very basic data and queries.
For simplicity we will use ejdb websocket network API which provides us a kind of interactive CLI. The same job can be done using pure C API too (ejdb2.h jql.h).
NOTE: Take a look into JQL test cases for more examples.
{
"firstName": "John",
"lastName": "Doe",
"age": 28,
"pets": [
{"name": "Rexy rex", "kind": "dog", "likes": ["bones", "jumping", "toys"]},
{"name": "Grenny", "kind": "parrot", "likes": ["green color", "night", "toys"]}
]
}Save json as sample.json then upload it the family collection:
# Start HTTP/WS server protected by some access token
./jbs -a 'myaccess01'
8 Mar 16:15:58.601 INFO: HTTP/WS endpoint at localhost:9191Server can be accessed using HTTP or Websocket endpoint. More info
curl -d '@sample.json' -H'X-Access-Token:myaccess01' -X POST http://localhost:9191/familyWe can play around using interactive wscat websocket client.
wscat -H 'X-Access-Token:myaccess01' -c http://localhost:9191
connected (press CTRL+C to quit)
> k info
< k {
"version": "2.0.0",
"file": "db.jb",
"size": 8192,
"collections": [
{
"name": "family",
"dbid": 3,
"rnum": 1,
"indexes": []
}
]
}
> k get family 1
< k 1 {
"firstName": "John",
"lastName": "Doe",
"age": 28,
"pets": [
{
"name": "Rexy rex",
"kind": "dog",
"likes": [
"bones",
"jumping",
"toys"
]
},
{
"name": "Grenny",
"kind": "parrot",
"likes": [
"green color",
"night",
"toys"
]
}
]
}Note about the k prefix before every command; It is an arbitrary key chosen by client and designated to identify particular
websocket request, this key will be returned with response to request and allows client to
identify that response for his particular request. More info
Query command over websocket has the following format:
<key> query <collection> <query>
So we will consider only <query> part in this document.
k query family /*
or
k query family /**
or specify collection name in query explicitly
k @family/*
We can execute query by HTTP POST request
curl --data-raw '@family/[firstName = John]' -H'X-Access-Token:myaccess01' -X POST http://localhost:9191
1 {"firstName":"John","lastName":"Doe","age":28,"pets":[{"name":"Rexy rex","kind":"dog","likes":["bones","jumping","toys"]},{"name":"Grenny","kind":"parrot","likes":["green color","night","toys"]}]}
k @family/* | limit 10
Element at index 1 exists in likes array within a pets sub-object
> k query family /pets/*/likes/1
< k 1 {"firstName":"John"...
Element at index 1 exists in likes array at any likes nesting level
> k query family /**/likes/1
< k 1 {"firstName":"John"...
From this point and below I will omit websocket specific prefix k query family and
consider only JQL queries.
In order to get documents by primary key the following options are available:
-
Use API call
ejdb_get()const doc = await db.get('users', 112);
-
Use the special query construction:
/=:?or@collection/=:?
Get document from users collection with primary key 112
> k @users/=112
Update tags array for document in jobs collection (TypeScript):
await db.createQuery('@jobs/ = :? | apply :? | count')
.setNumber(0, id)
.setJSON(1, { tags })
.completionPromise();Array of primary keys can also be used for matching:
await db.createQuery('@jobs/ = :?| apply :? | count')
.setJSON(0, [23, 1, 2])
.setJSON(1, { tags })
.completionPromise();Below is a set of self explaining queries:
/pets/*/[name = "Rexy rex"]
/pets/*/[name eq "Rexy rex"]
/pets/*/[name = "Rexy rex" or name = Grenny]
Note about quotes around words with spaces.
Get all documents where owner age greater than 20 and have some pet who like bones or toys
/[age > 20] and /pets/*/likes/[** in ["bones", "toys"]]
Here ** denotes some element in likes array.
ni is the inverse operator to in.
Get documents where bones somewhere in likes array.
/pets/*/[likes ni "bones"]
We can create more complicated filters
( /[age <= 20] or /[lastName re "Do.*"] )
and /pets/*/likes/[** in ["bones", "toys"]]
Note about grouping parentheses and regular expression matching using re operator.
~ is a prefix matching operator (Since ejdb v2.0.53).
Prefix matching can benefit from using indexes.
Get documents where /lastName starts with "Do".
/[lastName ~ Do]
Filter documents with likes array exactly matched to ["bones","jumping","toys"]
/**/[likes = ["bones","jumping","toys"]]
Matching algorithms for arrays and maps are different:
- Array elements are matched from start to end. In equal arrays all values at the same index should be equal.
- Object maps matching consists of the following steps:
- Lexicographically sort object keys in both maps.
- Do matching keys and its values starting from the lowest key.
- If all corresponding keys and values in one map are fully matched to ones in other
and vice versa, maps considered to be equal.
For example:
{"f":"d","e":"j"}and{"e":"j","f":"d"}are equal maps.
Find JSON document having firstName key at root level.
/[* = "firstName"]
I this context * denotes a key name.
You can use conditions on key name and key value at the same time:
/[[* = "firstName"] = John]
Key name can be either firstName or lastName but should have John value in any case.
/[[* in ["firstName", "lastName"]] = John]
It may be useful in queries with dynamic placeholders (C API):
/[[* = :keyName] = :keyValue]
APPLY section responsible for modification of documents content.
APPLY = ({'apply' | `upsert`} { PLACEHOLDER | json_object | json_array }) | 'del'
JSON patch specs conformed to rfc7386 or rfc6902 specifications followed after apply keyword.
Let's add address object to all matched document
/[firstName = John] | apply {"address":{"city":"New York", "street":""}}
If JSON object is an argument of apply section it will be treated as merge match (rfc7386) otherwise
it should be array which denotes rfc6902 JSON patch. Placeholders also supported by apply section.
/* | apply :?
Set the street name in address
/[firstName = John] | apply [{"op":"replace", "path":"/address/street", "value":"Fifth Avenue"}]
Add Neo fish to the set of John's pets
/[firstName = John]
| apply [{"op":"add", "path":"/pets/-", "value": {"name":"Neo", "kind":"fish"}}]
upsert updates existing document by given json argument used as merge patch
or inserts provided json argument as new document instance.
/[firstName = John] | upsert {"firstName": "John", "address":{"city":"New York"}}
Increments numeric value identified by JSON path by specified value.
Example:
Document: {"foo": 1}
Patch: [{"op": "increment", "path": "/foo", "value": 2}]
Result: {"foo": 3}
Same as JSON patch add but creates intermediate object nodes for missing JSON path segments.
Example:
Document: {"foo": {"bar": 1}}
Patch: [{"op": "add_create", "path": "/foo/zaz/gaz", "value": 22}]
Result: {"foo":{"bar":1,"zaz":{"gaz":22}}}
Example:
Document: {"foo": {"bar": 1}}
Patch: [{"op": "add_create", "path": "/foo/bar/gaz", "value": 22}]
Result: Error since element pointed by /foo/bar is not an object
Swaps two values of JSON document starting from from path.
Swapping rules
- If value pointed by
fromnot exists error will be raised. - If value pointed by
pathnot exists it will be set by value fromfrompath, then object pointed byfrompath will be removed. - If both values pointed by
fromandpathare presented they will be swapped.
Example:
Document: {"foo": ["bar"], "baz": {"gaz": 11}}
Patch: [{"op": "swap", "from": "/foo/0", "path": "/baz/gaz"}]
Result: {"foo": [11], "baz": {"gaz": "bar"}}
Example (Demo of rule 2):
Document: {"foo": ["bar"], "baz": {"gaz": 11}}
Patch: [{"op": "swap", "from": "/foo/0", "path": "/baz/zaz"}]
Result: {"foo":[],"baz":{"gaz":11,"zaz":"bar"}}