Storage Channel Configuration and related error #71

DhinchakDhoomDev · 2023-11-29T11:26:23Z

DhinchakDhoomDev
Nov 29, 2023

Q1 : I am using a FileNameProvider and Directory to save those entry in specific directories but after restarting the application I am getting error as : "Non-deleted non-empty data file not found: channel 1, file 1"
Source code is as follows :

            StorageFileNameProvider fileNameProvider = StorageFileNameProvider.Builder()
                    .setChannelDirectoryPrefix("channel_")
                    .setDataFilePrefix("data_")
                    .setDataFileSuffix(".databin")
                    .setTransactionsFilePrefix("events_")
                    .setTransactionsFileSuffix(".trbin")
                    .setTypeDictionaryFileName("typeDictionary.txt")
                    .createFileNameProvider();

            NioFileSystem fileSystem = NioFileSystem.New();
            StorageLiveFileProvider fileProvider = Storage
                    .FileProviderBuilder(fileSystem)
                    .setDirectory(fileSystem.ensureDirectoryPath("./saveDir"))
                    .setDeletionDirectory(fileSystem.ensureDirectoryPath("./deleteDir"))
                    .setTruncationDirectory(fileSystem.ensureDirectoryPath("./truncateDir"))
                    .setFileNameProvider(fileNameProvider)
                    .createFileProvider();
            EmbeddedStorageManager storage = EmbeddedStorage.Foundation(
                            Storage.ConfigurationBuilder()
                                    .setChannelCountProvider(Storage.ChannelCountProvider(2))
                                    .setStorageFileProvider(fileProvider)
                                    .createConfiguration()
                    )
                    .start();

I am using example from the following DOCS link

Q2 : There was a feature in Microstream Read-Only Storage Manager

Referenced from the following link :
https://docs.microstream.one/manual/storage/configuration/readonly.html

hg-ms · 2023-11-29T14:44:48Z

hg-ms
Nov 29, 2023
Collaborator

The problem seems to be caused by the dots in setDataFileSuffix and setTransactionsFileSuffix.
It should work if you specify the suffixes without the leading dots e.g.: “databin” and “trbin”.

TODO: check docu and code if there is a bug…

0 replies

hg-ms · 2023-11-30T06:48:46Z

hg-ms
Nov 30, 2023
Collaborator

Q2 : There was a feature in Microstream Read-Only Storage Manager

The StorageWriteControllerReadOnlyMode is still there. It has been moved to the package org.eclipse.store.storage.types

TODO: Docu seem to be incomplete

0 replies

hg-ms · 2023-11-30T07:04:55Z

hg-ms
Nov 30, 2023
Collaborator

@DhinchakDhoomDev: Many thanks for the hint to the missing / wrong documentation.

2 replies

DhinchakDhoomDev Nov 30, 2023
Author

One more thing if if you could help me out from the docs shared in MicroStream project. Below is the snippet I would to quote

It is possible to have a Storage Manager which operates in a read-only mode. Since there is a limitation in the Open Source version of MicroStream that you can have only one manager that is connected to one data storage, you can bypass this restriction with this read-only mode.

Q1 Is it possible to write from a thread/micro-service and read from another? (like publish-subscribe pattern)

Because it's been mentioned over here as limitation in the "Open Source" version. That limitation still exist in EclipseStore project.

8000

hg-ms Dec 1, 2023
Collaborator

There a two different case to distinguish between:

Only one storage manager opens a storage exclusively:
In that case writing with one thread and reading from another one is not a problem. You just need to take the standard java concurrency topics into account, e.g.: use reentrant read write locks. The storage manager is shared by all treads of your application.
The read access is done on the already loaded object graph in memory. Write operations require that you ensure that the object graph is not modified while the data to be persisted is collected.
The io operations (store/load) a done by the single storage manager that syncs io calls internally.
More the one storage manager instances access a storage:
This is the docu part you quoted. It is essential that there is only one storage manager that is allowed to perform modifications to the peristed data. So only one storage manager instance is allowed to have write access to the stored data. Other instances must ensure read only access. One reason for that is that the storage manager instances do not know each other and are not able to detect if another instance modified the persisted data.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Storage Channel Configuration and related error #71

Uh oh!

{{title}}

Uh oh!

Replies: 3 comments 2 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Storage Channel Configuration and related error #71

Uh oh!

DhinchakDhoomDev Nov 29, 2023

Replies: 3 comments · 2 replies

Uh oh!

hg-ms Nov 29, 2023 Collaborator

Uh oh!

Uh oh!

hg-ms Nov 30, 2023 Collaborator

Uh oh!

hg-ms Nov 30, 2023 Collaborator

Uh oh!

DhinchakDhoomDev Nov 30, 2023 Author

Uh oh!

hg-ms Dec 1, 2023 Collaborator

DhinchakDhoomDev
Nov 29, 2023

Replies: 3 comments 2 replies

hg-ms
Nov 29, 2023
Collaborator

hg-ms
Nov 30, 2023
Collaborator

hg-ms
Nov 30, 2023
Collaborator

DhinchakDhoomDev Nov 30, 2023
Author

hg-ms Dec 1, 2023
Collaborator