Description
In what area(s)?
/area runtime
Describe the proposal
There are a ton of component implementations for the Dapr state store building block. This is great and encourages adoption of the platform, making it easy to perform simple operations to store and retrieve data.
But there have been several iterations on this that have since stalled out due to the breadth of the application to such scope. State queries are a great example in that there are some state providers that just don't support them or might only support queries against keys and not values, or maybe they required the values formatted in some way that isn't standard, etc. The state building block doesn't support streaming which can lead to memory issues with sufficiently large objects.
I think there's a great case to be made for building out specialty state building blocks instead of trying to make a general purpose state store. There have been a number of proposals for specific state stores and I link to them below - my goal here is to specifically call attention away from the state management building block as it stands today and instead redirect future such work to each of the specialty state stores going forward to stand alongside the generic all-purpose option for basic get and set operations we have today.
There are a great many purpose-built state stores Dapr could provide specific support for:
- Centralized Cache - Proposed
- Queues
- Documents - Proposed
- Object storage - Proposed
- Blob storage
- Key/Value stores - Proposed
- Vector database (suggested in Discord)
- Graph database
- Event Store
- Search Engine database
- Time Series
- Relational here and here
- Schema registry (as described here and located here)
- High-performance Actor store (e.g. connection pooling, distributed cache support, etc.)
The current state store implementation looks much like a key/value store, but I'd propose it could be richer - support key querying and streaming (bidirectional paging requests and responses), and caching for higher performance reads. Don't support querying by value as this typically isn't supported for these stores. Focus on what is typically supported and generically support it.
Blob storage today is somewhat limited to smaller files because of the lack of streaming support. I propose putting this into a specialty building block that looks much like the one we have today, but supports streaming so this can read and write blobs of arbitrarily large sizes, delete blobs, set arbitrary metadata and list blobs (again with bidirectional stream-based paging as supported).
Documents are more likely to have more elaborate querying needs and this is an API widely requested by the community in a recent survey. But this is also an API that's likely to diverge from the others as queries are likely to span not only keys, but also values. NoSQL stores differ from relational stores, so it makes sense to me that the Dapr API for either should vary as well.
I'm eager to see more discussion around relational building blocks as those seems to be elaborate enough technologies that absent accepting raw input strings and parameters, I'm not sure how much technology-agnostic complexity can be shoved into a generic Dapr API that result in simply becoming a Go clone of EF or Dapper, but like I said, I'm eager to see how discussions around this go.
No doubt there are more state-storing opportunities - I think they should similarly be tackled by subject-matter specific (and separately versioned) building blocks as well instead of trying to make a one-size-fits-all generic API for them all necessarily and then limiting their individually purposed potential along the way.
Like I said in the key/value store proposal. I think there's plenty of room for keeping the existing state management store API as-is because it's an easy go-to for generic saving and retrieving of data, but I also think there's a great opportunity here also for more purpose-specific specializations as well.
Thank you for the consideration!