Usability Improvements

This body of work focuses on improving the experience around configuring, interacting with, and debugging ReadySet.

Projects

Reduce configuration complexity

The configuration updates aim to reduce the number of runtime arguments to a smaller set of well tested configurations. Default configuration values will also be updated so that in most cases when a user is testing ReadySet they will only have to configure their upstream database URL.

• Better docs for how to configure your RDS Postgres to work with ReadySet #399
• Metrics integration instructions #398
• Enable metrics in show proxied/caches command #394
• Clean up knobs that we don't need. #275

Improve caching experience

Support bulk downloading and uploading of queries so that a user can export a set of caches that they have tested into a production deployment.

Support exporting tables referenced by caches to enable users to then pass them in via --replication-tables so as to avoid snapshotting tables that will not be used.

• Bulk Caching #279
• DROP CACHE - return num queries / success status #340
• Create a CREATE CACHE CONCURRENTLY command #258
• Limit concurrent migrations #250
• Task panicked; aborting panic_message=Maximum offset must be present after snapshot #195
• Export tables used by caches #249
• Fail starting up if ulimit is too low #86

Improve debuggability

Expose additional system state and metrics through ReadySet SQL extensions. Expand the information readily available on dataflow state and snapshotting/replication status. This system status information is intended to convey whether or not a specific subsystem is healthy and surface any error or restart information.

We'll also add more system status information to specificaly facilitate troubleshooting errors (for example, surfacing the specific reason that ReadySet is unable to begin replication).

Since ReadySet maintains state across server restarts, we will also ensure that system status persists across restarts.

• Expose adapter metrics via status commands #260
• Expand Snapshotting/ Replication status #261
• Expose additional domain/worker stats #262
• Expose Controller/Dataflow Status #264
• Update health check endpoints with upstream connectivity #259
• Build replicator -> upstream status check for troubleshooting #324
• Clean up ReadySet logs #265
• Allow querying configuration #271
• Update EVICTION_WORKER_HEAP_ALLOCATED_BYTES outside of doing evictions #194
• Add total memory usage to SHOW READYSET STATUS #291
• Metrics summary quantiles are incorrect #251