8000 Performance/Asynchronous database backend by carljohnsen · Pull Request #6360 · duplicati/duplicati · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Performance/Asynchronous database backend #6360

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 355 commits into
base: master
Choose a base branch
from
Open

Performance/Asynchronous database backend #6360

wants to merge 355 commits into from

Conversation

carljohnsen
Copy link
Contributor
@carljohnsen carljohnsen commented Jun 19, 2025 8000 •
edited
Loading

This pull request migrates the database backend to Microsoft.Data.Sqlite, unlocking improved performance and enabling full support for asynchronous operations.

While it's not a proper benchmark, on my machine, running the test suite went from ~1600 seconds on the current master branch to ~1300 seconds on this branch, demonstrating a ~1.23x performance improvement purely from the new database backend.

Major changes

  • Full Async Refactor: Migrated all database operations to async to improve scalability and responsiveness, especially under high load or I/O-bound scenarios.
  • Deferred Transactions: Switched to deferred transactions to allow greater concurrency and reduce locking contention, improving reliability.
  • Removed IDb Usage: Eliminated the IDb interface to simplify the codebase and leverage features specific to Microsoft.Data.Sqlite.
  • Factory Instantiation Pattern: Adopted factory methods for object creation to support async initialization, making setup explicit and avoiding heavy work in constructors.
  • Property Refactor: Refactored properties to be lightweight and synchronous, moving any heavy or async logic to methods for clarity and maintainability.
  • .Cast() Fixes: Corrected improper .Cast<object>() usage to prevent runtime errors and ensure type safety.
  • Constructor Chain Parity: Ensured new factory methods replicate the logic of previous constructor chains to maintain consistent behavior.
  • Transaction Disposal: Verified that disposing transactions still triggers rollbacks as before, preserving data integrity.
  • Livelock Fix: Added locking in FileProcessor to resolve a livelock during restore, improving reliability.
  • ConfigureAwait(false): Applied ConfigureAwait(false) to async database calls to avoid deadlocks and improve context management.
  • Explicit Transaction Handling: Internalized transaction management for safety, while still allowing external commits as needed, simplifying usage.
  • Docstrings: Enhanced or added docstrings for better documentation and developer understanding.
  • SQL Query Formatting: Standardized SQL queries to use double quotes for identifiers, ensuring consistency and compatibility.
  • Async Propagation: Updated calling functions to propagate async changes, including use of await foreach for collections, to fully leverage async benefits.
  • Async Disposal: Adopted await using for async disposables to ensure proper resource cleanup.
  • Chained Command Execution: Chained multiple database commands for improved clarity and correctness.
  • CancellationToken Propagation: Propagated CancellationToken parameters to support cooperative cancellation and responsive shutdowns.
  • Database Pragmas: Updated default pragmas for better performance and reliability based on best practices.
  • SQLite Page Cache: Removed page cache setting, delegating configuration to environment variables for flexibility.

carljohnsen added 30 commits May 21, 2025 07:40
@carljohnsen
Removed FormatInvariant as it's now handeld by the extension methods
712d75a
@carljohnsen
Ensured all calls to sqlite in LocalRecreateDatabase uses extension m…
003faef 
…ethods and are properly async
@carljohnsen
Restructered SQL queries in LocalRecreateDatabase
4affe46
@carljohnsen
Found more factory construction inconsistencies
d45e977
@carljohnsen
Removed whitespace
8c11cc9
@carljohnsen
Changed LocalTestDatabase to use factory construction
60b8df6
@carljohnsen
Purged IDb from LocalTestDatabase
602a30d
@carljohnsen
Removed FormatInvariant as it's now handled by the extension methods
d688956
@carljohnsen
Made everything in LocalTestDatabase async
ef2f856
@carljohnsen
Restructured SQL queries in LocalTestDatabase
e4cffd5
@carljohnsen
Removed transactions as arguments as part of the transition to implic…
edf1d93 
…it transactions
@carljohnsen
Reintroduced the helper method for constructing a
cc2f974 
LocalDeleteDatabase.BlockQuery
@carljohnsen
Fixed a method being synchronous in LocalTestDatabase
5ffff0e
@carljohnsen
Removed whitespace
96449bc
@carljohnsen
Updated BackupDatabase async interface to call the new async database…
c7f7b2c 
… methods
@carljohnsen
Removed whitespace
6e26d68
@carljohnsen
Updated DatabaseCommon async interface to use the new async database …
8d00d24 
…methods
@carljohnsen
Updated the internal processes of the reworked restore flow to use th…
85c645c 
…e new async database methods
@carljohnsen
Removed whitespace
6dc40fc
@carljohnsen
Updated the BackupHandler to use the new async database methods
d02608f
@carljohnsen
Removed whitespace
99dddd4
@carljohnsen
Updated CompactHandler to use the new async database methods
f4981ec
@carljohnsen
Removed whitespace
b7beea4
@carljohnsen
Updated CreateBugReportHandler to use the new async database methods
e9bb718
@carljohnsen
Removed whitespace
300d80f
@carljohnsen
Updated DeleteHandler to use the new async database methods
43e5e8f
@carljohnsen
Removed whitespace
63141d4
@carljohnsen
Updated the FilelistProcessor operation to use the new async database…
996f2b1 
… methods
@carljohnsen
Removed whitespace
1f8c174
@carljohnsen
Updated TestHandler to use the new async database methods
40bc75b
carljohnsen added 30 commits June 18, 2025 14:37
@carljohnsen
Made the ChangeStatistics methods accept a cancellation token
87dc978
@carljohnsen
Made the LocalDatabase methods accept a cancellation token
3b3a780
@carljohnsen
Made the LocalBackupDatabase methods accept a cancellation token
8039cbb
@carljohnsen
Made the LocalBugReportDatabase methods accept a cancellation token
ccce18b
@carljohnsen
Made the LocalDeleteDatabase methods accept a cancellation token
c496d0b
@carljohnsen
Made the LocalListAffectedDatabase methods accept a cancellation token
936742b
@carljohnsen
Made the LocalListBrokenFilesDatabase methods accept a cancellation t…
42877eb 
…oken
@carljohnsen
Made the LocalListChangesDatabase methods accept a cancellation token
28ca30f
@carljohnsen
Made the LocalListDatabase methods accept a cancellation token
3ea8733
@carljohnsen
Made the LocalPurgeDatabase methods accept a cancellation token
736b52c
@carljohnsen
Made the LocalRecreateDatabase methods accept a cancellation token
1c103c1
@carljohnsen
Made the LocalRepairDatabase methods accept a cancellation token
04c5201
@carljohnsen
Made the LocalRestoreDatabase methods accept a cancellation token
488d531
@carljohnsen
Made the LocalTestDatabase methods accep a cancellation token
4f7760f
@carljohnsen
ReusableTransaction.RollbackAsync now also accepts cancellation tokens
b995228
@carljohnsen
Added some extra overloads to the async database extension methods
7c425c4
@carljohnsen
Updated all of the handler calls to database methods to forward their…
32efa4b 
… cancellation token
@carljohnsen
Removed whitespace
0f0da82
@carljohnsen
All of the non-internal-library methods calling the database function…
78d649f 
…s forwards CancellationToken.None
@carljohnsen
Found some methods in LocalBackupDatabase that didn't accept a cancel…
7e979a4 
…lation token
@carljohnsen
Forwarded cancellation tokens, added some missing .ConfigureAwait() a…
31c5441 
…nd added some missing docstrings
@carljohnsen
Removed whitespace
9b0d893
@carljohnsen
Removed the sqlite page cache option. It should be set through the en…
2412800 
…vironment variable, or some other way of providing a custom database configurotion
@carljohnsen
Implemented a lightweight parser to the CUSTOMSQLITEOPTIONS environme…
322e4f1 
…nt variable along with new default options.
@carljohnsen
Merge remote-tracking branch 'upstream/master' into performance/mssql…
f6ac30e 
…ite-async
@carljohnsen
Post merge with master fixes
26c48c7 < 6D47 /code>
@carljohnsen
Assert.Throws did not like async lambda functions
089dc8c
@carljohnsen
Previous version was using 'null' as a transaction, which implies a c…
b9fa8bd 
…ommit after each query.
@carljohnsen
SQL query inside GetEmptyMetadataBlocksetId had not only moved files,…
0a047a3 
… but had also changed
@carljohnsen
Added missing double quotes to the SQL queries in LocalRepairDatabase…
6460e33 
….FixEmptyMetadatasets()
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@carljohnsen
0