Persist cache on disk #1010

noomly · 2025-05-21T14:28:38Z

noomly
May 21, 2025

Hi, I'm trying to integrate Foyer in an app I'm building which involves fetching many audio files from a remote location and using them locally. I'm thinking of using Foyer to store these audio files on disk so they are available between sessions, and also keep some in memory for faster access. I'm having trouble to test disk persistence though, it seems I can't get Foyer to actually commit things to disk. Take this example, using Foyer 0.17.2:

use std::sync::Arc;

use foyer::{DirectFsDeviceOptions, Engine, HybridCache, HybridCacheBuilder};

#[tokio::main]
async fn main() {
    let init_cache = || async {
        let cache: HybridCache<String, String> = HybridCacheBuilder::new()
            .with_flush_on_close(true)
            .with_policy(foyer::HybridCachePolicy::WriteOnInsertion)
            .memory(10 * 1024 * 1024)
            .storage(Engine::Large)
            .with_admission_picker(Arc::new(foyer::AdmitAllPicker::default()))
            .with_device_options(
                DirectFsDeviceOptions::new("./cache").with_capacity(1_000 * 1024 * 1024),
            )
            .with_flush(true)
            .with_recover_mode(foyer::RecoverMode::Strict)
            .build()
            .await
            .unwrap();

        return cache;
    };

    let key = "mykey".to_string();
    let value = "value".to_string();

    {
        let cache = init_cache().await;

        let inserted = cache.insert(key.clone(), value.clone());
        assert!(*inserted.value() == value);

        let entry = cache.get(&key).await.unwrap().unwrap();
        assert!(*entry == value);

        cache.memory().evict_all();
        cache.close().await.unwrap();
    }

    {
        let cache = init_cache().await;

        let entry = cache.get(&key).await.unwrap().unwrap(); // <--- fails here with a cache miss
        assert!(*entry == value);

        cache.close().await.unwrap();
    }
}

I'm expecting the last assert to work but it always returns None. Foyer creates ./cache with a bunch of foyer-storage-direct-fs-* files inside, but I can't get any data to actually persist there it seems. What am I doing wrong?

Answered by MrCroxx

May 21, 2025

Hi, @noomly . Thanks for reporting.

While debugging, I printed the hash of the same key between the two runs.

2025-05-21T16:01:27.904183Z ERROR t: 67: <<<<<<<<<<<<<<<< v1=11976684267437382861
2025-05-21T16:01:27.906258Z ERROR t: 80: <<<<<<<<<<<<<<<< v1=9171236786011198560

The hash doesn't match each other.

According to the document of ahash:

/// Provides a [Hasher] factory. This is typically used (e.g. by [HashMap]) to create
/// [AHasher]s in order to hash the keys of the map. See `build_hasher` below.
///
/// [build_hasher]: ahash::
/// [Hasher]: std::hash::Hasher
/// [BuildHasher]: std::hash::BuildHasher
/// [HashMap]: std::collections::HashMap
///
/// There are multiple constructors …

View full answer

MrCroxx · 2025-05-21T16:13:21Z

MrCroxx
May 21, 2025
Maintainer

Hi, @noomly . Thanks for reporting.

While debugging, I printed the hash of the same key between the two runs.

2025-05-21T16:01:27.904183Z ERROR t: 67: <<<<<<<<<<<<<<<< v1=11976684267437382861
2025-05-21T16:01:27.906258Z ERROR t: 80: <<<<<<<<<<<<<<<< v1=9171236786011198560

The hash doesn't match each other.

According to the document of ahash:

/// Provides a [Hasher] factory. This is typically used (e.g. by [HashMap]) to create
/// [AHasher]s in order to hash the keys of the map. See `build_hasher` below.
///
/// [build_hasher]: ahash::
/// [Hasher]: std::hash::Hasher
/// [BuildHasher]: std::hash::BuildHasher
/// [HashMap]: std::collections::HashMap
///
/// There are multiple constructors each is documented in more detail below:
///
/// | Constructor   | Dynamically random? | Seed |
/// |---------------|---------------------|------|
/// |`new`          | Each instance unique|_[RandomSource]_|
/// |`generate_with`| Each instance unique|`u64` x 4 + [RandomSource]|
/// |`with_seed`    | Fixed per process   |`u64` + static random number|
/// |`with_seeds`   | Fixed               |`u64` x 4|
///
#[derive(Clone)]
pub struct RandomState {
    // ... ...
}

Each time the RandomState is created, the seed differs from each other. And the indices of the large object disk cache are wrong.

It is a bug from the foyer side, the default hash shouldn't return an unstable result. I'll fix it.

As a temporary solution, please use a seed to fix the hasher when building the memory cache.

e.g.

            .memory(10 * 1024 * 1024)
            .with_hash_builder(ahash::RandomState::with_seed(0))
            .storage(Engine::Large)

0 replies

MrCroxx · 2025-05-21T16:13:55Z

MrCroxx
May 21, 2025
Maintainer

I've created an issue based on this discussion. FYI: #1011

0 replies

noomly · 2025-05-21T20:52:55Z

noomly
May 21, 2025
Author

Thank you so much for the quick reply and for the fix! I'll be trying this tomorrow.

0 replies

noomly · 2025-06-07T08:56:31Z

noomly
Jun 7, 2025
Author

Hi @MrCroxx. Coming back to this, I've tried 0.17.3 and it correctly fixes the specific test I wrote originally. However, it seems that the cache is still unable to recover data from a previous instance. What I mean by this is that if you run the test above once, comment out the following block:

    {
   
8000
     let cache = init_cache().await;

        let inserted = cache.insert(key.clone(), value.clone());
        assert!(*inserted.value() == value);

        let entry = cache.get(&key).await.unwrap().unwrap();
        assert!(*entry == value);

        cache.memory().evict_all();
        cache.close().await.unwrap();
    }

and then run it again, it panics when we try to retrieve the data:

        let entry = cache.get(&key).await.unwrap().unwrap(); // <--- fails here with a cache miss

Even if the directory ./cache is created and persisted between the two executions. Is there some kind of in-memory static state that is persisted during the first execution that allows it to retrieve the data, that is not present in the second execution?

7 replies

noomly Jun 26, 2025
Author

Hi @MrCroxx, sorry for the delayed response. I tried your advice of only using flush_on_close and close on graceful shutdown. On my previous tests I was still not able to recover any entries from the cache after completely closing and re-executing the program. As for the actual app in which I'm still currently using Foyer, I'm observing the same behavior: even if cache is written on disk, no entries are able to be recovered from a session to another.

Our usecase for a cache system is to persist large downloaded data (audio files) on device between sessions while keeping a control on the total size of said cache. It seemed that Foyer was a good fit but I may have been mistaken? I would really appreciate it if you could shed some light on this.

MrCroxx Jun 26, 2025
Maintainer

Hi, noonmly. Don't worry. I'll take a look tonight. Stay tuned! 🙌

MrCroxx Jun 27, 2025
Maintainer

I reproduced the bug. May need some time to fix. 🙌

MrCroxx Jun 27, 2025
Maintainer

I see. The bug is still caused by the randomness of ahash::AHasher.

The document is a little bit misleading.

/// Provides a default [Hasher] with fixed keys.
/// This is typically used in conjunction with [BuildHasherDefault] to create
/// [AHasher]s in order to hash the keys of the map.
///
/// Generally it is preferable to use [RandomState] instead, so that different
/// hashmaps will have different keys. However if fixed keys are desirable this
/// may be used instead.
///
/// # Example
/// ```
/// use std::hash::BuildHasherDefault;
/// use ahash::{AHasher, RandomState};
/// use std::collections::HashMap;
///
/// let mut map: HashMap<i32, i32, BuildHasherDefault<AHasher>> = HashMap::default();
/// map.insert(12, 34);
/// ```
/
8000
//
/// [BuildHasherDefault]: std::hash::BuildHasherDefault
/// [Hasher]: std::hash::Hasher
/// [HashMap]: std::collections::HashMap
impl Default for AHasher {
    /// Constructs a new [AHasher] with fixed keys.
    /// If `std` is enabled these will be generated upon first invocation.
    /// Otherwise if the `compile-time-rng`feature is enabled these will be generated at compile time.
    /// If neither of these features are available, hardcoded constants will be used.
    ///
    /// Because the values are fixed, different hashers will all hash elements the same way.
    /// This could make hash values predictable, if DOS attacks are a concern. If this behaviour is
    /// not required, it may be preferable to use [RandomState] instead.
    ///
    /// # Examples
    ///
    /// ```
    /// use ahash::AHasher;
    /// use std::hash::Hasher;
    ///
    /// let mut hasher_1 = AHasher::default();
    /// let mut hasher_2 = AHasher::default();
    ///
    /// hasher_1.write_u32(1234);
    /// hasher_2.write_u32(1234);
    ///
    /// assert_eq!(hasher_1.finish(), hasher_2.finish());
    /// ```
    #[inline]
    fn default() -> AHasher {
        RandomState::with_fixed_keys().build_hasher()
    }
}

But even with BuildHasherDefault<AHasher>, it only guarantees consistent results in the same run, not between runs.

I'm going to use xxhash or murmurhash by default.

MrCroxx Jun 27, 2025
Maintainer

Hi, @noomly . I just released foyer v0.17.4, which fixed your problem. 🙌

Persist cache on disk #1010

Uh oh!

Uh oh!

noomly May 21, 2025

Replies: 4 comments · 8000 7 replies

Uh oh!

MrCroxx May 21, 2025 Maintainer

Uh oh!

MrCroxx May 21, 2025 Maintainer

Uh oh!

noomly May 21, 2025 Author

Uh oh!

noomly Jun 7, 2025 Author

Uh oh!

noomly Jun 26, 2025 Author

Uh oh!

MrCroxx Jun 26, 2025 Maintainer

Uh oh!

MrCroxx Jun 27, 2025 Maintainer

Uh oh!

MrCroxx Jun 27, 2025 Maintainer

Uh oh!

MrCroxx Jun 27, 2025 Maintainer

noomly
May 21, 2025

Replies: 4 comments 8000 7 replies

MrCroxx
May 21, 2025
Maintainer

MrCroxx
May 21, 2025
Maintainer

noomly
May 21, 2025
Author

noomly
Jun 7, 2025
Author

noomly Jun 26, 2025
Author

MrCroxx Jun 26, 2025
Maintainer

MrCroxx Jun 27, 2025
Maintainer

MrCroxx Jun 27, 2025
Maintainer

MrCroxx Jun 27, 2025
Maintainer