shuf: panic due to capacity overflow when allocating a vector #1420

daihy8759 · 2019-08-22T00:51:35Z

I use shuf for a large txt file(about 1Gb),throw capacity overflow,detail:

thread 'main' panicked at 'capacity overflow', src\liballoc\raw_vec.rs:754:5
stack backtrace:
   0: std::sys_common::backtrace::print
             at /rustc/bea0372a1a7a31b81f28cc4d9a83a2dc9a79d008\/src\libstd\sys_common\backtrace.rs:36
   1: std::panicking::default_hook::{{closure}}
             at /rustc/bea0372a1a7a31b81f28cc4d9a83a2dc9a79d008\/src\libstd\panicking.rs:200
   2: std::panicking::default_hook
             at /rustc/bea0372a1a7a31b81f28cc4d9a83a2dc9a79d008\/src\libstd\panicking.rs:214
   3: uucore::panic::install_sigpipe_hook
   4: std::panicking::rust_panic_with_hook
             at /rustc/bea0372a1a7a31b81f28cc4d9a83a2dc9a79d008\/src\libstd\panicking.rs:481
   5: std::panicking::continue_panic_fmt
             at /rustc/bea0372a1a7a31b81f28cc4d9a83a2dc9a79d008\/src\libstd\panicking.rs:384
   6: std::panicking::rust_begin_panic
             at /rustc/bea0372a1a7a31b81f28cc4d9a83a2dc9a79d008\/src\libstd\panicking.rs:311
   7: core::panicking::panic_fmt
             at /rustc/bea0372a1a7a31b81f28cc4d9a83a2dc9a79d008\/src\libcore\panicking.rs:85
   8: core::panicking::panic
             at /rustc/bea0372a1a7a31b81f28cc4d9a83a2dc9a79d008\/src\libcore\panicking.rs:49
   9: alloc::raw_vec::capacity_overflow
             at /rustc/bea0372a1a7a31b81f28cc4d9a83a2dc9a79d008\/src\liballoc\raw_vec.rs:754
  10: alloc::raw_vec::RawVec<T,A>::reserve
  11: uu_shuf::uumain
  12: hashbrown::raw::RawTable<T>::reserve_rehash
  13: <std::collections::hash::map::DefaultHasher as core::hash::Hasher>::write
  14: std::panicking::try
             at /rustc/bea0372a1a7a31b81f28cc4d9a83a2dc9a79d008\/src\libstd\panicking.rs:275
  15: std::panic::catch_unwind
             at /rustc/bea0372a1a7a31b81f28cc4d9a83a2dc9a79d008\/src\libstd\panic.rs:394
  16: std::rt::lang_start_internal
             at /rustc/bea0372a1a7a31b81f28cc4d9a83a2dc9a79d008\/src\libstd\rt.rs:48
  17: main
  18: BaseThreadInitThunk
  19: RtlGetAppContainerNamedObjectPath
  20: RtlGetAppContainerNamedObjectPath

The text was updated successfully, but these errors were encountered:

Arcterus · 2019-08-22T01:10:12Z

Can you provide the file? I should have time this weekend to figure out what's wrong (and deal with some of the other issues that have stacked up...).

stale · 2021-06-08T21:13:56Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

jfinkels · 2022-03-25T02:16:08Z

This is not tied to a specific operating system. I am on Ubuntu 20.04, GNU coreutils v8.30.

GNU shuf:

$ shuf -i1-1844674407370955161 -n 1
840424655081420769

uutils shuf:

$ RUST_BACKTRACE=1 ./target/debug/shuf -i1-1844674407370955161 -n 1
thread 'main' panicked at 'capacity overflow', library/alloc/src/raw_vec.rs:518:5
stack backtrace:
   0: rust_begin_unwind
             at /rustc/9d1b2106e23b1abd32fce1f17267604a5102f57a/library/std/src/panicking.rs:498:5
   1: core::panicking::panic_fmt
	     at /rustc/9d1b2106e23b1abd32fce1f17267604a5102f57a/library/core/src/panicking.rs:116:14
   2: core::panicking::panic
             at /rustc/9d1b2106e23b1abd32fce1f17267604a5102f57a/library/core/src/panicking.rs:48:5
   3: alloc::raw_vec::capacity_overflow
             at /rustc/9d1b2106e23b1abd32fce1f17267604a5102f57a/library/alloc/src/raw_vec.rs:518:5
   4: alloc::raw_vec::RawVec<T,A>::allocate_in
             at /rustc/9d1b2106e23b1abd32fce1f17267604a5102f57a/library/alloc/src/raw_vec.rs:178:27
   5: alloc::raw_vec::RawVec<T,A>::with_capacity_in
             at /rustc/9d1b2106e23b1abd32fce1f17267604a5102f57a/library/alloc/src/raw_vec.rs:132:9
   6: alloc::vec::Vec<T,A>::with_capacity_in
             at /rustc/9d1b2106e23b1abd32fce1f17267604a5102f57a/library/alloc/src/vec/mod.rs:609:20
   7: alloc::vec::Vec<T>::with_capacity
             at /rustc/9d1b2106e23b1abd32fce1f17267604a5102f57a/library/alloc/src/vec/mod.rs:470:9
   8: <alloc::vec::Vec<T> as alloc::vec::spec_from_iter_nested::SpecFromIterNested<T,I>>::from_iter
             at /rustc/9d1b2106e23b1abd32fce1f17267604a5102f57a/library/alloc/src/vec/spec_from_iter_nested.rs:48:33
   9: <alloc::vec::Vec<T> as alloc::vec::spec_from_iter::SpecFromIter<T,I>>::from_iter
             at /rustc/9d1b2106e23b1abd32fce1f17267604a5102f57a/library/alloc/src/vec/spec_from_iter.rs:33:9
  10: <alloc::vec::Vec<T> as core::iter::traits::collect::FromIterator<T>>::from_iter
             at /rustc/9d1b2106e23b1abd32fce1f17267604a5102f57a/library/alloc/src/vec/mod.rs:2541:9
  11: core::iter::traits::iterator::Iterator::collect
             at /rustc/9d1b2106e23b1abd32fce1f17267604a5102f57a/library/core/src/iter/traits/iterator.rs:1745:9
  12: uu_shuf::uumain::uumain
             at ./src/uu/shuf/src/shuf.rs:107:24
  13: uu_shuf::uumain
             at ./src/uu/shuf/src/shuf.rs:56:1
  14: shuf::main
             at ./src/uucore/src/lib/lib.rs:90:24
  15: core::ops::function::FnOnce::call_once
             at /rustc/9d1b2106e23b1abd32fce1f17267604a5102f57a/library/core/src/ops/function.rs:227:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

This is happening because we are creating a vector of all integers in the given interval before shuffling. (This test case is using the "shuffle numbers in a given range" mode, but the same thing will happen with a sufficiently large file because all lines are read into memory before shuffling.)

Arcterus added I - Crash U - shuf labels Jun 8, 2020

stale bot added the wontfix label Jun 8, 2021

stale bot removed the wontfix label Mar 25, 2022

jfinkels changed the title ~~shuf capacity overflow on Windows~~ shuf: panic due to capacity overflow when allocating a vector Mar 25, 2022

BenWiederhake mentioned this issue Feb 17, 2024

shuf: Fix OOM crash for huge number ranges #5980

Merged

cakebaker linked a pull request Feb 17, 2024 that will close this issue

shuf: Fix OOM crash for huge number ranges #5980

Merged

sylvestre closed this as completed in #5980 Feb 25, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

shuf: panic due to capacity overflow when allocating a vector #1420

shuf: panic due to capacity overflow when allocating a vector #1420

Uh oh!

Uh oh!

Uh oh!

Uh oh!

shuf: panic due to capacity overflow when allocating a vector #1420

U 8000 h oh!

Uh oh!

Uh oh!

Uh oh!