Description
🐛 Describe the bug
Hi,
This is more of a mild annoyance rather than a show-stopping issue. This issue occurs when on Linux and when using an NFS-mounted directory as the temporary directory.
When finished iterating over a DataLoader object, I get the following errors:
Traceback (most recent call last):
File "/usr/lib64/python3.9/multiprocessing/util.py", line 300, in _run_finalizers
finalizer()
File "/usr/lib64/python3.9/multiprocessing/util.py", line 224, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/usr/lib64/python3.9/multiprocessing/util.py", line 133, in _remove_temp_dir
rmtree(tempdir)
File "/usr/lib64/python3.9/shutil.py", line 734, in rmtree
_rmtree_safe_fd(fd, path, onerror)
File "/usr/lib64/python3.9/shutil.py", line 690, in _rmtree_safe_fd
onerror(os.unlink, fullname, sys.exc_info())
File "/usr/lib64/python3.9/shutil.py", line 688, in _rmtree_safe_fd
os.unlink(entry.name, dir_fd=topfd)
OSError: [Errno 16] Device or resource busy: '.nfs8b2479d03841bd4400015e16'
Traceback (most recent call last):
File "/usr/lib64/python3.9/multiprocessing/util.py", line 300, in _run_finalizers
finalizer()
File "/usr/lib64/python3.9/multiprocessing/util.py", line 224, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/usr/lib64/python3.9/multiprocessing/util.py", line 133, in _remove_temp_dir
rmtree(tempdir)
File "/usr/lib64/python3.9/shutil.py", line 734, in rmtree
_rmtree_safe_fd(fd, path, onerror)
File "/usr/lib64/python3.9/shutil.py", line 690, in _rmtree_safe_fd
onerror(os.unlink, fullname, sys.exc_info())
File "/usr/lib64/python3.9/shutil.py", line 688, in _rmtree_safe_fd
os.unlink(entry.name, dir_fd=topfd)
OSError: [Errno 16] Device or resource busy: '.nfs17203ac1c489d74f00015e15'
Code to reproduce:
from torch.utils.data import DataLoader, Dataset
class ExampleDataset(Dataset):
def __len__(self):
return 100
def __getitem__(self, index):
return index
dataset = ExampleDataset()
dl = DataLoader(dataset, num_workers=2)
for i in dl:
print(i)
I believe this is related to shutdown/cleanup of multiprocessing managers/workers python/cpython#58186. The error occurs precisely when shutting down the workers https://github.com/pytorch/pytorch/blob/main/torch/utils/data/dataloader.py#L1582, but I don't understand enough about how the dataloader works to suggest a fix.
I know in most cases it's easier to just use a local directory as tmp, but our cluster (academic HPC) is setup such that each node has minimal local disk space and local disk space is shared by multiple users.
Thanks,
Ed
Versions
Collecting environment information...
PyTorch version: 1.13.1+cu117
Is debug build: False
CUDA used to build PyTorch: 11.7
ROCM used to build PyTorch: N/A
OS: Red Hat Enterprise Linux 9.1 (Plow) (x86_64)
GCC version: (GCC) 11.4.1 20231218 (Red Hat 11.4.1-3)
Clang version: Could not collect
CMake version: version 3.26.5
Libc version: glibc-2.34
Python version: 3.9.18 (main, Jul 3 2024, 00:00:00) [GCC 11.4.1 20231218 (Red Hat 11.4.1-3)] (64-bit runtime)
Python platform: Linux-5.14.0-162.23.1.el9_1.x86_64-x86_64-with-glibc2.34
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True
CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 45 bits physical, 48 bits virtual
Byte Order: Little Endian
CPU(s): 48
On-line CPU(s) list: 0-47
Vendor ID: GenuineIntel
Model name: Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz
CPU family: 6
Model: 79
Thread(s) per core: 1
Core(s) per socket: 1
Socket(s): 48
Stepping: 1
BogoMIPS: 5187.98
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon nopl xtopology tsc_reliable nonstop_tsc cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti ssbd ibrs ibpb stibp fsgsbase tsc_adjust bmi1 avx2 smep bmi2 invpcid rdseed adx smap xsaveopt arat md_clear flush_l1d arch_capabilities
Hypervisor vendor: VMware
Virtualization type: full
L1d cache: 1.5 MiB (48 instances)
L1i cache: 1.5 MiB (48 instances)
L2 cache: 12 MiB (48 instances)
L3 cache: 1.6 GiB (48 instances)
NUMA node(s): 2
NUMA node0 CPU(s): 0-23
NUMA node1 CPU(s): 24-47
Vulnerability Itlb multihit: KVM: Mitigation: VMX unsupported
Vulnerability L1tf: Mitigation; PTE Inversion
Vulnerability Mds: Mitigation; Clear CPU buffers; SMT Host state unknown
Vulnerability Meltdown: Mitigation; PTI
Vulnerability Mmio stale data: Mitigation; Clear CPU buffers; SMT Host state unknown
Vulnerability Retbleed: Mitigation; IBRS
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; IBRS, IBPB conditional, RSB filling, PBRSB-eIBRS Not affected
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Not affected
Versions of relevant libraries:
[pip3] numpy==1.26.4
[pip3] nvidia-cublas-cu11==11.10.3.66
[pip3] nvidia-cuda-nvrtc-cu11==11.7.99
[pip3] nvidia-cuda-runtime-cu11==11.7.99
[pip3] nvidia-cudnn-cu11==8.5.0.96
[pip3] >
[pip3] >
[pip3] torch==1.13.1
[pip3] torchvision==0.14.1
[conda] Could not collect