8000 multiprocessing on py>=3.7 causes threading.RLock pickling exception · Issue #920 · tqdm/tqdm · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

multiprocessing on py>=3.7 causes threading.RLock pickling exception #920

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
6 of 8 tasks
KingDarBoja opened this issue Mar 20, 2020 · 6 comments · Fixed by #1046
Closed
6 of 8 tasks

multiprocessing on py>=3.7 causes threading.RLock pickling exception #920

KingDarBoja opened this issue Mar 20, 2020 · 6 comments · Fixed by #1046
Assignees
@casperdcl
Labels
p0-bug-critical ☢ Exception rasing submodule ⊂ Periphery/subclasses synchronisation ⇶ Multi-thread/processing to-fix ⌛ In progress
Milestone
Non-breaking

Comments

@KingDarBoja
Copy link
  • I have marked all applicable categories:
    • exception-raising bug
    • visual output bug
    • documentation request (i.e. "X is missing from the documentation." If instead I want to ask "how to use X?" I understand StackOverflow#tqdm is more appropriate)
    • new feature request
  • I have visited the source website, and in particular
    read the known issues
  • I have searched through the issue tracker for duplicates
  • I have mentioned version numbers, operating system and
    environment, where applicable: 
    import tqdm, sys
print(tqdm.__version__, sys.version, sys.platform)

The current parallel-bars example doesn't work as it is raising the following exception on the anaconda prompt terminal:

Multi-processing
Traceback (most recent call last):
  File "D:\Anaconda3\envs\sample-project\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "D:\Anaconda3\envs\sample-project\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "D:\Programming\personal_projects\sample-project\sample_tdqm.py", line 38, in <module>
    p = Pool(initializer=tqdm.set_lock, initargs=(tqdm.get_lock(),))
  File "D:\Anaconda3\envs\sample-project\lib\multiprocessing\context.py", line 119, in Pool
    context=self.get_context())
  File "D:\Anaconda3\envs\sample-project\lib\multiprocessing\pool.py", line 176, in __init__
    self._repopulate_pool()
  File "D:\Anaconda3\envs\sample-project\lib\multiprocessing\pool.py", line 241, in _repopulate_pool
    w.start()
  File "D:\Anaconda3\envs\sample-project\lib\multiprocessing\process.py", line 112, in start
    self._popen = self._Popen(self)
  File "D:\Anaconda3\envs\sample-project\lib\multiprocessing\context.py", line 322, in _Popen
    return Popen(process_obj)
  File "D:\Anaconda3\envs\sample-project\lib\multiprocessing\popen_spawn_win32.py", line 89, in __init__
    reduction.dump(process_obj, to_child)
  File "D:\Anaconda3\envs\sample-project\lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
TypeError: can't pickle _thread.RLock objects

However, this one from an old comment in an old PR does work.

Environment info

4.43.0 3.7.3 | packaged by conda-forge | (default, Jul  1 2019, 22:01:29) [MSC v.1900 64 bit (AMD64)] win32
@cgebbe
Copy link
cgebbe commented Sep 21, 2020

Can confirm the error Windows.

  • Windows 10
  • Python Virtual Environment (not anaconda)
  • tqdm versions 4.42, 4.48, 4.49 using python version 3.7.0

Interesting side note: It works when I downgrade python to version before 3.7 (tested 3.6.0 and even 3.5.x I believe)

@prisae prisae mentioned this issue Oct 7, 2020
Closed
@prisae
Copy link
prisae commented Oct 7, 2020

I can confirm this error. Code works fine on Linux, but fails on Windows with Python 3.8.

No problem when simply using concurrent.futures.ProcessPoolExecutor, but failing with TypeError: cannot pickle '_thread.RLock' object when using tqdm.contrib.concurrent.process_map.

@casperdcl
Copy link
Member

Ah so looks like before running multiprocessing code you need to:

from multiprocessing import RLock
tqdm.set_lock(RLock())

That's unfortunate. In any case tqdm.contrib.concurrent.process_map should do this automatically; will release a fix soon...

@casperdcl casperdcl self-assigned this Oct 7, 2020
@casperdcl casperdcl added p0-bug-critical ☢ Exception rasing submodule ⊂ Periphery/subclasses synchronisation ⇶ Multi-thread/processing to-fix ⌛ In progress labels Oct 7, 2020
@casperdcl casperdcl added this to the Non-breaking milestone Oct 7, 2020
casperdcl added a commit that referenced this issue Oct 7, 2020
@casperdcl
contrib.concurrent.process_map: fix threading.RLock pickling error
572a8e8 
- fixes #920
@casperdcl casperdcl mentioned this issue Oct 7, 2020
Merged
@prisae
Copy link
prisae commented Oct 7, 2020

Awesome, thanks @casperdcl !

@casperdcl casperdcl changed the title Current Parallel Bars Example doesn't work on Windows 10 + Anaconda multiprocessing on py>=3.7 causes threading.RLock pickling exception Oct 7, 2020
@casperdcl casperdcl mentioned this issue Oct 8, 2020
Merged
@cgebbe
Copy link
cgebbe commented Oct 9, 2020

Works, thanks a ton @casperdcl

@vertexclique
Copy link

This still carries on. Hasn't been fixed. Example code:

                from multiprocessing import RLock
                t = tqdm.tqdm(queries)
                t.set_lock(RLock())

and end result:

multiprocessing.pool.MaybeEncodingError: Error sending result: '<multiprocessing.pool.ExceptionWithTraceback object at 0x7faf058ae900>'. Reason: 'TypeError("cannot pickle '_thread.RLock' object")'

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@casperdcl casperdcl

Labels
p0-bug-critical ☢ Exception rasing submodule ⊂ Periphery/subclasses synchronisation ⇶ Multi-thread/processing to-fix ⌛ In progress
Projects
None yet
Milestone
Non-breaking
Development

Successfully merging a pull request may close this issue.

next release v4.50.2 tqdm/tqdm
5 participants
@vertexclique @cgebbe @prisae @casperdcl @KingDarBoja
0