8000 Memory issues with Dask and Mac OS Sequoia >=15.4.1 · Issue #1032 · PSLmodels/OG-Core · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Memory issues with Dask and Mac OS Sequoia >=15.4.1 #1032

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
rickecon opened this issue Apr 30, 2025 · 1 comment
8000
Open

Memory issues with Dask and Mac OS Sequoia >=15.4.1 #1032

rickecon opened this issue Apr 30, 2025 · 1 comment

Comments

@rickecon
Copy link
Member

We have encountered memory issues using Dask and with Mac OS Sequoia >= 15.4.1. In particular, we are getting distributed error outs in OG-USA with the TMD microdata file as well as black screens. Google Gemini search answered the search prompt "dask issues with new macos sequoia" with the following information.

Yes, macOS Sequoia (version 15.0) has been reported to cause issues with Dask and other network-based tools. The primary problem stems from changes to the built-in Application Firewall that now incorrectly blocks incoming internet connections, even when explicitly allowed by the user. This behavior can lead to inconsistent connectivity or complete connection drops for applications like Dask, which rely on network communication. (ref 1, ref 2)

Here's a more detailed breakdown:

Issue:

Firewall Changes:
macOS Sequoia's new Application Firewall has been modified, particularly how it handles application settings in property files. (ref 1)

Blocking Incoming Connections:
This change causes the firewall to automatically block incoming internet connections for certain applications, including Dask, even if the user has explicitly granted permission. (ref 1, ref 2)

Persistent Settings:
The firewall settings are not persistent across reboots, meaning the blocked connection behavior can reappear after a restart. (ref 1)

Impact on Dask:
Dask relies on network communication for various tasks, including distributed computing. The blocked connections interfere with Dask's ability to function correctly. (ref 2)

Solutions and Workarounds:

  1. Update Dask: Ensure you are using the latest version of Dask, as developers may have released patches to address these issues. (ref 2, ref 3)
  2. Firewall Settings: Carefully review and adjust your firewall settings to ensure that Dask and related applications are allowed to receive incoming connections. Remember that these settings may revert after a reboot. (ref 1)
  3. Disable Firewall: As a temporary workaround, you can disable the Application Firewall entirely. However, this will compromise your system's security, so use this option with caution.
  4. Monitor for Updates: Apple may release future updates to address this issue. Keep your macOS up-to-date to benefit from any bug fixes.

cc: @jdebacker
@jdebacker
Copy link
Member

I am noticing this only when estimating tax functions of the HSV form with a large microdataset. It would be worth running a memory profiler on smaller data to see where bottlenecks are. A particular place to look would be the matrix inversion when estimating the HSV functions with OLS.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@rickecon @jdebacker
0