0.113.2 - Home Assistant is starting, not everything will be available until it is finished. #38274
Labels
Comments
|
Same issue. Have to roll-back 0.112.
|
|
Same issue on my end as well, and same problem with zwave not auto-starting on boot. |
|
Can you guys turn on debug logging and look for something about waiting for a task to finish? You should see some logging about that. |
|
I've already rolled back to 0.112.4, this was breaking my entire home automation via NodeRed and unfortunately I can't have everything be down :( |
|
@frenck This isn't specific to Zwave, it's a general startup problem.... zwave is just one of the many things that is not getting started
Me thinks, @danecreekphotography I feel your pain, none of the automatons I have work either @dshokouhi my log is 48MB after three hours w/ the default log level, it already shows DEBUG messages, I don't think it's practical to do TRACE logging. Did grep for |
|
The debug log should show up around when HA starts up and seems it should be from Heres the PR that describes what to look for: #38134 I dont think it will catch all integrations but in order to fix the issue we need to find out which integration is causing start up from being blocked. Double check that all of your custom components are also disabled for the check, during the beta the team found one that blocked start up so thats a good thing to check too. |
|
@dshokouhi, per your suggestion I changed my logging to this: and much like @jurgenweber, when I don't know if I should close this or not... |
|
@bdraco any thoughts on this? |
Debug being on won't affect startup. He just got lucky that whichever integration was blocking startup didn't happen to get stuck that time around. We have some integrations that don't have proper timeouts that reschedule a reconnect or update before the first one times out which means there is never a time window where startup can continue unblocked. We probably need to adjust the logging to start showing which tasks are being waited for after we have blocked for 60 seconds. The original logging only shows if we are waiting for a single task for 60 seconds. It's a chain of tasks that never finish that is the issue in this case. |
|
@bdraco I agree with you in principle that the logging level wouldn't be doing this but I was switching it on and off, disabling integrations and it wouldn't start up completely. Automations/zwave were the two obvious parts that just never turned on (discussed lots on discord on Saturday). The only thing I realised in the end is I was switching the logging on and off all the time (i'd turn it on, think it was working... turn it off... then it wouldn't work again, drove me nuts :)). But currently, I have it that core logging at warning and it worked....... argh.... I stopped at this point because my house was up again and that is all I wanted. :) I let it sit for 40minutes once and nothing happened. The debug logging did not help. :( So I am running the docker container, 0.113.1, I have two custom integrations (nsw fire and aarlo) and then I have the following integrations configured, I curled '/api/config' for this list "binary_sensor.updater", I removed in my testing the custom integrations, cast, transmission, daikin and unifi mostly because these are the integrations I have had issues with before but I was able to restart it eventually with all of them working/setup. That said, this is without a doubt the best HASS release I have experienced in my 3 years of using it. Everything is lighting fast, my zwave starts up in 5 minutes instead of 10. The automation fixes and changes are the best and I couldn't be happier.. Just a little afraid to restart atm. :) A big shout out to the team for all their efforts. If you need me to do anything feel free to ping me on discord or here. Thanks |
|
Its probably |
|
I have onvif integration in my config, so that's consistent with at least one other person who hit this. |
|
yeah, I did note my ONVIF was a bit weird, I have two devices and one of them the camera works but all of the sensors are unavailable. |
|
I don't use Onvif bit still get the issue. Have 5 Hass upgraded to 0.113 without issue, only one stuck with startup problem. |
|
is that all of them? Maybe then we close this ticket as a parent ticket for 'lots of integrations are stopping start up'.... in 0.113.0 and 0.113.1 and fixes are in 0.113.2.... |
I'm sure there are more. This isn't a new issue as these integrations have always had the risk of blocking startup because they can create a long running task that never finishes or a chain of these tasks that block forever. The chance of it happening increases as core gets faster since it more likely they will be doing the thing that blocks startup before bootstrap is completed.
Only |
|
makes sense and thank you for the explaination. |
|
I can confirm the ONVIF issue, I deleted it and restarted.. it works. |
|
I'm terrified of rolling to any newer build of HomeAssistant if all it takes is one random integration taking too long/hanging to cause my entire home automation system to die... |
Any integration can monopolize the event loop, run the system out of memory, or overwhelm the cpu. This is why we have code review to catch these type of things. The good news is that as core gets faster, its more likely we find these type of problems and squash them for good instead of having them randomly pop up and go unsolved. |
and as already pointed out, these problems have always existed that just didn't become problems until the performance work was completed. So we will have this dark period of a bunch of integrations with issues and then it will become the norm and its over. |
🤔 |
|
I don't have any Here's what I get loading by running
Also this zha_event continuously reoccurs in the log @danecreekphotography @bdraco
To be fair, a lot of the older components were built when the coding standards / objectives were a bit different. |
|
you have speedtesdotnet which @bdraco lists as a problem in #38274 (comment) |
|
Yup, I saw that removed and restarted, but I'm still seeing same problem... :( |
Did you try the logging suggested in this PR #38134. ? Apologizes if you already have as I'm having trouble keeping track. |
|
@bdraco I applied the debug logging, but didn't patch core.py... not sure if I can patch core within a docker container without rebuilding the whole thing... On another note, my last Home Assistant run has finally gotten to startup in just over 2 hours... The only error I see relating to core is: |
You shouldn't have to patch core as since the logging changes are in 0.113.1 (not in .0) |
|
Oh ok... I grepped the home-assistant.log Went into the container and verified that the patch was there and it is... My log config is set to debug for homeassistant.core.... so not sure why that didn't work. |
So it's likely the problem is a chain of tasks that never finish instead of a single one that takes more than 60 seconds. I'll work on a new PR to add additional logging for this case. |
|
New PR is here |
|
I noticed looking through my logs that wink and qnap throw SLOW_SETUP_MAX_WAIT, so I'm going to remove them from my configuration as well. |
|
@deftdawg |
|
With the extended debugging in 0.113.2 I was able the figure out why HA wasn't starting up. I found several messages regarding waiting for a task which pointed me to my fritzbox device tracker. Interestingly the fritzbox device tracker itself seemed to be working before as my devices had the correct state and were also updated correctly but HA ever could start up complete. |
|
With 0.113.2, I'm seeing logging now but it's not obvious to me what the bottleneck to getting finished start up is... |
|
yeah, I hear you. Maybe wait longer? I upgraded and it all works but my ONVIF was still un configured. I put it back in (after upgrade/restart), but yeah. So far so good. |
|
@jurgenweber 5 hours and 328K tasks later... 🤣 😭 |
|
remove device_tracker stuff and upnp? ALl I got.. :0 |
|
I had the same problem. But it started working again when I removed |
|
Fresh install with VMware VM, same problem here. Tried skipping all integrations and ignoring discovered ones too |
|
have the same problem after update to 0.113.2 |
|
After trying many things I found out the problem: |
@sivero Thanks for reporting this. @cgtobi is working hard to improve the |
|
Restarting
10000
from |
|
@wernerhp Please turn on debug logging for |
|
Similar issue for me: #38453 |
Solved for me: was this custom component not updated: https://github.com/custom-components/sensor.mitemp_bt |
@danecreekphotography An intelligent timeout handler has been added to 0.114 in #38329 that should prevent most integrations from blocking startup. Additionally the other known cases should be fixed by #38601 which should close out this issue for good. |
This was
linked to
pull requests
Aug 7, 2020
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
The problem
Home Assistant is starting, not everything will be available until it is finished.This message persists despite home-assistant being started for over 4 hours:
Something is getting stuck during startup and Home Assistant cannot official complete startup (see additional info for reasons I believe that)
Environment
Problem-relevant
configuration.yamlI don't know if there is a specific section of configuration that applied to this...
Traceback/Error logs
There's nothing specific I can see from the logs, if someone wants to suggest things to filter or grep for, I'm happy to do that.
Additional information
Since updating to 0.113.1:
The text was updated successfully, but these errors were encountered: