8000 After Docker daemon restart, containers do not gracefully recover · Issue #237 · StackStorm/st2-docker · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content
After Docker daemon restart, containers do not gracefully recover #237
Open
@greenmoss

Description

@greenmoss

Describe the problem

  • Bring up st2-docker
  • Restart the Docker service
  • Some of the containers do not recover properly, and throw stack traces

Versions

  • Host OS: Ubuntu 20.04
  • docker: version 20.10.2, build 2291f61
  • docker-compose: version 1.29.2, build 5becea4c
  • stackstorm version: 3.6.0

To Reproduce
Bring up cluster
docker-compose up -d

Look for exited containers

docker-compose ps -a | grep Exit
st2-docker_st2chatops_1            /st2chatops-startup.sh           Exit 0
st2-docker_st2makesecrets_1        /makesecrets.sh                  Exit 0

Stop/restart docker service
service docker restart

Look for exited containers

docker-compose ps -a | grep Exit
st2-docker_mongo_1                 docker-entrypoint.sh mongod      Exit 0
st2-docker_rabbitmq_1              docker-entrypoint.sh rabbi ...   Exit 0
st2-docker_redis_1                 docker-entrypoint.sh redis ...   Exit 0
st2-docker_st2actionrunner_1       /opt/stackstorm/st2/bin/st ...   Exit 0
st2-docker_st2chatops_1            /st2chatops-startup.sh           Exit 0
st2-docker_st2makesecrets_1        /makesecrets.sh                  Exit 0
st2-docker_st2scheduler_1          /opt/stackstorm/st2/bin/st ...   Exit 0
st2-docker_st2web_1                /bin/bash -c if [ ${ST2WEB ...   Exit 0
st2-docker_st2workflowengine_1     /opt/stackstorm/st2/bin/st ...   Exit 0

Follow/tail Docker logs

docker-compose logs -f --tail 20

Note all the errors. It seems all of them are some variant of pymongo.errors.ServerSelectionTimeoutError: mongo:27017: [Errno -2] Name or service not known.

To recover:

docker-compose down
docker-compose up -d

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions

      0