Description
Description
We are managing containers via Docker remote API. Everything works fine except one issue: when calling "exec" and "pause" on several containers in parallel for several times, Docker hangs on command "docker ps". This issue is easy to reproduce by some testing scripts I created.
Steps to reproduce the issue:
- Run docker on a Linux machine with remote API opened on port 4243
- Logon to the Linux machine, create several Ubuntu containers (or other simple images):
# for i in {0..10}; do docker run -itd --net=host ubuntu; done
- Invoke 100 times parallel "exec" and "pause" on these containers (where node03 is the hostname of the Linux machine, install command "parallel" on the Linux machine for call bash functions in parallel):
# ./pt.sh node03 100
pt.sh
#!/bin/bash
host=$1
num=$2
./do.while.sh $host $num &
./exec.while.sh $host $num &
do.while.sh
#!/bin/bash
host=$1
num=$2
for ((i=0; i<$num; i++)); do
./do.sh -u $host
./do.sh -p $host
done
exec.while.sh
#!/bin/bash
host=$1
num=$2
for ((i=0; i<$num; i++)); do
./exec.sh $host
done
do.sh
#!/bin/bash
param=$1
host=$2
ids=`curl -X GET -H "Content-Type: application/json" http://$host:4243/containers/json | ./JSON.sh -b | grep Id | sed '21,$d' | sed 's/["]$//g;s/[[].*["]//g' | xargs`
if [[ $param == "-p" ]]
then
operation="pause"
elif [[ $param == "-u" ]]; then
operation="unpause"
fi
function work() {
echo "curl -X POST http://$host:4243/containers/$1/$operation"
curl -X POST http://$host:4243/containers/$1/$operation
}
export -f work
export host=$host
export operation=$operation
ids=($ids)
parallel work ::: ${ids[@]}
exec.sh
#!/bin/bash
host=$1
ids=`curl -X GET -H "Content-Type: application/json" http://$host:4243/containers/json | ./JSON.sh -b | grep Id | sed '21,$d' | sed 's/["]$//g;s/[[].*["]//g' | xargs`
function work() {
execid=`curl -H "Content-Type: application/json" -X POST -d '{"AttachStderr":true,"AttachStdin":false,"AttachStdout":true,"Cmd":["date"],"Tty":false}' http://$host:4243/containers/$1/exec | ./JSON.sh -b | grep Id | sed '21,$d' | sed 's/["]$//g;s/[[].*["]//g'`
curl -H "Content-Type: application/json" -X POST -d '{"Detach":false,"Tty":false}' http://$host:4243/exec/$execid/start
}
export -f work
export host=$host
ids=($ids)
parallel work ::: ${ids[@]}
All scripts are published at https://gist.github.com/deathspeeder/e301b248458e95aef5fcd30ad5744a49
JSON.sh is a script formatting JSON string in shell. Download from https://github.com/dominictarr/JSON.sh/blob/master/JSON.sh
Describe the results you received:
Docker hangs on command "docker ps", and will not recover for a long time (several days) untill restarting docker daemon.
docker.log is attached at https://gist.github.com/deathspeeder/e301b248458e95aef5fcd30ad5744a49#file-docker-log
sending SIGUSR1 to dockerd process dumped the trace https://gist.github.com/deathspeeder/e301b248458e95aef5fcd30ad5744a49#file-trace-log
Describe the results you expected:
"docker ps" works fine.
Additional information you deem important (e.g. issue happens only occasionally):
Output of docker version:
# docker version
Client:
Version: 1.12.1
API version: 1.24
Go version: go1.6.3
Git commit: 23cf638
Built: Thu Aug 18 05:22:43 2016
OS/Arch: linux/amd64
Server:
Version: 1.12.1
API version: 1.24
Go version: go1.6.3
Git commit: 23cf638
Built: Thu Aug 18 05:22:43 2016
OS/Arch: linux/amd64
Output of docker info:
# docker info
Containers: 11
Running: 0
Paused: 11
Stopped: 0
Images: 6
Server Version: 1.12.1
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 98
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: null bridge host overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: apparmor
Kernel Version: 4.2.0-35-generic
Operating System: Ubuntu 14.04.4 LTS
OSType: linux
Architecture: x86_64
CPUs: 56
Total Memory: 251.9 GiB
Name: node03
ID: RK67:42MB:7PWA:W7HE:EOYO:2LZD:XIDF:DQVM:5UPZ:PVPE:3MTV:KBHZ
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
File Descriptors: 24
Goroutines: 44
System Time: 2016-10-11T09:07:48.708508545+08:00
EventsListeners: 0
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Insecure Registries:
10.10.1.2:15000
127.0.0.0/8
Additional environment details (AWS, VirtualBox, physical, etc.):
I am using physical Linux machine with Ubuntu server 14.04.1