Not able to pass healthcheck #719

mbajur · 2024-03-10T13:11:52Z

mbajur
Mar 10, 2024

Hello! I have a perfectly fine working Dockerfile with production config for my app. I'm able to run it locally and everything seems to be fine with the image itself but once i use it with kamal, my healtcheck starts complaining without any helpfull output displayed.

All i can see is:

Ensure app can pass healthcheck...
  INFO [db9b8c4e] Running docker run --detach --name healthcheck-xxx-36014653139f534d2728d2656a36bc6e573c3d60 --publish 3999:3000 --label service=healthcheck-xxx -e KAMAL_CONTAINER_NAME="healthcheck-xxx" --env-file .kamal/env/roles/xxx-web.env --health-cmd "ls" --health-interval "1s" --volume /home/app/xxx-cache:/app/tmp/cache --volume /home/app/xxx-shared:/app/shared --volume /home/app/xxx-storage:/app/storage --network "private" ghcr.io/mbajur/xxx:36014653139f534d2728d2656a36bc6e573c3d60 on x.x.x.x
  INFO [db9b8c4e] Finished in 6.367 seconds with exit status 0 (successful).
  INFO [b08e73ca] Running docker container ls --all --filter name=^healthcheck-xxx-36014653139f534d2728d2656a36bc6e573c3d60$ --quiet | xargs docker inspect --format '{{if .State.Health}}{{.State.Health.Status}}{{else}}{{.State.Status}}{{end}}' on x.x.x.x
  INFO [b08e73ca] Finished in 1.611 seconds with exit status 0 (successful).
  INFO container not ready (unhealthy), retrying in 1s (attempt 1/7)...
  INFO [9338dbbb] Running docker container ls --all --filter name=^healthcheck-xxx-36014653139f534d2728d2656a36bc6e573c3d60$ --quiet | xargs docker inspect --format '{{if .State.Health}}{{.State.Health.Status}}{{else}}{{.State.Status}}{{end}}' on x.x.x.x
  INFO [9338dbbb] Finished in 0.277 seconds with exit status 0 (successful).
  INFO container not ready (unhealthy), retrying in 2s (attempt 2/7)...
  INFO [9f8534bf] Running docker container ls --all --filter name=^healthcheck-xxx-36014653139f534d2728d2656a36bc6e573c3d60$ --quiet | xargs docker inspect --format '{{if .State.Health}}{{.State.Health.Status}}{{else}}{{.State.Status}}{{end}}' on x.x.x.x
  INFO [9f8534bf] Finished in 0.291 seconds with exit status 0 (successful).
  INFO container not ready (unhealthy), retrying in 3s (attempt 3/7)...
  INFO [9e1cf807] Running docker container ls --all --filter name=^healthcheck-xxx-36014653139f534d2728d2656a36bc6e573c3d60$ --quiet | xargs docker inspect --format '{{if .State.Health}}{{.State.Health.Status}}{{else}}{{.State.Status}}{{end}}' on x.x.x.x
  INFO [9e1cf807] Finished in 0.283 seconds with exit status 0 (successful).
  INFO container not ready (unhealthy), retrying in 4s (attempt 4/7)...
  INFO [a0eec7d7] Running docker container ls --all --filter name=^healthcheck-xxx-36014653139f534d2728d2656a36bc6e573c3d60$ --quiet | xargs docker inspect --format '{{if .State.Health}}{{.State.Health.Status}}{{else}}{{.State.Status}}{{end}}' on x.x.x.x
  INFO [a0eec7d7] Finished in 0.282 seconds with exit status 0 (successful).
  INFO container not ready (unhealthy), retrying in 5s (attempt 5/7)...
  INFO [c3e00218] Running docker container ls --all --filter name=^healthcheck-xxx-36014653139f534d2728d2656a36bc6e573c3d60$ --quiet | xargs docker inspect --format '{{if .State.Health}}{{.State.Health.Status}}{{else}}{{.State.Status}}{{end}}' on [185](https://github.com/mbajur/xxx/actions/runs/8222074109/job/22483228721#step:7:186).25.150.[187](https://github.com/mbajur/xxx/actions/runs/8222074109/job/22483228721#step:7:188)
  INFO [c3e00218] Finished in 0.285 seconds with exit status 0 (successful).
  INFO container not ready (unhealthy), retrying in 6s (attempt 6/7)...
  INFO [e246f316] Running docker container ls --all --filter name=^healthcheck-xxx-36014653139f534d2728d2656a36bc6e573c3d60$ --quiet | xargs docker inspect --format '{{if .State.Health}}{{.State.Health.Status}}{{else}}{{.State.Status}}{{end}}' on 185.25.150.187
  INFO [e246f316] Finished in 0.281 seconds with exit status 0 (successful).
  INFO container not ready (unhealthy), retrying in 7s (attempt 7/7)...
  INFO [56bd0913] Running docker container ls --all --filter name=^healthcheck-xxx-36014653139f534d2728d2656a36bc6e573c3d60$ --quiet | xargs docker inspect --format '{{if .State.Health}}{{.State.Health.Status}}{{else}}{{.State.Status}}{{end}}' on 185.25.150.187
  INFO [56bd0913] Finished in 0.282 seconds with exit status 0 (successful).
  INFO [01ccc6a2] Running docker container ls --all --filter name=^healthcheck-xxx-36014653139f534d2728d2656a36bc6e573c3d60$ --quiet | xargs docker logs --tail 50 2>&1 on x.x.x.x
  INFO [01ccc6a2] Finished in 0.433 seconds with exit status 0 (successful).
 ERROR Switch to inspect mode.
 ERROR {
  "Status": "unhealthy",
  "FailingStreak": 0,
  "Log": [
  ]
}

Because the Log is empty, i'm not really able to tell what's wrong with the container itself. All i was able to see was that, during the deployment, there is a container for web being created with irb command (which is nothing i defined, my healthcheck command is bundle exec puma -C config/puma.rb so not sure where irb is coming from?).

Also, when running Dockerfile locally, i can see it produces output so i'm not sure why there is no output when kamal tries to run a healthcheck. Here is my deploy.yml file:

service: xxx

image: mbajur/xxx

volumes:
  - "/home/app/xxx-cache:/app/tmp/cache"
  - "/home/app/xxx-shared:/app/shared"
  - "/home/app/xxx-storage:/app/storage"

servers:
  web:
    hosts:
      - x.x.x.x
    labels:
      traefik.http.routers.xxx.entrypoints: websecure
      traefik.http.routers.xxx.rule: Host(`xxx.xxx.xxx`)
      traefik.http.routers.xxx.tls.certresolver: letsencrypt
    options:
      network: "private"

registry:
  server: ghcr.io
  username: mbajur

  password:
    - KAMAL_REGISTRY_PASSWORD

# Inject ENV variables into containers (secrets come from .env).
# Remember to run `kamal env push` after making changes!
env:
  clear:
    HOSTNAME: xxx.xxx.xxx
    DB_HOST: x.x.x.x
    RAILS_SERVE_STATIC_FILES: true
    RAILS_LOG_TO_STDOUT: true
  secret:
    - POSTGRES_PASSWORD
    - RAILS_MASTER_KEY

ssh:
  user: app

builder:
  dockerfile: Dockerfile.production
  multiarch: false
  cache:
    type: registry

accessories:
  db:
    image: postgres:15
    host: x.x.x.x
    port: 5432
    env:
      clear:
        POSTGRES_USER: "xxx"
        POSTGRES_DB: 'xxx_production'
      secret:
        - POSTGRES_PASSWORD
    files:
      - config/init.sql:/docker-entrypoint-initdb.d/setup.sql
    directories:
      - data:/var/lib/postgresql/data
    options:
      network: "private"

traefik:
  args:
    accesslog: true
  options:
    network: "private"
    publish:
      - "443:443"
    volume:
      - "/letsencrypt/acme.json:/letsencrypt/acme.json"
  args:
    entryPoints.web.address: ":80"
    entryPoints.websecure.address: ":443"
    entryPoints.web.http.redirections.entryPoint.to: websecure # We want to force https
    entryPoints.web.http.redirections.entryPoint.scheme: https
    entryPoints.web.http.redirections.entrypoint.permanent: true
    certificatesResolvers.letsencrypt.acme.email: "xxx@xxx.com"
    certificatesResolvers.letsencrypt.acme.storage: "/letsencrypt/acme.json" # Must match the path in `volume`
    certificatesResolvers.letsencrypt.acme.httpchallenge: true
    certificatesResolvers.letsencrypt.acme.httpchallenge.entrypoint: web

healthcheck:
  path: /health/ready
  port: 4000
  cmd: bundle exec puma -C config/puma.rb

Please let me know if there is anything i can do to help you help me or if there is anything obvious i'm missing in here? :)

sebastienbourda · 2024-03-20T09:26:47Z

sebastienbourda
Mar 20, 2024

I have same issue. Did you find a solution ?

5 replies

RomanKrasavtsev Apr 24, 2024

Try to increase value for max_attempts or interval https://kamal-deploy.org/docs/configuration/healthchecks/

intrip Apr 26, 2024
Collaborator

Why you are using bundle exec puma as health check instead of checking the health of the app via curl?

You can ssh in the machine and see what's going on in the Healthcheck container via docker logs -f ID

Also if you are stuck you can disable the health check, but you'll lose the zero-downtime deployment feature.

jmcbri Jul 6, 2024

Thanks. How do you disable the healthcheck? Something like:

healthcheck: false

intrip Jul 8, 2024
Collaborator

You can use:

healthcheck:
  cmd: true

jmcbri Jul 13, 2024

Thanks @intrip ! In the above, do disable it would be cmd: true Or would it be cmd: false ?

imWildCat · 2024-07-10T18:06:41Z

imWildCat
Jul 10, 2024

did you add this to your routes.rb if it's rails?

  # Reveal health status on /up that returns 200 if the app boots with no exceptions, otherwise 500.
  # Can be used by load balancers and uptime monitors to verify that the app is live.
  get "up" => "rails/health#show", as: :rails_health_check

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Not able to pass healthcheck #719

Uh oh!

{{title}}

Uh oh!

Replies: 2 comments 5 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Not able to pass healthcheck #719

Uh oh!

mbajur Mar 10, 2024

Replies: 2 comments · 5 replies

Uh oh!

sebastienbourda Mar 20, 2024

Uh oh!

RomanKrasavtsev Apr 24, 2024

Uh oh!

intrip Apr 26, 2024 Collaborator

Uh oh!

jmcbri Jul 6, 2024

Uh oh!

intrip Jul 8, 2024 Collaborator

Uh oh!

jmcbri Jul 13, 2024

Uh oh!

imWildCat Jul 10, 2024

mbajur
Mar 10, 2024

Replies: 2 comments 5 replies

sebastienbourda
Mar 20, 2024

intrip Apr 26, 2024
Collaborator

intrip Jul 8, 2024
Collaborator

imWildCat
Jul 10, 2024