8000 Reverse Proxy `URL_PREFIX` connection error · Issue #18173 · PrefectHQ/prefect · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Reverse Proxy URL_PREFIX connection error #18173

New issue

Have a question 8000 about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
Pk13055 opened this issue May 27, 2025 · 1 comment
Closed

Reverse Proxy URL_PREFIX connection error #18173

Pk13055 opened this issue May 27, 2025 · 1 comment
Labels
bug Something isn't working

Comments

@Pk13055
Copy link
Pk13055 commented May 27, 2025 •
edited
Loading

Bug summary

The worker and deploy scripts are unable to communicate with the API server (in docker compose with nginx reverse proxy), ie, UI: /prefect and API /prefect/api

Compose file:

services:
  ### Prefect Database
  database:
    image: postgres:alpine
    restart: always
    environment:
      - POSTGRES_USER=${POSTGRES_USER}
      - POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
      - POSTGRES_DB=${POSTGRES_DB}
    networks:
      - prefect_network
    volumes:
      - ./data/db:/var/lib/postgresql/data

  ### Nginx Reverse Proxy
  nginx:
    image: nginx:alpine
    restart: always
    ports:
      - 8000:80
    networks:
      - ui_network
    volumes:
      - ./config/nginx/:/etc/nginx/conf.d/
    depends_on:
      - server

  ### Prefect Server API and UI
  server:
    image: prefecthq/prefect:3-python3.12
    restart: always
    volumes:
      - prefect:/root/.prefect
    entrypoint: ["/opt/prefect/entrypoint.sh", "prefect", "server", "start"]
    environment:
      - PREFECT_UI_URL=http://0.0.0.0:8000/prefect
      - PREFECT_API_URL=http://0.0.0.0:8000/prefect/api
      - PREFECT_UI_SERVE_BASE=/prefect
      # If you want to access Prefect Server UI from anywhere other than the Docker host machine, you will need to change
      # PREFECT_UI_URL and PREFECT_API_URL to match the external hostname/IP of the host machine. For example:
      #- PREFECT_UI_URL=http://external-ip/prefect
      #- PREFECT_API_URL=http://external-ip/prefect/api
      - PREFECT_SERVER_API_HOST=${PREFECT_SERVER_API_HOST}
      - PREFECT_API_DATABASE_CONNECTION_URL=${PREFECT_API_DATABASE_CONNECTION_URL}
      # - EXTRA_PIP_PACKAGES=${EXTRA_PIP_PACKAGES}
    depends_on:
      - database
    networks:
      - ui_network
      - prefect_network

  ### Auto-discovery and deployment service
  deploy:
    image: prefecthq/prefect:3-python3.12
    restart: "no"
    working_dir: "/root/flows"
    volumes:
      - "./flows:/root/flows"
      - "./deployments:/root/deployments"
    environment:
      # - PREFECT_API_URL=${PREFECT_API_URL:-http://0.0.0.0:8000/prefect/api}
      - PREFECT_API_URL=http://0.0.0.0:8000/prefect/api
      - WORK_POOL_NAME=${WORK_POOL_NAME}
      - WORK_POOL_TYPE=${WORK_POOL_TYPE}
    depends_on:
      - server
    networks:
      - ui_network
      - prefect_network
    entrypoint:
      - bash
      - -c
      - |
        echo 'Waiting for Prefect server to be ready...'
        sleep 30
        echo 'Creating work pool...'
        prefect work-pool create $${WORK_POOL_NAME} --type $${WORK_POOL_TYPE} || echo 'Work pool already exists'
        echo 'Deploying flows...'
        cd /root/flows

        # List all Python files for debugging
        echo 'Found Python files:'
        find . -name "*.py" -not -path "./__pycache__/*" | head -10

        # Find all flow functions and deploy them using xargs
        echo 'Discovering and deploying flows...'

        # Create a function to extract flow entrypoints
        extract_flows() {
            for py_file in $$(find . -name "*.py" -not -path "./__pycache__/*"); do
                if grep -q "@flow" "$$py_file"; then
                    # Extract flow function names that follow @flow decorator
                    grep -A 1 "@flow" "$$py_file" | grep "def " | sed 's/def \([^(]*\).*/\1/' | while read flow_func; do
                        if [ ! -z "$$flow_func" ]; then
                            # Output in format: filename:function_name
                            echo "$${py_file#./}:$$flow_func"
                        fi
                    done
                fi
            done
        }

        # Extract all flows and deploy them using xargs
        extract_flows | while read entrypoint; do
            if [ ! -z "$$entrypoint" ]; then
                flow_name=$$(echo "$$entrypoint" | cut -d':' -f2)
                deployment_name="$$flow_name-deployment"

                echo "Deploying: $$entrypoint as $$deployment_name"

                prefect deploy \
                    --name "$$deployment_name" \
                    --pool "$${WORK_POOL_NAME}" \
                    "$$entrypoint" || echo "Failed to deploy $$entrypoint"
            fi
        done
        echo 'Auto-discovery and deployment completed!'

  ## Prefect Worker (Process-based for local execution)
  worker:
    image: prefecthq/prefect:3-python3.12
    restart: always
    entrypoint:
      [
        "/opt/prefect/entrypoint.sh",
        "prefect",
        "worker",
        "start",
        "--pool",
        "${WORK_POOL_NAME}",
        "--type",
        "${WORK_POOL_TYPE}",
      ]
    environment:
      - PREFECT_API_URL=http://0.0.0.0:8000/prefect/api
      # - PREFECT_API_URL=${PREFECT_API_URL:-http://0.0.0.0:8000/prefect/api}
      # - EXTRA_PIP_PACKAGES=${EXTRA_PIP_PACKAGES}
    networks:
      - ui_network
      - prefect_network
    volumes:
      - ./flows:/root/flows
    depends_on:
      - deploy

volumes:
  prefect:

networks:
  ui_network:
  prefect_network:

NGINX config:

upstream prefect_server {
    server server:4200;
}

# now we declare our main server
server {

    listen 80;
    server_name _;
    client_max_body_size 10M;

    location /prefect/api {
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $host;
        proxy_redirect off;
        rewrite ^/prefect/api/?(.*)$ /api/$1 break;
        proxy_pass http://prefect_server;
    }
    location /prefect/ {
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $host;
        proxy_redirect off;
        proxy_pass http://prefect_server;
    }
}

.env vars:

COMPOSE_PROJECT_NAME=scheduler
# Database Configuration
POSTGRES_USER=prefect
POSTGRES_PASSWORD=prefect123
POSTGRES_DB=prefect

# Prefect Server Configuration
PREFECT_UI_URL=http://0.0.0.0:8000/prefect
PREFECT_API_URL=http://0.0.0.0:8000/prefect/api
PREFECT_SERVER_API_HOST=0.0.0.0
PREFECT_API_DATABASE_CONNECTION_URL=postgresql+asyncpg://prefect:prefect123@database:5432/prefect

# Work Pool Configuration
WORK_POOL_NAME=default-pool
WORK_POOL_TYPE=process

# EXTRA_PIP_PACKAGES=

Version info

Version:             3.4.3
API version:         0.8.4
Python version:      3.12.10
Git commit:          1c2ba7a4
Built:               Thu, May 22, 2025 10:03 PM
OS/Arch:             linux/aarch64
Profile:  
8000
           ephemeral
Server type:         server
Pydantic version:    2.11.5
Integrations:
  prefect-redis:     0.2.2

Additional context

Traceback:

 Traceback (most recent call last):
   File "/usr/local/lib/python3.12/site-packages/httpx/_transports/default.py", line 101, in map_httpcore_exceptions
     yield
   File "/usr/local/lib/python3.12/site-packages/httpx/_transports/default.py", line 394, in handle_async_request
     resp = await self._pool.handle_async_request(req)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.12/site-packages/httpcore/_async/connection_pool.py", line 256, in handle_async_request
     raise exc from None
   File "/usr/local/lib/python3.12/site-packages/httpcore/_async/connection_pool.py", line 236, in handle_async_request
     response = await connection.handle_async_request(
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.12/site-packages/httpcore/_async/connection.py", line 101, in handle_async_request
     raise exc
   File "/usr/local/lib/python3.12/site-packages/httpcore/_async/connection.py", line 78, in handle_async_request
     stream = await self._connect(request)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.12/site-packages/httpcore/_async/connection.py", line 124, in _connect
     stream = await self._network_backend.connect_tcp(**kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.12/site-packages/httpcore/_backends/auto.py", line 31, in connect_tcp
     return await self._backend.connect_tcp(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.12/site-packages/httpcore/_backends/anyio.py", line 113, in connect_tcp
     with map_exceptions(exc_map):
          ^^^^^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.12/contextlib.py", line 158, in __exit__
     self.gen.throw(value)
   File "/usr/local/lib/python3.12/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
     raise to_exc(exc) from exc
 httpcore.ConnectError: All connection attempts failed

 The above exception was the direct cause of the following exception:

 Traceback (most recent call last):
   File "/usr/local/lib/python3.12/site-packages/prefect/cli/_utilities.py", line 44, in wrapper
     return fn(*args, **kwargs)
            ^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.12/site-packages/prefect/cli/_types.py", line 156, in sync_fn
     return asyncio.run(async_fn(*args, **kwargs))
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.12/asyncio/runners.py", line 195, in run
     return runner.run(main)
            ^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.12/asyncio/runners.py", line 118, in run
     return self._loop.run_until_complete(task)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.12/asyncio/base_events.py", line 691, in run_until_complete
     return future.result()
            ^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.12/site-packages/prefect/cli/worker.py", line 116, in start
     is_paused = await _check_work_pool_paused(work_pool_name)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.12/site-packages/prefect/cli/worker.py", line 181, in _check_work_pool_paused
     work_pool = await client.read_work_pool(work_pool_name=work_pool_name)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.12/site-packages/prefect/client/orchestration/_work_pools/client.py", line 416, in read_work_pool
     response = await self.request(
                ^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.12/site-packages/prefect/client/orchestration/base.py", line 53, in request
     return await self._client.send(request)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.12/site-packages/prefect/client/base.py", line 330, in send
     response = await self._send_with_retry(
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.12/site-packages/prefect/client/base.py", line 250, in _send_with_retry
     response = await send(request, *send_args, **send_kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
8000
^^^^^
   File "/usr/local/lib/python3.12/site-packages/httpx/_client.py", line 1629, in send
     response = await self._send_handling_auth(
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.12/site-packages/httpx/_client.py", line 1657, in _send_handling_auth
     response = await self._send_handling_redirects(
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.12/site-packages/httpx/_client.py", line 1694, in _send_handling_redirects
     response = await self._send_single_request(request)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.12/site-packages/httpx/_client.py", line 1730, in _send_single_request
     response = await transport.handle_async_request(request)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.12/site-packages/httpx/_transports/default.py", line 393, in handle_async_request
     with map_httpcore_exceptions():
          ^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.12/contextlib.py", line 158, in __exit__
     self.gen.throw(value)
   File "/usr/local/lib/python3.12/site-packages/httpx/_transports/default.py", line 118, in map_httpcore_exceptions
     raise mapped_exc(message) from exc
 httpx.ConnectError: All connection attempts failed
 An exception occurred.
@Pk13055 Pk13055 added the bug Something isn't working label May 27, 2025
@zzstoatzz
Copy link
Collaborator

hi @Pk13055 - this appears to be a configuration issue, in particular

- PREFECT_API_URL=http://0.0.0.0:8000/prefect/api

port 8000 isn't available for inter-container communication per the above config

feel free to reopen with a description of what's violating your expectations if you think I'm mistaken

docs

another example

@zzstoatzz zzstoatzz closed this as not planned Won't fix, can't repro, duplicate, stale May 27, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
< 3420 div class="js-socket-channel js-updatable-content" data-channel="eyJjIjoiaXNzdWU6MzA5MzI0OTgzNSIsInQiOjE3NDg5OTk3MDF9--d59b7d7a6c881d7ae1653965f7f34e2275c4a47d792eb4e4b92f1543c90acec9" data-gid="I_kwDOCEwExM64Xz8r" data-url="/PrefectHQ/prefect/issues/closing_references/partials/sidebar?source_id=3093249835&source_type=ISSUE" data-channel-event-name="issue_references_updated" >
Development

No branches or pull requests
2 participants
@Pk13055 @zzstoatzz
0