docs: Explain corp vpn and proxy, fixup FAQ, fixes #7048 #7061

rfay · 2025-03-10T01:47:53Z

The Issue

Add better documentation on how to use a proxy #7048

We occasionally have people with VPN or proxy situations that need help and reference about those

How This PR Solves The Issue

Add VPN and Proxy info
Add more on DNS problems
Add name info to FAQ

TODO

Update ddev/ddev-proxy-support to show supported and explain the different techniques and point to doc.
Add info about name line to remote-config if it's not already there
Add a link to @darrenoh article, https://darren.oh.name/node/81
Verify that ddev-proxy-support can work when pulling new images (it probably can't)
Related discussion for Global Protect with fix mentioned
Docker Desktop docs, additional blog
OrbStack issue
Extensive slack writeup by @shelane, who should review changes.

Manual Testing Instructions

Review (mostly) at https://ddev--7061.org.readthedocs.build/en/7061/users/usage/networking/

Network test setup rendered at https://ddev--7061.org.readthedocs.build/en/7061/developers/network-test-environments/

FAQ is at https://ddev--7061.org.readthedocs.build/en/7061/users/usage/faq/#i-want-to-use-the-same-code-with-various-project-names

docs/content/users/usage/faq.md

docs/content/users/usage/networking.md

stasadev

8000

Looks good to me.

docs/content/users/usage/networking.md

rfay · 2025-03-11T13:32:42Z

This still needs a bit of work and community review:

Test out proxying again. Can it work with image pulls using the add-on or not?
Community review of Zscaler has been requested, and @darrenoh has given his recent experience.

docs/content/users/usage/networking.md

stasadev · 2025-03-14T15:29:35Z

Rebased to pick up buildkite skip for docs:

build: skip testing with buildkite if diff is not from that branch #7064

rfay · 2025-03-14T18:36:12Z

Here is @shelane writeup.

Adding Custom CA Certificates to Your DDEV Project

When working with DDEV in environments that require custom CA certificates (e.g., internal networks, VPNs, or corporate proxies), it is essential to properly configure these certificates to ensure secure communication. This guide provides step-by-step instructions for obtaining, converting, and installing custom CA certificates in your DDEV project.

Steps to Add Custom CA Certificates

Obtain the CA Certificates

Obtain the .crt files for your custom Certificate Authorities (CAs) from your IT department, vendor, or other trusted source.
Ensure the certificates are provided in .crt format. If they are in a different format (e.g., .der), you will need to convert them to the .crt (PEM) format. See Step 2 for details.
Verify and Convert Certificates to PEM Format

To ensure compatibility, all CA certificates must be in PEM format. PEM format certificates are human-readable and start with -----BEGIN CERTIFICATE-----.

Check if a certificate is in PEM format: Open the .crt file in a text editor (e.g., nano or vim). If the file starts with -----BEGIN CERTIFICATE-----, it is already in PEM format, and no conversion is needed.
Convert DER to PEM format: If the certificate is in DER format (binary), you can convert it to PEM format using OpenSSL:

openssl x509 -inform der -in <input-file>.crt -out <output-file>.crt

Replace .crt with the path to your DER certificate and .crt with the desired output file name.
3. Place the Certificates in the .ddev/web-build Directory

Copy all the PEM-formatted .crt files into your DDEV project's .ddev/web-build directory. For example:

my-ddev-project/.ddev/web-build/
├── my-custom-cert1.crt
├── my-custom-cert2.crt

Create a Custom Dockerfile for Certificate Installation

In the .ddev/web-build directory, create a file named Dockerfile.vpn (or a similarly descriptive name).
Add the following content to the Dockerfile.vpn to copy the certificates into the container and update the CA store:

# Copy custom certificates into the container's CA directory
COPY *.crt /usr/local/share/ca-certificates/

# Update the CA certificates store
RUN update-ca-certificates --fresh

Restart DDEV to Apply Changes

Restart your DDEV project to rebuild the container and apply the custom certificates:

ddev restart

Test the Certificates

Verify that the certificates have been installed correctly by running a curl command inside the DDEV container:

ddev exec curl -I https://www.google.com

Replace https://www.google.com with a URL that requires the custom CA certificates. A successful response will include a 200 OK status.

Troubleshooting

Common Issues

Certificate Not in PEM Format:

If the certificate is not in PEM format, you will see errors when updating the CA certificates. Ensure all certificates are properly converted to PEM format using the openssl command in Step 2.

Certificate Not Recognized:

Double-check that the .crt files are correctly placed in the .ddev/web-build directory and that the Dockerfile.vpn is properly configured.

Connection Issues Persist:

Verify that the CA certificates provided are correct and up-to-date. Contact your IT department or vendor if you suspect the certificates are outdated or invalid.

Debugging Tips

To check if the certificates were installed in the container, you can list the installed certificates:

ddev exec ls /etc/ssl/certs

To inspect the CA certificate store for your custom certificates:

ddev exec cat /etc/ssl/certs/ca-certificates.crt | grep "Your-Certificate-Name"

Example Directory Structure

After completing the steps above, your .ddev/web-build directory should look like this:

.ddev/
├── config.yaml
├── web-build/
│   ├── Dockerfile.vpn
│   ├── my-custom-cert1.crt
│   ├── my-custom-cert2.crt

Summary

By following this guide, you can successfully add and configure custom CA certificates in your DDEV project. Ensuring the certificates are in PEM format and properly installed in the container is critical for secure communication with internal or external services.

maciazek · 2025-03-15T20:10:47Z

Hello,

just my 5 cents here as I was working with DDEV behind corporate proxy for last few months, struggling with no previous experience with proxies at all (long story short: proxy info inside /etc/environment, /etc/apt/apt.conf, ~/.bashrc, /etc/systemd/system/docker.service.d/http-proxy.conf combined with ddev/ddev-proxy-support addon was enough to run MariaDB-based project).
But recently I've encountered startup connection errors (during installation of APT packages) with PostgreSQL-based project and after some research here on GitHub I figured out that I need to set up proxy in ~/.docker/config.json (and downgrade DDEV to 1.23.5). After that I was able to run both MariaDB- and PostgreSQL-based projects, but only IP access was working - I couldn't open it via domain name (myapp.ddev.site). This comment was very helpful - after adding ddev-myapp-web (and ddev-myapp-phpmyadmin) to noProxy in ~/.docker/config.json, I was finally able to run both my projects without problems (and without addon).

I'm writing this because I see that you're going to add info about ~/.docker/config.json in documentation, so maybe these container names in noProxy should also be included? *

I was doing all this research this week in my office, but today I was able to reproduce same situation on my personal PC with two VirtualBox machines (one Debian with Squid as proxy, and one Linux Mint with DDEV, connected together via 'internal' network).

rfay · 2025-03-15T21:30:33Z

@maciazek thanks for checking in!

your comment is about a proxy only, right, not a VPN situation?

Was your problem with the ddev-router not becoming ready as in the issue you link to?

If so, have you tested DDEV HEAD with the fix mentioned in that issue? https://ddev.readthedocs.io/en/stable/developers/building-contributing/#testing-latest-commits-on-head

If you have suggestions for this PR based on your work (and using DDEV HEAD) they would be very welcome!

maciazek · 2025-03-16T15:46:54Z

Yes, I'm talking about proxy only, not VPN as I'm working on-site.
We need to set up proxy config to access internet (in OS settings, in Firefox etc.)
Also we need to authenticate through proxy, so our proxy config looks like this:

http://username:password@192.168.1.10:3128

If there is some special character, it has to be url-encoded (with double %):

http://username:p%%40ssword@192.168.1.10:3128

And yes, after setting up my proxy correctly (in ~/.docker/config.json) ddev-router couldn't start because of traefik healthcheck errors (that's why I downgraded to DDEV 1.23.5).

ddev-router failed to become ready; log=, err=health check timed out after 1m0s

Today I upgraded to 1.24.3 and ddev-router couldn't start again. Then I tried HEAD version (ddev version v1.24.3-25-g419560521) as described in docs and now ddev-router is starting without problems (I was testing in my local reproduced environment). This week I'll try to test it at my workplace, but I assume it will behave the same way.

So, in my opinion there are 2 things that should be added to docs (from my environment's perspective, of course):

Proxy config in /etc/systemd/system/docker.service.d/http-proxy.conf - as without this, Docker couldn't pull any image (or maybe simply include links to docker documentation as proxy config is described here and here). Both CLI (~/.docker/config.json) and daemon (/etc/systemd/system/docker.service.d/http-proxy.conf) config files are required in my case to make it all work right.
Adding container names in noProxy in ~/.docker/config.json as described in my previous post. BTW I'm not sure why I can't use wildcards (ddev-*-web,ddev-*-phpmyadmin) - I need to setup names for every project, so my ~/.docker/config.json looks like this:

{
  "proxies": {
    "default": {
       "httpProxy": "http://username:p%%40ssword@192.168.1.10:3128",
       "httpsProxy": "http://username:p%%40ssword@192.168.1.10:3128",
       "noProxy": "localhost,127.0.0.1/8,::1,*.ddev.site,ddev-myapp1-web,ddev-myapp1-phpmyadmin,ddev-myapp2-web,ddev-myapp2-phpmyadmin"
    }
  }
}

Maybe there is some simpler way to set these container names once for all projects?

rfay · 2025-03-16T15:58:41Z

Thanks @maciazek -

Proxy config in /etc/systemd/system/docker.service.d/http-proxy.conf

I've had varying approaches to this, and am a bit confused about it. You're sure that the /etc/systemd stuff is required, as opposed to just putting it in the user's .docker, true?

Adding container names in noProxy in ~/.docker/config.json

That seems enormously awkward and wrong of course.

One note: I've seen a mention that *.<domain> is sometimes invalid, and .domain is correct. However, I haven't confirmed that and don't know that Docker doesn't handle both.

Thanks so much for your careful work and help on this!

maciazek · 2025-03-16T17:38:00Z

I've had varying approaches to this, and am a bit confused about it. You're sure that the /etc/systemd stuff is required, as opposed to just putting it in the user's .docker, true?

I deleted file /etc/systemd/system/docker.service.d/http-proxy.conf, then daemon-reload & restart docker, then ddev delete images --all and when I tried ddev start got this:

Error response from daemon: Get "https://registry-1.docker.io/v2/": dial tcp: lookup registry-1.docker.io on 127.0.0.53:53: server misbehaving
Unable to pull Docker images: exit status 1

I have HTTP_PROXY env vars set in ~/.bashrc and /etc/environment all the time.

I also tried "Daemon configuration" approach described in the docs, but I was able to pull images only when I set up /etc/docker/daemon.json config.

Config set via both 'rootless modes' described in docs (~/.config/docker/daemon.json or ~/.config/systemd/user/docker.service.d/http-proxy.conf) where giving me the same error mentioned above. So one way or another, I need to set up proxy in /etc... (/etc/systemd/system/docker.service.d/http-proxy.conf or /etc/docker/daemon.json) to be able to pull images. Or maybe there is some another method? Unfortunately I'm not very familiar with Docker (that's why I started to work with DDEV 😊) But to answer your question: proxy config inside ~/docker/config.json was not sufficient for me.

That seems enormously awkward and wrong of course.

One note: I've seen a mention that *. is sometimes invalid, and .domain is correct. However, I haven't confirmed that and don't know that Docker doesn't handle both.

I did some more tests (in ~/docker/config.json):

"noProxy": "127.0.0.0/8,::1,localhost,.ddev.site,.ddev.local"
I couldn't access both my app and phpmyadmin through domain

"noProxy": "127.0.0.0/8,::1,localhost,*.ddev.site,*.ddev.local,.ddev.site,.ddev.local,ddev-,ddev-*,ddev-*-web,ddev-*-phpmyadmin"
Same as above

"noProxy": "127.0.0.0/8,::1,localhost,*.ddev.site,*.ddev.local,.ddev.site,.ddev.local,ddev-,ddev-*,ddev-*-web,ddev-*-phpmyadmin,ddev-myapp-phpmyadmin"
I could only access phpmyadmin via domain - my app was still not accessible.

Errors are coming from Squid ((111) Connection refused) - for me it looks like Traefik uses proxy data from ~/docker/config.json and tries to access these containers via my proxy (when i use docker logs ddev-router I can see these container names there):

172.18.0.1 - - [16/Mar/2025:17:11:26 +0000] "GET / HTTP/2.0" 503 4348 "-" "-" 1 "myapp-web-80-https@file" "http://ddev-myapp-web:80" 92ms
172.18.0.1 - - [16/Mar/2025:17:11:28 +0000] "GET / HTTP/2.0" 503 4373 "-" "-" 2 "myapp-phpmyadmin-80-https@file" "http://ddev-myapp-phpmyadmin:80" 2ms

rfay · 2025-03-16T17:49:05Z

Thanks.

Just a note that DDEV does not support rootless docker.

stasadev · 2025-03-18T13:06:31Z

@maciazek and @mac-dev,

Please check this PR and leave a review there:

fix: set NO_PROXY=* in ddev-router to allow internal connections, use default Traefik config file location #7118 (comment)

rfay · 2025-04-09T14:51:48Z

I'd appreciate everybody's review again, https://github.com/ddev/ddev/pull/7061/files

I think this now covers an enormous range of possibilities.

The important rendered version is at https://ddev--7061.org.readthedocs.build/en/7061/users/usage/networking/

If you're interested in testing setup, https://ddev--7061.org.readthedocs.build/en/7061/developers/network-test-environments/

stasadev

Wow, so much information!

I'm not sure where squid.host-only:3128 comes from, I replaced it with localhost:3128 to make it work for me locally.

docs/content/users/usage/networking.md

stasadev · 2025-04-09T19:57:10Z

docs/content/users/usage/networking.md

+  "proxies": {
+    "http-proxy": "http://squid.host-only:3128",
+    "https-proxy": "http://squid.host-only:3128",
+    "no-proxy": "localhost,127.0.0.0/8"


This should look like NO_PROXY=localhost,127.0.0.1,::1,*.ddev.site ?

In this situation it doesn't matter, because it's all about getting to the registry, and doesn't affect our containers. At least that's what I think.

docs/content/developers/network-test-environments.md

rfay · 2025-04-09T20:47:20Z

On squid.host-only: I used a Parallels VM named squid, freestanding Ubuntu. I limited access to it via the host-only network to prove that proxying was happening. I decided to leave it that way here because in the testbed page... people will have to adjust for their environment anyway.

Co-authored-by: Stanislav Zhuk <stasadev@gmail.com>

rfay · 2025-04-09T21:08:37Z

The intent of the "Network Test Environments" page is just so that we have a way to set this up again; it's not intended for any user; hopefully the bump on the top keeps it out of the search results mostly.

stasadev

Looks good to me.

rfay · 2025-04-10T14:34:17Z

For reviewers, this is now in https://ddev.readthedocs.io/en/latest/users/usage/networking/ and we'd love your feedback as you discover it, just open an issue or click the pencil in the upper right!

github-actions bot added the documentation label Mar 10, 2025

rfay requested review from stasadev and tyler36 March 10, 2025 01:53

rfay marked this pull request as ready for review March 10, 2025 01:55

rfay requested review from a team as code owners March 10, 2025 01:55

tyler36 requested changes Mar 10, 2025

View reviewed changes

docs/content/users/usage/faq.md Outdated Show resolved Hide resolved

docs/content/users/usage/networking.md Outdated Show resolved Hide resolved

docs/content/users/usage/networking.md Outdated Show resolved Hide resolved

stasadev approved these changes Mar 10, 2025

View reviewed changes

docs/content/users/usage/networking.md Outdated Show resolved Hide resolved

docs/content/users/usage/networking.md Outdated Show resolved Hide resolved

docs/content/users/usage/networking.md Outdated Show resolved Hide resolved

darrenoh reviewed Mar 10, 2025

View reviewed changes

docs/content/users/usage/networking.md Outdated Show resolved Hide resolved

tyler36 approved these changes Mar 11, 2025

View reviewed changes

stasadev mentioned this pull request Mar 11, 2025

build: skip testing with buildkite if diff is not from that branch #7064

Merged

rfay marked this pull request as draft March 11, 2025 13:32

rfay commented Mar 14, 2025

View reviewed changes

docs/content/users/usage/networking.md Outdated Show resolved Hide resolved

stasadev force-pushed the 20250309_rfay_docs_vpn_proxy branch from c2b2f12 to 92e89ad Compare March 14, 2025 15:29

stasadev mentioned this pull request Mar 18, 2025

fix: set NO_PROXY=* in ddev-router to allow internal connections, use default Traefik config file location #7118

Merged

rfay modified the milestone: v1.24.4 Mar 31, 2025

rfay mentioned this pull request Apr 2, 2025

Custom root CA not applied anymore in DDEV 1.24.4 #7180

Closed

4 tasks

rfay force-pushed the 20250309_rfay_docs_vpn_proxy branch from 92e89ad to da3f8b7 Compare April 7, 2025 20:09

docs: Explain corp vpn and proxy, fixes ddev#7048

b81d47a

rfay force-pushed the 20250309_rfay_docs_vpn_proxy branch from ea1461e to 68575f5 Compare April 9, 2025 04:46

rfay added 10 commits April 9, 2025 07:01

apply chatGPT suggestions [skip buildkite]

af4eeaf

add section about .cer and .pem files, aided by chatgpt [skip buildkite]

2b27932

testbed setup for crt file [skip buildkite]

f41f81a

More minor edits from chatgpt [skip buildkite]

c8861a5

Minor edits

eeb4d98

Add small boost back in

78dab51

Another review of test environments [skip buildkite]

c1b7d50

pacify linter [skip buildkite]

0468d7d

add note that this is not typical [skip buildkite]

06ffc32

spellcheck fixes [skip buildkite]

d80ab22

rfay marked this pull request as ready for review April 9, 2025 14:48

rfay requested review from darrenoh, tyler36 and stasadev April 9, 2025 14:49

pacify markdownlint [skip buildkite]

643e04b

darrenoh approved these changes Apr 9, 2025

View reviewed changes

stasadev requested changes Apr 9, 2025

View reviewed changes

rfay and others added 3 commits April 9, 2025 14:53

@stasadev suggestions, thanks

f958431

Co-authored-by: Stanislav Zhuk <stasadev@gmail.com>

Use less headings in network lab page [skip buildkite]

dc03a49

Call off textlint on Internet [skip buildkite]

47e1f9c

stasadev approved these changes Apr 9, 2025

View reviewed changes

Add note about checking with IT [skip buildkite]

60e094e

rfay force-pushed the 20250309_rfay_docs_vpn_proxy branch from 970f629 to 60e094e Compare April 9, 2025 21:48

rfay merged commit 4573008 into ddev:main Apr 10, 2025
10 checks passed

rfay deleted the 20250309_rfay_docs_vpn_proxy branch April 10, 2025 14:31

Uh oh!

docs: Explain corp vpn and proxy, fixup FAQ, fixes #7048 #7061

docs: Explain corp vpn and proxy, fixup FAQ, fixes #7048 #7061

Uh oh!

Conversation

Uh oh!

The Issue

How This PR Solves The Issue

TODO

Manual Testing Instructions

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Adding Custom CA Certificates to Your DDEV Project

Steps to Add Custom CA Certificates

Troubleshooting

Common Issues

Debugging Tips

Summary

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!