Unable to deploy to Digital Ocean - 401 Unauthorized #1199

codepilotsf · 2024-11-06T23:28:01Z

codepilotsf
Nov 6, 2024

For some reason, after authenticating with doctl auth init --context myapp I'm hitting this error:

 > exporting to image:
------
ERROR: failed to solve: failed to push registry.digitalocean.com/myapp-registry/myapp:17f228b9bd594c42e92dfcaa593d770d210090b3: failed to authorize: failed to fetch oauth token: unexpected status from GET request to https://api.digitalocean.com/v2/registry/auth?scope=repository%3Acodepilotsf%2Ftestme%3Apull&scope=repository%3Amyapp-registry%2Fmyapp%3Apull%2Cpush&service=registry.digitalocean.com: 401 Unauthorized

The really weird thing is that the URL in the error seems to show old connection info "codepilotsf/testme" in addition to the new one. This is all such a black box to me. I thought Kamal was supposed to simplify deployment. :-)

Any clues?

codepilotsf · 2024-11-07T18:17:46Z

codepilotsf
Nov 7, 2024
Author

I wanted to answer my own question he 8000 re (sorta) in case it's helpful to someone else. What I've learned is that DO container registry is a pain in the ass. I think almost all of my initial frustration with Kamal was due to issues with doctl and various related auth issues. I switched over to DockerHub and everything worked great straight away. Then I learned that my self-hosted GitLab also has a built in container registry service which is just as simple, works perfectly, and costs nothing. Tldr; Love DO, but their container registry service kinda blows.

0 replies

rschultheis · 2024-12-31T01:00:36Z

rschultheis
Dec 31, 2024

OK, I have made progress but I still do not understand this issue.

I am sometimes getting an error when pushing registry.digitalocean.com that looks like this:

#22 ERROR: failed to push registry.digitalocean.com/xxxx/yyyy:c7420fb0197c08e7d29d3ebb15b295af533f1fb0: failed to authorize: failed to fetch anonymous token: unexpected status from GET request to https://api.digitalocean.com/v2/registry/auth?scope=repository%xxxx%yyyy%3Apull%2Cpush&service=registry.digitalocean.com: 401 Unauthorized

I can successfully kamal registry login so my digitalocean registry config is good. But when I do a kamal deploy or kamal build push I get that blasted 401 error above.

Major clue: I am using OSX and Docker Desktop (by chance were you too @codepilotsf ?).

When I change kamal config so it uses remote linux builder, everything works. I change nothing else, just add the remote: line to the builder config and it just works (like it used to on my OSX machine).

I believe there is some funny business around the "credStore" config feature in docker for OSX. Unfortunately, it seems to be very very difficult if not impossible to disable that feature now in the newest versions of docker desktop, so I went way down the rabbit hole of trying to figure how to disable it and ultimately it was never confident that it was actually disabled. I am not a super docker expert nor do I really want to be one.

Anyways.... my workaround is to just use a remote linux builder and everything works perfectly. I am moving on... I really tried to get to the bottom of this. If there are any Docker OSX experts out there, this could use a look (maybe @nickhammond ?). I don't think this is a kamal issue at this point but my confidence in that is low.

I love Kamal, but man this docker issue was a major PITA!

0 replies

codepilotsf · 2024-12-31T16:59:53Z

codepilotsf
Dec 31, 2024
Author

@rschultheis Yes. MacOS with Docker Desktop over here. But here's an important point: When I switched away from DO's registry and tried using DockerHub, then GitLab's built-in registry, then later using Gitea's built-in registry, all three of those non-DO options worked flawlessly even with MacOS Docker Desktop.

0 replies

rschultheis · 2024-12-29T22:27:22Z

rschultheis
Dec 29, 2024

I am having this same apparent issue with DigitalOcean registry.

When I attempt a kamal deploy or more simply a kamal build push it fails with:

#22 ERROR: failed to push registry.digitalocean.com/xxxx/yyyy:c7420fb0197c08e7d29d3ebb15b295af533f1fb0: failed to authorize: failed to fetch anonymous token: unexpected status from GET request to https://api.digitalocean.com/v2/registry/auth?scope=repository%3Axxxx%2Fyyyy%3Apull%2Cpush&service=registry.digitalocean.com: 401 Unauthorized

When I do kamal registry login it works no problem.

This was all working so nicely the other day!

What I know so far: It does not matter if I use kamal 2.3.0 or 2.4.0, I still get the same result.

@codepilotsf did you find some kind of cache file or other thing relating to DO API/doctl that helps here? I would prefer not to switch registries.

0 replies

nickhammond · 2024-12-29T23:25:38Z

nickhammond
Dec 29, 2024

@codepilotsf Kamal doesn't use doctl, it's just using docker push commands with the given server, username, and password. This thread has a good full example of using DO as your registry #804 (comment).

0 replies

rschultheis · 2024-12-30T02:16:37Z

rschultheis
Dec 30, 2024

I've got a few apps on kamal, all setup basically the same with DigitalOcean, just different accounts. I have them setup like the example you link to @nickhammond and its been working fine for months. When I moved to a new laptop and set up all the .kamal/secrets only one is now not working. the .kamal/secrets is identical to the one on my my old laptop where it does work! (shasum .kamal/secrets gives idential sha)

This is the error

#22 pushing layers
#22 pushing layers 1.0s done
#22 ERROR: failed to push registry.digitalocean.com/xxxx/yyyy:c7420fb0197c08e7d29d3ebb15b295af533f1fb0: failed to authorize: failed to fetch anonymous token: unexpected status from GET request to https://api.digitalocean.com/v2/registry/auth?scope=repository%xxxx%yyyy%3Apull%2Cpush&service=registry.digitalocean.com: 401 Unauthorized
------
 > exporting to image:
------
ERROR: failed to solve: failed to push registry.digitalocean.com/xxxx/yyyy:c7420fb0197c08e7d29d3ebb15b295af533f1fb0: failed to authorize: failed to fetch anonymous token: unexpected status from GET request to https://api.digitalocean.com/v2/registry/auth?scope=repository%xxxx%2Fyyyy%3Apull%2Cpush&service=registry.digitalocean.com: 401 Unauthorized

I am able to use the exact same credentials that are not working in kamal to do a manual docker login registry.digitalocean.com and a docker push. I am using the account email as the username.

I think there is something weird about the digitalocean registry here. I remember when I was initially setting this app up I had a similar issue and just fiddled and retried and it eventually worked. I wasn't sure what acually ended up fixing it.

I'll update back here if I get a solution.

0 replies

rschultheis · 2025-01-02T20:24:15Z

rschultheis
Jan 2, 2025

I think the difference with DigitalOcean is that every private repo uses the same url: registry.digitalocean.com where as GitLab gives you a distinct URL for each private repo?

I have multiple kamal projects hosted on DigitalOcean, and they use distinct DigitalOcean registries. Somewhere, something is caching login credentials for registry.digitalocean.com and not accounting for the fact that we have multiple accounts for multiple different repos.

My best guess is the credStore part of Docker Desktop, but I just can't pin down the exact place this cacheing is happening.

0 replies

itsgucci · 2025-06-02T07:52:20Z

itsgucci
Jun 2, 2025

i had this problem where every time i tried kamal deploy it would give me a 401 Unauthorized error.

i solved it by using doctl auth remove --context default
then going in to digital ocean token page
and ensuring there was a token with 2 scopes for registry read and registry update.
then doctl auth init and putting in the token that was issued.
then i tried kamal deploy again and it worked.

when i was looking at the amount of tokens generated by DO, there were at least 100. all 10 pages were full. all with the same permutation of my registry, and all with 2 scopes. i did delete about 20 of them, but i am not sure that this helped or effected the situation at all. it seems like somehow i am generating tons of these tokens. the likely culprit is my lack of understanding of all of this and doctl somehow doing it.

0 replies

drewhoffer · 2025-06-07T21:11:59Z

drewhoffer
Jun 7, 2025

Just ran into this where:

I had an original project (ex. free-tier-nyc3-registry), did a whole bunch of fricking around to get working.
After needing to bump the size for DO (now ex. paid-tier-nyc3-registry), ran into a bunch of issues.

I reset dockers login, doctls login/auth, cleared everything out of ALMOST everywhere. I could NOT for the life of me get to deploy (401 errors only with Kamal). I kept seeing the name of my new beefed up container registry url(paid-tier-nyc3-registry) and the old free one(free-tier-nyc3-registry) and knew this was causing my 401's.

Then, copilot suggested something called a buildx builder (someone smarter than me may be able to explain these further). I listed those with docker buildx ls and saw: kamal-local-docker-container

I removed with docker buildx rm kamal-local-docker-container, reran and evertyhing is now working

(Note at least as of June 7, 2025 DO uses just the pat for both username and password)

0 replies

Unable to deploy to Digital Ocean - 401 Unauthorized #1199

Uh oh!

Replies: 9 comments

Uh oh!

codepilotsf Nov 7, 2024 Author

Uh oh!

Uh oh!

Uh oh!

codepilotsf Dec 31, 2024 Author

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codepilotsf
Nov 7, 2024
Author

codepilotsf
Dec 31, 2024
Author