net/http, x/net/http2: rapid stream resets can cause excessive work (CVE-2023-39325) #63417

neild · 2023-10-06T15:52:23Z

A malicious HTTP/2 client which rapidly creates requests and
immediately resets them can cause excessive server resource consumption.
While the total number of requests is bounded to the
http2.Server.MaxConcurrentStreams setting, resetting an in-progress
request allows the attacker to create a new request while the existing
one is still executing.

HTTP/2 servers now bound the number of simultaneously executing
handler goroutines to the stream concurrency limit. New requests
arriving when at the limit (which can only happen after the client
has reset an existing, in-flight request) will be queued until a
handler exits. If the request queue grows too large, the server
will terminate the connection.

This issue is also fixed in golang.org/x/net/http2 v0.17.0,
for users manually configuring HTTP/2.

The default stream concurrency limit is 250 streams (requests)
per HTTP/2 connection. This value may be adjusted using the
golang.org/x/net/http2 package; see the Server.MaxConcurrentStreams
setting and the ConfigureServer function.

This is CVE-2023-39325 and Go issue https://go.dev/issue/63417.
This is also tracked by CVE-2023-44487.

This is a PRIVATE issue for CVE-2023-39325, tracked in http://b/303836512.

/cc @golang/security and @golang/release

neild · 2023-10-06T20:37:47Z

@gopherbot please open backport issues

gopherbot · 2023-10-06T20:38:49Z

Backport issue(s) opened: #63426 (for 1.20), #63427 (for 1.21).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://go.dev/wiki/MinorReleases.

gopherbot · 2023-10-10T15:42:20Z

Change https://go.dev/cl/534215 mentions this issue: http2: limit maximum handler goroutines to MaxConcurrentStreams

gopherbot · 2023-10-10T15:43:12Z

Change https://go.dev/cl/534235 mentions this issue: [release-branch.go1.21] net/http: regenerate h2_bundle.go

gopherbot · 2023-10-10T15:43:15Z

Change https://go.dev/cl/534255 mentions this issue: [release-branch.go1.20] net/http: regenerate h2_bundle.go

dmitshur · 2023-10-10T15:46:31Z

Reopening to track bundling into the main tree, for Go 1.22.

Pull in a security fix from x/net/http2: http2: limit maximum handler goroutines to MaxConcurrentStreamso For #63417 Fixes #63427 Fixes CVE-2023-39325 Change-Id: I70626734e6d56edf508f27a5b055ddf96d806eeb Reviewed-on: https://team-review.git.corp.google.com/c/golang/go-private/+/2047402 Reviewed-by: Tatiana Bradley <tatianabradley@google.com> Run-TryBot: Damien Neil <dneil@google.com> Reviewed-by: Ian Cottrell <iancottrell@google.com> Reviewed-on: https://go-review.googlesource.com/c/go/+/534235 Auto-Submit: Dmitri Shuralyov <dmitshur@google.com> Reviewed-by: Michael Pratt <mpratt@google.com> Reviewed-by: Damien Neil <dneil@google.com> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>

gopherbot · 2023-10-10T16:18:10Z

Change https://go.dev/cl/534295 mentions this issue: all: pull in x/net v0.17.0 and its dependencies

Pull in a security fix from x/net/http2: http2: limit maximum handler goroutines to MaxConcurrentStreamso For #63417 Fixes #63426 Fixes CVE-2023-39325 Change-Id: I6e32397323cd9b4114c990fcc9d19557a7f5f619 Reviewed-on: https://team-review.git.corp.google.com/c/golang/go-private/+/2047401 Reviewed-by: Tatiana Bradley <tatianabradley@google.com> TryBot-Result: Security TryBots <security-trybots@go-security-trybots.iam.gserviceaccount.com> Run-TryBot: Damien Neil <dneil@google.com> Reviewed-by: Ian Cottrell <iancottrell@google.com> Reviewed-on: https://go-review.googlesource.com/c/go/+/534255 Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> Reviewed-by: Damien Neil <dneil@google.com> TryBot-Bypass: Dmitri Shuralyov <dmitshur@google.com> Reviewed-by: Michael Pratt <mpratt@google.com> Auto-Submit: Dmitri Shuralyov <dmitshur@google.com>

gopherbot · 2023-10-10T17:10:30Z

Change https://go.dev/cl/534218 mentions this issue: [internal-branch.go1.21-vendor] http2: limit maximum handler goroutines to MaxConcurrentStreams

gopherbot · 2023-10-10T17:13:40Z

Change https://go.dev/cl/534236 mentions this issue: [internal-branch.go1.20-vendor] http2: limit maximum handler goroutines to MaxConcurrentStreams

…es to MaxConcurrentStreams When the peer opens a new stream while we have MaxConcurrentStreams handler goroutines running, defer starting a handler until one of the existing handlers exits. For golang/go#63417. For golang/go#63427. For CVE-2023-39325. Change-Id: If0531e177b125700f3e24c5ebd24b1023098fa6d Reviewed-on: https://team-review.git.corp.google.com/c/golang/go-private/+/2047391 Reviewed-by: Tatiana Bradley <tatianabradley@google.com> Run-TryBot: Damien Neil <dneil@google.com> Reviewed-by: Ian Cottrell <iancottrell@google.com> Reviewed-on: https://go-review.googlesource.com/c/net/+/534218 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> Auto-Submit: Dmitri Shuralyov <dmitshur@google.com> Reviewed-by: Michael Pratt <mpratt@google.com>

…es to MaxConcurrentStreams When the peer opens a new stream while we have MaxConcurrentStreams handler goroutines running, defer starting a handler until one of the existing handlers exits. For golang/go#63417. For golang/go#63426. For CVE-2023-39325. Change-Id: If0531e177b125700f3e24c5ebd24b1023098fa6d Reviewed-on: https://team-review.git.corp.google.com/c/golang/go-private/+/2047553 Reviewed-by: Ian Cottrell <iancottrell@google.com> Run-TryBot: Damien Neil <dneil@google.com> Reviewed-by: Tatiana Bradley <tatianabradley@google.com> TryBot-Result: Security TryBots <security-trybots@go-security-trybots.iam.gserviceaccount.com> Reviewed-on: https://go-review.googlesource.com/c/net/+/534236 Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Michael Pratt <mpratt@google.com> Auto-Submit: Dmitri Shuralyov <dmitshur@google.com>

gopherbot · 2023-10-10T17:29:43Z

Change https://go.dev/cl/534297 mentions this issue: [release-branch.go1.20] all: tidy dependency versioning after release

gopherbot · 2023-10-10T17:42:53Z

Change https://go.dev/cl/534415 mentions this issue: [release-branch.go1.21] all: tidy dependency versioning after release

See golang/go#63417 Signed-off-by: Andrey Smirnov <andrey.smirnov@siderolabs.com>

This change mitigates CVE-2023-44487 by disabling HTTP2 by default and forcing HTTP/1.1 until the Go standard library and golang.org/x/net are fully fixed. Right now, it is possible for authenticated and unauthenticated users to hold open HTTP2 connections and consume huge amounts of memory. It is possible to revert back the change by using the `--web.enable-http2` argument. Before this change: ``` curl -kv https://localhost:8443/metrics * Trying 127.0.0.1:8443... * Connected to localhost (127.0.0.1) port 8443 (#0) * ALPN: offers h2,http/1.1 [...] * ALPN: server accepted h2 [...] * using HTTP/2 * h2h3 [:method: GET] * h2h3 [:path: /metrics] * h2h3 [:scheme: https] * h2h3 [:authority: localhost:8443] * h2h3 [user-agent: curl/8.0.1] * h2h3 [accept: */*] * Using Stream ID: 1 (easy handle 0x5594d4614b10) [...] > GET /metrics HTTP/2 [...] ``` After this change: ``` curl -kv https://localhost:8443/metrics * Trying 127.0.0.1:8443... * Connected to localhost (127.0.0.1) port 8443 (#0) * ALPN: offers h2,http/1.1 [...] * ALPN: server accepted http/1.1 [...] * using HTTP/1.1 > GET /metrics HTTP/1.1 > Host: localhost:8443 > User-Agent: curl/8.0.1 > Accept: */* [...] < HTTP/1.1 200 OK [...] ``` See also: * kubernetes/kubernetes#121120 * kubernetes/kubernetes#121197 * golang/go#63417 (comment) Signed-off-by: Simon Pasquier <spasquie@redhat.com> (cherry picked from commit a62e814)

ATP3530 · 2023-11-06T10:44:58Z

Does this CVE-2023-39325 impacts Go version 1.18? If yes is there a plan to backport required fix into version 1.18? Or we have to move to the latest version of go (1.20 or 1.21) to have this CVE fixed

neild · 2023-11-06T17:41:11Z

1.18 is affected. There are no plans to backport to 1.18, which is out of the support window; see: https://go.dev/doc/devel/release#policy

As an alternative to upgrading to a recent Go version, you can use a recent version of the golang.org/x/net/http2 package and http2.ConfigureServer to replace the standard library's bundled HTTP/2 implementation, although I'd really recommend upgrading to a Go version that's still getting security fixes.

This change mitigates CVE-2023-44487 by disabling HTTP2 by default and forcing HTTP/1.1 until the Go standard library and golang.org/x/net are fully fixed. Right now, it is possible for authenticated and unauthenticated users to hold open HTTP2 connections and consume huge amounts of memory. It is possible to revert back the change by using the `--web.enable-http2` argument. Before this change: ``` curl -kv https://localhost:8443/metrics * Trying 127.0.0.1:8443... * Connected to localhost (127.0.0.1) port 8443 (#0) * ALPN: offers h2,http/1.1 [...] * ALPN: server accepted h2 [...] * using HTTP/2 * h2h3 [:method: GET] * h2h3 [:path: /metrics] * h2h3 [:scheme: https] * h2h3 [:authority: localhost:8443] * h2h3 [user-agent: curl/8.0.1] * h2h3 [accept: */*] * Using Stream ID: 1 (easy handle 0x5594d4614b10) [...] > GET /metrics HTTP/2 [...] ``` After this change: ``` curl -kv https://localhost:8443/metrics * Trying 127.0.0.1:8443... * Connected to localhost (127.0.0.1) port 8443 (#0) * ALPN: offers h2,http/1.1 [...] * ALPN: server accepted http/1.1 [...] * using HTTP/1.1 > GET /metrics HTTP/1.1 > Host: localhost:8443 > User-Agent: curl/8.0.1 > Accept: */* [...] < HTTP/1.1 200 OK [...] ``` See also: * kubernetes/kubernetes#121120 * kubernetes/kubernetes#121197 * golang/go#63417 (comment) Signed-off-by: Simon Pasquier <spasquie@redhat.com>

# AWS EKS Backported To: go-1.19.13-eks Backported On: Thu, 12 Oct 2023 Backported By: rcrozean@amazon.com Backported From: release-branch.go1.20 Source Commit: golang@e175f27 # Original Information Pull in a security fix from x/net/http2: http2: limit maximum handler goroutines to MaxConcurrentStreamso For golang#63417 Fixes golang#63426 Fixes CVE-2023-39325 Change-Id: I6e32397323cd9b4114c990fcc9d19557a7f5f619 Reviewed-on: https://team-review.git.corp.google.com/c/golang/go-private/+/2047401 Reviewed-by: Tatiana Bradley <tatianabradley@google.com> TryBot-Result: Security TryBots <security-trybots@go-security-trybots.iam.gserviceaccount.com> Run-TryBot: Damien Neil <dneil@google.com> Reviewed-by: Ian Cottrell <iancottrell@google.com> Reviewed-on: https://go-review.googlesource.com/c/go/+/534255 Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> Reviewed-by: Damien Neil <dneil@google.com> TryBot-Bypass: Dmitri Shuralyov <dmitshur@google.com> Reviewed-by: Michael Pratt <mpratt@google.com> Auto-Submit: Dmitri Shuralyov <dmitshur@google.com>

aojea · 2023-12-26T11:15:13Z

FYI, this fix is not sufficient for Kube, and possibly other projects. We are working on this in kubernetes/kubernetes#121120. pprof svg from a single client with a single connection attempting to DOS a Kube API server that has this fix and max streams set to 100 (memory usage grew to 5 GB in a few mins before it stabilized - with multiple connections the API server would have easily OOM'd):

@enj from the pprof attached the bulk of the memory consumed comes from the 4kb buffer allocated to the responseWriter that is created after the Headers frame is received. The buffer is also obtained from the sync.Pool that will make the memory usage more efficient.

https://github.com/golang/net/blob/f12db26b1c9293fa3eb95c936e548d2c1fba4ba9/http2/server.go#L69-L75

In this case, since that the attack is based only on headers frames, postponing the allocation of the responseWriter will reduce a lot the impact on memory usage, however, I've been checking around the code and I don't know how feasible is to do that, based on the rfc7540

Sending or receiving a HEADERS frame causes the stream to become "open"

A stream in the "open" state may be used by both peers to send frames of any type. In this state, sending peers observe
advertised stream-level flow-control limits (Section 5.2)

Pull in a security fix from x/net/http2: http2: limit maximum handler goroutines to MaxConcurrentStreamso For golang#63417 Fixes golang#63427 Fixes CVE-2023-39325 Change-Id: I70626734e6d56edf508f27a5b055ddf96d806eeb Reviewed-on: https://team-review.git.corp.google.com/c/golang/go-private/+/2047402 Reviewed-by: Tatiana Bradley <tatianabradley@google.com> Run-TryBot: Damien Neil <dneil@google.com> Reviewed-by: Ian Cottrell <iancottrell@google.com> Reviewed-on: https://go-review.googlesource.com/c/go/+/534235 Auto-Submit: Dmitri Shuralyov <dmitshur@google.com> Reviewed-by: Michael Pratt <mpratt@google.com> Reviewed-by: Damien Neil <dneil@google.com> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>

Done with: go get golang.org/x/net@internal-branch.go1.21-vendor go mod tidy go mod vendor go generate net/http # zero diff since CL 534235 already did this For golang#63417. For golang#63427. For CVE-2023-39325. Change-Id: Ib258e0d8165760a1082e02c2f4c5ce7d2a3c3c90 Reviewed-on: https://go-review.googlesource.com/c/go/+/534415 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> Auto-Submit: Dmitri Shuralyov <dmitshur@google.com> Reviewed-by: Michael Pratt <mpratt@google.com>

Pull in a security fix from x/net/http2: http2: limit maximum handler goroutines to MaxConcurrentStreamso For golang#63417 Fixes golang#63427 Fixes CVE-2023-39325 Change-Id: I70626734e6d56edf508f27a5b055ddf96d806eeb Reviewed-on: https://team-review.git.corp.google.com/c/golang/go-private/+/2047402 Reviewed-by: Tatiana Bradley <tatianabradley@google.com> Run-TryBot: Damien Neil <dneil@google.com> Reviewed-by: Ian Cottrell <iancottrell@google.com> Reviewed-on: https://go-review.googlesource.com/c/go/+/534235 Auto-Submit: Dmitri Shuralyov <dmitshur@google.com> Reviewed-by: Michael Pratt <mpratt@google.com> Reviewed-by: Damien Neil <dneil@google.com> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>

Done with: go get golang.org/x/net@internal-branch.go1.21-vendor go mod tidy go mod vendor go generate net/http # zero diff since CL 534235 already did this For golang#63417. For golang#63427. For CVE-2023-39325. Change-Id: Ib258e0d8165760a1082e02c2f4c5ce7d2a3c3c90 Reviewed-on: https://go-review.googlesource.com/c/go/+/534415 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> Auto-Submit: Dmitri Shuralyov <dmitshur@google.com> Reviewed-by: Michael Pratt <mpratt@google.com>

tuk2 · 2024-08-20T19:29:48Z

The issue with HTTP/2 rapid reset is still active. When an HTTP/2 server is attacked using rapid resets, it results in memory leaks (up to peak memory usage). This problem is especially noticeable when using net/http/httputil (reverse proxy). IMPORTANT: The memory leak occurs ONLY during rapid reset attacks.

tuk2 · 2024-08-20T20:48:59Z

@bradfitz

- Ensure HTTP/2 is disabled when the enable-http2 flag is set to false (default). - Disabling HTTP/2 mitigates vulnerabilities associated with: - HTTP/2 Stream Cancellation (GHSA-qppj-fm5r-hxr3) - HTTP/2 Rapid Reset (GHSA-4374-p667-p6c8) - While CVE fixes exist, they remain insufficient; disabling HTTP/2 helps reduce risks. For details, see: - GHSA-qppj-fm5r-hxr3 - GHSA-4374-p667-p6c8 - golang/go#63417

neild self-assigned this Oct 6, 2023

seankhliao added Security NeedsFix The path to resolution is known, but the work has not been done. release-blocker labels Oct 6, 2023

seankhliao added this to the Go1.22 milestone Oct 6, 2023

gopherbot mentioned this issue Oct 6, 2023

security: fix CVE-2023-39325 [1.20 backport] #63426

Closed

gopherbot mentioned this issue Oct 6, 2023

security: fix CVE-2023-39325 [1.21 backport] #63427

Closed

gopherbot closed this as completed in golang/net@b225e7c Oct 10, 2023

dmitshur reopened this Oct 10, 2023

gopherbot closed this as completed in f34964a Oct 10, 2023

dmitshur changed the title ~~security: fix CVE-2023-39325~~ net/http, x/net/http2: rapid stream resets can cause excessive work (CVE-2023-39325) Oct 10, 2023

This was referenced Oct 10, 2023

go@1.20 1.20.10 Homebrew/homebrew-core#150459

Merged

go 1.21.3 Homebrew/homebrew-core#150460

Merged

piroux mentioned this issue Oct 10, 2023

HTTP/2 Rapid Reset : CVE-2023-44487 caddyserver/caddy#5877

Closed

smira added a commit to smira/tools that referenced this issue Oct 10, 2023

feat: update Go to 1.21.3

35948af

See golang/go#63417 Signed-off-by: Andrey Smirnov <andrey.smirnov@siderolabs.com>

slashpai mentioned this issue Nov 3, 2023

OCPBUGS-21637: fix: disable HTTP2 connections by default openshift/prometheus-operator#252

Merged

5 tasks

gsoltis mentioned this issue Nov 3, 2023

fix(Turborepo): Bump x/net, go mod tidy vercel/turborepo#6355

Merged

yeya24 mentioned this issue Nov 5, 2023

Update go version to 1.21.3 cortexproject/cortex#5630

Merged

3 tasks

lucianomda mentioned this issue Nov 6, 2023

Bump golang.org/x/net from 0.8.0 to 0.17.0 go-playground/validator#1179

Closed

MichaelSnowden mentioned this issue Nov 18, 2023

Patch x/net CVE fix temporalio/ringpop-go#13

Merged

racheljpg mentioned this issue Dec 15, 2023

Bump golang.org/x/net to v0.19.0 kubernetes/cloud-provider-alibaba-cloud#389

Closed

iwata mentioned this issue Feb 8, 2024

Update vulnerable package versions cloudspannerecosystem/wrench#103

Merged

jmlMetaswitch mentioned this issue Mar 8, 2024

Bump golang.org/x/net from 0.7.0 to 0.17.0 antchfx/xmlquery#110

Closed

This was referenced Aug 27, 2024

[Apiiro] 2 risks shukibakbuki/dingdong#133

Open

[Apiiro] SCA OSS Vulnerabilities - High CVSS score · High Risk shukibakbuki/dingdong#135

Open

[Apiiro] SCA OSS Vulnerabilities - High CVSS score · High Risk shukibakbuki/dingdong#137

Open

camilamacedo86 mentioned this issue Dec 13, 2024

🐛 Disable HTTP/2 by Default for Webhooks to Mitigate CVE Risks operator-framework/catalogd#484

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

net/http, x/net/http2: rapid stream resets can cause excessive work (CVE-2023-39325) #63417

net/http, x/net/http2: rapid stream resets can cause excessive work (CVE-2023-39325) #63417

net/http, x/net/http2: rapid stream resets can cause excessive work (CVE-2023-39325) #63417

net/http, x/net/http2: rapid stream resets can cause excessive work (CVE-2023-39325) #63417

Comments