8000 `decode_malformed_zlib_message` test fails when built against `zlib-ng` · Issue #23095 · vectordotdev/vector · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

decode_malformed_zlib_message test fails when built against zlib-ng #23095

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
strophy opened this issue May 23, 2025 · 4 comments
Closed

decode_malformed_zlib_message test fails when built against zlib-ng #23095

strophy opened this issue May 23, 2025 · 4 comments
Labels
type: bug A code related bug.

Comments

@strophy
Copy link
strophy commented May 23, 2025

A note for the community

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

I'm trying to package vector 0.47.0 for Alpine Linux and build against system zlib-ng libraries. My patch looks like this:

diff --git a/Cargo.toml b/Cargo.toml
index 3453d2307..ad490ffe2 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -139,7 +139,7 @@ cfg-if = { version = "1.0.0", default-features = false }
 chrono = { version = "0.4.40", default-features = false, features = ["clock", "serde"] }
 chrono-tz = { version = "0.10.3", default-features = false, features = ["serde"] }
 clap = { version = "4.5.34", default-features = false, features = ["derive", "error-context", "env", "help", "std", "string", "usage", "wrap_help"] }
-flate2 = { version = "1.0.35", default-features = false, features = ["default"] }
+flate2 = { version = "1.0.35", default-features = false, features = ["zlib-ng"] }
 futures = { version = "0.3.31", default-features = false, features = ["compat", "io-compat", "std"], package = "futures" }
 glob = { version = "0.3.2", default-features = false }
 hickory-proto = { version = "0.25.1", default-features = false, features = ["dnssec-ring"] }
@@ -392,7 +392,7 @@ thread_local = { version = "1.1.8", default-features = false, optional = true }
 typetag = { version = "0.2.20", default-features = false }
 url = { version = "2.5.4", default-features = false, features = ["serde"] }
 warp = { version = "0.3.7", default-features = false }
-zstd = { version = "0.13.0", default-features = false }
+zstd = { version = "0.13.0", default-features = false, features = ["pkg-config"] }
 arr_macro = { version = "0.2.1" }
 
 # depending on fork for bumped nix dependency
@@ -468,7 +468,7 @@ default-cmake = ["api", "api-client", "enrichment-tables", "rdkafka?/cmake_build
 # Default features for *-pc-windows-msvc
 # TODO: Enable SASL https://github.com/vectordotdev/vector/pull/3081#issuecomment-659298042
 default-msvc = ["api", "api-client", "enrichment-tables", "rdkafka?/cmake_build", "sinks", "sources", "transforms", "secrets"]
-default-musl = ["api", "api-client", "enrichment-tables", "rdkafka?/cmake_build", "sinks", "sources", "sources-dnstap", "transforms", "unix", "rdkafka?/gssapi-vendored", "secrets"]
+default-musl = ["api", "api-client", "enrichment-tables", "rdkafka?/dynamic-linking", "sinks", "sources", "sources-dnstap", "transforms", "unix", "rdkafka?/gssapi", "secrets"]
 default-no-api-client = ["api", "enrichment-tables", "sinks", "sources", "sources-dnstap", "transforms", "unix", "rdkafka?/gssapi-vendored", "secrets"]
 default-no-vrl-cli = ["api", "sinks", "sources", "sources-dnstap", "transforms", "unix", "rdkafka?/gssapi-vendored", "secrets"]
 tokio-console = ["dep:console-subscriber", "tokio/tracing"]

We build like this:

prepare() {
	default_prepare

	# Rust target triple.
	local target=$(rustc -vV | sed -n 's/host: //p')

	# Build against system-provided zlib-ng
	mkdir -p .cargo
	cat >> .cargo/config.toml <<-EOF
		[target.$target]
		z-ng = { rustc-link-lib = ["z-ng"], rustc-cfg = ["zng"] }
	EOF

	# "cargo auditable build" still tries to pull dependencies for other platforms
	cargo fetch --locked
}

build() {
	cargo auditable build \
		--release \
		--frozen \
		--no-default-features \
		--features default-musl,component-validation-runner

	# cargo-nextest drops the release build, so back it up here
	cp -r target/release releasebuild
}

This results in the following error in the test phase:

  TRY 1 FAIL [   0.018s] codecs decoding::framing::chunked_gelf::tests::decode_malformed_zlib_message
──── TRY 1 STDOUT:       codecs decoding::framing::chunked_gelf::tests::decode_malformed_zlib_message
running 1 test
test decoding::framing::chunked_gelf::tests::decode_malformed_zlib_message ... FAILED
failures:
failures:
    decoding::framing::chunked_gelf::tests::decode_malformed_zlib_message
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 144 filtered out; finished in 0.00s
──── TRY 1 STDERR:       codecs decoding::framing::chunked_gelf::tests::decode_malformed_zlib_message
thread 'decoding::framing::chunked_gelf::tests::decode_malformed_zlib_message' panicked at lib/codecs/src/decoding/framing/chunked_gelf.rs:1073:14:
decoding should fail: Some(b"\x113")
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Why does this test fail and how can I fix it? Should test be updated to consider different possible errors from different zlib implementations?

Configuration

Version

0.47.0

Debug Output

Example Data

No response

Additional Context

No response

References

No response
@strophy strophy added the type: bug A code related bug. label May 23, 2025
@jorgehermo9
Copy link
Contributor
jorgehermo9 commented May 25, 2025 •
edited
Loading

There was a discussion related to this: #22533 (comment). The test you are pointing was changed in #22533 and the change included in Vector 0.48.0, so I suggest to update your Vector version to 0.48.0 first.

There was a bug in miniz_oxide that for some input (the one that we were using on the test you pointed) miniz_oxide returned an error but other backends succeed (for example, zlib-ng, that's why you are seeing that error). You can find the fix for that here, were miniz_oxide would succeed for the input we were using and not fail. Therefore, the decode_malformed_zlib_message had to be changed to another payload that failed, work which was done in #22533.

I think this issue should be closed

@strophy
Copy link
Author
strophy commented May 26, 2025

Thanks @jorgehermo9 #22533 looks like it should address the issue I'm seeing as well. 0.48.0 isn't released yet, so I'll wait for that release (or make a patch) and report back with my results here.

@jorgehermo9
Copy link
Contributor

My mistake, sorry. The flate2 update I'm referring to was released on 0.47.0. I saw on your diff that the flate2 version that you are using is 1.0.35

-flate2 = { version = "1.0.35", default-features = false, features = ["default"] }

And therefore I thought you were not using the latest Vector release. But, for Vector 0.47.0 (the version your are using), you should be using flate2 version 1.1.1. Your Cargo.toml is not updated with the Vector's v0.47.0 branch's tip

Please, see this https://github.com/vectordotdev/vector/blob/v0.47/Cargo.toml#L143 and try to update your diff to something like

+flate2 = { version = "1.1.1", default-features = false, features = ["zlib-ng"] }

@strophy
Copy link
Author
strophy commented May 26, 2025

Thanks for the clarification @jorgehermo9 I agree that I did something wrong with my patch against 0.47.0, likely because I started this work against 0.46.1. The build succeeds now, so the fix you explained earlier also works with zlib-ng. Thanks for the help!

@strophy strophy closed this as completed May 26, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type: bug A code related bug.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jorgehermo9 @strophy
0