Warning in Kafka logs Invalid receive (size 369295618 larger than 104857600) #8956

bharat15-ml · 2023-08-07T09:10:17Z

bharat15-ml
Aug 7, 2023

HI,

Need Expedite help on identifying the possible cause for warning in Kafka logs.

Setup - Kafka operator 0.28.0 and deployed three Kafka pods with version 3.1.0 enabled only plaintext communication for Kafka listeners on port 9092 also having JMX Prometheus Exporter enabled to expose metrics.

All went well but now I continuously see the below warnings in Kafka logs -

[2023-08-03 05:17:48,262] WARN [SocketServer listenerType=ZK_BROKER, nodeId=0] Unexpected error from /100.98.238.7; closing 
connection (org.apache.kafka.common.network.Selector) org.apache.kafka.common.network.InvalidReceiveException: Invalid receive (size 369295618 larger than 104857600)
    at org.apche.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:105)
    at org.apche.kafka.common.KafkaChannel.receive(KafkaChannel.java:452)
    at org.apche.kafka.common.KafkaChannel.read(KafkaChannel.java:402)
    at org.apche.kafka.common.network.Selector.attemptRead(Selector.java:674)
    at org.apche.kafka.common.network.Selector.pollSelectionKeys(Selector.java:576)
    at org.apche.kafka.common.network.Selector.poll(Selector.java:481)
    at kafka.network.Processor.poll(SocketServer.scala:989)
    at kafka.network.Processor.run(SocketServer.scala:892)
    at java.base/java.lang.Thread.run(Thread.java:829)

For possible reasons have checked below threads -

https://stackoverflow.com/questions/57141350/apache-kafka-invalid-receive-size - which basically means changes in kafka server.properties like - socket.request.max.bytes=300000000
Also was checking this thread - WARN [SocketServer brokerId=1] Unexpected error from /IPaddress; closing connection (org.apache.kafka.common.network.Selector) #3462

I do understand there are multiple clients connecting to the Kafka cluster, but unable to identify source Ip 100.98.238.7 in our k8s cluster from where this request is coming. Pls, provide insight/clues on this issue to get rid of this warning.

scholzj · 2023-08-07T12:23:59Z

scholzj
Aug 7, 2023
Maintainer

It is a bit hard to comment on it based on a single log line. One thing which can cause this is when a client without TLS connects to TLS based listener or the other way around. It could be a message size as well -> Kafka is designed for small messages.

5 replies

bharat15-ml Aug 9, 2023
Author

@scholzj I have updated the logs, pls check once. If you could suggest something based on the available information.
basically, need help with whether it will require changes to the Kafka configuration or need to check client configurations (which is hard to track because of multiple client applications consuming the Kafka cluster).
Additionally running Kafka connect pods as well with multiple source/sink connectors, but no issue was identified on connect pod logs, and all connectors also running fine.

winfriedgerlach Jun 24, 2025

Thanks @scholzj, in my case it was a Producer configured with TLS trying to connect to a broker without TLS - quite misleading error message!

suchak-dwivedi Jun 26, 2025

@winfriedgerlach I am observing the same symptoms but I am not sure if its because producer (in my case kafka connect) is configured with TLS. I know that kafka broker has TLS disabled and connect has it enabled. Can you please share if you had similar setup when you say "Producer configured with TLS trying to connect to a broker without TLS"?

winfriedgerlach Jun 26, 2025

@suchak-dwivedi Yes, exactly as you describe. I had a producer (from kafka-clients Java library) with security.protocol=SSL, and all of ssl.truststore.location, ssl.truststore.password etc. set and tried to connect to both an apache/kafka:4.0.0 and later a confluentinc/cp-kafka:8.0.0 container with default settings (=only PLAINTEXT endpoints). And instead of an SSL/TLS warning, Kafka presented the Invalid receive (size 369......) error (which would mean that somebody tried to send a 300 MB message...).

ppatierno Jun 26, 2025
Maintainer

I don't think it's a size related error, despite what the error log says.
It's about using a TLS client with a NO TLS broker. The TLS client is starting the TLS handshake but the broker is interpreting the coming bytes in a totally different way (expecting a packet size I guess). So it's wrongly decoding what is a packet for TLS handshake in reality and it drives to this error.

Strimzi

Warning in Kafka logs Invalid receive (size 369295618 larger than 104857600) #8956

Uh oh!

Uh oh!

bharat15-ml Aug 7, 2023

Replies: 1 comment · 5 replies

Uh oh!

Uh oh!

scholzj Aug 7, 2023 Maintainer

Uh oh!

Uh oh!

bharat15-ml Aug 9, 2023 Author

Uh oh!

winfriedgerlach Jun 24, 2025

Uh oh!

suchak-dwivedi Jun 26, 2025

Uh oh!

Uh oh!

winfriedgerlach Jun 26, 2025

Uh oh!

ppatierno Jun 26, 2025 Maintainer

bharat15-ml
Aug 7, 2023

Replies: 1 comment 5 replies

scholzj
Aug 7, 2023
Maintainer

bharat15-ml Aug 9, 2023
Author

ppatierno Jun 26, 2025
Maintainer