8000 Produce to topic: avro with nullable can't be produced · Issue #407 · tchiotludo/akhq · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Produce to topic: avro with nullable can't be produced #407

New issue

Have a question abo 8000 ut this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
zaghdous opened this issue Sep 22, 2020 · 4 comments · Fixed by #741
Closed

Produce to topic: avro with nullable can't be produced #407

zaghdous opened this issue Sep 22, 2020 · 4 comments · Fixed by #741
Labels
backend Need a backend update bug Something isn't working topic data Kafka Topic data

Comments

@zaghdous
Copy link

Dear Ludovic, I reopen the issue that you closed because I really like your product and like to use it in my organization.
My goal is not to flood you with issues ! I think that the "produce to topic" function is cool but does not work since any given bad encoded JSON cannot be handled. Try to take the json you display and use it to produce a new message, you will get a "org.apache.avro.AvroTypeException" !

So here is the avro spec (link that I already added in the previous ticket) :

http://avro.apache.org/docs/current/spec.html#json_encoding

And an example of serde using avro-tools (we always learn even if you are a specialist):

test.asvc :
{
"type" : "record",
"name" : "recordName",
"namespace" : "name.space",
"fields" : [ {
"name" : "id",
"type" : ["null", "int"]
} ]
}

bad.json:

{
"id" : 141
}

good.json:
{
"id" : {"int" : 141}
}

$ java -jar avro-tools-1.10.0.jar fromjson --schema-file test.asvc bad.json > test.avro
Exception in thread "main" org.apache.avro.AvroTypeException: Expected start-union. Got VALUE_NUMBER_INT

$ java -jar avro-tools-1.10.0.jar fromjson --schema-file test.asvc good.json > test.avro
OK

$ java -jar avro-tools-1.10.0.jar tojson test.avro > test.json

test.json :

{
"id" : {"int" : 141}
}
@zaghdous
Copy link
Author

I tested your app, and works exactly like avro-tools :

  • unable to serialize bad.json to avro => AvroTypeException
  • good.json is accepted and rendered in the same form !

So the pb is present when avro is not generated from json !

@tchiotludo
Copy link
Owner

Sorry to close the previous one !
Doing some test with your example, and for now, you need to produce good.json from akhq !
I really think it was not the case and bad.json should be accepted IMO.

Your goal is to produce the good.json or the bad.json from akhq ?

@zaghdous
Copy link
Author

🙂
Since bad.json is not accepted, the avro should decode to good.json which is accepted.
My goal is to copy a json, change data and resend it to the topic.
Thanks a lot !

@tchiotludo tchiotludo changed the title Bad json encoding of avro data 2 Produce to topic: avro with nullable can't be produced Sep 23, 2020
@tchiotludo tchiotludo added backend Need a backend update bug Something isn't working topic data Kafka Topic data labels Sep 23, 2020
tchiotludo added a commit that referenced this issue Jun 21, 2021
@tchiotludo
feat(topic-data): allow to consume & produce any kind of avro types
b11b6ca 
close #409
close #739
close #407
close #387

relate #352
@tchiotludo tchiotludo mentioned this issue Jun 21, 2021
Merged
tchiotludo added a commit that referenced this issue Jun 21, 2021
@tchiotludo
feat(topic-data): allow to consume & produce any kind of avro types
e319b33 
close #409
close #739
close #407
close #387

relate #352
tchiotludo added a commit that referenced this issue Jun 21, 2021
@tchiotludo
feat(topic-data): allow to consume & produce any kind of avro types
ff2c8fd 
close #409
close #739
close #407
close #387

relate #352
tchiotludo added a commit that referenced this issue Jun 21, 2021
@tchiotludo
feat(topic-data): allow to consume & produce any kind of avro types
4891953 
close #409
close #739
close #407
close #387

relate #352
tchiotludo added a commit that referenced this issue Jun 22, 2021
@tchiotludo
feat(topic-data): allow to consume & produce any kind of avro types
ec8b992 
close #409
close #739
close #407
close #387

relate #352
tchiotludo added a commit that referenced this issue Jun 22, 2021
@tchiotludo
feat(topic-data): allow to consume & produce any kind of avro types
13b9a73 
close #409
close #739
close #407
close #387

relate #352
tchiotludo added a commit that referenced this issue Jun 22, 2021
@tchiotludo
feat(topic-data): allow to consume & produce any kind of avro types
a2fdf91 
close #409
close #739
close #407
close #387
close #737

relate #352
tchiotludo added a commit that referenced this issue Jun 22, 2021
@tchiotludo
feat(topic-data): allow to consume & produce any kind of avro types
c34526d 
close #409
close #739
close #407
close #387
close #737

relate #352
@tchiotludo
Copy link
Owner

must be done on dev, can you try ?

@amccague amccague mentioned this issue Jul 5, 2022
Open
@tchiotludo tchiotludo moved this to Done in Backlog Jun 29, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
backend Need a backend update bug Something isn't working topic data Kafka Topic data
Projects
Backlog
Status: Done
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feat(topic-data): allow to consume & produce any kind of avro types tchiotludo/akhq
2 participants
@tchiotludo @zaghdous
0