8000 wit/bindgen: support JSON encoding of WIT types · Issue #239 · bytecodealliance/go-modules · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

wit/bindgen: support JSON encoding of WIT types #239

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
2 of 11 tasks
lxfontes opened this issue Nov 7, 2024 · 14 comments
Open
2 of 11 tasks

wit/bindgen: support JSON encoding of WIT types #239

lxfontes opened this issue Nov 7, 2024 · 14 comments
Assignees
@lxfontes
Labels
enhancement New feature or request

Comments

@lxfontes
Copy link
Member
lxfontes commented Nov 7, 2024 •
edited by ydnar
Loading

This started in #225

Goal: JSON Serialize / Deserialize WIT Records based on WIT field names.

Progress

Approach

Centralize JSON serialization in the cm package, keeping codegen changes to a minimum.

Given a complex record type:

  record response {
      headers: list<tuple<string, list<string>>>,
      http-code: u16,
      body: response-body
  }
  variant response-body {
    ok(list<list-element>),
    err(function-error),
    platform-err(platform-error)
  }
  record list-element {
      optional-int: option<u64>,
      optional-bool: option<bool>,
  }
  record function-error {
      error: string
  }
  record platform-error {
    code: string,
    message: string
  }

and filling it up:

	hdrVals := cm.ToList([]string{"value1", "value2"})
	hdrTuple := cm.Tuple[string, cm.List[string]]{
		F0: "header-name",
		F1: hdrVals,
	}
	headers := cm.ToList([]cm.Tuple[string, cm.List[string]]{hdrTuple})
	v := somefunctioninterface.Response{
		Headers:  headers,
		HTTPCode: 200,
		Body:     somefunctioninterface.ResponseBodyErr(somefunctioninterface.FunctionError{Error: "failed"}),
	}

We should serialize it to JSON as:

{"headers":[["header-name",["value1","value2"]]],"http-code":200,"body":{"err":{"error":"failed"}}}

For comparison, this is what is produced today:

{"Headers":{},"Body":{"RequiredParam":"required","OptionalParam":{}}}

Type encoding

Whenever possible, reuse standard mappings. string -> string, u32 -> uint32, etc.

Tuple handling

Tuples are encoded as json arrays with explicit nulls.

Tuple[string,int] -> [ "some string", 42 ]
Tuple3[string, *string, int] -> [ "some string", null, 42 ]

Variant handling

Variants are encoded as json dictionaries, so they can carry the variant data.

 record somerecord {
   myvariant: somevar,
 }
  variant somevar {
    ok,
    err(string),
  }

// JSON
{ "myvariant": { "ok": true }}
{ "myvariant": { "error": "error value" }}

Option handling

Options rely on explicit null for the "None" case.

{ "myoptional": null } // cm.None
{ "myoptional": "this" } // cm.Option[string]

Questions

Zero Value Variants

Today they end up with tag = 0, this impacts de-serialization of variants. We need the ability to distinct between:

  • var v Variant
  • v := NewVariantWithZeroTagValue()
  • v := Some(NewVariantWithZeroTagValue)

atm only the Some() path de-serializes correctly ( pointers ).
@lxfontes lxfontes mentioned this issue Nov 7, 2024
Closed
@lxfontes lxfontes self-assigned this Nov 7, 2024
@ydnar ydnar changed the title marshal & unmarshal for record types wit/bindgen: JSON serialization support for record types Nov 7, 2024
@ydnar
Copy link
Collaborator
ydnar commented Nov 10, 2024
< 8000 /div>

Can we move the design and task list for this to an issue?

Then we can have discrete PRs that implement JSON support for different types?

This was referenced Dec 9, 2024
Merged
Closed
@ydnar ydnar changed the title wit/bindgen: JSON serialization support for record types wit/bindgen: support JSON encoding of WIT types Dec 28, 2024
@ydnar ydnar mentioned this issue Jan 4, 2025
Merged
@ydnar
Copy link
Collaborator
ydnar commented Jan 5, 2025 •
edited
Loading

Proposal for option, result, and variant JSON:

Option

The current proposal specifies null for option<T> when it is the none case. If option values are nested, this creates an ambiguity as to which option is some versus none, e.g. option<option<T>>.

An alternate proposal would be to specify {"none": ...} instead of null to disambiguate nested options.

For an option<string> with the some case:

{"some":"foo"}

For an option<string> with the none case:

{"none":null}

For an option<option<string>> where the second option is the none case:

{"some":{"none":null}}

Result

result<T, E> values would serialize similarly to option as a simplified variant.

For result<string> in the OK case (no error type):

{"ok": "some OK value"}

For result<string> in the error case:

{"error": null}

@ydnar
Copy link
Collaborator
ydnar commented Jan 5, 2025

For resource, stream, future, and error-context types, I think JSON serialization should be an error, similar to how chan and func types cannot be marshaled into JSON in Go today.

@ydnar ydnar added the enhancement New feature or request label Jan 27, 2025
@brooksmtownsend brooksmtownsend mentioned this issue Mar 7, 2025
Open
@ydnar
Copy link
Collaborator
ydnar commented Mar 8, 2025

Inspired by #301, I think option values should be represented as:

  • None: {}
  • Some: {"some": 123}

@brooksmtownsend
Copy link
brooksmtownsend commented Mar 10, 2025 •
edited
Loading

@ydnar would love to discuss that a bit, I think that representing option values in that way gives us the best ability to understand when it's a Some(t) value, my concern would be that optional fields from other tools (e.g. OpenAPI / JSON schema) aren't encoded in this way.

Generally what I would expect an optional field to be in JSON is either omitting the field entirely or having the field be set to null, and when present it would just be the value itself. Does this make sense / am I missing something that would bite us during unmarshaling?

What I have in my PR corresponds to the following behavior:

  • Marshal Some(T) -> {"field": T}
  • Marshal None(T) -> {"field": null}
  • Unmarshal {"field": null} -> None
  • Unmarshal {} -> None
  • Unmarshal {"field": T} -> Some(T)

@ydnar
Copy link
Collaborator
ydnar commented Mar 10, 2025 •
edited
Loading

Is it a goal to support symmetrical round-trip of WIT values to JSON? If so, then this would require unambiguous encoding of option<T> values.

Given a WIT type option<option<T>, the following have different semantic, and in-memory value:

  1. None(T)
  2. Some(None(T)

How should these be serialized?

  1. null
  2. null

Or:

  1. {}
  2. {"some":{}}

Using null to represent the none case loses information, and cannot be symmetrically encoded and decoded.

@brooksmtownsend
Copy link
brooksmtownsend commented Mar 13, 2025 •
edited
Loading

Is it a goal to support symmetrical round-trip of WIT values to JSON?

Well, honestly I'm not sure. I know that practically from a WIT perspective we would want to symmetrically marshal/unmarshal, but from a JSON perspective there's not a way to represent nested optionals so we either have to have a WIT-specific structure or lose information.

My goal personally in the contribution / thinking is to marshal the option type in a way that would be compatible with existing tools as much as possible. Ideally, a None variant would just be omitted from serialization entirely if omitempty was supplied. And the presence of a field would be unmarshaled into the Some(T) variant. In normal Go I would just use a pointer to represent a nullable value, but the value that comes from wit-bindgen-go from an optional WIT type is the Option.

Do you think that this is a reasonable tradeoff to accept into the marshaling implementation or do we need to have symmetric marshaling/unmarshaling? It may be valuable to consider your proposal for the result type as well, since of course JSON doesn't have that concept.

@ydnar
Copy link
Collaborator
ydnar commented Mar 15, 2025

but from a JSON perspective there's not a way to represent nested optionals

Can you explain what you mean? A mechanism for representing nested options is proposed earlier in this thread: #239 (comment)

@brooksmtownsend
Copy link

@ydnar I don't mean to say it's not possible, just that I'm not aware of existing libraries that would parse the {"some": <value>} without extra information. I was hoping that marshaling a structure with an Option[T] would result in a JSON object that could be parsed just like a nullable field

@ydnar
Copy link
Collaborator
ydnar commented Mar 17, 2025

The client will need to model result types as well. This is just a form of that.

Noted Serde has long-standing issues with double options:

serde-rs/serde#1042

https://docs.rs/serde_with/latest/x86_64-apple-darwin/serde_with/rust/double_option/index.html

@brooksmtownsend
Copy link

Seems like it would be great to be able to decide what kind of tagging to use for marshaling/unmarshaling in an ideal world. https://serde.rs/enum-representations.html

I'm looking for untagged here for compatibility, but externally tagged allows for symmetry. I don't know if there's a way to mark a specific field to change the tagging, would this be possible to specify to wit-bindgen-go in some way? Ultimately looking to enable the flexibility for a user of wit-bindgen-go to choose

@brooksmtownsend
Copy link

@ydnar since there's a couple of different goals for JSON de/serialization, do you think it would be possible for us to allow specifying options to wit-bindgen-go at generation time to customize the marshaling and unmarshaling behavior? Similar to how the cm library is a possible flag with generation.

Ideally it would be great to supply the best marshaling behavior based on the use case, and what JSON tools you're working with on the other end. I'd be happy to help contribute this if this feels reasonable

@ydnar
Copy link
Collaborator
ydnar commented Apr 10, 2025

I think we should start with a proposal. Ideally in the component model repository, as a Markdown file that describes a serialization format, goals, non-goals, etc.

Then we can align this code generator to that.

@devigned
Copy link

My goal personally in the contribution / thinking is to marshal the option type in a way that would be compatible with existing tools as much as possible. Ideally, a None variant would just be omitted 6AC6 from serialization entirely if omitempty was supplied. And the presence of a field would be unmarshaled into the Some(T) variant. In normal Go I would just use a pointer to represent a nullable value, but the value that comes from wit-bindgen-go from an optional WIT type is the Option.

I actually ran into a problem using a WIT generated data structure, which I was using to represent a OpenAI API request. I noticed the json tags generated and was delighted to use them to marshal the generated structure to json for use in the HTTP API request to OAI. Unfortunately the optional fields were not tagged with omitempty, which caused a 400 response from the service. The presence of the field cause the request to be misshapen.

It would be nice to be able to either default to optionals being omitempty or be able to provide the generator some additional information to tell it to omitempty on fields.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@lxfontes lxfontes

Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@ydnar @lxfontes @devigned @brooksmtownsend
0