Description
More concise description (the bulk of this conversation was debugging a niche GPU rendering issue)
Tauri currently supports sending binary data from the FE to Rust with array buffers.
It would be great if the event system (Rust -> FE) could skip this se 5E78 rialization cost as well and send array buffers.
Some potential workarounds for those that need this functionality now:
- Rust pings the FE to make a request for a command which sends an array buffer
- Long polling
Describe the problem
I'm prototyping an app that does heavy image processing. Images can easily be 50 megapixels, and the processing pipeline treats each pixel component as a 32 bit float. (this is a common approach. See Darktable) The issue though, is that the payload for a single image in a raw format can easily be up to 50_000_000 * 3 * 4 bytes (600 MB) If that sounds scary, it already takes 200ms to send only 3MB of data over IPC. The image data could be represented in 16 bit color (8 bit is insufficient for many cases) but that doesn't make things much better.
It's well documented that IPC in Tauri is slow for large payloads, but workarounds or knowledge about how this could be improved today or in the future is hard to find. It seems that there are a few approaches:
Potential improvements today:
- Implement streaming. working example: https://github.com/tauri-apps/tauri/tree/dev/examples/streaming
- This example mentions an already built in
assetprotocol but no mention of how to use it / where to learn more
- This example mentions an already built in
- Shrink payload (even at 100x smaller this would be too slow)
Potential improvements in the future:
- Expose a more developer friendly API for sending binary data
- Shared memory. It sounds like there's security concerns with implementing this
- Use a memory mapped file (similar security concerns?)
Describe the solution you'd like
A faster way to share / get large amounts of binary data to the front-end, or if it's possible to draw directly from Rust to a canvas element. This is possible if rust is running in WASM, but is likely much more difficult / impossible when Rust is running natively and a separate process.
Alternatives considered
With Tauri:
- Creating a separate window exclusively controlled by Rust and GPU. Would this work across Windows/Linux/MacOS?
- I'd still want to send smaller thumbnails, histograms, etc to the web UI
- Streaming (couldn't get this working. probably still too slow with current architecture)
- Rendering directly to a canvas element from Rust (not possible?)
Without Tauri:
- JS + Rust WASM & WGPU
- JS & WGPU
Additional context
I'm an experienced FE developer and very new to Rust.
I've scoured over a dozen Tauri threads over this already. It's starting to feel like building an image editing app based on Tauri is not possible with current technology, and the only viable alternative if I want to leverage Web UI is to do everything in the browser (eg: Adobe's browser version of Photoshop) but then I'm hit with RAM limitations.
Similar thread: #4177