8000 Missing texture replace · Issue #44 · sammyfreg/netImgui · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Missing texture replace #44

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
lemantisee opened this issue May 17, 2023 · 18 comments
Closed

Missing texture replace #44

lemantisee opened this issue May 17, 2023 · 18 comments

Comments

@lemantisee
Copy link
Contributor

Hi! In our project client app provides the video stream witch I would like to embed in ui on server side. Previously we implement it very roughly:

  1. On client app draws texture with ImGui::Image and send texture and texture id via third party tool (NDI)
  2. On server side receive texture and texture id from third party tool
  3. Provides to netimgui on server side texture data by texture id which was used in ImGui::Image on client side

So, I interested is netimgui project interesting in this feature? I can make PR with describes the idea.
Seems it very close to #10 but difference in that I'm want to provide texture in each frame on server side.
@sammyfreg
Copy link
Owner
sammyfreg commented May 17, 2023 via email •
edited
Loading

@lemantisee
Copy link
Contributor Author

Oh, sorry for bad describing our pipeline:) Our case is more common.
We have two applications - server(local or dedicated) and client. On server we use imgui (and NetImgui) to draw interface and shows it on client app with NetImgui. Server continuously renders some image which we want to show in client app. For that we use third party tool to stream this image to client app. We don't want use transferring texture via NetImgui (NetImgui::SendDataTexture) because video streaming requires complicated compression and obliviously not a task for NetImgui.

@sammyfreg
Copy link
Owner
sammyfreg commented May 17, 2023 •
edited
Loading

Ok, if I understood correctly (because there a lot of server and client mixed in :) )

You have 'Server A' application with a 'Netimgui Client B' connection.
You have 'Client A' PC that's running 'NetImgui Server B' application.

Your Server A is running some logic, then sending the imgui results to the NetImgui Server B.

Given that it's a video stream, I agree that sending it uncompressed would be a problem. An easier solution than what you propose, could be to send the compressed image frame as a new texture format, and then have the NetImgui server decompres it, maybe by using a known library, or what you are already using.

@lemantisee
Copy link
Contributor Author
lemantisee commented May 17, 2023 •
edited
Loading

You have 'Server A' application with a 'Netimgui Client B' connection.
You have 'Client A' PC that's running 'NetImgui Server B' application.

Yes, totally correct.

Given that it's a video stream, I agree that sending it uncompressed would be a problem. An easier solution than what you propose, could be to send the compressed image frame as a new texture format, and then have the NetImgui server decompres it, maybe by using a known library, or what you are already using.

Actually we made something similar. We send video stream from 'Server A'. After receiving it on 'Client A' we upload it to GPU and obtain 'Video texture id'. After that when NetImguiServer (on 'NetImgui Server B') replaces imgui texture ids by own texture ids (NetImguiServer_RemoteClient.cpp:250) we check if requested texture id is for our video stream and replace it with 'Video texture id'. It's implemented in way which breaks smooth update of NetImgui to next version because we changed NetImguiServer_RemoteClient.cpp file. Maybe it's possible to add some callback to NetImguiServer which will requests missed textures id (which not found in NetImguiServer::Client::mvTextures vector)?

@sammyfreg
Copy link
Owner
sammyfreg commented May 18, 2023 via email

@lemantisee
Copy link
Contributor Author

This solution is good. It actually much better than my callback idea for our future plans about video streaming.
But can we not create texture every time in NetImgui Server when it receives 'Custom' Texture update and just update existed? On first call for custom texture user will creates it, for next calls just updates existed texture.

@sammyfreg
Copy link
Owner

The idea would be that the NetImgui Client keep sending video frame using the same textureid. Then, the NetImgui Server would recreate a new texture GPU ressource, but at the same TextureID. This would be simple.

It might be possible to setup the GPU texture to be updated rather than recreated, but this is something that would need to happen in the custom texture creation function (so handled by you).

@lemantisee
Copy link
Contributor Author

Sounds good. It's possible also to pass some user data to custom texture creation function? Or it will be to specific? User data can helps to determinate is creating or updating needed.

@sammyfreg
Copy link
Owner
sammyfreg commented May 19, 2023 via email

@lemantisee
Copy link
Contributor Author

Maybe it's too specific case, but some times user can wants to change processing texture command data depends on some external parameters. In this case some user data in custom texture callback may helps.

@sammyfreg
Copy link
Owner

I was thinking that this would be up to the user to create their own data format inside the texture data.

I'm doing a custom texture branch, and once I have something, you can try it out and let me know what works and doesn't

@sammyfreg
Copy link
Owner

Submit bc7d34e in the Task_CustomTexture branch give an idea of how I was imagining the feature to work.

The ProcessTexture_Custom function will be moved to a standalone file, so user can freely modify it, and name might change.

Let me know if it looks like something that would work for you.

@lemantisee
Copy link
Contributor Author
lemantisee commented May 22, 2023 •
edited
Loading

Thank you! It's totally works for me.
But want to describe how exactly I use this feature now. I still use third party lib to transfer texture data. Can't send texture data via NetImgui because texture compressing is not yet done in our code. From NetImgui Client I send some small dummy texture data only for calling ProcessTexture_Custom callback. In callback I just set valid texture id to serverTexture which I created on NetImgui Server side after receiving texture data. It's works perfectly because there are no checks (size, bits per pixel and etc) for data validation in NetImgui texture synchronization process. So if it possible to keep it in same way it will be perfect.

My ProcessTexture_Custom implementation looks like

bool ProcessTexture_Custom(const NetImgui::Internal::CmdTexture& cmdTextureUpdate, ServerTexture& serverTexture)
{
    auto eTexFmt = static_cast<NetImgui::eTexFormat>(cmdTextureUpdate.mFormat);
    if (eTexFmt != NetImgui::eTexFormat::kTexFmtCustom) {
        return false;
    }

    if (cmdTextureUpdate.mTextureId != remoteTextureIdReplace || serverTexture.mImguiId != remoteTextureIdReplace) {
        return false;
    }
  
    serverTexture.mpHAL_Texture = reinterpret_cast<void *>(static_cast<uint64_t>(localTextureIdReplace));
    serverTexture.mSize[0] = 1024;
    serverTexture.mSize[1] = 1024;
    return false;
}

where
remoteTextureIdReplace is a texture id which I use in NetImgui::SendDataTexture to send dummy texture
localTextureIdReplace is a texture id with valid texture data which I created on NetImgui Client side

In some future I will use hardware h264 encoding for video and send texture data via NetImgui as it designed now.

@sammyfreg
Copy link
Owner
sammyfreg commented May 24, 2023 •
edited
Loading

I have updated the Sample in the CustomTexture branch. Let me know if this solves your issues. I will then clean it up and integrate it to the DevBranch.

if (cmdTextureUpdate.mTextureId != remoteTextureIdReplace || serverTexture.mImguiId != remoteTextureIdReplace) {
        return false;
    }

So I guess you send a texture update everyframe, but only update the texture if you detect you have received a new one from the video server that you are using? I guess there's not much choice, if you can't be sure on the netimgui client, when the video server has received a new texture and is ready to display it on the NetImguiServer.

So for now, your code won't change much, but you have the option of streaming directly from the Texture command now, and your code changes will be isolated to a separate file (once I move the function).

@lemantisee
Copy link
Contributor Author

Everything good. Thank you!
Please correct me if I understand it wrong. Will ProcessTexture_Custom function goes to HAL section on feature release?

@sammyfreg
Copy link
Owner

I finalized the work on custom texture support and imported it in the 'dev' branch. (see beb5dd4).

The custom texture handling is done inside NetImguiServer_App_Custom.cpp and there's a sample demonstrating the usage inside the Texture Sample, and the Server Application. (look for the TEXTURE_CUSTOM_SAMPLE define)

Once you have confirmed that everything works as expected, I will close this issue.

@sammyfreg
Copy link
Owner

Note: I never worked with textures updated every frame (like a video frame), but there might be a better way of handling this than Destroy/Create texture gpu resource each time (like the sample is doing). It might be that creating the texture gpu resource as 'updatable' and then updating it instead of recreating it, could be faster.

This is something that you can explore, but would need to implement your own HAL_CreateTexture to handle it.

Just a thought.

@lemantisee
Copy link
Contributor Author

Yeah. In OpenGL there is glTexSubImage2D for updating whole texture or only a part of it.
Thank you for this feature!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@sammyfreg @lemantisee
0