8000 Segmentation fault in mlua userdata caused by storing a Lua VM handle in a userdata with generic parameters · Issue #552 · mlua-rs/mlua · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Segmentation fault in mlua userdata caused by storing a Lua VM handle in a userdata with generic parameters #552

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

8000 Closed
cheesycod opened this issue Apr 2, 2025 · 8 comments

Comments

@cheesycod
Copy link

Decided to make this a sep issue from the Github Discussion as its pretty big.

The latest mlua build (with luau 667 and thread callbacks) now gives me segmentation faults

@cheesycod
Copy link
Author

Here's the backtrace btw @khvzak

Thread 18 "lua-vm-threadpo" received signal SIGILL, Illegal instruction.
[Switching to Thread 0x7fffef2006c0 (LWP 620293)]
0x0000555559606482 in luaU_freeudata (L=0x7fff8ce89f20, u=0x7fff8dac2ee8, page=0x7fff8dac0fc0)
at /home/antiraid/.cargo/registry/src/index.crates.io-6f17d22bba15001f/luau0-src-0.13.0+luau667/luau/VM/src/ludata.cpp:42
42 luaM_freegco(L, u, sizeudata(u->len), u->memcat, page);
(gdb)

@cheesycod
Copy link
Author

Image

Full backtrace

@cheesycod
Copy link
Author

I did some debugging and the segfault happens due to it trying to destroy the same userdata twice????

Destroying userdata of type: Event
Destroying userdata of type: TemplateContext
Destroying userdata of type: TemplateContext```

@cheesycod
Copy link
Author
cheesycod commented Apr 2, 2025

@khvzak did more testing and tracked down the issue to having a Lua VM handle in a userdata that also has a generic parameter. This causes the UserData to be freed multiple times during lua garbage collection (likely due to a bug in mlua’s Drop(??) implementation) leading to a double free and hence a segmentation fault

Wrapping the Lua VM in a ManuallyDrop or removing the mlua::Lua solves the issue making it very likely a bug in mlua::Lua’s Drop implementation conflicting with how userdata is destructed somehow leading to a double free.

@cheesycod cheesycod changed the title Segmentation fault in mlua Segmentation fault in mlua userdata caused by storing a Lua VM handle in a userdata with generic parameters Apr 2, 2025
@cheesycod
Copy link
Author
cheesycod commented Apr 2, 2025

Managed to get a nice minimal reproducible example @khvzak

use mlua::prelude::*;

pub struct MyUserData<T: 'static + Clone> {
    pub lua: mlua::Lua,
    v: T
}

impl<T: 'static + Clone> LuaUserData for MyUserData<T> {}

fn main() {
    let lua = mlua::Lua::new();

    let my_userdata: MyUserData<String> = MyUserData {
        lua: lua.clone(),
        v: "Hello".to_string(),
    };

    let mv = (10, my_userdata).into_lua_multi(&lua).unwrap();

    for i in 1..10 {
        load(&lua, mv.clone());
    }
}

fn load(lua: &mlua::Lua, mv: LuaMultiValue) {
    lua.load("return 1")
    .call::<()>(mv)
    .unwrap();
}

Note that the generics above as well as into_lua_multi() are optional, so the below also Illegal Instructions (or segfaults sometimes) as well:

use mlua::prelude::*;

#[derive(Clone, Debug)]
pub struct MyUserData {
    lua: Lua,
    v: String,
}

impl LuaUserData for MyUserData {}

fn main() {
    let lua = mlua::Lua::new();

    let my_userdata: MyUserData = MyUserData {
        lua: lua.clone(),
        v: "Hello".to_string(),
    };

    let mv = (10, my_userdata);

    for i in 1..10 {
        load(&lua, mv.clone());
    }
}

fn load(lua: &mlua::Lua, mv: impl IntoLuaMulti) {
    lua.load("return 1")
    .call::<()>(mv)
    .unwrap();
}

This specific example doesn't seem to segfault in release but its still possible to encounter using some slight modifications in release mode as well

@khvzak
Copy link
Member
khvzak commented Apr 2, 2025

The minimal example would be as simple as:

    let lua = Lua::new();
    lua.create_any_userdata(lua.clone())?;

The crash is happens because when Lua (main Lua specifically) is dropped, it triggers garbage collection.
But when it placed inside UserData, then double GC cycle is happening:

  1. GC when UserData is dropped
  2. GC triggered by Lua inside the first GC when Lua has dropped

Lua (VM) not designed to handle this case unfortunately. Seems I need to remove enforced garbage collection on Lua drop.

@khvzak
Copy link
Member
khvzak commented Apr 5, 2025

The problem is more complex than I anticipated.
Unlike classic Lua, Luau cannot handle any panics (unwinding) in UserData destructors. This cause use-after-free and other memory related issues.

To address this issue, I made few changes:

  1. Lua clones will not trigger GC collection on Drop
  2. mlua will panic on any Lua operations inside userdata destructors
  3. Any panics in userdata destructors will trigger abort

@khvzak khvzak closed this as completed Apr 5, 2025
@khvzak
Copy link
Member
khvzak commented Apr 6, 2025

Btw, I added WeakLua type that is the best way to include reference to Lua to Lua managed types.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants
0