10000 crash in bps tree on OOM · Issue #11326 · tarantool/tarantool · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

crash in bps tree on OOM #11326

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
nshy opened this issue Apr 2, 2025 · 1 comment · Fixed by #11331
Closed

crash in bps tree on OOM #11326

nshy opened this issue Apr 2, 2025 · 1 comment · Fixed by #11331
Assignees
@nshy
Labels
3.3 Target is 3.3 and all newer release/master branches bug Something isn't working crash

Comments

@nshy
Copy link
Contributor
nshy commented Apr 2, 2025 •
edited
Loading

Bug description

Tarantool crashed with backtrace:

#4  0x000064c034cbf04e in crash_signal_cb (signo=11, siginfo=0x64c05c919f30,
    context=0x64c05c919e00) at /home/shiny/dev/tarantool/src/lib/core/crash.c:204
#5  <signal handler called>
#6  0x000064c034a1f428 in NS_USE_HINT::bps_tree_memtx_tree_garbage_push (
    tree=0x64c05c9b4f38, block=0x7c5457120200, id=1)
    at /home/shiny/dev/tarantool/src/lib/salad/bps_tree.h:3170
#7  0x000064c034a1f6aa in NS_USE_HINT::bps_tree_memtx_tree_reserve_blocks (
    tree=0x64c05c9b4f38, count=2)
    at /home/shiny/dev/tarantool/src/lib/salad/bps_tree.h:3262
#8  0x000064c034a26b78 in NS_USE_HINT::bps_tree_memtx_tree_insert (t=0x64c05c9b4f38,
    new_elem=..., replaced=0x7c5467080890, successor=0x7c54670808a0,
    inserted_iterator=0x0, offset=0x0)
    at /home/shiny/dev/tarantool/src/lib/salad/bps_tree.h:5825
#9  NS_USE_HINT::memtx_tree_insert (t=0x64c05c9b4f38, new_elem=...,
    replaced=0x7c5467080890, successor=0x7c54670808a0)
    at /home/shiny/dev/tarantool/src/lib/salad/bps_tree.h:5880
#10 0x000064c034a2a2d0 in memtx_tree_index_replace<true> (base=0x64c05c9b4f00,
    old_tuple=0x0, new_tuple=0x7c54560078cc, mode=DUP_INSERT, result=0x7c5467080930,
    successor=0x7c5467080948) at /home/shiny/dev/tarantool/src/box/memtx_tree.cc:1521
#11 0x000064c034a572aa in index_replace (index=0x64c05c9b4f00, old_tuple=0x0,
    new_tuple=0x7c54560078cc, mode=DUP_INSERT, result=0x7c5467080930,
    successor=0x7c5467080948) at /home/shiny/dev/tarantool/src/box/index.h:1020
#12 0x000064c034a58cf1 in memtx_space_replace_all_keys (space=0x64c05c9b4b70,
    old_tuple=0x0, new_tuple=0x7c54560078cc, mode=DUP_INSERT, result=0x7c54670809c8)
    at /home/shiny/dev/tarantool/src/box/memtx_space.c:327
#13 0x000064c034a58f43 in memtx_space_replace_tuple (space=0x64c05c9b4b70,
    stmt=0x7c546742d2b0, old_tuple=0x0, new_tuple=0x7c54560078cc, mode=DUP_INSERT)
    at /home/shiny/dev/tarantool/src/box/memtx_space.c:389
#14 0x000064c034a59122 in memtx_space_execute_replace (space=0x64c05c9b4b70,
    txn=0x7c546742d038, request=0x7c5467080b80, result=0x7c5467080af8)
    at /home/shiny/dev/tarantool/src/box/memtx_space.c:435
#15 0x000064c034ae6a9c in space_execute_dml (space=0x64c05c9b4b70,
    txn=0x7c546742d038, request=0x7c5467080b80, result=0x7c5467080af8)
    at /home/shiny/dev/tarantool/src/box/space.c:1450
$ ./build-dev/src/tarantool --version
Tarantool 3.4.0-entrypoint-216-gdef445684d
Target: Linux-x86_64-Debug
Build options: cmake . -DCMAKE_INSTALL_PREFIX=/usr/local -DENABLE_BACKTRACE=TRUE
Compiler: GNU-14.2.1
C_FLAGS: -fexceptions -funwind-tables -fasynchronous-unwind-tables -fno-common -msse2 -Wformat -Wformat-security -Werror=format-security -fstack-protector-strong -fPIC -fmacro-prefix-map=/home/shiny/dev/tarantool=. -std=c11 -Wall -Wextra -Wno-gnu-alignof-expression -fno-gnu89-inline -Wno-cast-function-type -Werror -g -ggdb -O0
CXX_FLAGS: -fexceptions -funwind-tables -fasynchronous-unwind-tables -fno-common -msse2 -Wformat -Wformat-security -Werror=format-security -fstack-protector-strong -fPIC -fmacro-prefix-map=/home/shiny/dev/tarantool=. -std=c++11 -Wall -Wextra -Wno-invalid-offsetof -Wno-gnu-alignof-expression -Wno-cast-function-type -Werror -g -ggdb -O0

Steps to reproduce

Repro script:

local fiber = require('fiber')
box.cfg{log_level = 'warn', wal_mode = 'none'}
local s = box.schema.create_space('test')
s:create_index('pk')
s:insert({1})
box.error.injection.set("ERRINJ_INDEX_ALLOC", true)
box.error.injection.set("ERRINJ_SNAP_COMMIT_DELAY", true)
fiber.create(function()
    box.snapshot()
end)
s:insert({2})
os.exit()

Same as https://github.com/tarantool/tarantool-ee/issues/1180.
@nshy nshy added the bug Something isn't working label Apr 2, 2025
@nshy nshy self-assigned this Apr 2, 2025
@nshy nshy added crash 3.3 Target is 3.3 and all newer release/master branches labels Apr 2, 2025
@nshy
Copy link
Contributor Author
nshy commented Apr 2, 2025

Introduced by 3.3.0-entrypoint-216-g51c56d9b33 (51c56d9). Before it we don't have any reservation inside bps tree and just do some heuristic reservation on insert using memtx_index_extent_reserve() and should fail gracefully for the given script. Note that heuristic was deleted only in 32b0678, the above commit introduced reservation inside bps tree.

nshy added a commit to nshy/tarantool that referenced this issue Apr 3, 2025
@nshy
salad: fix crash on insertion into bps tree with OOM conditions
77828a5 
On insertion into bps tree we first reserve blocks for the case we
need to insert new leaf during rebalancing. Next there can be a free
block on last extent in matras but no more free extents in tuple arena.
So `bps_tree_reserve_blocks()` is able to allocate a block but then
`bps_tree_garbage_push()` is called that touches a block. As there no
free extents touch is failed which is unexpected and causes a crash.

We don't need to touch the block on this path as block is newly allocated
and is not shared with any read view. We don't need to touch block on
other paths too (disposing a leaf or a inner node) because we move data
from this node before so it is already should be touched.

So just don't touch the block in `bps_tree_garbage_push()`.

The issue is introduced in the commit 51c56d9 ("salad: reserve
extents for matras_touch on BPS tree operations") and unlocked in the
commit 32b0678 ("memtx: reserve extents in the RTREE index itself")
where we removed crutches where we heuristically reserve memory for
index operations.

Closes tarantool#11326

NO_DOC=bugfix
This was referenced Apr 3, 2025
Merged
Open
github-actions bot pushed a commit that referenced this issue Apr 4, 2025
@nshy @github-actions
salad: fix crash on insertion into bps tree with OOM conditions
539c9ff 
On insertion into bps tree we first reserve blocks for the case we
need to insert new leaf during rebalancing. Next there can be a free
block on last extent in matras but no more free extents in tuple arena.
So `bps_tree_reserve_blocks()` is able to allocate a block but then
`bps_tree_garbage_push()` is called that touches a block. As there
8DD1
 no
free extents touch is failed which is unexpected and causes a crash.

We don't need to touch the block on this path as block is newly allocated
and is not shared with any read view. We don't need to touch block on
other paths too (disposing a leaf or a inner node) because we move data
from this node before so it is already should be touched.

So just don't touch the block in `bps_tree_garbage_push()`.

The issue is introduced in the commit 51c56d9 ("salad: reserve
extents for matras_touch on BPS tree operations") and unlocked in the
commit 32b0678 ("memtx: reserve extents in the RTREE index itself")
where we removed crutches where we heuristically reserve memory for
index operations.

Closes #11326

NO_DOC=bugfix

(cherry picked from commit 46cc98b)
@TarantoolBot TarantoolBot mentioned this issue Apr 4, 2025
Merged
sergepetrenko pushed a commit that referenced this issue Apr 4, 2025
@nshy @sergepetrenko
salad: fix crash on insertion into bps tree with OOM conditions
82c23f2 
On insertion into bps tree we first reserve blocks for the case we
need to insert new leaf during rebalancing. Next there can be a free
block on last extent in matras but no more free extents in tuple arena.
So `bps_tree_reserve_blocks()` is able to allocate a block but then
`bps_tree_garbage_push()` is called that touches a block. As there no
free extents touch is failed which is unexpected and causes a crash.

We don't need to touch the block on this path as block is newly allocated
and is not shared with any read view. We don't need to touch block on
other paths too (disposing a leaf or a inner node) because we move data
from this node before so it is already should be touched.

So just don't touch the block in `bps_tree_garbage_push()`.

The issue is introduced in the commit 51c56d9 ("salad: reserve
extents for matras_touch on BPS tree operations") and unlocked in the
commit 32b0678 ("memtx: reserve extents in the RTREE index itself")
where we removed crutches where we heuristically reserve memory for
index operations.

Closes #11326

NO_DOC=bugfix

(cherry picked from commit 46cc98b)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@nshy nshy

Labels
3.3 Target is 3.3 and all newer release/master branches bug Something isn't working crash
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

salad: fix crash on insertion into bps tree with OOM conditions nshy/tarantool
1 participant
@nshy
0