Interactive transactions in IPROTO #2016

kostja · 2016-12-28T18:09:43Z

A client transaction can be started with CALL("box.begin").

Vinyl doesn't abort a transaction at fiber yield. But the same fiber may serve different connections of the binary protocol. It means that queries from different connections may end up in the same transaction, or that a subsequent query which is intended for a multi-statement transaction ends up committed prematurely.

All this trouble happens because struct txn is now stored in struct fiber, not in struct session, as well as gc pool.

So it seems time has come to move struct txn, struct region, and struct diag from struct fiber to struct session. Then we will either need to create a technical session for every fiber, or make sure that the rest of the code, which uses fibers, works without struct region gc and struct diag.

How to repeat:

begin a transaction on the client, over the binary protocol
create another connection
send a batch of two queries over the same connection; the queries must use the transactrion and yield
observe the second query execute before the first one.

Implementation

The binary protocol

We will extend the binary protocol header with a new key, IPROTO_TRANSACTION_ID.
Each request missing this key, or with key value 0, will run in its own transaction, or, if it's CALL/EVAL, will be able to manually control the transaction flow. In other words, for old clients and drivers, the semantics will stay the same.

We will also add commands for explicit transaction control flow:
IPROTO_BEGIN
IPROTO_COMMIT
IPROTO_ROLLBACK

These transaction will pass IPROTO_TRANSACTION_ID in the header, so they will affect the transaction identified by the supplied id.

IPROTO_BEGIN may miss IPROTO_TRANSACTION_ID. In this case, as well as in case of CALL/EVAL request explicitly starting a transaction by invoking box.begin() and leaving it open, the server will pass an automatically generated id back to the client.

Since, with this enhancement, the client will get an opportunity to flood the server with infinitely many open transaction, a new box dynamic configuration variable needs to be introduced: transaction_max, limiting the number of active transactions.
We also need to reflect the number of active transactions in a statistical variable.

Lua concole

We need a way to store the active transaction id in a Lua console. Since the id can not be stored in the session (a single session may multiplex many transactions), we need a way to store it someplace else - perhaps in the console itself. For that purpose, we perhaps need to extend box.session package to return transaction id, so that Lua console can query it explicitly and save.

Runtime

fiber->gc needs to become a pointer, so that setting active transaction will update it.
Disconnect trigger needs to roll back all transactions active in the connection.
There should be a way to get a list of all active transactions in Lua and explicitly roll them back (by the
system administrator).

Binary log and replication

We may need to introduce transaction id to the binary log to implement synchronous replication. For now, though, since we never have partial transactions in the binary log, and there is a binary log transaction (physical one), we don't need to change anything.

The text was updated successfully, but these errors were encountered:

See #2016

To make remote transactions we need to save a transaction with its data after the fiber already yielded or rescheduled. Fiber at the end of the work destroyes its region, so we can't store a transaction in its region - the transaction must have its own region. Need for #2016

Use mempool of struct txn objects to allocate new transactions. It allows to store transactions independetly from fibers. Need for #2016

Part of the #2016

See #2016

To make remote transactions we need to save a transaction with its data after the fiber already yielded or rescheduled. Fiber at the end of the work destroyes its region, so we can't store a transaction in its region - the transaction must have its own region. Need for #2016

Use mempool of struct txn objects to allocate new transactions. It allows to store transactions independetly from fibers. Need for #2016

Part of the #2016

Tests for #2016

See #2016

To make remote transactions we need to save a transaction with its data after the fiber already yielded or rescheduled. Fiber at the end of the work destroyes its region, so we can't store a transaction in its region - the transaction must have its own region. Need for #2016

Use mempool of struct txn objects to allocate new transactions. It allows to store transactions independetly from fibers. Need for #2016

Part of the #2016

Tests for #2016

See #2016

With introduction of mvcc, both memtx and vinyl engines are allowed to yield during a transaction. This makes possible to implement interactive transactions: when txn statements don't have to be sent at once. Current implementation allows to have one transaction per connection. More flexible approach is to multiplex transactions in one connection using e.g. idea of streams. This is going to be implemented later. Closes #2016

kyukhin · 2021-03-29T10:03:47Z

Superceded by #5860

Commit 3decf8b78017d86f4b593b28789775aaa930ba97 ("Disable test bank-multitable-lua") tried to disable test by removing it from a list of tests (standard workloads). First of all patch disabled incorrect workload, it should be "bank-multitable-lua" and not "bank-multitable". Secondly there is a better way to do it - using "workloads-expected-to-pass" that removes workloads that broken for some reasons. Now "bank-multitable-lua" has been fixed and no reasons to disable it, see commit e0a27dd "Fix test bank with accounts in separate tables". Tests "bank-multitable" and "bank" are still broken because Tarantool doesn't support interactive transactions, see Tarantool issue [1] and thus these tests excluded in "workloads-expected-to-pass". Additionally to use working workloads on Tarantool CI option "workloads-expected-to-pass" enabled by default when "test-all" used. 1. tarantool/tarantool#2016 2. #83 Follows up tarantool/tarantool#5848

Commit 3decf8b78017d86f4b593b28789775aaa930ba97 ("Disable test bank-multitable-lua") tried to disable test by removing it from a list of tests (standard workloads). First of all patch disabled incorrect workload, it should be "bank-multitable-lua" and not "bank-multitable". Secondly there is a better way to do it - using "workloads-expected-to-pass" that removes workloads that broken for some reasons. Now "bank-multitable-lua" has been fixed and no reasons to disable it, see commit 978dd81 "Fix test bank with accounts in separate tables". Tests "bank-multitable" and "bank" are still broken because Tarantool doesn't support interactive transactions, see Tarantool issue [1] and thus these tests excluded in "workloads-expected-to-pass". Additionally to use working workloads on Tarantool CI option "workloads-expected-to-pass" enabled by default when "test-all" used. 1. tarantool/tarantool#2016 2. #83 Follows up tarantool/tarantool#5848

kostja added the bug Something isn't working label Dec 28, 2016

kostja modified the milestones: 1.7.4, 1.7.5 Dec 28, 2016

kostja added the prio2 label Jan 24, 2017

kostja assigned Gerold103 Jan 24, 2017

Gerold103 added a commit that referenced this issue Jan 31, 2017

Add transaction identifier to request

52f6b34

See #2016

Gerold103 added a commit that referenced this issue Feb 1, 2017

Add transaction identifier to request

a934259

See #2016

Gerold103 added a commit that referenced this issue Feb 1, 2017

iproto: implement remote txn_begin

fd09938

See #2016

Gerold103 added a commit that referenced this issue Feb 4, 2017

Add transaction identifier to request

adaa67c

See #2016

Gerold103 added a commit that referenced this issue Feb 4, 2017

iproto: implement remote txn_begin

733a5ab

See #2016

Gerold103 added a commit that referenced this issue Feb 4, 2017

Attach existing transaction to the remote request

9ac764b

See #2016

Gerold103 added a commit that referenced this issue Feb 15, 2017

Add transaction identifier to request

e4f4815

See #2016

Gerold103 added a commit that referenced this issue Feb 15, 2017

Allocate transaction objects from pool

9617245

Use mempool of struct txn objects to allocate new transactions. It allows to store transactions independetly from fibers. Need for #2016

Gerold103 added a commit that referenced this issue Feb 15, 2017

Use iproto transactions for multiple requests

6988f85

Part of the #2016

Gerold103 added a commit that referenced this issue Feb 15, 2017

Use iproto transactions for multiple requests

d5dca83

Part of the #2016

Gerold103 added a commit that referenced this issue Feb 16, 2017

Add transaction identifier to request

f633a8d

See #2016

Gerold103 added a commit that referenced this issue Feb 16, 2017

Allocate transaction objects from pool

0c157db

Use mempool of struct txn objects to allocate new transactions. It allows to store transactions independetly from fibers. Need for #2016

Gerold103 added a commit that referenced this issue Feb 16, 2017

Use iproto transactions for multiple requests

f820e9d

Part of the #2016

Gerold103 added a commit that referenced this issue Feb 16, 2017

vinyl: add tests on remote transactions

65865fb

Tests for #2016

Gerold103 added a commit that referenced this issue Feb 16, 2017

Add transaction identifier to request

8cb5bb3

See #2016

Gerold103 added a commit that referenced this issue Feb 16, 2017

Allocate transaction objects from pool

c785fd9

Use mempool of struct txn objects to allocate new transactions. It allows to store transactions independetly from fibers. Need for #2016

Gerold103 added a commit that referenced this issue Feb 16, 2017

Use iproto transactions for multiple requests

8188dab

Part of the #2016

Gerold103 added a commit that referenced this issue Feb 16, 2017

vinyl: add tests on remote transactions

3abb55e

Tests for #2016

Gerold103 added a commit that referenced this issue Feb 20, 2017

Add transaction identifier to request

0890395

See #2016

mary3000 mentioned this issue Dec 7, 2020

Introduce interactive transactions #5595

Closed

This was referenced Dec 19, 2020

Bank test in multi-table configuration failed on a synchronous cluster #5513

Closed

Add tube:put_many() tarantool/queue#124

Open

kyukhin added the tmp label Dec 29, 2020

Mons removed the tmp label Jan 11, 2021

alyapunov added the tmp label Jan 11, 2021

alyapunov added the blocked Not ready to be implemented label Jan 20, 2021

kyukhin modified the milestones: 2.7.1, 2.8.1 Feb 12, 2021

kyukhin removed the tmp label Feb 12, 2021

mary3000 mentioned this issue Mar 1, 2021

Interactive transactions and streams in iproto/http2 #5860

Closed

kyukhin added the tmp label Mar 29, 2021

kyukhin unassigned mary3000 Mar 29, 2021

kyukhin modified the milestones: 2.8.1, 8000 2.9.1 Mar 29, 2021

kyukhin closed this as completed Mar 29, 2021

kyukhin removed this from the 2.9.1 milestone Mar 29, 2021

kyukhin removed the tmp label Apr 22, 2021

ligurio mentioned this issue May 24, 2021

'BankClient' and 'MultiBankClient' are broken tarantool/jepsen.tarantool#83

Open

Totktonada mentioned this issue Dec 13, 2021

How to do transaction without have to write it in Lua? tarantool/go-tarantool#101

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Interactive transactions in IPROTO #2016

Interactive transactions in IPROTO #2016

Uh oh!

Interactive transactions in IPROTO #2016

Interactive transactions in IPROTO #2016

Comments

Uh oh!

Implementation

The binary protocol

Lua concole

Runtime

Binary log and replication

Uh oh!

Uh oh!