8000 Max byte check by alijnmerchant21 · Pull Request #1384 · cometbft/cometbft · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Max byte check #1384

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 14 commits into from
Sep 22, 2023
Merged

Max byte check #1384

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
9d53787
Update go version
alijnmerchant21 Sep 21, 2023
7685e79
Update go.mod to reflect go version and module
alijnmerchant21 Sep 21, 2023
46049a4
Update to 0.38
alijnmerchant21 Sep 21, 2023
7e786e7
Update formatting
alijnmerchant21 Sep 21, 2023
21c6cb2
Update to 0.38
alijnmerchant21 Sep 21, 2023
fef32e0
Update formatting
alijnmerchant21 Sep 21, 2023
9ad4558
Refactor PrepareProposal to handle transaction size limit
alijnmerchant21 Sep 21, 2023
917908e
Update go version
alijnmerchant21 Sep 21, 2023
c4fc4bb
Update go.mod to reflect go version and module
alijnmerchant21 Sep 21, 2023
fbd4e2b
Update to 0.38
alijnmerchant21 Sep 21, 2023
617f95d
Update formatting
alijnmerchant21 Sep 21, 2023
14811b1
Refactor PrepareProposal to handle transaction size limit
8000 alijnmerchant21 Sep 21, 2023
d78184b
Fix passage sequence
alijnmerchant21 Sep 22, 2023
fc597c0
Fix passage sequence
alijnmerchant21 Sep 22, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 32 additions & 20 deletions docs/guides/go-built-in.md
View file
Open in desktop
8000
Original file line number Diff line number Diff line change
Expand Up @@ -40,14 +40,13 @@ guarantees as two processes would be communicating via established binary protoc
CometBFT will not have access to application's state.
If that is the way you wish to proceed, use the [Creating an application in Go](./go.md) guide instead of this one.


## 1.1 Installing Go

Verify that you have the latest version of Go installed (refer to the [official guide for installing Go](https://golang.org/doc/install)):

```bash
$ go version
go version go1.21.0 darwin/amd64
go version go1.21.1 darwin/amd64
```

## 1.2 Creating a new Go project
Expand Down Expand Up @@ -93,12 +92,12 @@ After running the above commands you will see two generated files, `go.mod` and
The go.mod file should look similar to:

```go
module github.com/me/example
module kvstore

go 1.21
go 1.21.1

require (
github.com/cometbft/cometbft v0.38.0
github.com/cometbft/cometbft v0.38.0
)
```

Expand All @@ -115,7 +114,7 @@ go build
CometBFT communicates with the application through the Application
BlockChain Interface (ABCI). The messages exchanged through the interface are
defined in the ABCI [protobuf
file](https://github.com/cometbft/cometbft/blob/v0.37.x/proto/tendermint/abci/types.proto).
file](https://github.com/cometbft/cometbft/blob/v0.38.x/proto/tendermint/abci/types.proto).

We begin by creating the basic scaffolding for an ABCI application by
creating a new type, `KVStoreApplication`, which implements the
Expand Down Expand Up @@ -380,7 +379,6 @@ The application state may have changed between the initial execution of `CheckTx

Other methods, such as `Query`, rely on a consistent view of the application's state, the application should only update its state by committing the Badger transactions when the full block has been delivered and the Commit method is invoked.


The `Commit` method tells the application to make permanent the effects of
the application transactions.
Let's update the method to terminate the pending Badger transaction and
Expand Down Expand Up @@ -462,16 +460,33 @@ execution of the block once accepted.
In the following code, the application simply returns the unmodified group of transactions:

```go
func (app *KVStoreApplication) PrepareProposal(_ context.Context, proposal *abcitypes.RequestPrepareProposal) (*abcitypes.ResponsePrepareProposal, error) {
return &abcitypes.ResponsePrepareProposal{Txs: proposal.Txs}, nil
}
```
func (app *KVStoreApplication) PrepareProposal(_ context.Context, proposal *abcitypes.RequestPrepareProposal) (*abcitypes.ResponsePrepareProposal, error) {
totalBytes := int64(0)
txs := make([]byte, 0)

for _, tx := range proposal.Txs {
totalBytes += int64(len(tx))
txs = append(txs, tx...)

Once a proposed block is received by a node, the proposal is passed to the application to give
its blessing before voting to accept the proposal.
if totalBytes > int64(proposal.MaxTxBytes) {
Comment on lines +463 to +471
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The use of totalBytes is redundant and could instead just use len(txs) but the even better check would be to firstly check if the final added value is greater than proposal.MaxTxBytes

txs := make([]byte, 0, len(proposal.Txs)*100)
max := int64(proposal.MaxTxBytes)

for _, tx := range proposal.Txs {
   if n := len(txs) + len(tx); n > max || n <= 0 {
       break
   }

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indeed, @odeke-em Thanks for pointing it out.
Given that this PR is now merged, would you mind opening a new PR with your suggestions?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't the problem here that the txs array is not of the right type? (is []byte, should be [][]byte)
Then we are "coalescing" all transactions into a byte array... but that's not what we should be doing, is it?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are right, @sergio-mena, but @odeke-em has a point in that the overflow check should be done before the last transaction has been added, otherwise it is too late to break the loop.

break
}
}

return &abcitypes.ResponsePrepareProposal{Txs: proposal.Txs}, nil
}
```

This code snippet iterates through the proposed transactions and calculates the `total bytes`. If the `total bytes` exceeds the `MaxTxBytes` specified in the `RequestPrepareProposal` struct, the loop breaks and the transactions processed so far are returned.

Note: It is the responsibility of the application to ensure that the `total bytes` of transactions returned does not exceed the `RequestPrepareProposal.max_tx_bytes` limit.

Once a proposed block is received by a node, the proposal is passed to the application to give
its blessing before voting to accept the proposal.

This mechanism may be used for different reasons, for example to deal with blocks manipulated
by malicious nodes, in which case the block should not be considered valid.

This mechanism may be used for different reasons, for example to deal with blocks manipulated
by malicious nodes, in which case the block should not be considered valid.
The following code simply accepts all proposals:

```go
Expand Down Expand Up @@ -692,7 +707,7 @@ signal.Notify(c, os.Interrupt, syscall.SIGTERM)

Our application is almost ready to run, but first we'll need to populate the CometBFT configuration files.
The following command will create a `cometbft-home` directory in your project and add a basic set of configuration files in `cometbft-home/config/`.
For more information on what these files contain see [the configuration documentation](https://github.com/cometbft/cometbft/blob/v0.37.x/docs/core/configuration.md).
For more information on what these files contain see [the configuration documentation](https://github.com/cometbft/cometbft/blob/v0.38.x/docs/core/configuration.md).

From the root of your project, run:

Expand Down Expand Up @@ -749,7 +764,6 @@ The blocks, as you can see from the `num_valid_txs=0` part, are empty, but let's
Let's try submitting a transaction to our new application.
Open another terminal window and run the following curl command:


```bash
curl -s 'localhost:26657/broadcast_tx_commit?tx="cometbft=rocks"'
```
Expand Down Expand Up @@ -786,6 +800,4 @@ echo cm9ja3M=" | base64 -d

## Outro

I hope everything went smoothly and your first, but hopefully not the last,
CometBFT application is up and running. If not, please [open an issue on
Github](https://github.com/cometbft/cometbft/issues/new/choose).
Hope you could run everything smoothly. If you have any difficulties running through this tutorial, reach out 8000 to us via [discord](https://discord.com/invite/cosmosnetwork) or open a new [issue](https://github.com/cometbft/cometbft/issues/new/choose) on Github.
42 changes: 26 additions & 16 deletions docs/guides/go.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ Verify that you have the latest version of Go installed (refer to the [official

```bash
$ go version
go version go1.21.0 darwin/amd64
go version go1.21.1 darwin/amd64
```

## 1.2 Creating a new Go project
Expand Down Expand Up @@ -92,9 +92,9 @@ After running the above commands you will see two generated files, `go.mod` and
The go.mod file should look similar to:

```go
module github.com/me/example
module kvstore

go 1.21
go 1.21.1

require (
github.com/cometbft/cometbft v0.38.0
Expand All @@ -109,13 +109,12 @@ go get
go build
```


## 1.3 Writing a CometBFT application

CometBFT communicates with the application through the Application
BlockChain Interface (ABCI). The messages exchanged through the interface are
defined in the ABCI [protobuf
file](https://github.com/cometbft/cometbft/blob/v0.37.x/proto/tendermint/abci/types.proto).
file](https://github.com/cometbft/cometbft/blob/v0.38.x/proto/tendermint/abci/types.proto).

We begin by creating the basic scaffolding for an ABCI application by
creating a new type, `KVStoreApplication`, which implements the
Expand Down Expand Up @@ -379,11 +378,8 @@ This can happen if the application state is used to determine transaction validi

Other methods, such as `Query`, rely on a consistent view of the application's state, the application should only update its state by committing the Badger transactions when the full block has been delivered and the `Commit` method is invoked.


The `Commit` method tells the application to make permanent the effects of
the application transactions.
Let's update the method to terminate the pending Badger transaction and
persist the resulting state:
The `Commit` method tells the application to make permanent the effects of the application transactions.
Let's update the method to terminate the pending Badger transaction and persist the resulting state:

```go
func (app KVStoreApplication) Commit(_ context.Context, commit *abcitypes.RequestCommit) (*abcitypes.ResponseCommit, error) {
Expand Down Expand Up @@ -460,15 +456,31 @@ In the following code, the application simply returns the unmodified group of tr

```go
func (app *KVStoreApplication) PrepareProposal(_ context.Context, proposal *abcitypes.RequestPrepareProposal) (*abcitypes.ResponsePrepareProposal, error) {
totalBytes := int64(0)
txs := make([]byte, 0)

for _, tx := range proposal.Txs {
totalBytes += int64(len(tx))
txs = append(txs, tx...)

if totalBytes > int64(proposal.MaxTxBytes) {
break
}
}

return &abcitypes.ResponsePrepareProposal{Txs: proposal.Txs}, nil
}
```

Once a proposed block is received by a node, the proposal is passed to the application to give
its blessing before voting to accept the proposal.
This code snippet iterates through the proposed transactions and calculates the `total bytes`. If the `total bytes` exceeds the `MaxTxBytes` specified in the `RequestPrepareProposal` struct, the loop breaks and the transactions processed so far are returned.

Note: It is the responsibility of the application to ensure that the `total bytes` of transactions returned does not exceed the `RequestPrepareProposal.max_tx_bytes` limit.

Once a proposed block is received by a node, the proposal is passed to the application to determine its validity before voting to accept the proposal.

This mechanism may be used for different reasons, for example to deal with blocks manipulated
by malicious nodes, in which case the block should not be considered valid.

The following code simply accepts all proposals:

```go
Expand Down Expand Up @@ -587,7 +599,7 @@ signal.Notify(c, os.Interrupt, syscall.SIGTERM)

Our application is almost ready to run, but first we'll need to populate the CometBFT configuration files.
The following command will create a `cometbft-home` directory in your project and add a basic set of configuration files in `cometbft-home/config/`.
For more information on what these files contain see [the configuration documentation](https://github.com/cometbft/cometbft/blob/v0.37.x/docs/core/configuration.md).
For more information on what these files contain see [the configuration documentation](https://github.com/cometbft/cometbft/blob/v0.38.x/docs/core/configuration.md).

From the root of your project, run:

Expand Down Expand Up @@ -699,6 +711,4 @@ echo cm9ja3M=" | base64 -d

## Outro

I hope everything went smoothly and your first, but hopefully not the last,
CometBFT application is up and running. If not, please [open an issue on
Github](https://github.com/cometbft/cometbft/issues/new/choose).
Hope you could run everything smoothly. If you have any difficulties running through this tutorial, reach out to us via [discord](https://discord.com/invite/cosmosnetwork) or open a new [issue](https://github.com/cometbft/cometbft/issues/new/choose) on Github.
319B
0