8000 bench: Initial wasm benchmarks by simar7 · Pull Request #790 · aquasecurity/tracee · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

bench: Initial wasm benchmarks #790

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 13 commits into from
Jun 30, 2021
Merged

bench: Initial wasm benchmarks #790

merged 13 commits into from
Jun 30, 2021

Conversation

simar7
Copy link
Member
@simar7 simar7 commented Jun 24, 2021 •
edited
Loading

image

➜  benchmark git:(wasm-benchmarks) ✗ go test -bench=. -benchmem -benchtime=100x  | prettybench -no-passthrough
benchmark                                     iter           time/iter     bytes alloc             allocs
---------                                     ----           ---------     -----------             ------
BenchmarkWasm_SimpleRule-8                     100      13646.00 ns/op       1684 B/op       30 allocs/op
BenchmarkRegoGoLibrary_SimpleRule-8            100      16221.00 ns/op       8040 B/op      123 allocs/op
BenchmarkWasm_RealRule-8                       100      30889.00 ns/op       1992 B/op       30 allocs/op
BenchmarkRegoGoLibrary_RealRule-8              100      40142.00 ns/op      17720 B/op      365 allocs/op
BenchmarkOnEventCodeInjectionRuleRego-8        100      40350.00 ns/op      18022 B/op      367 allocs/op
BenchmarkOnEventCodeInjectionRuleGo-8          100        814.90 ns/op        560 B/op        4 allocs/op
BenchmarkOnEventCodeInjectionRuleWASM-8        100      24217.00 ns/op       1912 B/op       30 allocs/op
BenchmarkEngineWithCodeInjectionRuleRego-8     100   13143482.00 ns/op    5483945 B/op   116421 allocs/op
BenchmarkEngineWithCodeInjectionRuleGo-8       100    1264845.00 ns/op     210097 B/op     2008 allocs/op
BenchmarkEngineWithCodeInjectionRuleWASM-8     100   30494157.00 ns/op    1725378 B/op    25678 allocs/op
BenchmarkEngineWithMultipleRulesRegoAndGo-8    100   16792141.00 ns/op   10575072 B/op   224289 allocs/op

Signed-off-by: Simar simar@linux.com

@simar7 simar7 marked this pull request as ready for review June 25, 2021 23:05
@simar7 simar7 requested a review from danielpacak June 25, 2021 23:05
danielpacak
danielpacak suggested changes Jun 28, 2021
View reviewed changes
Copy link
Contributor
@danielpacak danielpacak left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @simar7 I left a few comments but the main thing is to see if we can use opa.Target option instead of github.com/open-policy-agent/golang-opa-wasm module. By doing that we could benchmark same Rego rules with different targets (rego vs wasm)

simar7 added 11 commits June 29, 2021 16:17
@simar7
bench: Initial wasm benchmarks
b46bddc 
Signed-off-by: Simar <simar@linux.com>
@simar7
bench: Add WASM benchmark with Target option
2184cff 
Signed-off-by: Simar <simar@linux.com>
@simar7
engine: Revert channel closure
0110509 
Signed-off-by: Simar <simar@linux.com>
@simar7
engine: Revert debug changes
ac3cf56 
Signed-off-by: Simar <simar@linux.com>
@simar7
refactor: Cleanup wasm/signature code
a1d43c6 
Signed-off-by: Simar <simar@linux.com>
@simar7
bench: Integrate WASM benchmarks with the rest
50fccd0 
Signed-off-by: Simar <simar@linux.com>
@simar7
bench: Remove benchmark_reference_test.go in favour of benchmark_test.go
21b1b71 
Signed-off-by: Simar <simar@linux.com>
@simar7
refactor/bench: Refactor code with sub-benchmarks and a table driven …
ef39ef6 
…tests

Signed-off-by: Simar <simar@linux.com>
@simar7
cleanup: remove aio.rego
a15d70c
@simar7
rebase on main
f25430c 
Signed-off-by: Simar <simar@linux.com>
@simar7
bench: Rebase on main and add table driven for NSignatures
769a2f6 
Signed-off-by: Simar <simar@linux.com>
@simar7
Copy link
Member Author
simar7 commented Jun 30, 2021

I cleaned up and refactored tests as per feedback. Now we have table driven tests for each type along with subtests.

go test -tags=opa_wasm -bench=. -benchmem -benchtime=10x  | prettybench -no-passthrough


benchmark                                                     iter             time/iter      bytes alloc               allocs
---------                                                     ----             ---------      -----------               ------
BenchmarkOnEventCodeInjectionRule/rego-8                        10        49014.00 ns/op       18886 B/op        382 allocs/op
BenchmarkOnEventCodeInjectionRule/golang-8                      10          843.80 ns/op         784 B/op          6 allocs/op
BenchmarkOnEventCodeInjectionRule/wasm-8                        10       355581.00 ns/op       47793 B/op       1548 allocs/op
BenchmarkEngineWithCodeInjection/rego-8                         10     14332547.00 ns/op     5783922 B/op     122534 allocs/op
BenchmarkEngineWithCodeInjection/golang-8                       10      1386668.00 ns/op      229213 B/op       2027 allocs/op
BenchmarkEngineWithCodeInjection/wasm-8                         10    509183955.00 ns/op    49738114 B/op    1636090 allocs/op
BenchmarkEngineWithMultipleRules/rego_and_golang-8              10     13236959.00 ns/op     5903897 B/op     121787 allocs/op
BenchmarkEngineWithMultipleRules/wasm_and_golang-8              10    602044118.00 ns/op    50179214 B/op    1642625 allocs/op
BenchmarkEngineWithMultipleRules/rego_and_wasm-8                10    546562624.00 ns/op    55432076 B/op    1760155 allocs/op
BenchmarkEngineWithMultipleRules/rego_and_golang_and_wasm-8     10    682285338.00 ns/op    55946471 B/op    1765420 allocs/op
BenchmarkEngineWithNSignatures/rego/2Signatures-8               10     25229206.00 ns/op    11297500 B/op     239605 allocs/op
BenchmarkEngineWithNSignatures/rego/4Signatures-8               10     62644894.00 ns/op    22705860 B/op     481871 allocs/op
BenchmarkEngineWithNSignatures/rego/8Signatures-8               10    132168955.00 ns/op    45258380 B/op     961040 allocs/op
BenchmarkEngineWithNSignatures/rego/16Signatures-8              10    247037004.00 ns/op    90983470 B/op    1932369 allocs/op
BenchmarkEngineWithNSignatures/rego/32Signatures-8              10    668971862.00 ns/op   181030222 B/op    3843827 allocs/op
BenchmarkEngineWithNSignatures/rego/64Signatures-8              10   1306239333.00 ns/op   366165934 B/op    7768299 allocs/op
BenchmarkEngineWithNSignatures/rego/128Signatures-8             10   1933996468.00 ns/op   739737259 B/op   15714503 allocs/op
BenchmarkEngineWithNSignatures/golang/2Signatures-8             10      2367042.00 ns/op      414360 B/op       3951 allocs/op
BenchmarkEngineWithNSignatures/golang/4Signatures-8             10      4178486.00 ns/op      837276 B/op       8080 allocs/op
BenchmarkEngineWithNSignatures/golang/8Signatures-8             10      7220736.00 ns/op     1658461 B/op      15904 allocs/op
BenchmarkEngineWithNSignatures/golang/16Signatures-8            10     12930749.00 ns/op     3320600 B/op      32101 allocs/op
BenchmarkEngineWithNSignatures/golang/32Signatures-8            10     26639562.00 ns/op     6648820 B/op      64310 allocs/op
BenchmarkEngineWithNSignatures/golang/64Signatures-8            10     48356388.00 ns/op    13005915 B/op     126428 allocs/op
BenchmarkEngineWithNSignatures/golang/128Signatures-8           10    117156119.00 ns/op    26090901 B/op     253538 allocs/op
BenchmarkEngineWithNSignatures/wasm/2Signatures-8               10   1095853289.00 ns/op    98904970 B/op    3227186 allocs/op

@simar7
Copy link
Member Author
simar7 commented Jun 30, 2021

I also left some remarks on this PR: #796 regarding memory profile for WASM #796 (comment) and CPU profile #796 (comment)

@danielpacak
Go mod tidy
63052f1 
Signed-off-by: Daniel Pacak <pacak.daniel@gmail.com>
@danielpacak danielpacak merged commit 662a668 into aquasecurity:main Jun 30, 2021
@yanivagman yanivagman mentioned this pull request Jul 3, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@itaysk itaysk itaysk left review comments

@danielpacak danielpacak danielpacak approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@simar7 @itaysk @danielpacak
0