Given a list of posts, compute the top 5 related posts for each post based on the number of shared tags.
- Read the posts JSON file.
- Iterate over the posts and populate a map containing:
tag -> List<int>, with the int representing the post index of each post with that tag. - Iterate over the posts and for each post:
- Create a map:
PostIndex -> intto track the number of shared tags - For each tag, Iterate over the posts that have that tag
- For each post, increment the shared tag count in the map.
- Create a map:
- Sort the related posts by the number of shared tags.
- Write the top 5 related posts for each post to a new JSON file.
./run.sh go | rust | python | all
# windows (powershell)
./run.ps1 go | rust | python | all
or
pwsh ./run.ps1 go | rust | python | all
Rules
- FFI (including assembly inlining)
- Unsafe code blocks
- Custom benchmarking
- Disabling runtime checks (bounds etc)
- Specific hardware targeting
- Parse json at runtime
- Not hardcode number of posts
- Support up to 100 tags
- Use a stable release of the compiler/runtime
- Represent tags as strings
Updated Results from github workflow (raw data)
VM Specs
NB: The benchmark runs on the free tier of github workflow.- CPU: 2 vCPUs
- RAM: 7GB
- OS: Ubuntu 22.04
| Language | Processing Time | Total (PT + I/O) |
|---|---|---|
| Go | 26.62 ms | 57.2 ms |
| Rust | 30.95 ms | 48.7 ms |
| Java (GraalVM) | 36.38 ms | 69.6 ms |
| Zig | 38.00 ms | 79.0 ms |
| Nim | 39.00 ms | 67.1 ms |
| Julia | 48.60 ms | 3.337 s |
| F# | 50.47 ms | 304.0 ms |
| Odin | 53.73 ms | 302.3 ms |
| Vlang | 59.03 ms | 396.9 ms |
| Swift | 65.70 ms | 442.9 ms |
| Crystal | 68.44 ms | 124.8 ms |
| C# | 81.07 ms | 147.1 ms |
| Dart VM | 109.08 ms | 579.9 ms |
| LuaJIT | 117.53 ms | 388.4 ms |
| Dart AOT | 140.62 ms | 283.9 ms |
| JS (Node) | 193.54 ms | 290.4 ms |
| JS (Deno) | 218.46 ms | 292.5 ms |
| Java (JIT) | 258.77 ms | 542.3 ms |
| Numpy | 372.44 ms | 589.8 ms |
| JS (Bun) | 738.92 ms | 780.7 ms |
| Python | 1.70 s | 1.769 s |
| Lua | 2365.36 ms | 3.025 s |
| Language | Processing Time | Total (PT + I/O) |
|---|---|---|
| Rust Concurrent | 15.70 ms | 32.9 ms |
| Go Concurrent | 21.63 ms | 54.4 ms |
| F# Concurrent | 40.53 ms | 859.2 ms |
| Swift Concurrent | 41.05 ms | 429.8 ms |
Old Results with details (on my machine)
| Language | Processing Time | Total (+ I/O) | Details |
|---|---|---|---|
| Rust | - | 4.5s | Initial |
| Rust v2 | - | 2.60s | Replace std HashMap with fxHashMap by phazer99 |
| Rust v3 | - | 1.28s | Preallocate and reuse map and unstable sort by vdrmn and Darksonn |
| Rust v4 | - | 0.13s | Use Post index as key instead of Pointer and Binary Heap by RB5009 |
| Rust v5 | 38ms | 52ms | Rm hashing from loop and use vec[count] instead of map[index]count by RB5009 |
| Rust v6 | 23ms | 36ms | Optimized Binary Heap Ops by scottlamb |
| Rust Rayon | 9ms | 22ms | Parallelize by masmullin2000 |
| Rust Rayon | 8ms | 22ms | Remove comparison out of hot loop |
| ⠀ | ⠀ | ⠀ | ⠀ |
| Go | - | 1.5s | Initial |
| Go v2 | - | 80ms | Add rust optimizations |
| Go v3 | 56ms | 70ms | Use goccy/go-json |
| Go v3 | 34ms | 55ms | Use generic binaryheap by DrBlury |
| Go v4 | 26ms | 50ms | Replace binary heap with custom priority queue |
| Go v5 | 20ms | 43ms | Remove comparison out of hot loop |
| Go Con | 10ms | 33ms | Go concurrency by tirprox and DrBlury |
| Go Con v2 | 5ms | 29ms | Use arena, use waitgroup, rm binheap by DrBlury |
| ⠀ | ⠀ | ⠀ | ⠀ |
| Python | - | 7.81s | Initial |
| Python v2 | 1.35s | 1.53s | Add rust optimizations by dave-andersen |
| Numpy | 0.57s | 0.85s | Numpy implementation by Copper280z |
| ⠀ | ⠀ | ⠀ | ⠀ |
| Crystal | 50ms | 96ms | Inital w/ previous optimizations |
| Crystal v2 | 33ms | 72ms | Replace binary heap with custom priority queue |
| ⠀ | ⠀ | ⠀ | ⠀ |
| Odin | 110ms | 397ms | Ported from golang code |
| Odin v2 | 104ms | 404ms | Remove comparison out of hot loop |
| ⠀ | ⠀ | ⠀ | ⠀ |
| Dart VM | 125ms | 530ms | Ported frog golang code |
| Dart bin | 274ms | 360ms | Compiled executable |
| ⠀ | ⠀ | ⠀ | ⠀ |
| Vlang | 339ms | 560ms | Ported from golang code |
| ⠀ | ⠀ | ⠀ | ⠀ |
| Zig | 80ms | 110ms | Provided by akhildevelops |