remove simd, improves perf on arm64 cpu by x2 #11
Conversation
|
Good morning. Thank you for the feedback. Does this happen on the 8-bit quantization models too? On x86, SIMD helps improve performance significantly on Q8. Anyways, Q4 is still being improved and I will test today on my x86 machine if I get the same results. |
|
Good question on Q8, I'll see if it makes a similar difference 👍 |
|
On Q8 it gives a smaller improvement. 2.90 t/s -> 3.70 t/s. Edit2: Doing the same for matmul f32 slows it down. 4.2 t/s -> 2.8 t/s using llama 3.2 1B model. |
|
Without having other people testing on arm64 systems it's hard to say if it's justified adding a target flag for the Q4 and Q8 matmuls. The reason I used the wide crate was that it falls back to normal instructions even when SIMD isn't available (which shouldn't be your case), so I'm not sure what's causing the slowdown. |
|
Just to be sure, are you using the |
|
I wasn't, however rerunning all of the above on |
On Arm64 cpu, removing the SIMD in matmul allows the compiler to o 8000 ptimise it for arm64. The performance in my experiments goes from around ~2 t/s to ~4.5 t/s.
I'm not sure if this has any performance implications for running on x86, as I don't have one to test on. If it does make it slower, than I'm happy to put this version behind a target flag for arm64 e.g. https://docs.rs/core_arch/latest/core_arch/#static-cpu-feature-detection
Before
After