perf: timestampCache causes GC-related slowdown #16796
Comments
|
Try zeroing the How much of the Go memory was used by the timestamp cache in these tests? It's usually the biggest single consumer of memory on the go side, in which case it might not be surprising that eliminating it cuts GC costs nearly in half. If that's the case, our options are limited. We could allocate a big slab of memory, copy the keys into it, and store indexes into that slab instead of the pointers in |
"zero all" corresponds to Go memory was in the 300-600MB range during these tests. The timestamp cache was at capacity (i.e. using the full 64 MB), but it is a bit difficult to tell how much Go memory was being used by the timestamp cache. Presumably somewhat more than 64 MB due to the btree and other overhead that isn't accounted for. Moving the cache to C++ would get rid of the GC issues, but introduce Cgo overhead. |
|
As a sanity check about the feasibility of moving the timestamp cache to C++, |
Running a read-only workload on a single node cluster shows performance degradation from ~19k ops/sec to a steady state of ~13.5k ops/sec. Investigation eventually pointed to the
timestampCache. Specifically, iftimestampCache.AddRequestis disabled, steady state performance is ~18k ops/sec. The workload under investigation is:Interestingly, the manipulation of the
timestampCache.requestsbtree byAddRequestdoesn't seem to be the problem. IfAddRequestis tweaked to add and immediately delete the request, performance is good.Part of the problem seems to be the btree. Replacing the btree with a fixed size ring buffer of approximately the same size results in steady state performance of ~15k ops/sec. The ring buffer experiment isn't a drop in replacement. More work would be required to flesh out its functionality and that additional functionality might end up negating the modest improvement it is providing.
Another experiment was to zero the
cacheRequestbefore inserting it. This brought steady state performance back up to ~17k ops/sec.My current suspicion is that the non-zeroed
cacheRequestsare causing additional GC pressure. EnablingGODEBUG=gctrace=1shows:Note that there are two fields in
cacheRequestthat zeroing affects with this workload:cacheRequest.spanandcacheRequest.reads. I'm not sure what to do here. We need the spans in order to later expand the requests into the interval tree. It is surprising to me that merely holding on to these keys is causing additional GC CPU usage. Suggestions are welcome.The text was updated successfully, but these errors were encountered: