Open
Description
Hello! I wrote a prehook&posthook addon for karton consumers to automatically exit (to cause restart in docker / systemctl) if process uses too much RAM.
https://github.com/rakovskij-stanislav/karton_memory_watcher
I noticed that some forensic libraries can cause a slow memory leak (in my case ~500 MB for 2-3 month), so the restart is required. To make it softly (for not losing current task) I am making a decicion of restarting in post-hook logic.
It has 4 configurable rules for restart:
proceed_tasks: count of tasks to proceed for restart (if you know that some dependency will make mistakes out of nowhere if it proceeded, for example, 500 files;elapsed_time: how many seconds should pass till the start of first job (if you know that there is some time-based issues in your dependencies)extra_consumed_memory_percent: extra memory used in % (100% means twice of memory compared to point before starting first task). To deal with memory leaks.extra_consumed_megabytes: extra memory used in megabytes (e.g. your service at start uses 60MB and you need to kill it if it consumes extra 500+ MB). Also to deal with memory leaks.
Usage is simple:
from karton.core import Consumer
from karton.memory_watcher import implant_watcher, RestartBehavior, RestartRule
...
class YourConsumer(Consumer):
pass
if __name__ == "__main__":
foobar = YourConsumer()
implant_watcher(
foobar,
RestartRule(
extra_consumed_memory_percent=80,
# call_before_exit=(close_db_connections, )
)
)
foobar.loop()
I hope this post-hook will help in long-term runs :)
Metadata
Metadata
Assignees
Labels
No labels