8000 crash on one of our instances -- source sequence is illegal/malformed utf-8 · Issue #82 · steelThread/redmon · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

crash on one of our instances -- source sequence is illegal/malformed utf-8 #82

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
SteveDevOps opened this issue Mar 3, 2016 · 4 comments
< 8000 div class="js-sticky js-sticky-offset-scroll top-0 gh-header-sticky">
Open

crash on one of our instances -- source sequence is illegal/malformed utf-8 #82

SteveDevOps opened this issue Mar 3, 2016 · 4 comments

Comments

@SteveDevOps
Copy link

any ideas? It loops through rescue, never resolves, and I have to manually stop it.

[16-03-03 13:28:05] listening on http#0.0.0.0:4567
[16-03-03 13:28:15] !!! Redmon has shit the bed, restarting... source sequence is illegal/malformed utf-8
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon/worker.rb:27:in `to_json'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon/worker.rb:27:in `stats'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon/worker.rb:13:in `record_stats'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon/worker.rb:7:in `block in run!'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/eventmachine-1.0.9.1/lib/em/timers.rb:56:in `fire'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/eventmachine-1.0.9.1/lib/eventmachine.rb:193:in `call'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/eventmachine-1.0.9.1/lib/eventmachine.rb:193:in `run_machine'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/eventmachine-1.0.9.1/lib/eventmachine.rb:193:in `run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:20:in `start_em'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:9:in `run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:15:in `rescue in run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:8:in `run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:15:in `rescue in run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:8:in `run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:15:in `rescue in run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:8:in `run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:15:in `rescue in run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:8:in `run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:15:in `rescue in run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:8:in `run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:15:in `rescue in run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:8:in `run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:15:in `rescue in run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:8:in `run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:15:in `rescue in run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:8:in `run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:15:in `rescue in run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:8:in `run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:15:in `rescue in run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:8:in `run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:15:in `rescue in run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:8:in `run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:15:in `rescue in run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:8:in `run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:15:in `rescue in run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:8:in `run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:15:in `rescue in run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:8:in `run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:15:in `rescue in run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:8:in `run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:15:in `rescue in run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:8:in `run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/bin/redmon:78:in `<top (required)>'^C
[root@evest-redis-1 redmon]# service redmon_4567 stop
@SteveDevOps SteveDevOps changed the title crash on one of our instances -- source sequence is illega/malformed utf-8 crash on one of our instances -- source sequence is illegal/malformed utf-8 Mar 3, 2016
@SteveDevOps
Copy link
Author

we traced through your code and found where it's trying to load stats json from the associated redis key which we identified as:

redmon:redis:<ip address>:6379:stats

We then attempted to clear it out from the redis-cli

del redmon:redis:<ip address>:6379:stats

and restart redmon, but ui fails to load and we re-enter this crash cycle. We are running redmon standalone, no integrations. Other things to try?

@SteveDevOps
Copy link
Author

I have been able to determine that the cause of the break in run is slowlog contents. There are some HMGETs included in the slowlog get and these end with (the following for example) :

835510:516818
839761:520340
843986:524044
654885:20391
... (995 more arguments)

which I believe is what is breaking the json convert. For the moment we are running without slowlog data but would like to know a workaround?

@steelThread
Copy link
Owner

@SteveDevOps - I haven't really touched this code for a couple of years. I can take a look this weekend and get back with you. Thanks for reporting this.

@SteveDevOps
Copy link
Author

Greatly appreciated.. let me know. Thx!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@steelThread @SteveDevOps
0