Description
SUMMARY
When writing out structured log data using the GelfFormatter, periodically the field _content_length is output with a value of: "_content_length": Infinity. This is invalid JSON and causes an error when parsing from external logging tools such as FluentD
ISSUE TYPE
- Bug Report
STACKSTORM VERSION
st2 3.0.1, on Python 2.7.5OS / ENVIRONMENT / INSTALL METHOD
OS = Red Hat 7.6
Install = puppet-st2
STEPS TO REPRODUCE
Configure GELF logging as described here: https://stackstorm.com/2017/08/22/stackstorm-centralized-logging-graylog/
Monitor the /var/log/st2/st2stream.gelf.log and periodically lines will be written out with "_content_length": Infinity"
st2stream.gelf.log:{"_path": "/v1/stream", "version": "1.1", "level": 6, "timestamp": 1561477579, "_content_length": Infinity, "_python": {"name": "st2.st2common.middleware.logging", "process": 81491, "module": "logging", "funcName": "__call__", "processName": "MainProcess", "lineno": 118, "filename": "logging.py"}, "_request_id": "2e85dac9-1cb2-46d0-969a-2c93efba10fa", "timestamp_f": 1561477579.465585, "host": "stackstorm.domain.tld", "full_message": "2e85dac9-1cb2-46d0-969a-2c93efba10fa - 200 inf 15.51ms", "_remote_addr": "127.0.0.1", "_runtime": 15.51, "_status": 200, "_method": "GET", "short_message": "2e85dac9-1cb2-46d0-969a-2c93efba10fa - 200 inf 15.51ms"}
st2stream.gelf.log:{"_path": "/v1/stream", "version": "1.1", "level": 6, "timestamp": 1561479377, "_content_length": Infinity, "_python": {"name": "st2.st2common.middleware.logging", "process": 81491, "module": "logging", "funcName": "__call__", "processName": "MainProcess", "lineno": 118, "filename": "logging.py"}, "_request_id": "46199a0b-c917-4221-b826-4380e919fd5c", "timestamp_f": 1561479377.730839, "host": "stackstorm.domain.tld", "full_message": "46199a0b-c917-4221-b826-4380e919fd5c - 200 inf 12.296ms", "_remote_addr": "127.0.0.1", "_runtime": 12.296, "_status": 200, "_method": "GET", "short_message": "46199a0b-c917-4221-b826-4380e919fd5c - 200 inf 12.296ms"}
st2stream.gelf.log:{"_path": "/v1/stream", "version": "1.1", "level": 6, "timestamp": 1561481182, "_content_length": Infinity, "_python": {"name": "st2.st2common.middleware.logging", "process": 81491, "module": "logging", "funcName": "__call__", "processName": "MainProcess", "lineno": 118, "filename": "logging.py"}, "_request_id": "60bdfd46-c49a-481e-a591-dc6a5ad7eeff", "timestamp_f": 1561481182.687065, "host": "stackstorm.domain.tld", "full_message": "60bdfd46-c49a-481e-a591-dc6a5ad7eeff - 200 inf 14.903ms", "_remote_addr": "127.0.0.1", "_runtime": 14.903, "_status": 200, "_method": "GET", "short_message": "60bdfd46-c49a-481e-a591-dc6a5ad7eeff - 200 inf 14.903ms"}EXPECTED RESULTS
Valid JSON be written to these files so that it can be parsed from an external tool
ANALYSIS
The only place in the code where i can find content_length being set to Infinity is here: https://github.com/StackStorm/st2/blob/master/st2common/st2common/middleware/logging.py#L102