KS2011: Structured error logging

Posted Oct 25, 2011 7:08 UTC (Tue) by Cyberax (✭ supporter ✭, #52523)
In reply to: KS2011: Structured error logging by dlang
Parent article: KS2011: Structured error logging

So, let's see - a record from our code:
============
[Sat 26 Oct 2011] "Joe User" requested lmpp://myserver.dc=some.dc=com/service , "SUCCESS" has been returned.
============

How should we parse it? Well, let's start with the date. It can be parsed by regexps, but it's already some amount of code.

Then there's user name. It can't be parsed by regexps at all (because of quoting, for example "Joe \"the mad\" User"). Then there's URL, which also can not be reliably parsed by regexps. And then finally the exit code which luckily is just a pre-defined string.

It's _really_ _really_ easy to make log unparseable accidentally. And given that quite a lot of log messages are printed only during exceptional/error conditions you might not discover it until it's too late.

So something which just CAN NOT be misused is sorely needed.

KS2011: Structured error logging

Posted Oct 25, 2011 7:50 UTC (Tue) by dlang (guest, #313) [Link]

you can also botch rules for any structured format that you make as well

the fact that your program outputs bogus logs now is a problem with your program, not with the logging protocol.

KS2011: Structured error logging

Posted Oct 25, 2011 7:56 UTC (Tue) by liljencrantz (guest, #28458) [Link] (4 responses)

Minor nit: You can easily parse quoted strings using something like:

"([^"]|\\.)*"

This will work since regexps choose the longest mathing string. Or am I missing something?

KS2011: Structured error logging

Posted Oct 25, 2011 8:21 UTC (Tue) by l0b0 (guest, #80670) [Link] (3 responses)

You also need to account for the fact that you might have an even or odd number of backslashes before the quote:

echo '"foo \"bar\" baz"' | grep -E '"([^"]|\\.)*"' # Succeeds
echo '"foo \"bar\\" baz"' | grep -E '"([^"]|\\.)*"' # Ouch, that's a literal backslash, not an escaped quote!

To fix it, we would need to check that any quotes are preceded by an *odd* number of backslashes: "([^"]|(?<=\\(\\\\)*)")*" Unfortunately this doesn't work with grep -P ("lookbehind assertion is not fixed length"). I don't know if any other regex engines support this.

KS2011: Structured error logging

Posted Oct 25, 2011 8:53 UTC (Tue) by iq-0 (subscriber, #36655) [Link] (2 responses)

Try this one: "(\\\\|\\[^\\]|[^\\"])*"

KS2011: Structured error logging

Posted Oct 25, 2011 9:35 UTC (Tue) by liljencrantz (guest, #28458) [Link] (1 responses)

Or "(\\.|[^\\"])*"

KS2011: Structured error logging

Posted Oct 25, 2011 10:02 UTC (Tue) by nix (subscriber, #2304) [Link]

This I think is proof that you need a proper parser rather than just regex matching. Regexps are not a parser, they are (the core of) a tokenizer.