Skip to main content
Sumo Logic

Parse Keyvalue Formatted Logs

Typically, log files contain information that follow a key-value pair structure. The keyvalue operator allows you to get values from a log message by specifying the key paired with each value.

For example, a log could contain some the following keys (highlighted):

From that log message, you can use the keyvalue operator to get the values for one or more keys. For example, if you'd like to see information just about the "remote_ip" value, running this query:

... | keyvalue "remote_ip" ...

would produce these results:

The keyvalue operator can also be used in two explicit modes: the default inference mode, and Regular Expression mode.

 

Inference mode syntax

When used in the default inference mode, the keyvalue operator uses an internal list of regular expressions to determine how to extract the value for a given key. This greatly simplifies the syntax.

For example, you could extract the keys "module" and "thread" and their values from a log message by running this query:

* | keyvalue infer "module", "thread"

to produce these results:

Regular Expression mode syntax

In Regular Expression mode, you must explicitly match keys and values based on a regular expression. This allows for greater flexibility than inference mode. For example, to extract the values for the keys "serviceinfo_IP", "loggingcontext.region", and "request.method" fields from a log message, use this query:

* | keyvalue regex "=(.*?)[,|}]" keys "serviceinfo.IP", "loggingcontext.region", "request.method" as ip, region, method

This provides the following results:

The keyvalue operator also supports regular expressions that contain a single match group. (You may notice an improvement in performance by running queries with a single match group.) For example, you could run this query to get the same results as the previous query:

* | kv regex "=(.*?)," "serviceinfo.IP", "loggingcontext.region", "request.method"

In the above case, the operator first finds the key itself in the message (first occurrence), and then finds the closest match of the regular expression to the location in the message where the key was found.

The number of fields specified with the "as" clause must match the number of key-value pairs specified. You can omit the clause if you'd like the operator to automatically create the field names for the extracted values. To do this, keyvalue replaces every character (other than a..zm A..Z, 0..9, or _) with an underscore (_).

Abbreviated syntax

The keyvalue operator can be abbreviated in either mode (Inference or Regular Expression). For example, running this query:

* | keyvalue infer keys "a", "b"keyvalue infer keys "a", "b"

will produce the same results as running this query:

* | keyvalue "a", "b"keyvalue "a", "b"

Also, keyvalue can be abbreviated to "kv". For example:

* | kv "a", "b"

If you are using the keyvalue operator with delim, "keys" must come before "delim" in the query, or you will get an error.keyvalue operator with delim, "keys" must come before "delim" in the query, or you will get an error.

Auto extracting key-value pairs

The keyvalue operator supports an optional auto mode if there are no flags.

* | keyvalue auto ...keyvalue auto ...

With this mode enabled, the operator can extract up to N non-referenced fields (fields that are not explicitly included in the form "keys foo,bar,..."). N is 100 by default. If the message includes more than N key value pairs, the operator extracts the first N key value pairs from left to right in the message. If there are duplicates, the last occurrence is extracted.

You can separate the key and value in a key-value pair using any of these delimiters:

: = -> =>

If you want to be able to use the keys later in the query, they must be referred to specifically.

Additional options

Aliases (renaming) using as are supported. For example:

* | keyvalue auto keys "key1", "key2" as alias1, alias2keyvalue auto keys "key1", "key2" as alias1, alias2

Refonly extracts only referenced keys. If this option is not used, keyvalue auto extracts all other fields it finds in the message.

* | kv auto keys "key1", "key2" refonly

For example:

"foo": "bar"

"my_key" => 'my value with spaces'

'my key' -> "my value"

'\"#$&@_--34234\"" -> '\'@#@!$!*\' '