Typically, log files contain information that follow a key-value pair structure. The keyvalue operator allows you to get values from a log message by specifying the key paired with each value.
For example, a log could contain the following keys (highlighted):
From that log message, you can use the keyvalue operator to get the values for one or more keys. For example, if you'd like to see information just about the "remote_ip" value, running this query:
... | keyvalue "remote_ip" ...
would produce these results:
The keyvalue operator can also be used in two explicit modes:
- Default inference mode. The keyvalue operator uses an internal list of regular expressions to determine how to extract the value for a given key
- Regular Expression mode. You explicitly match keys and values based on a regular expression
Inference mode syntax
When used in the default inference mode, the keyvalue operator uses an internal list of regular expressions to determine how to extract the value for a given key. This greatly simplifies the syntax.
For example, you could extract the keys "module" and "thread" and their values from a log message by running this query:
* | keyvalue infer "module", "thread"
to produce these results:
Regular Expression mode syntax
In Regular Expression mode, you must explicitly match keys and values based on a regular expression. This allows for greater flexibility than inference mode. For example, to extract the values for the keys "serviceinfo_IP", "loggingcontext.region", and "request.method" fields from a log message, use this query:
* | keyvalue regex "=(.*?)[,|}]" keys "
.region", "request.method" as
, region, method
This provides the following results:
The keyvalue operator also supports regular expressions that contain a single match group. (You may notice an improvement in performance by running queries with a single match group.) For example, you could run this query to get the same results as the previous query:
regex "=(.*?)," "
In the above case, the operator first finds the key itself in the message (first occurrence), and then finds the closest match of the regular expression to the location in the message where the key was found.
The number of fields specified with the "as" clause must match the number of key-value pairs specified. You can omit the clause if you'd like the operator to automatically create the field names for the extracted values. To do this, keyvalue replaces every character (other than a..z, A..Z, 0..9, or _) with an underscore (_).
The keyvalue operator can be abbreviated in either mode (Inference or Regular Expression). For example, running this query:
* | keyvalue infer keys "a", "b"
will produce the same results as running this query:
* | keyvalue "a", "b"
Also, keyvalue can be abbreviated to "kv". For example:
If you are using the keyvalue operator with delim, "keys" must come before "delim" in the query, or you will get an error.
Auto extracting key-value pairs
The keyvalue operator supports an optional auto mode if there are no flags.
* | keyvalue auto ...keyvalue auto ...
With this mode enabled, the operator can extract up to N non-referenced fields (fields that are not explicitly included in the form "keys foo,bar,..."). N is 100 by default. If the message includes more than N key value pairs, the operator extracts the first N key value pairs from left to right in the message. If there are duplicates, the last occurrence is extracted.
You can separate the key and value in a key-value pair using any of these delimiters:
: = -> =>
If you want to be able to use the keys later in the query, they must be referred to specifically.
Aliases (renaming) using as are supported. For example:
* | keyvalue auto keys "key1", "key2" as alias1, alias2
Refonly extracts only referenced keys. If this option is not used, keyvalue auto extracts all other fields it finds in the message.
* | kv auto keys "key1", "key2" refonly
"my_key" => 'my value with spaces'
'my key' -> "my value"
'\"#$&@_--34234\"" -> '\'@#@!$!*\' '