Elastic search_ Any idea about parsing the below mentioned data into fields in kibana using logstash config file? suggestion grok filter

Help
1

03:17:32.8372 Trace {“message”:“condition for user skipped before check invalid users”,“level”:“Verbose”,“logType”:“User”,“timeStamp”:“2019-10-23T03:17:32.8382364-05:00”,“fingerprint”:“50f2262a-03d5-49e2-bd66-f3903c23ca43”,“windowsIdentity”:“usedev\samplebot”,“machineName”:“USDbac9889”,“processName”:sampleFinalBots",“processVersion”:“1.0.6927.16110”,“jobId”:“5551849a-b749-46d9-b8e9-aaf668d99032”,“robotName”:“usedev\samplebot”,“machineId”:0}

2

Didn’t test it but something like this should work:

filter{
 dissect {   # first we split the 3 fields
     mapping => {
       "message" => "%{time} %{loglevel} %{jsonstring}"
     }
  }
 json {   # next we parse the JSON data
   source => "[jsonstring]"
   target => "data"
 }

}

2 Likes
3

Hi @virgilp,

I’m new to logstash and your answer has helped me a lot!!!
Thanks!!!

Do you have more ways to filter on the data? like a complete “.conf”?

For instance, I want to know how to create a field to capture the “runtime” of each run. Do you have any idea how to achieve it?

4

I’m not sure I understand what you try to achieve. Perhaps you can show me an example log line, and explain better what you mean by “create a field to capture the runtime of each run” ?

I can’t give a complete .conf since it’s highly dependent on environment & what you’re trying to achieve, but Logstash actually has pretty good documentation, and you have config examples here: Logstash configuration examples | Logstash Reference [8.4] | Elastic

5

Thanks for the help! I’ll read the document first.
Probably figure something out :slight_smile:

Just to explain what I meant there, in case you are wondering:

  • if a process has 5 segments and I want to time each segment, so when using Kibana, there could be a per-segment view on how much time it takes to process. And the success rate per-segment.
6

Not sure what are the “segments”, but if you organize them as 5 transactions, then each “Transactin End” log message is going to have a “transactionExecutionTime” (and other useful information, like whether if failed or succeeded). And you can use that to create a per-transaction view in Kibana. See fields here: Logging and Log Levels (make sure to expand the collapsed sections of interest/ look in the right-hand-side “table of contents”).

If you don’t use transactions, you can still apply the same principle using custom logs (just create a “segment start”/ “segment end” pair of events, and put the “segment duration” as a property on the “segment end” log).

1 Like