Anyone out there doing Kibana insights?

logs
kibana
elasticsearch

#1

Hi all,

I’ve just discovered the magnificence of Elastic Search + Kibana, so I have some big new ideas.

I’m wondering how many of you are already using “Log Message” to write structured bits of real-time data into your logs from which you can later glean metrics and insights? Is there a better approach?

I’d be curious to hear about some use cases and/or successes…


Ui path databases
#2

Hello,

I recommend you to have a look on the activity Add log field" and use those custom log fields to make clear segregations between your logs (ex: transaction status, exception type). Using them will be sligthly more robust than only considering messages only.

Cheers


#3

Thanks for the affirmation. That’s definitely in my plan.

I was just curious to know whether anyone is already doing this kind of thing… i.e. emitting JSON-type entities into your ElasticSearch logs that can later be interrogated via Kibana queries & aggregations… or even interrogated in real-time via API to make AI-type decisions during robot execution.


#4

Heys,

I’m currently using Kibana for just that. Within our automation processes we use the add log fields and log message activities to send specific bits of data to Kibana. From there we are creating dashboards and visualisations to send out to stakeholders.

Are you looking for anything specific?

Thanks


#5

Great to hear. Since I’m pretty new to ES & Kibana… I was hoping to identify some role models to spark ideas and help affirm my approach.

So… given that the most common ElasticSearch use case is to index NOSQL “documents” for the sake of providing search & insights in the context of an application… I’m wondering whether you are essentially indexing structured “documents” from your robots via Log Message + extra fields?

I know that log message will add a string into the log, and since I haven’t used the extra fields yet, I don’t know what kind of additional structure that introduces into the log message. I’m guessing the normal Log Message item is already JSON format behind the scenes. Do you have a sample of what that format looks like? I don’t have access to our ES instance yet to see it.

When we add new fields, are they just sibling elements next to the original log message field (within the same object), or are they child elements of the main log message element? I’ll be trying it myself soon… but if you had a quick sample of what a message with extra fields will look like, it would help me wrap my brain around it.

I’m trying to imagine whether I have any ability to create a hierarchical relationship (nested objects) using “add log fields”, or if I would need to create my own JSON objects in the standard “log message” call to accomplish that.

Before I learned about extra fields I was imagining that I might create my own JSON string (having multiple NVP) that describes a searchable entity… and then simply logging that.


#6

Heys,

So I’ve attached a preview of a test log message (some parts ommitted) of the way that it looks as a table and as json. On this you can see three fields, called argument1 ,2, and 3. These 3 fields have been added using the add log fields activity.

When you run a log message after the add log fields activity this will then push a new document to the elastic search database. The message field is the specific log message and then any add log fields are added as extra fields in that document. (The 3 arguments in this case)

When we are using this for data logging we make sure that the documents we want to count have a specific log message in and then filter for that message.

hope that makes sense.

Thanks


#7

Hey, I am working on a “best practices” document about logging and reporting with Kibana. It is following the same principles as what Jake describes above, and should contain all the information you need to know to squeeze the maximum out of logs/Kibana. Once we review it internally we want to make it public to all users.
Best


#8

@Sorin_Stania Looking forward to this document


#9

Kibana RPA how-to version 1.pdf (1.2 MB)

This is the Kibana document. I have also finished the logging best practices document, but we are still validating it internally.


#10

Thanks! Will check it out later today.


#11

Thank you


#12

Hi Sorin,

Have you finished the section regarding logging ? I would love to give it a read,

best regards,
Markus


#13

Hello - Is this available anywhere?


#14

Hey,

We are currently working on integrating this into the product to make it easier to use, and also into the ReFramework. Once this is integrated, most of the things mentioned in this document will happen automatically, in the background. Are you using the ReFramework by any chance ?


#15

Markus,

Please see the next reply. We have been working a lot on the methodology and integrating it into the product, so it’s still a moving target - basically I need to write a new version that is simplified in terms of implementation, though the principles are the same… so please stay tuned. You can use the ReFramework in the meantime; please ping me again in a few weeks, I might have some updates. Thanks for your patience !


#16

Not yet, but we have plans once we upgrade to 2018.X. Currently we are working on enhancing our reports using Kibana/Elastic Search. In that process, I’m trying to research the best ways the Robot can perform logging, so that the reporting can be easily accomplished.


#17

That was my goal as well