[WIP][App] LogSync - Load multiple log files and view side-by-side, synchronized by timestamps

Post your working scripts, libraries and tools
User avatar
evilC
Posts: 3685
Joined: 27 Feb 2014, 12:30

[WIP][App] LogSync - Load multiple log files and view side-by-side, synchronized by timestamps

20 Jan 2017, 18:16

My company writes some software that involves a server and a client, and part of my job is to debug issues in communication between them.
Hence lots of looking at logs, and trying to piece together the communication between the two, and CMTrace just wasn't doing it for me.
I had a google around, but turned up nothing, so I decided to see if I could hack something together.

Here is an initial proof-of-concept. There's a bunch of borrowed code - AutXYWH and some listview scrolling synch code I found and adapted, but as a start, it seems quite promising.

You will need two log files, the ones I used varied slightly in format:

Code: [Select all] [Download] GeSHi © Codebox Plus

20 Jan 2017 12:50:34.702 [17308] INFO  - Some Client v3.0.0.12
12:50:14.510 Some Server: v3.0.0.24 (Jan 19 2017)
2017-01-20 12:54:32,438 INFO Some other text


For now, it ignores the date part, so in the above example, the first example would just register as 12:50:34.702
Basically, as long as your log has HH:MM:SS.mil type timestamps, the logic should work, although the regex may need some tweaking

Image

Code: [Select all] [Expand] [Download] GeSHi © Codebox Plus

Last edited by evilC on 15 Nov 2017, 17:16, edited 10 times in total.
guest3456
Posts: 2120
Joined: 09 Oct 2013, 10:31

Re: [WIP][App] LogSync - Load two log files and synchronize the view

20 Jan 2017, 21:10

so the sync is based on the time of the log entry?

User avatar
evilC
Posts: 3685
Joined: 27 Feb 2014, 12:30

Re: [WIP][App] LogSync - Load two log files and synchronize the view

21 Jan 2017, 08:01

yup
It Looks at the first line of each log, if one is lower it adds it to the listview, and adds a blank line to the other listview.
Only if the timestamps are identical will you get something on the same row in both listviews.
User avatar
evilC
Posts: 3685
Joined: 27 Feb 2014, 12:30

Re: [WIP][App] LogSync - Load two log files and synchronize the view

21 Jan 2017, 09:29

Updated code.

1.0.1
Improved log sync logic, it now successfully syncs more logs.
Added Progress window when syncing
Added support for keyboard scrolling - arrow keys, pgup/dn, home etc.

I don't really understand what most of the code in ListScrolled() is doing, eg I am not really sure what startlogged is for, any ideas?
User avatar
SnowFlake
Posts: 365
Joined: 28 Apr 2015, 05:41
Google: floowsnaake
GitHub: floowsnaake
Contact:

Re: [WIP][App] LogSync - Load two log files and synchronize the view

21 Jan 2017, 10:59

pretty cool :D
:yawn:
User avatar
evilC
Posts: 3685
Joined: 27 Feb 2014, 12:30

Re: [WIP][App] LogSync - Load two log files and synchronize the view

21 Jan 2017, 12:12

Would love to expand this to be able to support more than two logs, the app I am working on has 7 really important ones, and to be able to view the communication between all the services would be awesome.

The hurdles to doing this that I see are:
Implementing a dynamic gui (Allow adding of new ListViews). I have done it in AHK_H, but am not sure if AutoXYWH will cope with guis in guis etc.
Synchronizing that many listviews (I think the code I have will handle it, the code it derived from used 3 listviews).

If possible, I would also like to implement tailing of live logs.

Searching will also need to be implemented, and I would also like to implement line highlighting based upon regexes. (Highlight line this color if regex matches).

If anyone has any suggestions, I would love to hear them.
Helgef
Posts: 2587
Joined: 17 Jul 2016, 01:02
Contact:

Re: [WIP][App] LogSync - Load two log files and synchronize the view

21 Jan 2017, 13:28

Looks good.
evilC wrote:If anyone has any suggestions, I would love to hear them.

Maybe you could use only one listview? Add a column for each log. It will save space when syncing many logs. You'll need only one Column for timestamps, and only one scroll bar.
log.png
log.png (50.7 KiB) Viewed 874 times
guest3456
Posts: 2120
Joined: 09 Oct 2013, 10:31

Re: [WIP][App] LogSync - Load two log files and synchronize the view

21 Jan 2017, 14:03

evilC wrote:Implementing a dynamic gui (Allow adding of new ListViews). I have done it in AHK_H, but am not sure if AutoXYWH will cope with guis in guis etc.


why are you adamanat about maintaining AHK_L compatibility? why not just make the script for AHK_H only? plenty of scripts were AHK_L only, back in the day

User avatar
evilC
Posts: 3685
Joined: 27 Feb 2014, 12:30

Re: [WIP][App] LogSync - Load two log files and synchronize the view

21 Jan 2017, 19:01

Helgef wrote:Maybe you could use only one listview? Add a column for each log. It will save space when syncing many logs. You'll need only one Column for timestamps, and only one scroll bar.
log.png

Hmm, interesting idea. takes all the complexity of that part out.
I am not 100% sure it would work though, as AFAIK in a ListView, all lines must be the same height, so you cannot have wrapping.
This may present an issue without a horizontal scrollbar for each log.

guest3456 wrote:
evilC wrote:Implementing a dynamic gui (Allow adding of new ListViews). I have done it in AHK_H, but am not sure if AutoXYWH will cope with guis in guis etc.


why are you adamanat about maintaining AHK_L compatibility? why not just make the script for AHK_H only? plenty of scripts were AHK_L only, back in the day

Not adamant, but I am aware that most people don't use it, and if there are other alternatives that do not add too much complexity, I would like to take them.
Helgef
Posts: 2587
Joined: 17 Jul 2016, 01:02
Contact:

Re: [WIP][App] LogSync - Load two log files and synchronize the view

22 Jan 2017, 08:03

evilC wrote:This may present an issue without a horizontal scrollbar for each log.

You can drag the column width, and double click the edge to expand to fit content.


Just some other random ideas, another way to orginise the view, add a listview below to display the (in this case double-clicked) selected row.
2017-01-22_13-54-40.gif
2017-01-22_13-54-40.gif (696.93 KiB) Viewed 810 times
User avatar
evilC
Posts: 3685
Joined: 27 Feb 2014, 12:30

Re: [WIP][App] LogSync - Load two log files and view side-by-side, synchronized by timestamps

22 Jan 2017, 10:12

I couldn't be bothered to debug what ListScrolled was doing, so I sat down and coded my own version.

Seems like a much simpler way of doing things, although it tries to scroll with every interaction. I just need to have it cache the positions and only scroll if it needs to.
Edit: It now only issues scrolls when it needs to.

Code: [Select all] [Expand] [Download] GeSHi © Codebox Plus

User avatar
evilC
Posts: 3685
Joined: 27 Feb 2014, 12:30

Re: [WIP][App] LogSync - Load two log files and view side-by-side, synchronized by timestamps

22 Jan 2017, 12:02

So I got to thinking about the fundamental way the list synchronization was handled - basically flags would be set to indicate the "Current LV", and any scroll messages coming in from other LVs get ignored.
The problem arises when the GUI and lv 1 already has focus, and you drag the scrollbar of lv2.
To the existing code, this was indistinguishable from it seeing lv2 scrolled programatically as a result of the user moving lv1.

So I tried a more elegant approach - when you programatically scroll a lv, turn off it's g-label, do the scroll, then turn the g-label back on.

Also, I wrapped the whole thing up into a class, and now you can have any number of listviews.
Plus, when you select a line in one LV, it becomes selected in all LVs

Image

Code: [Select all] [Expand] [Download] GeSHi © Codebox Plus

User avatar
evilC
Posts: 3685
Joined: 27 Feb 2014, 12:30

Re: [WIP][App] LogSync - Load two log files and view side-by-side, synchronized by timestamps

23 Jan 2017, 10:45

Aww yeah, I just figured out how to deal with logs missing dates (ie having only HH:MM:SS timestamps).
Scan the first line of each log, if it is missing a date, get the file creation time. If the time portion matches the timestamp of the first line of the log (it usually does), then you can be sure that the date portion of the file creation time is correct.
Then, iterate thru each line of the log, adding in the date. If the time of the current line is lower than the time of the previous line, then the day has ticked over, so add 1 day to the date we got from the creation time.

So, given a log file, we can generally work out the date portion for every line of the log, without any other files.
The above system would only fail for a log covering 3 or more days where you had nothing logged for a whole day.
This is not gonna happen in my use-case, so screw everyone else :P
User avatar
evilC
Posts: 3685
Joined: 27 Feb 2014, 12:30

Re: [WIP][App] LogSync - Load two log files and view side-by-side, synchronized by timestamps

26 Jan 2017, 17:56

So I decided to set the bar higher.
Why only 2 logs, when you can have as many as you want?

Here is a POC for the new version. It is AHK_H ONLY. It may well stay that way also.
Be aware that for the moment, you must call LogSync.AddParser once for each of the filenames you might load, and tell it which parser to use.
This will be guified at some point.
Parsers are classes that read the log, chunk it into lines, extract the date and ensure that it is in the correct format.
Three parsers are provided. Samples of the kind of log they can process are commented next to them.

For now, the time column is left in the raw format used for processing. I am thinking of adding options to use the original string, or convert it to some human readable - Useful eg if you want to convert all the timestamps to a localized, consistent format.
I am thinking of maybe event adding time parser classes or something so you can customize that part easily too.

At some point, parsers will probably be dynamically included from a plugins folder, like I do in UCR. This will probably mean that this version will remain AHK_H only.
However, in this case I can just package it as an EXE, so non AHK_H users can still use it. Everything will remain uncompiled, I will just ship a renamed AHK_H Autohotkey.exe with it.

Image

Code: [Select all] [Expand] [Download] GeSHi © Codebox Plus

elmo
Posts: 54
Joined: 09 Oct 2013, 09:08

Re: [WIP][App] LogSync - Load two log files and view side-by-side, synchronized by timestamps

09 Nov 2017, 08:10

Hello EvilC,

This looks quite valuable. Am I reading correctly that it will not run under Autohotkey_L ?

May I trouble you for some insight as to the reason ?

I have been pondering Autohotkey_H but it seems confusing to implement; this might be the reason to try installing it again.

Thank you in advance for your time and attention.
User avatar
evilC
Posts: 3685
Joined: 27 Feb 2014, 12:30

Re: [WIP][App] LogSync - Load two log files and view side-by-side, synchronized by timestamps

09 Nov 2017, 08:47

The ListViews in the GUI are dynamically resizable.
AHK_H has built-in support for dynamically sizing GuiControls, AHK_L does not.
It could be done in AHK_L, but you would need the likes of AutoXYWH or Anchor
elmo
Posts: 54
Joined: 09 Oct 2013, 09:08

Re: [WIP][App] LogSync - Load two log files and view side-by-side, synchronized by timestamps

09 Nov 2017, 10:20

So appreciate the timely response.
I understand and that makes sense.
To date AutoXYWH has worked nicely and no familiarity with Anchor.
Hmmm, while I have you on the line, is there a definitive AHK_H deployment guide that you could point me to?
For some reason "grokking" the sequence to switch over, especially if one wishes to preserve their code, has eluded me.
If not, that is cool; thank you again for your response.
User avatar
evilC
Posts: 3685
Joined: 27 Feb 2014, 12:30

Re: [WIP][App] LogSync - Load two log files and view side-by-side, synchronized by timestamps

09 Nov 2017, 10:49

All you need to do to "Install" AHK_H is to replace AutoHotkey.exe with the appropriate one from the AHK_H zip, and drop msvcr100.dll into your AHK folder.

You can also use AHK-EXE-Swapper

Drop the AHK_H zip into the import folder
Click Refresh
Select the zip from the bottom pane
Click Import
Select the AHK_H version from the upper list
Click "Replace with selected version"
elmo
Posts: 54
Joined: 09 Oct 2013, 09:08

Re: [WIP][App] LogSync - Load two log files and view side-by-side, synchronized by timestamps

09 Nov 2017, 17:00

jumpin jehosephat !!!
must have been kharmatic thang that caused me to ask the question about AHK_H deployment.
perhaps if i can get through this initial switch then i will be able to press forward with re-compiling with a different password.
thank you so much for your guidance.

Return to “Scripts and Functions”

Who is online

Users browsing this forum: No registered users and 9 guests