UIA Automation - Inspect.exe for AHK

Post your working scripts, libraries and tools
sancarn
Posts: 211
Joined: 01 Mar 2016, 14:52

UIA Automation - Inspect.exe for AHK

05 Mar 2017, 19:09

A bit of history

So I first came across UIA while looking for better ways to automate the user interfaces of other applications which were not based off Win32 forms. The thread I found was this one. The thread offers a method of finding text anywhere without the use of an OCR. Or rather, it can "see text" in many other places other than just Win32 forms like the AHK_Spy for example. After testing it on some non-win32 forms I found that it could extract information from these UIs that no other AHK function could. This peaked my interest as you can imagine.

In the thread Jethrow and nepter develop a IUIAutomation Interface. And I knew that when i was automating software of this kind I should refer to this library... A few months passed and here I am back looking at it. I've been extending an application that Nepter created - A GUI Cloner a lot like the Windows Inspect.exe which is provided in the Windows SDK. When I first adopted the application it was incredibly compact, but very messy and difficult to understand. It also lacked functionality I wanted. So I began to extend it.

Inspect.ahk

You can find the Inspect.ahk here:

https://github.com/sancarn/Inspect.exe_AHK

The original made by nepter can be found here.

Download both inspect.ahk and uia.ahk

To inspect a Window press {F8}.

Possible features for the future:
- Sample code generator for targeting specified element with the library.
- UIA TreeView to be generated on mouse move/hover (like AHK_Spy).
- More information to be extracted where useful.
- Draw a rectangle around selected tree element.

Even though this code is just reinventing the wheel, this is an extremely powerful learning tool and has the usual advantages of being written directly in AHK.

Happy scripting!
guest3456
Posts: 2403
Joined: 09 Oct 2013, 10:31

Re: UIA Automation - Inspect.exe for AHK

05 Mar 2017, 20:32

thank you, i will definitely be investigating this

guest3456
Posts: 2403
Joined: 09 Oct 2013, 10:31

Re: UIA Automation - Inspect.exe for AHK

05 Mar 2017, 22:00

i see so much information but can you show me an example how i can use this to read text that otherwise can't be read?

SOTE
Posts: 109
Joined: 15 Jun 2015, 06:21

Re: UIA Automation - Inspect.exe for AHK

05 Mar 2017, 23:00

This is a fantastic tool for automation. Good work.

I was thinking of doing something like this, where a tool gives a picture of all controls in a window and organizes it in a way that is similar to what it looks like visually. A kind of clone of the Window, but more of an x-ray, showing all the controls, menus, etc... Unlike AU3 Spy or Active Window Info, where you must put the mouse over the control to get information, you get information on all controls. Window Detective is another tool that kind of can give this type of picture, but not quite and is not written or customizable in AutoHotkey.

A tool that mirrors the controls in a window, helps greatly for writing macro/automation scripts, because you can see names and info and write the script accordingly. I would like to see this tool go deeper into the menus and the various hierarchies of the controls, to give that kind of x-ray view.

Another problem in writing macro/automation scripts is writing the logic for different conditions. If this control says this, do this. But if it says this, than do this instead. You don't know what window or dialog will come up next, unless you click on everything and play with it or you just have to wait until a certain condition happens, to see certain controls or windows. A tool that can look deeper, is of great help.
sancarn
Posts: 211
Joined: 01 Mar 2016, 14:52

Re: UIA Automation - Inspect.exe for AHK

06 Mar 2017, 02:34

@Guest3456

The initial complete example can be found here:

https://autohotkey.com/board/topic/9461 ... -anywhere/

However of course this does not use the library... I've tried recreating it below, however it is untested.

Code: [Select all] [Expand] [Download] GeSHi © Codebox Plus



Edit: Tested, it doesn't work :P Or at least, I can get the CurrentName() but for some reason all GetCurrentPropertyValue() functions return numbers... It's working in a completely different way to the code in Inspect.ahk... Even when moving to using caches I get the same issue... E.G.

Code: [Select all] [Expand] [Download] GeSHi © Codebox Plus



This might be actually because these patterns don't exist... Still not entirely sure. However you can already use it to get some information out of applications that you can't with AHK_Spy. E.G.

1. Open Internet Explorer
2. Navigate to google
3. Search for something
4. Hover over a link
5. Press ctrl+t

Code: [Select all] [Expand] [Download] GeSHi © Codebox Plus

Last edited by sancarn on 06 Mar 2017, 06:17, edited 5 times in total.
User avatar
RUNIE
Posts: 279
Joined: 03 May 2014, 14:50
GitHub: Run1e

Re: UIA Automation - Inspect.exe for AHK

06 Mar 2017, 04:51

Very nice.

If you want to detect when a new window is activated use this code:

Code: [Select all] [Expand] [Download] GeSHi © Codebox Plus

guest3456
Posts: 2403
Joined: 09 Oct 2013, 10:31

Re: UIA Automation - Inspect.exe for AHK

06 Mar 2017, 08:23

sancarn wrote:However you can already use it to get some information out of applications that you can't with AHK_Spy. E.G.

1. Open Internet Explorer
2. Navigate to google
3. Search for something
4. Hover over a link
5. Press ctrl+t

Code: [Select all] [Expand] [Download] GeSHi © Codebox Plus



But what IS that information? What do those numbers represent and why would I want them?

sancarn
Posts: 211
Joined: 01 Mar 2016, 14:52

Re: UIA Automation - Inspect.exe for AHK

06 Mar 2017, 08:58

guest3456 wrote:
But what IS that information? What do those numbers represent and why would I want them?


They're different names/contents for the UI Element given in different ways. 1 using the value pattern of the UI Element and the other 2 using the old legacy system for IAccessibility. I.E. more text to extract and thus stronger confidence that you've got the correct element.

For example I have seen (with inspect.exe) elements with no name but a Legacy value which is it's name, if that makes sense...
malcev
Posts: 94
Joined: 12 Aug 2014, 12:37

Re: UIA Automation - Inspect.exe for AHK

15 Aug 2018, 10:54

This script does not work with Microsoft Edge.
It shows errors:
ElementFromHandleBuildCache: One or more arguments are not valid error.
CompareElements: Pointer not valid error.
But Inspect.exe works fine.
Dont You know why?
Thank You!
sancarn
Posts: 211
Joined: 01 Mar 2016, 14:52

Re: UIA Automation - Inspect.exe for AHK

15 Aug 2018, 11:13

Afraid not! I haven't worked on this in a while, let alone in AHK...

Return to “Scripts and Functions”

Who is online

Users browsing this forum: No registered users and 20 guests