UIA Automation - Inspect.exe for AHK

[sancarn]

A bit of history

So I first came across UIA while looking for better ways to automate the user interfaces of other applications which were not based off Win32 forms. The thread I found was this one. The thread offers a method of finding text anywhere without the use of an OCR. Or rather, it can "see text" in many other places other than just Win32 forms like the AHK_Spy for example. After testing it on some non-win32 forms I found that it could extract information from these UIs that no other AHK function could. This peaked my interest as you can imagine.

In the thread Jethrow and nepter develop a IUIAutomation Interface. And I knew that when i was automating software of this kind I should refer to this library... A few months passed and here I am back looking at it. I've been extending an application that Nepter created - A GUI Cloner a lot like the Windows Inspect.exe which is provided in the Windows SDK. When I first adopted the application it was incredibly compact, but very messy and difficult to understand. It also lacked functionality I wanted. So I began to extend it.

Inspect.ahk

You can find the Inspect.ahk here:

https://github.com/sancarn/Inspect.exe_AHK

The original made by nepter can be found here.

Download both inspect.ahk and uia.ahk

To inspect a Window press {F8}.

Possible features for the future:
- Sample code generator for targeting specified element with the library.
- UIA TreeView to be generated on mouse move/hover (like AHK_Spy).
- More information to be extracted where useful.
- Draw a rectangle around selected tree element.

Even though this code is just reinventing the wheel, this is an extremely powerful learning tool and has the usual advantages of being written directly in AHK.

Happy scripting!

[guest3456]

thank you, i will definitely be investigating this

[guest3456]

i see so much information but can you show me an example how i can use this to read text that otherwise can't be read?

[SOTE]

This is a fantastic tool for automation. Good work.

I was thinking of doing something like this, where a tool gives a picture of all controls in a window and organizes it in a way that is similar to what it looks like visually. A kind of clone of the Window, but more of an x-ray, showing all the controls, menus, etc... Unlike AU3 Spy or Active Window Info, where you must put the mouse over the control to get information, you get information on all controls. Window Detective is another tool that kind of can give this type of picture, but not quite and is not written or customizable in AutoHotkey.

A tool that mirrors the controls in a window, helps greatly for writing macro/automation scripts, because you can see names and info and write the script accordingly. I would like to see this tool go deeper into the menus and the various hierarchies of the controls, to give that kind of x-ray view.

Another problem in writing macro/automation scripts is writing the logic for different conditions. If this control says this, do this. But if it says this, than do this instead. You don't know what window or dialog will come up next, unless you click on everything and play with it or you just have to wait until a certain condition happens, to see certain controls or windows. A tool that can look deeper, is of great help.

[sancarn]

@Guest3456

The initial complete example can be found here:

https://autohotkey.com/board/topic/9461 ... -anywhere/

However of course this does not use the library... I've tried recreating it below, however it is untested.

Code: Select all

#Include uia.ahk
$u:=new IUIAutomation
$e:=new IUIAutomationElement
$c:=new IUIAutomationCondition
$r:=new IUIAutomationCacheRequest
$t:=new IUIAutomationTreeWalker

^t::
MouseGetPos,x,y
pt := x|y<<32
Element := $u.ElementFromPoint(pt)
text := $e.(Element).CurrentName() . "`n"
text .= $e.(Element).GetCurrentPropertyValue(UIA_Property("ValueValue")) . "`n"
text .= $e.(Element).GetCurrentPropertyValue(UIA_Property("LegacyIAccessibleValue")) . "`n"
text .= $e.(Element).GetCurrentPropertyValue(UIA_Property("LegacyIAccessibleName")) . "`n"
text .= UIA_ControlType($e.(Element).GetCurrentPropertyValue(UIA_Property("ControlType")))
Msgbox, %text%
return

Edit: Tested, it doesn't work Or at least, I can get the CurrentName() but for some reason all GetCurrentPropertyValue() functions return numbers... It's working in a completely different way to the code in Inspect.ahk... Even when moving to using caches I get the same issue... E.G.

Code: Select all

---------------------------
SeeTextAnywhere.ahk
---------------------------
Desktop
65787000
65786968
65709496
UIA_ListControlTypeId
---------------------------
OK   
---------------------------

This might be actually because these patterns don't exist... Still not entirely sure. However you can already use it to get some information out of applications that you can't with AHK_Spy. E.G.

1. Open Internet Explorer
2. Navigate to google
3. Search for something
4. Hover over a link
5. Press ctrl+t

Code: Select all

---------------------------
Test.ahk
---------------------------
AutoHotkey - Wikipedia
66730344
66729336
66746680
UIA_TextControlTypeId
---------------------------
OK   
---------------------------

[runie]

Very nice.

If you want to detect when a new window is activated use this code:

Code: Select all

#SingleInstance,Force

Gui +LastFound
DllCall("RegisterShellHookWindow", UInt, WinExist())
OnMessage(DllCall("RegisterWindowMessage", Str, "SHELLHOOK"), "WinActiveChange")

WinActiveChange(wParam, hwnd) {
	static HSHELL_RUDEAPPACTIVATED := 32772
	
	if (wParam != HSHELL_RUDEAPPACTIVATED) ; only listen for HSHELL_RUDEAPPACTIVATED
		return
	
	WinGetTitle, title, A
	Tooltip % title
}

[guest3456]

However you can already use it to get some information out of applications that you can't with AHK_Spy. E.G.

1. Open Internet Explorer
2. Navigate to google
3. Search for something
4. Hover over a link
5. Press ctrl+t

Code: Select all

---------------------------
Test.ahk
---------------------------
AutoHotkey - Wikipedia
66730344
66729336
66746680
UIA_TextControlTypeId
---------------------------
OK   
---------------------------

But what IS that information? What do those numbers represent and why would I want them?

[sancarn]

But what IS that information? What do those numbers represent and why would I want them?

They're different names/contents for the UI Element given in different ways. 1 using the value pattern of the UI Element and the other 2 using the old legacy system for IAccessibility. I.E. more text to extract and thus stronger confidence that you've got the correct element.

For example I have seen (with inspect.exe) elements with no name but a Legacy value which is it's name, if that makes sense...

[malcev]

This script does not work with Microsoft Edge.
It shows errors:
ElementFromHandleBuildCache: One or more arguments are not valid error.
CompareElements: Pointer not valid error.
But Inspect.exe works fine.
Dont You know why?
Thank You!

[sancarn]

Afraid not! I haven't worked on this in a while, let alone in AHK...

[tamcl]

关于Uia的主题和介绍实在是太少了，我不知道如何用autohotkey来构建一个自动处理uia界面的脚本。但是pywinauto可以做这个事情


[Mod edit: Please use english in the general sections of our forums. Thank you!]
This is an automated translation of the message above:

There are so few topics and presentations about Uia that I don't know how to use autohotkey to build a script that automatically handles the uia interface. But pywinauto can do this