Interacting with specific elements in unique programs

[speedskis777]

Hi guys,

I write a lot of code simplifying repetitive tasks on a music notation software called Finale. For a long time, I've been using pix search and specific coordinates for interacting with items on the program, but as you know, all of the circumstances need to be perfect for correct code execution, and with new product updates every year or so, this has outdated a lot of my work.

I'd really like to programatically interact with this software, but at most I can only grab menu buttons listed with classNN in windows Spy. I'm pretty sure the program doesn't have COM built into it, although I'm new to this approach so I'm not completely sure.

Of course, unlike many programs where you scan for words on a page, the medium is notes:



I'm stuck on how to pull data out of this, such as where notes lie on a music stave, or what type they are. Is pixsearch for particular notes images and particular coordinates really my only option?

Any thoughts? How about for other programs which also don't support COM? Or should I look at this as a case-by-case basis? Are there any other interfaces I could look into which may help (I've heard about Selenium but from what I know it's only for browsers)?

Thanks in advance!

[speedskis777]

Anyone have any ideas for the above post?? I'm super curious. There's gotta be a faster and more reliable way to control elements in a program without having to use mouse coordinates, pixel searches, image searches, etc. etc.

[jeeswg]

- I tried experimenting with some music software once, MuseScore. I wrote the music in a custom shorthand and used key presses to input the data.
- Generally, if feasible, I would be more interested in editing the data in the saved file. And then what I would be interested in automating would be saving the file, and reopening the file/refreshing. Some programs accept an old-school message-based drag-and-drop approach, you can use the Open dialog, or I've been trying to get a new-school object-based drag-and-drop approach working for programs that don't support the old way. I have asked some COM experts to try and help me with this.
- I don't think that image searches is such a bad approach for this software, e.g. search for the circles at the top/bottom of notes.
- You could get a printscreen of the music, and analyse it, i.e. a custom OCR, using Gdip. I intend to release a script re. pixels and OCR perhaps within weeks.
- In theory, you could try doing things with the address space. In theory. I wouldn't recommend, but actually, it might not be a bad approach.
- When a note is focused, is there any numerical information displayed for that note? That could be one way. E.g. you press right, select another note, retrieve the information.
- It should be possible to automate clicking invoking menu items and toolbar buttons. I'll check re. toolbar buttons via messages, but otherwise via Acc.

Get Info from Context Menu (x64/x32 compatible) - AutoHotkey Community
https://autohotkey.com/boards/viewtopic.php?f=6&t=31971
Acc library (MSAA) and AccViewer download links - AutoHotkey Community
https://autohotkey.com/boards/viewtopic.php?f=6&t=26201
- It sounds like the sheet music industry hasn't got serious about the industrial-scale use of music software. I had wondered if/what standards would emerge re. digitally storing sheet music. Are there are any industry standard file formats or core principles?

[EDIT:] Further things:
- use Resource Hacker to check files:
e.g. C:\Windows\System32\notepad.exe
e.g. C:\Windows\System32\en-US\notepad.exe.mui
- check the registry for registry keys:
e.g. HKEY_CURRENT_USER\Software
- use a window spy e.g. Winspector, Microsoft Spy++, HotkeyP:
best utilities + best AutoHotkey scripts (+ useful tips) - AutoHotkey Community
https://autohotkey.com/boards/viewtopic.php?f=7&t=28149

[Guest]

BUTTON BAR

1 - If the buttons have a unique class NN that is actually good news, yes, indeed you would need to check the class NN after each update of the program but if you document it well and use variable it wouldn't be too much of a chore.

Code: Select all

; define classes for all buttons at the top of your script:
; for version x.y:
Class_New:="Button1"
Class_Open:="Button2"
; etc

; later in your script you can use

- ControlClick [, Control-or-Pos, WinTitle, WinText, WhichButton, ClickCount, Options, ExcludeTitle, ExcludeText]
- ControlSend [, Control, Keys, WinTitle, WinText, ExcludeTitle, ExcludeText]
; Use %Class_New% for "Control-or-Pos" or "Control" in the commands above

2 - the buttons are equally spaced, so you don't have to find each one as you can just calculate each position, the second button
is the position of the first button + pixels between buttons + the width of a button, do the same as above with the classes but for positions, calcluate them all at start up

MENU

1 - You can try and see if this script by Lexikos works for your program https://autohotkey.com/board/topic/9106 ... nu-search/
if it does, you can use that code to easily find all "messages" for each of the menu commands, that way you can run a separate script each time your App updates to set the variables for each message and simply use those in your script using SendMessage/PostMessage

2 - nearly all programs allow you to navigate the menus by sending alt-letters so Send !f should open the menu, a subsequent letter or Send {down} will navigate the menu and send {enter} execute it. Not too reliable but should work in many cases. If you don't see accelerator keys at first press Alt once to get underlined letters so you know which ones to 'send' - if you find it changes between versions I would also define these keys to be sent as variables at the top of your script for easy updating.

Finally, you could contact the developers of your program to see if they have any info, perhaps they have a list/wiki of updated "messages" or have tips for automation. Perhaps they can add it to a new version if it doesn't respond to "messages" yet.

[jeeswg]

TOOLBAR BUTTONS: GET COMMAND IDs

An example to get the toolbar button command IDs and text, in order to invoke toolbar menu items.

Code: Select all

;functions needed to run the script:
;[Acc functions]
;Acc library (MSAA) and AccViewer download links - AutoHotkey Community
;https://autohotkey.com/boards/viewtopic.php?f=6&t=26201
;GUIs via DllCall: get/set internal/external control text - AutoHotkey Community
;https://autohotkey.com/boards/viewtopic.php?f=6&t=40514

;e.g. Notepad++: must be hovered over control (to get text of all buttons)
;e.g. WordPad (Windows XP): must be hovered over control button (to get text of individual button)
q:: ;toolbar - get text and command IDs
MouseGetPos,,, hWnd, vCtlClassNN
;vCtlClassNN := "ToolbarWindow321"
if !InStr(vCtlClassNN, "ToolbarWindow32")
	return
ControlGet, hCtl, Hwnd,, % vCtlClassNN, % "ahk_id " hWnd
oArray := JEE_TBGetText(hCtl, -1, "`r`n", "c o")
vText := JEE_AccCtlGetText(hCtl)

vOutput := ""
VarSetCapacity(vOutput, StrLen(vText)*4)
Loop, Parse, vText, `n, `r
	vOutput .= RTrim(oArray[A_Index] A_LoopField) "`r`n"
Clipboard := vOutput
MsgBox, % vOutput
return

w:: ;notepad++ - trigger toolbar button: Open...
PostMessage, 0x111, 41002,,, A ;WM_COMMAND := 0x111
return

Note: Oftentimes, the toolbar buttons also have context menu item equivalents, which are easier to retrieve command IDs from:
Get Info from Context Menu (x64/x32 compatible) - AutoHotkey Community
https://autohotkey.com/boards/viewtopic.php?f=6&t=31971

E.g. Notepad2 toolbar info.

Code: Select all

;Notepad2: toolbar button command IDs and text
40700	New
40701	Open
40702	Browse
40703	Save
0
40704	Undo
40705	Redo
0
40706	Cut
40707	Copy
40708	Paste
0
40709	Find
40710	Replace
0
40711	Word Wrap
0
40712	Zoom In
40713	Zoom Out
0
40714	Select Scheme
40715	Customize Schemes
0
40716	Exit

E.g. Notepad++ toolbar info.

Code: Select all

;Notepad++: toolbar button command IDs and text
41001	New
41002	Open...
41006	Save
41007	Save All
41003	Close
41004	Close All
41010	Print...
0
42001	Cut
42002	Copy
42005	Paste
0
42003	Undo
42004	Redo
0
43001	Find...
43003	Replace...
0
44023	Zoom In (Ctrl+Mouse Wheel Up)
44024	Zoom Out (Ctrl+Mouse Wheel Down)
0
44035	Synchronize Vertical Scrolling
44036	Synchronize Horizontal Scrolling
0
44022	Word wrap
44019	Show All Characters
44020	Show Indent Guide
46150	Define your language...
44080	Document Map
44084	Function List
44085	Folder as Workspace
44097	Monitoring (tail -f)
0
42018	Start Recording
42019	Stop Recording
42021	Playback
42032	Run a Macro Multiple Times...
42025	Save Current Recorded Macro...