Interacting with specific elements in unique programs

Get help with using AutoHotkey and its commands and hotkeys
speedskis777
Posts: 30
Joined: 28 Jan 2016, 17:24

Interacting with specific elements in unique programs

08 Dec 2017, 19:53

Hi guys,

I write a lot of code simplifying repetitive tasks on a music notation software called Finale. For a long time, I've been using pix search and specific coordinates for interacting with items on the program, but as you know, all of the circumstances need to be perfect for correct code execution, and with new product updates every year or so, this has outdated a lot of my work.

I'd really like to programatically interact with this software, but at most I can only grab menu buttons listed with classNN in windows Spy. I'm pretty sure the program doesn't have COM built into it, although I'm new to this approach so I'm not completely sure.

Of course, unlike many programs where you scan for words on a page, the medium is notes:

Image

I'm stuck on how to pull data out of this, such as where notes lie on a music stave, or what type they are. Is pixsearch for particular notes images and particular coordinates really my only option?

Any thoughts? How about for other programs which also don't support COM? Or should I look at this as a case-by-case basis? Are there any other interfaces I could look into which may help (I've heard about Selenium but from what I know it's only for browsers)?

Thanks in advance!
speedskis777
Posts: 30
Joined: 28 Jan 2016, 17:24

Re: Interacting with specific elements in unique programs

12 Dec 2017, 17:13

Anyone have any ideas for the above post?? I'm super curious. There's gotta be a faster and more reliable way to control elements in a program without having to use mouse coordinates, pixel searches, image searches, etc. etc.
User avatar
jeeswg
Posts: 3335
Joined: 19 Dec 2016, 01:58
Location: UK

Re: Interacting with specific elements in unique programs

12 Dec 2017, 18:46

- I tried experimenting with some music software once, MuseScore. I wrote the music in a custom shorthand and used key presses to input the data.
- Generally, if feasible, I would be more interested in editing the data in the saved file. And then what I would be interested in automating would be saving the file, and reopening the file/refreshing. Some programs accept an old-school message-based drag-and-drop approach, you can use the Open dialog, or I've been trying to get a new-school object-based drag-and-drop approach working for programs that don't support the old way. I have asked some COM experts to try and help me with this.
- I don't think that image searches is such a bad approach for this software, e.g. search for the circles at the top/bottom of notes.
- You could get a printscreen of the music, and analyse it, i.e. a custom OCR, using Gdip. I intend to release a script re. pixels and OCR perhaps within weeks.
- In theory, you could try doing things with the address space. In theory. I wouldn't recommend, but actually, it might not be a bad approach.
- When a note is focused, is there any numerical information displayed for that note? That could be one way. E.g. you press right, select another note, retrieve the information.
- It should be possible to automate clicking invoking menu items and toolbar buttons. I'll check re. toolbar buttons via messages, but otherwise via Acc.

Get Info from Context Menu (x64/x32 compatible) - AutoHotkey Community
https://autohotkey.com/boards/viewtopic.php?f=6&t=31971
Acc library (MSAA) and AccViewer download links - AutoHotkey Community
https://autohotkey.com/boards/viewtopic.php?f=6&t=26201
- It sounds like the sheet music industry hasn't got serious about the industrial-scale use of music software. I had wondered if/what standards would emerge re. digitally storing sheet music. Are there are any industry standard file formats or core principles?

[EDIT:] Further things:
- use Resource Hacker to check files:
e.g. C:\Windows\System32\notepad.exe
e.g. C:\Windows\System32\en-US\notepad.exe.mui
- check the registry for registry keys:
e.g. HKEY_CURRENT_USER\Software
- use a window spy e.g. Winspector, Microsoft Spy++, HotkeyP:
best utilities + best AutoHotkey scripts (+ useful tips) - AutoHotkey Community
https://autohotkey.com/boards/viewtopic.php?f=7&t=28149
Last edited by jeeswg on 19 Dec 2017, 08:45, edited 1 time in total.
Guest

Re: Interacting with specific elements in unique programs

13 Dec 2017, 08:52

BUTTON BAR

1 - If the buttons have a unique class NN that is actually good news, yes, indeed you would need to check the class NN after each update of the program but if you document it well and use variable it wouldn't be too much of a chore.

Code: [Select all] [Expand] [Download] GeSHi © Codebox Plus



2 - the buttons are equally spaced, so you don't have to find each one as you can just calculate each position, the second button
is the position of the first button + pixels between buttons + the width of a button, do the same as above with the classes but for positions, calcluate them all at start up

MENU

1 - You can try and see if this script by Lexikos works for your program https://autohotkey.com/board/topic/9106 ... nu-search/
if it does, you can use that code to easily find all "messages" for each of the menu commands, that way you can run a separate script each time your App updates to set the variables for each message and simply use those in your script using SendMessage/PostMessage

2 - nearly all programs allow you to navigate the menus by sending alt-letters so Send !f should open the menu, a subsequent letter or Send {down} will navigate the menu and send {enter} execute it. Not too reliable but should work in many cases. If you don't see accelerator keys at first press Alt once to get underlined letters so you know which ones to 'send' - if you find it changes between versions I would also define these keys to be sent as variables at the top of your script for easy updating.

Finally, you could contact the developers of your program to see if they have any info, perhaps they have a list/wiki of updated "messages" or have tips for automation. Perhaps they can add it to a new version if it doesn't respond to "messages" yet.
User avatar
jeeswg
Posts: 3335
Joined: 19 Dec 2016, 01:58
Location: UK

Re: Interacting with specific elements in unique programs

18 Dec 2017, 08:58

TOOLBAR BUTTONS: GET COMMAND IDs

An example to get the toolbar button command IDs and text, in order to invoke toolbar menu items.

Code: [Select all] [Expand] [Download] GeSHi © Codebox Plus


Note: Oftentimes, the toolbar buttons also have context menu item equivalents, which are easier to retrieve command IDs from:
Get Info from Context Menu (x64/x32 compatible) - AutoHotkey Community
https://autohotkey.com/boards/viewtopic.php?f=6&t=31971

E.g. Notepad2 toolbar info.

Code: [Select all] [Expand] [Download] GeSHi © Codebox Plus


E.g. Notepad++ toolbar info.

Code: [Select all] [Expand] [Download] GeSHi © Codebox Plus


Return to “Ask For Help”

Who is online

Users browsing this forum: ark, smarq8, Spawnova, xmachinery and 70 guests