Jump to content

Sky Slate Blueberry Blackcurrant Watermelon Strawberry Orange Banana Apple Emerald Chocolate
Photo

Interest in getting text under mouse cursor


  • Please log in to reply
8 replies to this topic
vashk
  • Members
  • 11 posts
  • Last active: Apr 02 2008 01:21 AM
  • Joined: 07 Jan 2008
First post so I want to start off by saying AHK is a great tool and I've already written several useful scripts in a short time to help with RSI and other annoyances, so thank you Chris.

I'm trying to write a fast translation tool that grabs the text under the mouse and displays a tooltip in the target language*, but I'm missing a couple pieces. The biggest hurdle so far is sneaking the text off any part of the screen without resorting to OCR.

*by target language I mean if the text (based on its UTF-8 values) is in the user's native language it translates to a user-configurable language and if it is in any non-native language it translates to the user's native language

The idea of using MSAA (Microsoft Active Accessibility) has been thrown around, and I already found it on the Planned Features page:

http://www.autohotke...dingChanges.htm

I have another suggestion, which is to use a 3rd party DLL by Deskperience. My idea would be to use MouseGetPos command in AHK and pass the resulting coordinates and HWND to the DLL's GetTextFromPoint method.**

**The WordCapture part of the project has been completed using this method. Thanks for the useful suggestions. I'll update when the project is finished in case anyone's interested.
--v

majkinetor
  • Moderators
  • 4512 posts
  • Last active: Jul 29 2016 12:40 AM
  • Joined: 24 May 2006
Check out Active Accesibilty COM module.
Posted Image

vashk
  • Members
  • 11 posts
  • Last active: Apr 02 2008 01:21 AM
  • Joined: 07 Jan 2008
Thanks majkinetor...

Do you have a link for this Active Accessibility COM module? I tried searching the forums and AHK script sites but all I could find was this topic:

http://www.autohotke...cessibility com

I'll certainly be reading it in more detail.
I appreciate your help.

Edit:

If you mean Microsoft Active Accessibility itself, I'm already looking at that, as indicated in my first post. I modified the code Sean posted in the above topic to find everything MSAA could tell me about a control (using mouse hover and logging to file the results) and it's a step up from ControlGet, but it still doesn't provide enough information about what visible text actually is under the mouse.
--v

  • Guests
  • Last active:
  • Joined: --
<!-- m -->http://www.autohotke... ... bility com<!-- m -->

vashk
  • Members
  • 11 posts
  • Last active: Apr 02 2008 01:21 AM
  • Joined: 07 Jan 2008
Thanks for the link. It turns out MSAA does not have the single-word granularity I'm looking for. I was able to write an AHK script using the 3rd party DLL I discussed previously which captures words from any part of the screen. I think it might be non-commercially distributable. Here's hoping for a free tool that can be distributed with a future version of AHK.

For anyone who's interested here's the code... it's really just a proof-of-concept.

;----Aqua Deskperience----
;Version 1.3
;System requirements: 
;Windows 2000, Windows XP, Windows 2003 Server
;Internet Explorer 5.5+
;400MHz processor
;128MB RAM
;5 MB of hard-disk space

;------TCaptureX.dll------
;CLSID      : {13FE2FA1-EE8B-45B9-BBB4-08E5F2F43AC3}
;Text       : TextCaptureX Class
;Type       : InProcServer32
;Type Value : C:\Program Files\Deskperience\Aqua\TCaptureX.dll
;ProgID     : TCaptureX.TextCaptureX.1
;TypeLib    : {92657C70-D31B-4930-9014-379E3F6FB91A}

wordCapture()
;returns the word under the mouse
;requires TCaptureX.dll from Aqua Deskperience
;requires COM AHK library
{
    CoordMode Mouse, Screen ;sets Mouse mode to global coordinates
    MouseGetPos mX, mY, w_hWnd, c_hWnd, 2
    COM_CoInitialize()
    hModule := DllCall("LoadLibrary","str","TCaptureX.dll")
    suuid := "{13FE2FA1-EE8B-45B9-BBB4-08E5F2F43AC3}"
    pTcx := COM_CreateObject(suuid)
    ;guid is alias for CLSID_TextCaptureX
    ;guid := COM_GUID4String(CLSID_TextCaptureX,suuid)

    ;GetTextFromPoint requires (x,y) position in global coordinates
    ;GetTextFromPoint also requires handle to the parent window of text
    text := COM_Invoke(pTcx,"GetTextFromPoint",w_hWnd,mX,mY)
    
    COM_Release(pTcx)
    DllCall("FreeLibrary", "UInt", hModule)
    COM_CoUninitialize()
    return text
}

--v

Sakurako
  • Members
  • 194 posts
  • Last active: Feb 01 2011 06:13 AM
  • Joined: 10 May 2007
Why not just use language code? ~

Link: <!-- m -->http://www.autohotke...c/Languages.htm<!-- m -->

vashk
  • Members
  • 11 posts
  • Last active: Apr 02 2008 01:21 AM
  • Joined: 07 Jan 2008

Why not just use language code? ~

Link: <!-- m -->http://www.autohotke...c/Languages.htm<!-- m -->


Yes, I am using language code to determine the user's native language, but that's only a small part of the code.
--v

F1reW1re
  • Members
  • 6 posts
  • Last active: May 30 2009 01:12 AM
  • Joined: 25 Apr 2009
Hi guys,

I tried the script above with all necessary files...

COM.ahk (Sean)
RL.dll (Aqua Deskperience 1.3)
TCapture.dll (Aqua Deskperience 1.3)
TCaptureX.dll (Aqua Deskperience 1.3)
TCCustom.dll (Aqua Deskperience 1.3)

...but there is no text shown within the tooltip. The variable "text" is empty.

Also i got this error message:

---------------------------
COM Error Notification
---------------------------
Function Name:   "GetTextFromPoint"
ERROR:   The COM Object may not be a valid Dispatch Object!
   First ensure that COM Library has been initialized through COM_Init().
   ()

Will Continue?
---------------------------
Yes   No   
---------------------------

Can anybody help me?

The Script:

#Persistent
#Include COM.ahk

SetTimer, SetTimer_Text, 100
Return

SetTimer_Text:
wordCapture()
tooltip, %hModule%|%suuid%|%pTcx%|%GetTextFromPoint%|%w_hWnd%|%mX%|%mY%`nText: "%text%"
Return

;----Aqua Deskperience----
;Version 1.3
;System requirements:
;Windows 2000, Windows XP, Windows 2003 Server
;Internet Explorer 5.5+
;400MHz processor
;128MB RAM
;5 MB of hard-disk space

;------TCaptureX.dll------
;CLSID      : {13FE2FA1-EE8B-45B9-BBB4-08E5F2F43AC3}
;Text       : TextCaptureX Class
;Type       : InProcServer32
;Type Value : C:\Program Files\Deskperience\Aqua\TCaptureX.dll
;ProgID     : TCaptureX.TextCaptureX.1
;TypeLib    : {92657C70-D31B-4930-9014-379E3F6FB91A}

wordCapture()
;returns the word under the mouse
;requires TCaptureX.dll from Aqua Deskperience
;requires COM AHK library
{
    Global
    COM_Init()
   ;COM_Error(0)
    CoordMode Mouse, Screen ;sets Mouse mode to global coordinates
    MouseGetPos mX, mY, w_hWnd, c_hWnd, 2
    COM_CoInitialize()
    hModule := DllCall("LoadLibrary","str","TCaptureX.dll")
    suuid := "{13FE2FA1-EE8B-45B9-BBB4-08E5F2F43AC3}"
    pTcx := COM_CreateObject(suuid)
    ;guid is alias for CLSID_TextCaptureX
    ;guid := COM_GUID4String(CLSID_TextCaptureX,suuid)

    ;GetTextFromPoint requires (x,y) position in global coordinates
    ;GetTextFromPoint also requires handle to the parent window of text
    text := COM_Invoke(pTcx,"GetTextFromPoint",w_hWnd,mX,mY)

    COM_Release(pTcx)
    DllCall("FreeLibrary", "UInt", hModule)
    COM_CoUninitialize()
    return text
}


F1reW1re
  • Members
  • 6 posts
  • Last active: May 30 2009 01:12 AM
  • Joined: 25 Apr 2009
...found a better solution...

#Persistent
#Include ACC.ahk
#Include COM.ahk

SetTimer, SetTimer_Text, 100
Return

SetTimer_Text:
ACC_Init()
pacc := Acc_AccessibleObjectFromPoint()
if paccChild := acc_Child(pacc, _idChild_)
 TextUnderMouse := acc_Name(paccChild)
else
 TextUnderMouse := acc_Name(pacc, _idChild_)
Tooltip, %TextUnderMouse%
Return

Many thanks to Sean
<!-- m -->http://www.autohotke... ... bility com<!-- m -->