FindText - Capture screen image into text and then find it

[ahkHereToLearn]

Is there a way I could capture 15 or so images to text using the Find Text Tool, and then combine FindText seaches / FindText.Click for each of the individual strings into a single AHK file where I could trigger any of the searches independently using, for instance F1:: , F2::, F3:: etc ? I have a simple , time-wasting method right now where I have to find the location of each gui element at the beginning of my day, update the coordinates, and then I trigger them with F1, F2, F3, etc. If I move the windows around at all right now, of course, my script fails to click the right places. Since this library is able to find things anywhere on the page, it would be awesome if I could use it in the way I envision , gaining the freedom to move the GUI around however I want throughout the day. Thanks for any pointers, and of course for this incredible tool.

[SteveMylo]

This is mainly what Find Text is used for. What you describe is very simple. Are you saying that you don’t know how to write a simple script with find text?
Or am I missing something?

[ahkHereToLearn]

This is mainly what Find Text is used for. What you describe is very simple. Are you saying that you don’t know how to write a simple script with find text?
Or am I missing something?

Hi Steve,

I hadn't moved FindText.ahk into my my main lib folder (didn't even have one until today when I created it) until a short while ago. I have now started to get the first two images working by doing this:

Code: Select all

CoordMode, Mouse, Screen

#Include <FindText>

F1::

Text:="|<>*185$69.0000000000000000000000000000000000000000000000000000000000000000000000000000000064000z000000sU00680000074ST0lbXXkwywaH869YqPAaKYXNUyAq296nqYPBqFyQF8qSQXNUn80m96nlaHA69YWPAaSASNUlbXXkwn00000000E00000000002000000000000000000000000000000000000000000000000000000000000000000000000000000000004"

if (ok:=FindText(X, Y, 4266-150000, 2083-150000, 4266+150000, 2083+150000, 0, 0, Text))
{
   FindText().Click(X, Y, "L")
}

return

F3::

Text:="|<>*176$69.0000000000032000E0Dk068QE00201U00l3WC73lkA4HjDSH9AaH1UWMl/GF8YH0DYG69PHvxWC1UWEl/CE84EMA4G69Mn94aF1UmEl/6D73lkA3m39000000000004"

if (ok:=FindText(X, Y, 4254-150000, 2118-150000, 4254+150000, 2118+150000, 0, 0, Text))
{
   FindText().Click(X, Y, "L")
}

return

I'm sure there is a better / cleaner / smarter way, but it looks like if I continue grabbing the strings and assigning them to hotkeys, I will have gotten done what I needed. I am an almost complete newb, so try not to laugh too hard Thanks for taking the time to reply.

[SteveMylo]

@ahkHereToLearn after a quick look, that looks right.
What you can do is change the coordinates to your screen resolution e.g. if (ok:=FindText(X, Y, 0, 0, 1920, 1080, 0, 0, Text))
this can speed things up, unless you have more than one monitor, keep it the same

[ahkHereToLearn]

@ahkHereToLearn after a quick look, that looks right.
What you can do is change the coordinates to your screen resolution e.g. if (ok:=FindText(X, Y, 0, 0, 1920, 1080, 0, 0, Text))
this can speed things up, unless you have more than one monitor, keep it the same

Thank you for the optimization tip. It changed what was before about a 1 to 1.5 second search (depending on the text) into essentially instantaneous, even at 8k resolution. Very cool!

[SteveMylo]

@ahkHereToLearn Glad to here, also you can change Left Click to UP so only the mouse moves but doesn't click. e.g. FindText().Click(X, Y, "U") Just a friendly tip.
Also if you don't know already , you can just use one hotkey to search for all your images in one go, or just keep pressing the same hotkey everytime a new image pops up.
Exmple below.

Code: Select all

F3::

Text:="|<>*185$69.0000000000000000000000000000000000000000000000000000000000000000000000000000000064000z000000sU00680000074ST0lbXXkwywaH869YqPAaKYXNUyAq296nqYPBqFyQF8qSQXNUn80m96nlaHA69YWPAaSASNUlbXXkwn00000000E00000000002000000000000000000000000000000000000000000000000000000000000000000000000000000000004"
Text.="|<>*176$69.0000000000032000E0Dk068QE00201U00l3WC73lkA4HjDSH9AaH1UWMl/GF8YH0DYG69PHvxWC1UWEl/CE84EMA4G69Mn94aF1UmEl/6D73lkA3m39000000000004"

if (ok:=FindText(X, Y, 0, 0, 0, 0, 0, 0, Text))
{
   FindText().Click(X, Y, "L")
}
return

Notice the 1st one has a Text:= a Colon symbol after Text
And the second one (and any more you add) should have just a period/dot Text.=

Also as an experiment, try using 4 zero's as your search area, if (ok:=FindText(X, Y, 0, 0, 0, 0, 0, 0, Text)) I think it just searches your whole screen, should get the same fast speed. Let me know

[gilberto_san]

I have a script that can identify numbers on the screen, although they may vary in color, their format remains consistent. How can I use the FINDTEXT function to locate each number? I've tried different variations such as GRAY, GRAYDFF, COLORS, and others, but I haven't been successful because FINDTEXT only detects some of the numbers. Please see the image below for reference.

Look the image



Findtext configs

Code: Select all

Text:="|<2>**20$13.7zb0n0/25X2lVTVXUnUlUlVlVwU2E180bzm"

if (ok:=FindText(X, Y, 811-150000, 400-150000, 811+150000, 400+150000, 0, 0, Text))
{
  ; FindText().Click(X, Y, "L")
}

result:


In this image, there is some color variation in the numbers, but FindText only locates 4 results. Assuming that the colors can vary randomly but the number structure remains constant, how can I use FindText to assist me?

[ftlsxp]

Hello guys how are you doing?

I would like to know if someone can help me.

I've put together a little script that helps me perform a repetitive task of setting up a plugin that is track related on a DAW (Audio recording and editing software. The specific software is not relevant here).

Basically what it does is click at X coordinates to open a list containing my track number and the track name which should not be relevant here but it kind of is.

Since i can't (actually I don't know how to) paste images here, the list looks something like this: track_# track_name

In example:

1 ANTONY
2 RACHEL
3 JOSE
4 JOSE 2

What my script do is search for the track numbers in a loop using FindText and than it opens another list where it matches the found number. So to be clear it opens the first list, searches for number 1, clicks on found number coordinates, open another menu with an other list of numbers from 1 to 32 and clicks on the matching number and so on.

Everything works ok with one little issue that I can't solve. To my knowledge you can tell FindText the direction you want it to perform the search by adding a number from 1 to 9 after the 14th comma (I've found this info about the search direction numbers at Descolada's tutorial.) but it seems to me that even if I ask it to search from left to right, top to bottom (number 1 - Please check the spoiler) it will "read" from right to left which leads to errors in situations where there is a number in the track name like with "4 JOSE 2". Instead of getting the track number (4), it reads the number 2 first and sets the track number 4 as number 2.

I've tried putting the corresponding search direction after the 14th comma but it didn't solved the issue.

Code: Select all

(ok:=FindText(clockXA, clockYA, 0, 0, A_ScreenWidth, 200, 0, 0, Text,,,,,,1))

I've done some testing searching for 4 incidences of an equal image and it changes which one is found first depending on the search direction. Also tested finding the number 2 in a notepad text and it always finds the one to the right first. (111112111111112111111)

I'm not putting here my full code because the issue is not the code since it works 99% of the time (the only issue is with this specific thing) and also it is basically this, straight from the tutorials/examples and the FindText Tool:

Code: Select all

#SingleInstance force
#Include <FindText>
#NoTrayIcon

t1:=A_TickCount, X:=Y:=""

Text:="|<1>**80$5.DqDGZ+IdHk"

if (ok:=FindText(X, Y, 0, 0, 1279, 1078, 0, 0, Text,,,,,,1))
{
FindText().Click(X, Y, "L")
}

; ok:=FindText(X:="wait", Y:=3, 0,0,0,0,0,0,Text)    ; Wait 3 seconds for appear
; ok:=FindText(X:="wait0", Y:=-1, 0,0,0,0,0,0,Text)  ; Wait indefinitely for disappear

;~ MsgBox, 4096, Tip, % "Found:`t" Round(ok.Length())
  ;~ . "`n`nTime:`t" (A_TickCount-t1) " ms"
  ;~ . "`n`nPos:`t" X ", " Y
  ;~ . "`n`nResult:`t<" (Comment:=ok[1].id) ">"

;~ for i,v in ok  ; ok value can be get from ok:=FindText().ok
  ;~ if (i<=2)
    ;~ FindText().MouseTip(ok[i].x, ok[i].y)
ExitApp

I could try working around it by trimming my search area but then there is the issue where the plugin position on screen and therefore the menus location are not fixed to a specific coordinate which would bring me a whole new set of problems that I might not be able to solve.

Also there are some other automations that I want to do that trimming the search area is not an option. There is one specific situation where a hotkey performs a different action depending on a little check mark on the left side of a text. (✓ Option_Text). It always perform the same action because it seems that it is "reading" from right to left and ends up ignoring the check mark. And i can't search only for the check mark because the Option_Text is relevant.

I tried to read the code to find something about this that i could modify by myself but this is waaaaaaaay to advanced for me. I've learned just the basic of AHK v1 to write some macros to aid me with my productivity. I have no background on programming or such thing and unfortunately I don't have enough time to study anymore.

Has someone else noticed the same behavior? Am I crazy? Can someone help me?

Sorry if I wasn't specific/clear enough. Since english is not my main language (I'm brazilian so my mother language is Portuguese) it is difficult to not get lost in translation.

Thanks in advance for the time spent reading this and for any kind of help you can provide me.

Best regards to you all.

Spoiler

; FindText - Capture screen image into text and then find it
; Version : 9.0 (2023-04-15)
;--------------------------------
; returnArray := FindText(
; OutputX --> The name of the variable used to store the returned X coordinate
; , OutputY --> The name of the variable used to store the returned Y coordinate
; , X1 --> the search scope's upper left corner X coordinates
; , Y1 --> the search scope's upper left corner Y coordinates
; , X2 --> the search scope's lower right corner X coordinates
; , Y2 --> the search scope's lower right corner Y coordinates
; , err1 --> Fault tolerance percentage of text (0.1=10%)
; , err0 --> Fault tolerance percentage of background (0.1=10%)
; , Text --> can be a lot of text parsed into images, separated by '|'
; , ScreenShot --> if the value is 0, the last screenshot will be used
; , FindAll --> if the value is 0, Just find one result and return
; , JoinText --> if you want to combine find, it can be 1, or an array of words to find
; , offsetX --> Set the max text offset (X) for combination lookup
; , offsetY --> Set the max text offset (Y) for combination lookup
; , dir --> Nine directions for searching: up, down, left, right and center
; , zoomW --> Zoom percentage of image width (1.0=100%)
; , zoomH --> Zoom percentage of image height (1.0=100%)
; )
;
;-------------------------------- (I've just wrote this part so i could not forget about this since I forgot where I read this in the first place)
; Search Direction Helper
;
; 1 ==> ( Left to Right ) Top to Bottom
; 2 ==> ( Right to Left ) Top to Bottom
; 3 ==> ( Left to Right ) Bottom to Top
; 4 ==> ( Right to Left ) Bottom to Top
; 5 ==> ( Top to Bottom ) Left to Right
; 6 ==> ( Bottom to Top ) Left to Right
; 7 ==> ( Top to Bottom ) Right to Left
; 8 ==> ( Bottom to Top ) Right to Left
; 9 ==> From center outwards
;
;--------------------------------

[ftlsxp]

@gilberto_san

I think that might exist a more intelligent/elegant way to do it but I believe you could parse each variation by using (.=). That way the function will search for any of the variations. The only thing is that you would have to grind a bit more to write your code.

Code: Select all

Text:="|<2_red>**20$13.7zb0n0/25X2lVTVXUnUlUlVlVwU2E180bzm"
Text.="|<2_purple>**20$13.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
Text.="|<2_green>**20$13.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
Text.="|<2_pink>**20$13.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
Text.="|<2_white>**20$13.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
Text.="|<2_orange>**20$13.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"

if (ok:=FindText(X, Y, 811-150000, 400-150000, 811+150000, 400+150000, 0, 0, Text))
{
  ; FindText().Click(X, Y, "L")
}

You could also experiment changing the tolerance

returnArray := FindText(
; OutputX --> The name of the variable used to store the returned X coordinate
; , OutputY --> The name of the variable used to store the returned Y coordinate
; , X1 --> the search scope's upper left corner X coordinates
; , Y1 --> the search scope's upper left corner Y coordinates
; , X2 --> the search scope's lower right corner X coordinates
; , Y2 --> the search scope's lower right corner Y coordinates
; , err1 --> Fault tolerance percentage of text (0.1=10%)
; , err0 --> Fault tolerance percentage of background (0.1=10%)
; , Text --> can be a lot of text parsed into images, separated by '|'
; , ScreenShot --> if the value is 0, the last screenshot will be used
; , FindAll --> if the value is 0, Just find one result and return
; , JoinText --> if you want to combine find, it can be 1, or an array of words to find
; , offsetX --> Set the max text offset (X) for combination lookup
; , offsetY --> Set the max text offset (Y) for combination lookup
; , dir --> Nine directions for searching: up, down, left, right and center
; , zoomW --> Zoom percentage of image width (1.0=100%)
; , zoomH --> Zoom percentage of image height (1.0=100%)
; )

[Descolada]

@ftlsxp, the way FindText works is that it goes row by row, pixel per pixel, trying to find a match to your Text. The search direction determines how it goes about that: for example dir=4 means that it goes Right to Left and Bottom to Top, meaning first the bottom row of pixels going right to left, then the next row of pixels from bottom going right to left and so on.

The default search direction is Left to Right and Top to Bottom. Now imagine that you have image "4 JOSE 2" and your Text has both "4" and "2". If "4" is even 1 pixel higher than "2" in the image then if the search is going Top to Bottom, then "4" is always found before "2"! If you were to search Bottom to Top then whichever one is even 1 pixel lower than the other is found first. You need to think in the terms of pixels, not in terms of characters.

One possible solution is to search all matches for "2" and "4", and then loop through the results and use some error margins to filter the results. Eg if the difference in y coordinate is less than 5 then consider them to be at the same height and compare the x coordinates.
EDIT: totally forgot that this can be done with ok:=FindText().Sort(ok), see feiyue's comment down delow.

[gilberto_san]

Thank you, I had thought of this solution. But please note that there are 50 or more color variations and 10 different numbers.
The result would be more than 500 "Text:=<number_color>xxxxxx"

[feiyue]

1. The "ColorPos" mode is precisely for adapting to different colors.
2. Using sorting can ignore slight height differences and provide visual results.
eg: ok:=FindText().Sort(ok)

[feiyue]

Updated to 9.2 version - 2023/10/01
1. Modify: After successfully searching for images in a loop (X:='wait' or 'wait1'),
it is no longer necessary to wait for the images to stabilize before returning,
as many people need to return immediately for gaming purposes
2. Modify: When searching for image files, multiple transparent colors can be set.
You can use Text:="##10-RRGGBB-RRGGBB... $ d:\a.bmp", then the 0xRRGGBB(+/-10)... as transparent color.
You can use FindText().ImageSearch(,,,,,, "*10 *TransBlack-White-RRGGBB... d:\a.bmp")
3. Add: Added Snapshot() function, which can be used to save screenshot to clipboard or file.
eg: PrintScreen::FindText().Snapshot()

[bradgirman]

Hello! New here and I've written a few things to improve efficiency at work. I'm having a hard time finding information on how to do this but I'd like to write up my script to display a custom error message when it can't find the image it's looking for. Any ideas? And thank you ahead of time!

[pushmaxonthree]

Code: Select all

WinGetPos, pX, pY, pW, pH, XXXX - XXXXX

if (FindText(X, Y, pX, pY, pX+pW, pY+pH, 0, 0, Text))

What should I put
If I want to completely ignore the background? No matter what it is...
I want the text to be important, not the background...

[dostroll]

Hi,
Is it possible to specify the range of X1, Y1, X2, Y2 in multiple locations?
Currently, I am writing three codes for three ranges, but I would like to combine them into one.

for example,

Code: Select all

if (ok:=FindText(X, Y, 10/100/300, 10/100/300, 20/110/310, 20/110/310, 0, 0, Text))

[Descolada]

@pushmaxonthree, you should be able to just set err1 or err0 to 1 to ignore either the text or the background.

@dostroll, no, you need to do three separate FindText calls. Or you could create a function yourself that takes an array of coordinates, performs FindText with the screenshot argument set to 1 on the first call and set to 0 for the subsequent calls (to take a screenshot only on the first call, because this is the part that is slowest), and then merge all the results into one.

[dostroll]

@Descolada
Thank you, I learned a lot.
Probably "combination lookup".
I will try my best.

[feiyue]

Updated to 9.3 version - 2023/11/05
1. Modify: FindMultiColor search mode, which can manually enhance the generated Text parameters,
such as adding multiple colors for each point, adding their own color bias, and using exclusion colors.
2. Add: bin2hex(), DrawHBM(), you can draw lines and rectangles in memory images.
3. Add: New() function, you can use ft:=FindText().New() instead of ft:=new FindTextClass()
4. Modify: Machine code using base64 strings reduces script size.

[trialAdd]

Code: Select all

loop, 200
{
sleep,10
if (ok:=FindText(X, Y, 630-150000, 622-150000, 630+150000, 622+150000, 0, 0, Text))
{
  ; FindText().Click(X, Y, "L")
}
}

heavy CPU Usage like 50 to 60 %
why is that ?