FindText - Capture screen image into text and then find it

Post your working scripts, libraries and tools
feiyue
Posts: 59
Joined: 08 Aug 2014, 04:08

FindText - Capture screen image into text and then find it

14 May 2016, 12:10

I wrote a useful function to share with you.
It is used to find text or images on the screen.
Hope you like it ! :)

Detailed usage can refer to:
1、The introduction by ed1chandler.
2、The introduction by c4p (latest).

Code: [Select all] [Expand] [Download] GeSHi © Codebox Plus

Last edited by feiyue on 20 Oct 2017, 23:21, edited 136 times in total.
feiyue
Posts: 59
Joined: 08 Aug 2014, 04:08

Re: FindText - Catch screen image into text and then find it

14 May 2016, 12:25

This function is similar to the AHK built-in command "ImageSearch",
but it can convert images into strings, written into the script without
saving as an external image, so it is easier to use than "ImageSearch". :beer:
Last edited by feiyue on 30 Jun 2017, 17:19, edited 37 times in total.
User avatar
kwstas13
Posts: 43
Joined: 19 Mar 2016, 06:26

Re: FindText - Catch screen image into text and then find it

14 May 2016, 12:28

wow nice this is cool script thank you!! :bravo: :D
Guest888

Re: FindText - Catch screen image into text and then find it

14 May 2016, 17:43

:thumbup: :thumbup: :thumbup:
This could be very useful for automatic operating. Thank you very much for sharing.
:bravo: :bravo: :bravo: :bravo:
guest3456
Posts: 2024
Joined: 09 Oct 2013, 10:31

Re: FindText - Catch screen image into text and then find it

14 May 2016, 19:16

very interesting image search / OCR example. thanks a lot. good job

User avatar
BGM
Posts: 366
Joined: 20 Nov 2013, 20:56
GitHub: bgmCoder
Contact:

Re: FindText - Catch screen image into text and then find it

14 May 2016, 21:36

__________________________________________00000_______________00_________
_________________________________________00___________________00_________
_________________________________________00___________________00_________
_________________________________________00___________________00_________
________00____00_____0000______00000____000000__00____00____00_________
________00____00____00___0____00___00____00_____00____00____00_________
________00____00____00________00___00____00_____00____00____00_________
________00____00____00_______00____000___00_____00____00____00_________
________00____00____0000_____00_____00___00_____00____00____00_________
________00____00_____0000____000000000___00_____00____00____00_________
________00____00_______000___00__________00_____00____00____00_________
________00____00________000__00__________00_____00____00____00_________
________00___000________000___00_________00_____00___000____00_________
________00__0000___00___00____000___00___00_____00__0000____00_________
_________0000_00____00000_______00000____00______0000_00____00_________
Guest

Re: FindText - Catch screen image into text and then find it

18 May 2016, 01:11

This is such an useful script. Your idea and approach are brilliant! I register to say thank you FeiYue.

For those who haven't tried: this is not for OCR. But image search, rather.
It stands out from the internal ImageSearch command in that no ImageFile is required, and runs much faster.
The FindText function takes less than half a second to seach a full 1680x1050 screen.
The fact that it can search for both text/character and graph/picture won my vote for killer script of the year!
I use this to automate all sort of web pages and traditional applications. Great work! :superhappy:
jmeneses
Posts: 402
Joined: 28 Oct 2014, 11:09

Re: FindText - Catch screen image into text and then find it

18 May 2016, 03:05

Hi feiyue, excellent job and thanks for sharing
:thumbup: :thumbup:
feiyue
Posts: 59
Joined: 08 Aug 2014, 04:08

Re: FindText - Catch screen image into text and then find it

19 May 2016, 09:43

In addition to searching for screen text and images,
This function can also be used for simple text recognition.
Here is an example of a simple identification Numbers.
FindTextOCR() function can also be used to identify simple verification code.


Code: [Select all] [Expand] [Download] GeSHi © Codebox Plus

Last edited by feiyue on 25 Sep 2017, 00:13, edited 8 times in total.
ed1chandler
Posts: 15
Joined: 15 May 2015, 11:18

Re: FindText - Catch screen image into text and then find it

20 May 2016, 09:34

Want to make sure I'm understanding use case and usage:

1) Download script and run it. This pops up the "Catch Image To Text And Find Text Tool" box with "Catch", "Test", and "Copy" buttons.

2) Click the "Catch" button and the cursor is accompanied by a transparent red box.

3) Move the red box over the text you want to search for and click, then move the box more than 100 px in any direction to trigger capture.

4) This brings up the "Catch Image To Text" box with a magnified view of the captured text.

5) The lower left portion of that GUI has a series of "trimming" buttons, used to trim the edges of the captured region. I'm not sure what the abbreviations are supposed to stand for, but it appears that "LD" trims one pixel from the left edge, "LD3" trims three pixels, and the other buttons do the same for "up", "down", and "right." Use those buttons to trim the region down to what you really want to search for.

6) Click somewhere on the magnified capture to select a pixel with the color you want to use for matching. When you do, the text box next to the "Color2Two" button will populate with the hex value of the color corresponding with the pixel you chose.

It appears there are two modes: color and greyscale. If you want to use color, it appears to go something like this:
7a) After selecting a pixel (step 6), if you click the "Color2Two" button, it appears to convert the magnified image into a two-tone version where all pixels of the selected color are converted to black and all other pixels are converted to white.

And if you want to use greyscale, it goes something like this:
7b) After selecting a pixel (step 6), if you click the "Gray2Two" button, it appears to make a "best guess" at a threshold, then convert the magnified image into a two-tone black/white version where all pixels on one side of the threshold are black and all others are white. It also appears that you can change the threshold and click the button again if you want to adjust the program's guess.

8) Once you've got the two-tone image you want, click the "Ok" button. This returns a GUI with a "Text=..." block of text and a related function call. You can click the "Test" button to test that the search will actually find what you're looking for, then click the "Copy" button to copy the text block and the related function call to the clipboard. The "Exchange" button will negate the image and the "Load" button will reload the original capture.

9) Copy that block of text and the function call into the script from which the search will be run ... and be sure the script can "see" the FindText() function.

10) Modify and enjoy.

What I'm *not* quite following is how to modify the function parameters. The "w" and "h" parameters say that they're "offsets" but what does that *mean*? (i.e. What's the effect of changing them?) The default is 150, but the capture tool seeds these to 150000. What is this changing?
feiyue
Posts: 59
Joined: 08 Aug 2014, 04:08

Re: FindText - Catch screen image into text and then find it

20 May 2016, 13:36

@ed1chandler, your description for use is very detailed, very good! :bravo:
Because my English is not very good, so the message is less in front,
thank you for your description. I add a few points:

1, LD meaning is the abbreviation of Left-Del, appear to be quite strange. :D

2, You can cut the edge first, then Make the image into black and white,
Can also be, first to make the image into black and white, and then cut the edge.
I recommend the way back, Because click "Auto" button can automatically cut edge.

(Slight correction)the Gray2Two button don't use the Selected color, will automatically
calculate a threshold, if you want to customize can manually enter the threshold.

3, the parameters of X, Y, W, H to determine a range:
the upper left corner (X-W, Y-H), the lower right corner of (X+W, Y+H),
So X, Y is the center point of the range, W, H is the distance between the center point
of the offset distance. The advantage is that the location of the search results is returned
Text / image center, and the parameters can be compared to know the relative text / image
as the origin of the offset.

4, In the default generated code, I use the offset 150000, So it's all around the screen,
it is full screen search. The actual use of the time required to select the appropriate scope,
such as use 150, in accordance with (3) to determine the scope of the search.

(Machine automatic translation)
Mkonopko
Posts: 21
Joined: 25 Jan 2015, 07:30

Re: FindText - Catch screen image into text and then find it

20 May 2016, 14:01

I am not a programmer and I struggle to make these things work. Could you post an example of how you would run this code once it is generated?
I'm getting a "call to non-existing function" error
feiyue
Posts: 59
Joined: 08 Aug 2014, 04:08

Re: FindText - Catch screen image into text and then find it

20 May 2016, 14:25

Useage:
1. Catch the image to text string.
2. Test find the text string on full Screen.
3. When test is successful, copy the code
and paste it into your own script.
4. Copy the "FindText()" function and the function
of the back, paste it into your own script.

You didn't do the fourth step, to copy the functions to your application.

Actual use of the time, you may also need to modify the scope of the search ( W, H offset),
the default (offset 150000) full screen to find a longer time and find the location may not be what you need.
Last edited by feiyue on 20 May 2016, 15:13, edited 1 time in total.
Mkonopko
Posts: 21
Joined: 25 Jan 2015, 07:30

Re: FindText - Catch screen image into text and then find it

20 May 2016, 14:43

So I'm trying to write a standalone script that finds your name when I hit Win F3
How would I modify this?

#f3::

Text=
(
__000__________________________________
_00__________00________________________
_00____________________________________
0000___0000__00_00___00_00__00___0000__
_00___00__00_00__00__0__00__00__00__00_
_00___000000_00__00_00__00__00__000000_
_00___00_____00___0_00__00__00__00_____
_00___00___0_00___000___00__00__00___0_
_00____00000_00___000____00_00___00000_
__________________00___________________
__________________00___________________
)

if FindText(44,243,Text,"*177",150000,150000,X,Y)
{
CoordMode, Mouse
MouseMove, X, Y
}
return
feiyue
Posts: 59
Joined: 08 Aug 2014, 04:08

Re: FindText - Catch screen image into text and then find it

20 May 2016, 15:04

Find this line at the end of my code,
Copy and paste the following into your script.

“ ;======== Copy The Next Functions To Your Code ======== ”
Mkonopko
Posts: 21
Joined: 25 Jan 2015, 07:30

Re: FindText - Catch screen image into text and then find it

20 May 2016, 15:16

1-I clicked on catch and caught the text
2-I did the "test" and it worked fine
3=I selected "Copy" I did a ctrl-a to copy everything

I could not find
“ ;======== Copy The Next Functions To Your Code ======== ”
feiyue
Posts: 59
Joined: 08 Aug 2014, 04:08

Re: FindText - Catch screen image into text and then find it

20 May 2016, 15:21

Find this line at the end of My source code,
Copy and paste the following into your script.

My source code on Post 1,that is:
Code: [Select all] [Expand] [Download]
------------------------------------------
……
……
;======== Copy The Next Functions To Your Code ========
------------------------------------------
Mkonopko
Posts: 21
Joined: 25 Jan 2015, 07:30

Re: FindText - Catch screen image into text and then find it

20 May 2016, 15:59

I tried pasting this first at the beginning of the code then after the code. No luck
Mkonopko
Posts: 21
Joined: 25 Jan 2015, 07:30

Re: FindText - Catch screen image into text and then find it

20 May 2016, 16:07

Here is my code:

; Note: parameters of the X,Y is the center of the coordinates,
; and the W,H is the offset distance to the center

FindText(x,y,Text,c,w=150,h=150,ByRef rx="",ByRef ry="")
{
xywh2xywh(x-w,y-h,2*w+1,2*h+1,x,y,w,h,c)
if (w<1 or h<1)
Return, 0
v:=Trim(RegExReplace(Text,"[^0_\n]+"),"`n") . "`n"
w2:=InStr(v,"`n")-1, h2:=StrLen(v)//(w2+1)
fmt:=A_FormatInteger
SetFormat, IntegerFast, d
re1:="[0_]{" (w2+0) "}\n", re2:=".{" (w-w2) "}"
SetFormat, IntegerFast, %fmt%
if (w2<1 or RegExReplace(v,re1)!="")
Return, 0
bch:=A_BatchLines
SetBatchLines, -1
if InStr(c,"-")
re:=StrReplace(v,"_","1"), c:=StrReplace(c,"-")
else
re:=StrReplace(StrReplace(v,"0","1"),"_","0")
; Starting from the 1 and 0 mutants can speed up the match
pos:=re~="10|01", re:=SubStr(re,pos+=!pos,-1)
GetBitsFromScreen(x,y,w,h,Scan0,Stride,bits,c)
if !InStr(c,"*") ; Color Mode, it is very fast
ss:=StrGet(&bits,w*h,"utf-16")
else ; Gray Threshold Mode
{
; In order to speed up the speed,
; only to take the green as the gray value
ListLines, Off
Threshold:=StrReplace(c,"*"), i:=-3
VarSetCapacity(tmp,w*h+1,Asc("0")), k:=Asc("1")
Loop, % w*h
if NumGet(bits,i+=4,"uchar")<=Threshold
NumPut(k,tmp,A_Index,"char")
ListLines, On
ss:=StrGet(&tmp+1,w*h,0)
}
SetBatchLines, %bch%
if i:=(ss~=StrReplace(re,"`n",re2))
{
rx:=x+Mod(i-1,w)-Mod(pos-1,w2+1)+(w2-1)//2
ry:=y+(i-1)//w-(pos-1)//(w2+1)+(h2-1)//2
Return, 1
}
Return, 0
}

; Fixed the range to make it no more than screen borders

xywh2xywh(x1,y1,w1,h1,ByRef x,ByRef y,ByRef w,ByRef h,c="*")
{
; Use this rather than A_ScreenWidth etc can support multiple displays
SysGet, zx, 76
SysGet, zy, 77
SysGet, zw, 78
SysGet, zh, 79
left:=x1, right:=x1+w1-1, up:=y1, down:=y1+h1-1
left:=left<zx ? zx:left, right:=right>zx+zw-1 ? zx+zw-1:right
up:=up<zy ? zy:up, down:=down>zy+zh-1 ? zy+zh-1:down
x:=left, y:=up, w:=right-left+1, h:=down-up+1
; Color Mode --> if Mod(w,2)!=0 --> (Stride:=((w*16+31)//32)*4)!=w*2
if !InStr(c,"*")
w:=(w//2)*2
}

GetBitsFromScreen(x,y,w,h,ByRef Scan0,ByRef Stride,ByRef bits,c="*")
{
Ptr:=A_PtrSize ? "UPtr" : "UInt"
; Use this rather than GetDC(0) can support multiple displays
win:=DllCall("GetDesktopWindow", Ptr)
hDC:=DllCall("GetDC", Ptr,win, Ptr)
mDC1:=DllCall("CreateCompatibleDC", Ptr,hDC, Ptr)
hBM1:=DllCall("CreateCompatibleBitmap", Ptr,hDC, "int",w, "int",h, Ptr)
oBM1:=DllCall("SelectObject", Ptr,mDC1, Ptr,hBM1, Ptr)
DllCall("BitBlt", Ptr,mDC1, "int",0, "int",0, "int",w, "int",h
, Ptr,hDC, "int",x, "int",y, "uint",0xCC0020) ; SRCCOPY
if !InStr(c,"*") ; Color Mode
{
; Color c(RGB)-->COLORREF for WinApi=RGB(R,G,B)=BGR
BGR:=((c&0xFF)<<16)|(c&0xFF00)|((c&0xFF0000)>>16)
; All colors are converted to 1 or 0 utf-16 characters
c1:=16bppTo24bpp(Asc("1")), c0:=16bppTo24bpp(Asc("0"))
VarSetCapacity(Rect, 16)
NumPut(0, Rect, 0, "uint"), NumPut(0, Rect, 4, "uint")
NumPut(w, Rect, 8, "uint"), NumPut(h, Rect, 12, "uint")
mDC2:=DllCall("CreateCompatibleDC", Ptr,hDC, Ptr)
hBM2:=DllCall("CreateCompatibleBitmap", Ptr,hDC, "int",w, "int",h, Ptr)
oBM2:=DllCall("SelectObject", Ptr,mDC2, Ptr,hBM2, Ptr)
hBrush1:=DllCall("CreateSolidBrush", "uint",BGR^c1^c0, Ptr)
DllCall("FillRect", Ptr,mDC2, Ptr,&Rect, Ptr,hBrush1)
DllCall("DeleteObject", Ptr,hBrush1)
DllCall("msimg32.dll\TransparentBlt", Ptr,mDC2
, "int",0, "int",0, "int",w, "int",h, Ptr,mDC1
, "int",0, "int",0, "int",w, "int",h, "uint",BGR)
DllCall("BitBlt", Ptr,mDC2, "int",0, "int",0, "int",w, "int",h
, Ptr,mDC1, "int",0, "int",0, "uint",0x660046) ; SRCINVERT
; Color c-->BGR^(BGR^c1^c0)=c1^c0, the others-->X^X=0
hBrush2:=DllCall("CreateSolidBrush", "uint",c0, Ptr)
DllCall("FillRect", Ptr,mDC1, Ptr,&Rect, Ptr,hBrush2)
DllCall("DeleteObject", Ptr,hBrush2)
DllCall("BitBlt", Ptr,mDC1, "int",0, "int",0, "int",w, "int",h
, Ptr,mDC2, "int",0, "int",0, "uint",0x660046) ; SRCINVERT
; Color c-->(c1^c0)^c0=c1, the others-->0^c0=c0
DllCall("SelectObject", Ptr,mDC2, Ptr,oBM2)
DllCall("DeleteObject", Ptr,hBM2)
DllCall("DeleteDC", Ptr,mDC2)
}
DllCall("ReleaseDC", Ptr,win, Ptr,hDC)
VarSetCapacity(bits, w*h*4, 0), bpp:=!InStr(c,"*") ? 16:32
VarSetCapacity(bi, 40, 0)
NumPut(40, bi, 0, "int"), NumPut(w, bi, 4, "int")
NumPut(-h, bi, 8, "int"), NumPut(1, bi, 12, "short")
NumPut(bpp, bi, 14, "short"), NumPut(0, bi, 16, "int")
DllCall("GetDIBits", Ptr,mDC1, Ptr,hBM1
, "int",0, "int",h, Ptr,&bits, Ptr,&bi, "int",0)
DllCall("SelectObject", Ptr,mDC1, Ptr,oBM1)
DllCall("DeleteObject", Ptr,hBM1)
DllCall("DeleteDC", Ptr,mDC1)
Scan0:=&bits, Stride:=((w*bpp+31)//32)*4
}

16bppTo24bpp(c) {
B:=(c&0x1F)<<3, G:=((c>>5)&0x1F)<<3, R:=((c>>10)&0x1F)<<3
Return, (B<<16)|(G<<8)|R
}

;================= The End =================

Text=
(
_______________________________________________________________________
_______________________________________________________________________
_______________________________________________________________________
_______________________________________________________________________
_______________________________________________________________________
_00000__________0________0_____________________________________________
00___00__________________0_____________________________________________
0_____0__________________0_____________________________________________
0_____00_0___0__0___000__0__00____0000___000___000___000__000__000_____
0_____00_0___0__0__00____0_0_________0__00____00____0___0_0____0_______
0_____00_0___0__0__0_____000_______000__0_____0_____00000_00___00______
0_____0__0___0__0__0_____000______0__0__0_____0_____0______00___00_____
00___00__0___0__0__00____0_00_____0__0__00____00____00______00___00____
_00000____0000__0___000__0__00____0000___000___000___0000_000__000_____
______00_______________________________________________________________
_______________________________________________________________________
_______________________________________________________________________
_______________________________________________________________________
_______________________________________________________________________
_______________________________________________________________________
_______________________________________________________________________
_______________________________________________________________________
_______________________________________________________________________
_______________________________________________________________________
_______________________________________________________________________
)

if FindText(81,134,Text,"*147",1500,1500,X,Y)
{
CoordMode, Mouse
MouseMove, X, Y
}
Mkonopko
Posts: 21
Joined: 25 Jan 2015, 07:30

Re: FindText - Catch screen image into text and then find it

20 May 2016, 19:37

Got it!!! This will be very useful to me. I'd like to recommend that you automatically post the additional code to the file.
Anyway many thanks for your help.

Marty Konopko

Return to “Scripts and Functions”

Who is online

Users browsing this forum: No registered users and 9 guests