FindText - Capture screen image into text and then find it

Post your working scripts, libraries and tools for AHK v1.1 and older
feiyue
Posts: 348
Joined: 08 Aug 2014, 04:08

Re: FindText - Catch screen image into text and then find it

22 May 2016, 17:42

Updated to v4.0 version, through the use of MCode,
the two modes have reached the ultrafast speed ! :beer:
Last edited by feiyue on 27 May 2016, 10:41, edited 4 times in total.
guest3456
Posts: 3453
Joined: 09 Oct 2013, 10:31

Re: FindText - Catch screen image into text and then find it

22 May 2016, 18:16

where/how did you create the MCode?

feiyue
Posts: 348
Joined: 08 Aug 2014, 04:08

Re: FindText - Catch screen image into text and then find it

22 May 2016, 21:03

I use mingw64-gcc to create MCode.
Last edited by feiyue on 29 May 2016, 06:30, edited 19 times in total.
Mkonopko
Posts: 21
Joined: 25 Jan 2015, 07:30

Re: FindText - Catch screen image into text and then find it

22 May 2016, 21:54

guest3456 wrote:where/how did you create the MCode?
Paste the below code into your script up till ";================= The End =================

;"
Lastly, paste the output from findText as I did in my example.
This is a GREAT little program. I have so many uses for this.
Code starts below:


; Note: parameters of the X,Y is the center of the coordinates,
; and the W,H is the offset distance to the center

FindText(x,y,Text,c,w=150,h=150,ByRef rx="",ByRef ry="")
{
xywh2xywh(x-w,y-h,2*w+1,2*h+1,x,y,w,h,c)
if (w<1 or h<1)
Return, 0
v:=Trim(RegExReplace(Text,"[^0_\n]+"),"`n") . "`n"
w2:=InStr(v,"`n")-1, h2:=StrLen(v)//(w2+1)
fmt:=A_FormatInteger
SetFormat, IntegerFast, d
re1:="[0_]{" (w2+0) "}\n", re2:=".{" (w-w2) "}"
SetFormat, IntegerFast, %fmt%
if (w2<1 or RegExReplace(v,re1)!="")
Return, 0
bch:=A_BatchLines
SetBatchLines, -1
if InStr(c,"-")
re:=StrReplace(v,"_","1"), c:=StrReplace(c,"-")
else
re:=StrReplace(StrReplace(v,"0","1"),"_","0")
; Starting from the 1 and 0 mutants can speed up the match
pos:=re~="10|01", re:=SubStr(re,pos+=!pos,-1)
GetBitsFromScreen(x,y,w,h,Scan0,Stride,bits,c)
if !InStr(c,"*") ; Color Mode, it is very fast
ss:=StrGet(&bits,w*h,"utf-16")
else ; Gray Threshold Mode
{
; In order to speed up the speed,
; only to take the green as the gray value
ListLines, Off
Threshold:=StrReplace(c,"*"), i:=-3
VarSetCapacity(tmp,w*h+1,Asc("0")), k:=Asc("1")
Loop, % w*h
if NumGet(bits,i+=4,"uchar")<=Threshold
NumPut(k,tmp,A_Index,"char")
ListLines, On
ss:=StrGet(&tmp+1,w*h,0)
}
SetBatchLines, %bch%
if i:=(ss~=StrReplace(re,"`n",re2))
{
rx:=x+Mod(i-1,w)-Mod(pos-1,w2+1)+(w2-1)//2
ry:=y+(i-1)//w-(pos-1)//(w2+1)+(h2-1)//2
Return, 1
}
Return, 0
}

; Fixed the range to make it no more than screen borders

xywh2xywh(x1,y1,w1,h1,ByRef x,ByRef y,ByRef w,ByRef h,c="*")
{
; Use this rather than A_ScreenWidth etc can support multiple displays
SysGet, zx, 76
SysGet, zy, 77
SysGet, zw, 78
SysGet, zh, 79
left:=x1, right:=x1+w1-1, up:=y1, down:=y1+h1-1
left:=left<zx ? zx:left, right:=right>zx+zw-1 ? zx+zw-1:right
up:=up<zy ? zy:up, down:=down>zy+zh-1 ? zy+zh-1:down
x:=left, y:=up, w:=right-left+1, h:=down-up+1
; Color Mode --> if Mod(w,2)!=0 --> (Stride:=((w*16+31)//32)*4)!=w*2
if !InStr(c,"*")
w:=(w//2)*2
}

GetBitsFromScreen(x,y,w,h,ByRef Scan0,ByRef Stride,ByRef bits,c="*")
{
Ptr:=A_PtrSize ? "UPtr" : "UInt"
; Use this rather than GetDC(0) can support multiple displays
win:=DllCall("GetDesktopWindow", Ptr)
hDC:=DllCall("GetDC", Ptr,win, Ptr)
mDC1:=DllCall("CreateCompatibleDC", Ptr,hDC, Ptr)
hBM1:=DllCall("CreateCompatibleBitmap", Ptr,hDC, "int",w, "int",h, Ptr)
oBM1:=DllCall("SelectObject", Ptr,mDC1, Ptr,hBM1, Ptr)
DllCall("BitBlt", Ptr,mDC1, "int",0, "int",0, "int",w, "int",h
, Ptr,hDC, "int",x, "int",y, "uint",0xCC0020) ; SRCCOPY
if !InStr(c,"*") ; Color Mode
{
; Color c(RGB)-->COLORREF for WinApi=RGB(R,G,B)=BGR
BGR:=((c&0xFF)<<16)|(c&0xFF00)|((c&0xFF0000)>>16)
; All colors are converted to 1 or 0 utf-16 characters
c1:=16bppTo24bpp(Asc("1")), c0:=16bppTo24bpp(Asc("0"))
VarSetCapacity(Rect, 16)
NumPut(0, Rect, 0, "uint"), NumPut(0, Rect, 4, "uint")
NumPut(w, Rect, 8, "uint"), NumPut(h, Rect, 12, "uint")
mDC2:=DllCall("CreateCompatibleDC", Ptr,hDC, Ptr)
hBM2:=DllCall("CreateCompatibleBitmap", Ptr,hDC, "int",w, "int",h, Ptr)
oBM2:=DllCall("SelectObject", Ptr,mDC2, Ptr,hBM2, Ptr)
hBrush1:=DllCall("CreateSolidBrush", "uint",BGR^c1^c0, Ptr)
DllCall("FillRect", Ptr,mDC2, Ptr,&Rect, Ptr,hBrush1)
DllCall("DeleteObject", Ptr,hBrush1)
DllCall("msimg32.dll\TransparentBlt", Ptr,mDC2
, "int",0, "int",0, "int",w, "int",h, Ptr,mDC1
, "int",0, "int",0, "int",w, "int",h, "uint",BGR)
DllCall("BitBlt", Ptr,mDC2, "int",0, "int",0, "int",w, "int",h
, Ptr,mDC1, "int",0, "int",0, "uint",0x660046) ; SRCINVERT
; Color c-->BGR^(BGR^c1^c0)=c1^c0, the others-->X^X=0
hBrush2:=DllCall("CreateSolidBrush", "uint",c0, Ptr)
DllCall("FillRect", Ptr,mDC1, Ptr,&Rect, Ptr,hBrush2)
DllCall("DeleteObject", Ptr,hBrush2)
DllCall("BitBlt", Ptr,mDC1, "int",0, "int",0, "int",w, "int",h
, Ptr,mDC2, "int",0, "int",0, "uint",0x660046) ; SRCINVERT
; Color c-->(c1^c0)^c0=c1, the others-->0^c0=c0
DllCall("SelectObject", Ptr,mDC2, Ptr,oBM2)
DllCall("DeleteObject", Ptr,hBM2)
DllCall("DeleteDC", Ptr,mDC2)
}
DllCall("ReleaseDC", Ptr,win, Ptr,hDC)
VarSetCapacity(bits, w*h*4, 0), bpp:=!InStr(c,"*") ? 16:32
VarSetCapacity(bi, 40, 0)
NumPut(40, bi, 0, "int"), NumPut(w, bi, 4, "int")
NumPut(-h, bi, 8, "int"), NumPut(1, bi, 12, "short")
NumPut(bpp, bi, 14, "short"), NumPut(0, bi, 16, "int")
DllCall("GetDIBits", Ptr,mDC1, Ptr,hBM1
, "int",0, "int",h, Ptr,&bits, Ptr,&bi, "int",0)
DllCall("SelectObject", Ptr,mDC1, Ptr,oBM1)
DllCall("DeleteObject", Ptr,hBM1)
DllCall("DeleteDC", Ptr,mDC1)
Scan0:=&bits, Stride:=((w*bpp+31)//32)*4
}

16bppTo24bpp(c) {
B:=(c&0x1F)<<3, G:=((c>>5)&0x1F)<<3, R:=((c>>10)&0x1F)<<3
Return, (B<<16)|(G<<8)|R
}

;================= The End =================

;

Text=
(
_______________________________________________________________________
_______________________________________________________________________
_______________________________________________________________________
_______________________________________________________________________
_______________________________________________________________________
___00000__________0________0___________________________________________
__00___00__________________0___________________________________________
__0_____0__________________0___________________________________________
__0_____00_0___0__0___000__0__00____0000___000___000___000__000__000___
_00_____00_0___0__0__00____0_0_________0__00____00____0___0_0____0_____
__0_____00_0___0__0__0_____000_______000__0_____0_____00000_00___00____
__0_____0__0___0__0__0_____000______0__0__0_____0_____0______00___00___
__00___00__0___0__0__00____0_00_____0__0__00____00____00______00___00__
___00000____0000__0___000__0__00____0000___000___000___0000_000__000___
________00_____________________________________________________________
_______________________________________________________________________
_______________________________________________________________________
_______________________________________________________________________
_______________________________________________________________________
_______________________________________________________________________
_______________________________________________________________________
_______________________________________________________________________
_______________________________________________________________________
_______________________________________________________________________
_______________________________________________________________________
)

if FindText(79,134,Text,"*147",150,150,X,Y)
{
CoordMode, Mouse
MouseMove, X, Y
}
feiyue
Posts: 348
Joined: 08 Aug 2014, 04:08

Re: FindText - Catch screen image into text and then find it

23 May 2016, 09:53

guest3456 wrote:where/how did you create the MCode?
Although I don't like to use the MCode,
because it is difficult to understand and memory,
but when the need for a faster image processing,
and more powerful functions, to use it,
so abandon many skills, use only machine code is enough!
Last edited by feiyue on 20 Jun 2016, 04:28, edited 7 times in total.
User avatar
oldbrother
Posts: 265
Joined: 23 Oct 2013, 05:08

Re: FindText - Catch screen image into text and then find it

23 May 2016, 15:23

Hi Feiyue, Thank you very much for your great job.

Is it possible to add Fault-Tolerant to your FindText function? This way it will be easier to use a catched profile on different computers.

Here is an example, I catched the same area under different brightness, the difference is minor. Currently your program treat them differently, but I think they should be treated as the same.
Snap1.jpg
Snap2.jpg
guest3456
Posts: 3453
Joined: 09 Oct 2013, 10:31

Re: FindText - Catch screen image into text and then find it

23 May 2016, 18:18

why don't you just make a smaller image that is more likely to be the same across both computers? thats a pretty large image

feiyue
Posts: 348
Joined: 08 Aug 2014, 04:08

Re: FindText - Catch screen image into text and then find it

23 May 2016, 19:11

Updated to v4.5 version, Now already support Fault-Tolerant ! :superhappy:
User avatar
oldbrother
Posts: 265
Joined: 23 Oct 2013, 05:08

Re: FindText - Catch screen image into text and then find it

23 May 2016, 20:18

Wow! Thanks a lot!

Can a user set a maximum Fault-Tolerant rate? For example, 5% max error. It will be used automatically in case of nothing is found on the screen.
Mkonopko
Posts: 21
Joined: 25 Jan 2015, 07:30

Re: FindText - Catch screen image into text and then find it

23 May 2016, 20:33

As long as you are entertaining wishes... how about if found this will allow the trigger of an action like an "if found do this else do that" command.
guest3456
Posts: 3453
Joined: 09 Oct 2013, 10:31

Re: FindText - Catch screen image into text and then find it

24 May 2016, 11:26

Mkonopko wrote:As long as you are entertaining wishes... how about if found this will allow the trigger of an action like an "if found do this else do that" command.
it already does that. perhaps you should read the docs for AHK's if command:
https://autohotkey.com/docs/commands/IfExpression.htm

feiyue
Posts: 348
Joined: 08 Aug 2014, 04:08

Re: FindText - Catch screen image into text and then find it

24 May 2016, 12:05

I improved the search function of C code, greatly improving the search speed. :dance:
Last edited by feiyue on 25 May 2016, 13:16, edited 1 time in total.
feiyue
Posts: 348
Joined: 08 Aug 2014, 04:08

Re: FindText - Catch screen image into text and then find it

24 May 2016, 12:15

oldbrother wrote:Can a user set a maximum Fault-Tolerant rate? For example, 5% max error. It will be used automatically in case of nothing is found on the screen.
OK, Your idea has been realized. This is a good suggestion, and at the present time,
If the first find fails, it will automatically use 5% of the fault tolerance to find another.
You can modify it to suit your needs.
:thumbup:
Last edited by feiyue on 26 May 2016, 21:29, edited 4 times in total.
Mkonopko
Posts: 21
Joined: 25 Jan 2015, 07:30

Re: FindText - Catch screen image into text and then find it

24 May 2016, 15:01

guest3456 wrote:
Mkonopko wrote:As long as you are entertaining wishes... how about if found this will allow the trigger of an action like an "if found do this else do that" command.
it already does that. perhaps you should read the docs for AHK's if command:
https://autohotkey.com/docs/commands/IfExpression.htm
Ok so using this routine I have successfully found the word "hello" Assuming I have a program called abc.exe I would I get this to run?
I can see how to do this in AHK. I cant see a way to have finding "my search word" cause a program to run.
guest3456
Posts: 3453
Joined: 09 Oct 2013, 10:31

Re: FindText - Catch screen image into text and then find it

24 May 2016, 15:05

Mkonopko wrote:
guest3456 wrote:
Mkonopko wrote:As long as you are entertaining wishes... how about if found this will allow the trigger of an action like an "if found do this else do that" command.
it already does that. perhaps you should read the docs for AHK's if command:
https://autohotkey.com/docs/commands/IfExpression.htm
Ok so using this routine I have successfully found the word "hello" Assuming I have a program called abc.exe I would I get this to run?
I can see how to do this in AHK. I cant see a way to have finding "my search word" cause a program to run.
i'm not sure if you're serious or not. this has absolutely nothing to do with this FindText script. this is basic programming for any language

change this:

Code: Select all

if FindText(79,134,Text,"*147",150,150,X,Y)
{
   CoordMode, Mouse
   MouseMove, X, Y
}
to this:

Code: Select all

if FindText(79,134,Text,"*147",150,150,X,Y)
{
   Run, abc.exe
}

Mkonopko
Posts: 21
Joined: 25 Jan 2015, 07:30

Re: FindText - Catch screen image into text and then find it

24 May 2016, 16:37

guest3456 wrote:
Mkonopko wrote:
guest3456 wrote:
Mkonopko wrote:As long as you are entertaining wishes... how about if found this will allow the trigger of an action like an "if found do this else do that" command.
it already does that. perhaps you should read the docs for AHK's if command:
https://autohotkey.com/docs/commands/IfExpression.htm
Ok so using this routine I have successfully found the word "hello" Assuming I have a program called abc.exe I would I get this to run?
I can see how to do this in AHK. I cant see a way to have finding "my search word" cause a program to run.
i'm not sure if you're serious or not. this has absolutely nothing to do with this FindText script. this is basic programming for any language

Obviously not a programmer at all, but now that I see what you've down it makes perfect sense. Thank you.

change this:

Code: Select all

if FindText(79,134,Text,"*147",150,150,X,Y)
{
   CoordMode, Mouse
   MouseMove, X, Y
}
to this:

Code: Select all

if FindText(79,134,Text,"*147",150,150,X,Y)
{
   Run, abc.exe
}
ToKa
Posts: 26
Joined: 23 Apr 2016, 04:37

Re: FindText - Catch screen image into text and then find it

03 Jun 2016, 19:34

very nice and useful function, thanks for sharing
User avatar
jigga
Posts: 93
Joined: 24 Jan 2014, 00:31

Re: FindText - Catch screen image into text and then find it

24 Jun 2016, 19:22

This is super cool! Am I missing something, or is there no way to make the capture area bigger?
feiyue
Posts: 348
Joined: 08 Aug 2014, 04:08

Re: FindText - Catch screen image into text and then find it

25 Jun 2016, 13:08

jigga wrote:This is super cool! Am I missing something, or is there no way to make the capture area bigger?
; The capture range can be changed by adjusting the numbers :dance:
ww:=35, hh:=12
nW:=2*ww+1, nH:=2*hh+1
;----------------------------
User avatar
Relayer
Posts: 160
Joined: 30 Sep 2013, 13:09
Location: Delaware, USA

Re: FindText - Catch screen image into text and then find it

26 Jun 2016, 18:04

Reading the enthusiasm others have for this script, I assumed it may be a good way to capture screen text that can be then be found on screen later even if the "image" has changed slightly. No soap! If I take a snip of the screen and change its size be even 1% the algorithm cannot find it even though the result of the "catch" clearly displays the word. Same goes for fooling with the colors or doing inverse video. So given that, I prefer to use an image clipping tool, save that snipped image as a .png or .jpg and then simply use ImageSearch to find that image on the screen. It suffers from the same problem of being unable to find a slightly modified image but it is a whole lot less code and it only relies on a simple saved image.

Just my $0.02
Relayer

Return to “Scripts and Functions (v1)”

Who is online

Users browsing this forum: JoeWinograd, kashmirLZ, Shifted_Right and 75 guests