FindText - Capture screen image into text and then find it

Post your working scripts, libraries and tools
feiyue
Posts: 59
Joined: 08 Aug 2014, 04:08

Re: FindText - Catch screen image into text and then find it

27 Jun 2016, 00:05

Currently captured screen image text, to be used on other computers, you must use the same:
1. screen resolution
2. browser magnification
3. Text font
4. smooth screen font edge

So basically, the image is required to be consistent (Allows for a certain tolerance).
Because the text lines are very thin, if you zoom in and out (even if it's very slight),
Text lines will stagger, causing great error,
Even if the use of tolerance is not easy to use.
so the current does not apply to zoom in or out.

The advantage of it over the built-in command ImageSearch is that :
it does not have to use an external image, just use the script, and find faster.

sGuest

Re: FindText - Catch screen image into text and then find it

05 Feb 2017, 15:42

00000000000000000000000000000000000000000000000000000000000000000000000
00000000______0__0000000000000000__0________00000000000000000_000000000
00000000__00000000000000000000000__0000__0000000000000000000__000000000
00000000__00000__00_0____0000____0_0000__00000____00__000_______0000000
00000000__00000__00__000__00__000__0000__0000__00__00__0__00__000000000
00000000__00000__00__0000_00_0000__0000__0000_0000_000_0_000__000000000
00000000______0__00_00000_00_0000__0000__0000______0000_0000__000000000
00000000__00000__00_00000_00_0000__0000__0000_00000000___000__000000000
00000000__00000__00_00000_00_0000__0000__0000_00000000_0_000__000000000
00000000__00000__00_00000_00__000__0000__0000__000000_00__00__000000000
00000000__00000__00_00000_000____0_0000__00000_____0__000__00___0000000
00000000000000000000000000000000000000000000000000000000000000000000000
:think: hmm... interesting... *thinks of use-cases*
User avatar
DataLife
Posts: 188
Joined: 29 Sep 2013, 19:52

Re: FindText - Catch screen image into text and then find it

06 Feb 2017, 15:44

This is an amazing piece of code. I don't know how it works, but it is very fast and accurate.

I would recommend the opening post include this link to ed1chandlers explaination on how to use this code. https://autohotkey.com/boards/viewtopic ... ext#p87947
I don't think we can rely on everyone to read thru all of the replies to this thread to see e1chandles explaination.

I would never have been able to figure out how to use this code without a detailed explanation like ed1chandlers.

thank you feiyue for this great and useful code.
Check out my scripts. (MyIpChanger) (XPSnap) (SavePictureAs)
All my scripts are tested on Windows 7, AutoHotkey 32 bit Ansi unless otherwise stated.
User avatar
SpeedMaster
Posts: 82
Joined: 12 Nov 2016, 16:09

Re: FindText - Catch screen image into text and then find it

10 Feb 2017, 12:44

Hello,

You can also use it as a Hexadecimal Color Picker
feiyue
Posts: 59
Joined: 08 Aug 2014, 04:08

Re: FindText - Catch screen image into text and then find it

16 Feb 2017, 15:06

I added an option for similarity to the color mode,
which can be used in the game for those background transparent text.
Note: when the similarity is less than 1, the tolerance
of the background character (_) should be a little larger,
in some cases it can even be set to 1.
stealzy
Posts: 48
Joined: 01 Nov 2015, 13:43

Re: FindText - Catch screen image into text and then find it

17 Mar 2017, 01:06

I used your function in script for searching ~8 (common color) patterns in common area at one moment.
So, most time get GetBitsFromScreen(). Are you considering divide function in parts?

Code: [Select all] [Download] GeSHi © Codebox Plus

FindText(pattern) { ; simple one pattern usage
GetBitsFromScreen()
Find(pattern)
}
; multiple pattern usage
GetBitsFromScreen()
Find(pattern1)
Find(pattern2)
...
Find(patternN)
For auto release memory on exit from function and for ease of use reasons it can look like:

Code: [Select all] [Expand] [Download] GeSHi © Codebox Plus

Also I thinking about usage base64 for writing pattern.
Current form spectacularly and wow-effect :clap: , but script became too big :) .
feiyue
Posts: 59
Joined: 08 Aug 2014, 04:08

Re: FindText - Catch screen image into text and then find it

19 Mar 2017, 14:12

Your suggestion is very good, I updated script, realized your two needs.
c4p
Posts: 17
Joined: 18 Jan 2017, 18:38

Re: FindText - Catch screen image into text and then find it

24 Mar 2017, 12:42

Updated to accommodate version 5.1 (Code Cleanup by feiyue). Awesome script. Perfect for my needs of interacting with Citrix Windows. Here's my humble attempt to make a functional wrapper. The utility of this is for linking multiple FindText together. For example, You use Findtext to click a webpage link, then try to click a second link after page reload. If the code fails it is hard to determine if you tested too early or if there was a recognition problem. This function bakes in wait time, post find wait times, and verbose tooltip options.

Code: [Select all] [Expand] [Download] GeSHi © Codebox Plus

Last edited by c4p on 28 Jul 2017, 09:08, edited 22 times in total.
eekhelpspike
Posts: 10
Joined: 05 Jul 2015, 17:51

Re: FindText - Catch screen image into text and then find it

04 Apr 2017, 18:56

A program I am manipulating via AHK recently updated and controls are no longer used. Before I delve into trying to utilize this, can someone tell me if I can adapt it to click on icons on a toolbar based on text underneath the icon?
duzymichal
Posts: 13
Joined: 31 Mar 2017, 04:11

Re: FindText - Catch screen image into text and then find it

11 Apr 2017, 14:34

Hi, I'm a newbie who just built his first GUI. I'm really interested in this script and I'm sure once I attach it properly to what I've done I shouldn't have much problem adding the movement and clicking to the whole process. But I have a more basic problem of actually finding my text outside of the test function in the app.

This is what I have. It's purpose is to initiate mouseover when pressing ctrl+q:

Code: [Select all] [Expand] [Download] (Untitled.ahk)GeSHi © Codebox Plus




Could you please steer me in the right direction with this, such as a working example? Thank you in advance.
feiyue
Posts: 59
Joined: 08 Aug 2014, 04:08

Re: FindText - Catch screen image into text and then find it

12 Apr 2017, 05:01

Because the script is updated and is no longer compatible with the previous code,
Please re - capture the screen image, and generate new code. :xmas:
You can modify the "FindText" function to fit your own style. :beer:
c4p
Posts: 17
Joined: 18 Jan 2017, 18:38

Re: FindText - Catch screen image into text and then find it

02 May 2017, 21:59

Version 5.1 Help - credit to echandler's oppst

The author is does not speak English and his/her posts can be hard to follow. As such, I am trying to fully document this amazing function.

FindText Function - Although named FindText, it also works equally well at finding graphics. Other have suggest it should be called "FindMonochromePattern", and just to be cleart this does NOT have anything to do with OCR (text can be found but not interpreted). The function can be thought of as an alternative to native AHK Imagesearch Function. The native function requires saved graphic files, identical image matching, is hard to troubleshoot and performs in a relatively slow manner. AHK Imagesearch can commonly fail for reasons I will not get into. FindText() approaches the search differently. Think 1980's ASCII art. This function abstracts the screen's image into representative 0's and _'s. Because this is an abstraction, not and bit for bit comparison, it allows fast matching and easy adjustments of fault tolerance. The FindText function was rewritten in version 5, so the ASCII art mostly stays behind the scenes now.

General Usage:

1) Target Selection: Click the "Capture" button and the cursor is accompanied by a transparent red box. Move the red box over the text you want to search for and click. Then move the cursor away from the target to capture a clean target image.

2) Search Image Refinement: "Catch Image To Text" window shows a magnified view of the captured text. The buttons to the lower-left allows trimming of the images edges by 1 or 3 pixels in each direction. Auto does automatic trimming of useless whitespace around the edges, but only after Image has been abstracted. "Reset" resets the original Image, "Invert" inverts black and white (only after Color2Two or Grey2Two). The Modify Checkbox allow you to cleanup extraneous pixel from an image by clicking the pixels.

There two comparison modes: color and greyscale. You can choose either. These are slightly different ways to determine if each pixel color/greyscale in the target image becomes simplified to black or white and eventually a "0" or "_". The threshold is the divide between black and white determination. Threshold too low -> all white, Threshold to high -> all black and at these extremes matching accuracy is low.

  • Grey Mode: This one is the simpler of the two. Click the "Gray2Two" button to convert the magnified image into a black/white version. "Grey Threshold" has been automatically calculated and populated. Although not commonly needed, you can manually set the threshold and reclick Grey2Two button again to allow for more or less detail.
  • Color Mode: No automatic Threshold determination on this one. First you must identify a color to separate black and white. For text just click the a black pixel of the word. If it is a non-text graphic select a color that appears to be present commonly in the image by clicking on a pixel in the image. The corresponding hex value of the color selected populates into "Selected Color". When you click the "Color2Two" button, it converts the magnified image into a black and version. The Color Sensitivity slider allows manual adjustment of whether colors slightly darker or lighter than the target color become black or white. For example you do a Color2Two, and don't see much detail then you move the slider to the left, and redo Color2Two. It now shows more detail and is more likely for an accurate match. Slide too far to the left, and again your into less detail and less accuracy.

Optionally, at this point you can click "Auto" to automatically trim off surrounding whitespace. Also, as an option, you can add a comment. The code that is generated is cryptic and without this comment, you are going to have trouble knowing what it is finding. Commenting is suggested. Clicking OK takes you back to original Window now populated with the FindText data. Alternatively, you could have hit "Append" to append additional FindText search code into previously generated code. "Close" just returns to previous window with no changes.

3) Testing: You have been taken back to the original window. For fun you can click the long Text string in the lower pane, and the upper window will show the corresponding ASCIIart searchtext. You can now can click the "Test" button to test that the search will actually find what you're looking for. Click the "Copy" button to copy the text block and the related function call to the clipboard for you to paste into your own AHK code. The checkbox Insert FindText() in Copy allows you to include the needed function in your code (if its the first time) or exclude it if it's already there.

If you just want it to work, you done and you can stop reading now.....

More Advanced Understanding

FindText(X,Y,W,H,err0,err1,Text)

the parameters of X, Y, W, H to used determine a search area
X,Y are the center point about which search was conducted (where your red box was when clicked)
W,H are the offset distance from the center defining the width and height of search area (not the red box's width and height),
W and H are hardcoded to 150000 indicating a search of the entire screen.
So, effectively the search range is (X-W, Y-H)-->(X+W, Y+H).

X, Y, W, and H are calculated for you, and generally you're going to leave them alone.
If we are searching the whole screen, why even have the options? Consider 2 appropriate matches on the screen, Findtext will return the first
one found, but you want the second. Having X,Y allows you to get close to one desired, and constraining the W or H arguments can allow for
avoiding the other match and getting the desired match.

err1 is the character "0" fault-tolerant in percentage (0-1) Default 0.
err0 is the character "_" fault-tolerant in percentage.(0-1) Default 0.

Text is the String of data that represents the match. Text may include multiple of findtexts each separated by "|".

Here is an example of outputted code for a single find...

Code: [Select all] [Download] GeSHi © Codebox Plus

Text.="|<>*139$71.4n3800+95AU0t44E00IHue02G88U00cY5I"
if ok:=FindText(121,44,150000,150000,0,0,Text)
{
CoordMode, Mouse
X:=ok.1, Y:=ok.2, W:=ok.3, H:=ok.4, Comment:=ok.5
MouseMove, X+W//2, Y+H//2
}


Although we are using the same naming convention, these object properties are entirely different than the X,Y,H,W argument submitted to FindText()
X, Y represents top/left coordinates of the found FindText Area. (The above calculation gets you to the center (instead of X and Y being left and top).
H, W represents width and height of find text area (how big was the red search box),
Comment is self explanatory and comes from code between <>'s in Text.

If you sent more than one find strings (separated by |) in Findtext, you can identify the correct array element by thinking:
ArgsPerMatch := 5
Matches := ok.MaxIndex()//ArgsPerMatch
DesiredIndex := (ArgsPerMatch * (DesiredMatch - 1) + DesiredArgNum

or if we wanted 4th Match's X (1st Arg) --- > (5 * (4-1)) + 1 = 16th array index position

FindText failures are unlikely to occur at the time of creation are unusual. It is worth knowing that any match failures with default of 0 error tolerance are automatically retried with 5% error tolerance (0.05). Future failures may happen on a different computer, OS, resolution, text aliasing, etc. You can manually adjust the error tolerance in the Function by adjusting 5th ("o" fault tolerance) and/or 6th ("_" fault tolerance) argument to hopefully gain a match.

The size of the red capture rectangle can be adjusted in the code.
You must ask yourself why you are doing this? Keeping it small works just fine, and keeps it speedy.
Still want it bigger, change the code below in the function
;----------------------------
; The capture range can be changed by adjusting the numbers
ww:=35, hh:=12
nW:=2*ww+1, nH:=2*hh+1
;---------------------------
Last edited by c4p on 13 Jul 2017, 11:30, edited 25 times in total.
feiyue
Posts: 59
Joined: 08 Aug 2014, 04:08

Re: FindText - Catch screen image into text and then find it

04 May 2017, 15:50

You write a very detailed description of the use. Thank you very much! :bravo: :beer:
Because my English is not very good, Can only rely on software translation,
there should be a lot of grammatical errors, so I wrote very little instructions.

The official forum has a lot of enthusiastic net friend,
Make this tool continuously improved, so that it can be better used.
Last edited by feiyue on 07 May 2017, 18:21, edited 5 times in total.
c4p
Posts: 17
Joined: 18 Jan 2017, 18:38

Re: FindText - Catch screen image into text and then find it

04 May 2017, 20:31

feiyue - check your Personal Messages, I sent you an update for consideration.
feiyue
Posts: 59
Joined: 08 Aug 2014, 04:08

Re: FindText - Catch screen image into text and then find it

06 May 2017, 04:47

c4p wrote:feiyue - check your Personal Messages, I sent you an update for consideration.

@c4p, FindText() Updated to v5.1 version, Thank you for your help!
You've done a lot of work to make the software easier to use. :thumbup:
Luffy

Re: FindText - Capture screen image into text and then find it

20 May 2017, 10:39

feiyue, I want to say Thank you for your awesome genius work :thumbup: :bravo: :bravo:


It helped me a lot ,and as an ambitious human being I wanted to take it to the next level :lol: , To make it work in background.
how can I make it work in back ground with ControlClick/Move modes ?
I have tried it but I guess the problem is that If statement can't work in background.
how can I make it detect hidden windows ?
I also tried DetectHiddenWindows, On
I renamed Title to "This is a new title" to avoid conflict.
changed Click , MouseMove to ControlMove,ControlClick

It doesn't Click , only selects the browser. like so https://prntscr.com/fa28cy (Selected) , https://prntscr.com/fa28ky (not selected)

sometimes it brings the desired window "This is a new title" window to the top , but doesn't do anything else.

P.S I have no experience with coding so these are just mere newbie assumptions :D

a fraction of what I want to be done:

Code: [Select all] [Expand] [Download] GeSHi © Codebox Plus



Thank you :wave:
c4p
Posts: 17
Joined: 18 Jan 2017, 18:38

Re: FindText - Capture screen image into text and then find it

20 May 2017, 11:14

See the wrapper I wrote above. It has controlclick functionality. Hidden Windows will not recognize because findtext is searching the screen's graphics. It has been awhile since I treed it, but as I remember, Controlclick works when IE browser is active, but failsed if page controls within browser has focus. Try clicking on IE URL bar, or titlebar then running your script. If that fixes it, it should be a matter coding AHK putting focus on correct part.
feiyue
Posts: 59
Joined: 08 Aug 2014, 04:08

Re: FindText - Capture screen image into text and then find it

20 May 2017, 15:58

When I was in WinXP, I realized the function of searching the background window in FindText,
Because there is a good WinApi (PrintWindow), It can capture image of the hidden window.
But in Win7 and above, PrintWindow is not working well, so I remove this function again.
Last edited by feiyue on 21 May 2017, 09:30, edited 1 time in total.
stealzy
Posts: 48
Joined: 01 Nov 2015, 13:43

Re: FindText - Capture screen image into text and then find it

21 May 2017, 00:29

I use GetDCEx method instead PrintWindow, btw it faster.
Example screenshoter, using Gdip library (uncomment 6 line for deactivate window before screenshot):

Code: [Select all] [Expand] [Download] GeSHi © Codebox Plus


I not found reliable method for take border and titlebar, so I take only client area of window:

Code: [Select all] [Expand] [Download] GeSHi © Codebox Plus

Last edited by stealzy on 24 May 2017, 06:29, edited 1 time in total.

Return to “Scripts and Functions”

Who is online

Users browsing this forum: boiler and 12 guests