FindText - Capture screen image into text and then find it

Post your working scripts, libraries and tools
paulpma
Posts: 18
Joined: 08 Sep 2018, 22:05

Re: FindText - Capture screen image into text and then find it

13 Oct 2018, 21:21

feiyue, thank you so much, it made a difference... I haven't finished testing yet, but results look promising. Thank you.
ras
Posts: 9
Joined: 25 Dec 2016, 11:52
Facebook: rastinder

Re: FindText - Capture screen image into text and then find it

19 Oct 2018, 02:07

Can anybody give me examples
6-7 which includes tolrence
Banayat
Posts: 10
Joined: 08 Jul 2018, 14:29

Re: FindText - Capture screen image into text and then find it

19 Oct 2018, 08:03

Can anybody give me examples
6-7 which includes tolrence
In this video you will find a walk through of this script:
https://www.youtube.com/watch?v=aWRAtvJq9ZE

OR you simply create yourself an example:
1. Open the script
2. Click "Capture" button
3. Capture anything on the screen by right clicking it
4. Set "Color Similarity" to your desire
5. Left click a color within the captured picture
6. Click "Color2Two" or "Gray2Two"
7. Press "Ok" button
8. Press "Test" button
9. Wait a sec for Message Box
10. Wait a sec for mouse movement :xmas:
Last edited by Banayat on 19 Oct 2018, 09:09, edited 1 time in total.
paulpma
Posts: 18
Joined: 08 Sep 2018, 22:05

Re: FindText - Capture screen image into text and then find it

21 Oct 2018, 23:01

I have been working with this script for a couple weeks now and came to stump that I can't figure it out. I am using the following OCR function.

Code: Select all

FindTextOCR(nX, nY, nW, nH, err1, err0, Text, Interval=20)
{
  OCR:="", RightX:=nX+nW-1
  While (ok:=FindText(nX, nY, nW, nH, err1, err0, Text))
  {
    ; For multi text search, This is the number of text images found
    For k,v in ok
    {
      ; X is the X coordinates of the upper left corner
      ; and W is the width of the image have been found
      x:=v.1, y:=v.2, w:=v.3, h:=v.4, comment:=v.5
      ; We need the leftmost X coordinates
      if (A_Index=1 or x<LeftX)
        LeftX:=x, LeftY:=y, LeftW:=w, LeftH:=h, LeftOCR:=comment
      else if (x=LeftX)
      {
        Loop, 100
        {
          err:=A_Index/100
          if FindText(x, y, w, h, err, err, Text)
          {
            LeftX:=x, LeftY:=y, LeftW:=w, LeftH:=h, LeftOCR:=comment
            Break
          }
          if FindText(LeftX, LeftY, LeftW, LeftH, err, err, Text)
            Break
        }
      }
    }
    ; If the interval exceeds the set value, add "*" to the result
    OCR.=(A_Index>1 and LeftX-nX-1>Interval ? "*":"") . LeftOCR
    ; Update nX and nW for next search
    nX:=LeftX+LeftW-1, nW:=RightX-nX+1
  }
  return, OCR
}



And trying to read text from the following image for example:
michael.jpg
(1.33 KiB) Downloaded 237 times
I am using the following Text Characters:

Code: Select all

Text.="|<m>*130$9.UC3kR5cgdZAdWAFU"
Text.="|<i>*130$5.8EV248EV8"
Text.="|<c>*130$7.C8cA210UEIFm"
Text.="|<h>*130$7.UkMA7z1UkMA6"
Text.="|<a>*130$9.20c50c8V4Tm2UA1U"
Text.="|<e>*130$7.zk843z0UE87y"
Text.="|<l>*130$6.UUUUUUUUUzU"
The result comes to the following:
result.jpg
(1.11 KiB) Downloaded 237 times
I think it has to do something with letter i being similar to h and l, but i am using 0,0 for error handlers?

Any ideas? Any help is appreciated.

Thank you.
pedroabs
Posts: 5
Joined: 04 Oct 2017, 23:09

Re: FindText - Capture screen image into text and then find it

22 Oct 2018, 05:24

I have been working with this script for a couple weeks now and came to stump that I can't figure it out. I am using the following OCR function.

Code: Select all

FindTextOCR(nX, nY, nW, nH, err1, err0, Text, Interval=20)
{
 OCR:="", RightX:=nX+nW-1
 While (ok:=FindText(nX, nY, nW, nH, err1, err0, Text))
 {
 ; For multi text search, This is the number of text images found
 For k,v in ok
 {
 ; X is the X coordinates of the upper left corner
 ; and W is the width of the image have been found
 x:=v.1, y:=v.2, w:=v.3, h:=v.4, comment:=v.5
 ; We need the leftmost X coordinates
 if (A_Index=1 or x<LeftX)
 LeftX:=x, LeftY:=y, LeftW:=w, LeftH:=h, LeftOCR:=comment
 else if (x=LeftX)
 {
 Loop, 100
 {
 err:=A_Index/100
 if FindText(x, y, w, h, err, err, Text)
 {
 LeftX:=x, LeftY:=y, LeftW:=w, LeftH:=h, LeftOCR:=comment
 Break
 }
 if FindText(LeftX, LeftY, LeftW, LeftH, err, err, Text)
 Break
 }
 }
 }
 ; If the interval exceeds the set value, add "*" to the result
 OCR.=(A_Index>1 and LeftX-nX-1>Interval ? "*":"") . LeftOCR
 ; Update nX and nW for next search
 nX:=LeftX+LeftW-1, nW:=RightX-nX+1
 }
 return, OCR
}



And trying to read text from the following image for example:
michael.jpg
I am using the following Text Characters:

Code: Select all

Text.="|<m>*130$9.UC3kR5cgdZAdWAFU"
Text.="|<i>*130$5.8EV248EV8"
Text.="|<c>*130$7.C8cA210UEIFm"
Text.="|<h>*130$7.UkMA7z1UkMA6"
Text.="|<a>*130$9.20c50c8V4Tm2UA1U"
Text.="|<e>*130$7.zk843z0UE87y"
Text.="|<l>*130$6.UUUUUUUUUzU"
The result comes to the following:
result.jpg
I think it has to do something with letter i being similar to h and l, but i am using 0,0 for error handlers?

Any ideas? Any help is appreciated.

Thank you.
I figured a way to solve your problem. I am also using OCR in my script and it really works. you just need to learn to do steps that are unusual.

Ok. OCR works by checking from the list top to bottom. So if checks your example from M to L (MICHAEL). Upon checking at the area after C, it checks if I is present. It is not but it sees the first vertical line of H as I. so it resolves to I. then checks again and sees the second vertical line and it resolves to I. The same goes for L.

What I have done is arranged my library in such a way that the most unique letters/numbers go first. Last would be the common-shaped ones like I and 1.

This will solve most of the OCR problems.

Edit:
I backread and saw your problem with 3 and 9. I think 3 is being read in 9 also because 3 is just a 9 without another segment that closes the circle (at least in the calculator). What you can do is rearrange your library so 9 is checked first before 3. In my library, I would put 8 and 9 first and 1 and 7 last.
iseahound
Posts: 321
Joined: 13 Aug 2016, 21:04
GitHub: iseahound

Re: FindText - Capture screen image into text and then find it

23 Oct 2018, 10:02

Feiyue, some questions for you:

1) Why is the image selection so small?
2) Also color2two doesn't work very well. If I select a red square -> grey, sometimes it will match a green square or blue square.
3) Can I make some changes to your script? I hope you will put the project on GitHub
feiyue
Posts: 125
Joined: 08 Aug 2014, 04:08

Re: FindText - Capture screen image into text and then find it

23 Oct 2018, 13:19

@paulpma, FindTextOCR() is mainly used to identify validation codes.
If you want to identify English words and match it with 100%,
you can use 0.00001 instead of 0 as the fault-tolerant value,
avoiding an automatic 10% second attempt.

@iseahound, Updated to the v5.5 version, in order to identify a variety of color verification code,
I improved the color model, now it can adapt to various colors.

When upgrading to the v5.5 version, the specific color mode is modified to color location mode.
When upgrading to the v6.1 version, The specific color mode has been restored, so there are three modes.

Welcome to improve my code. I'm lazy, so the script's interface and functionality haven't improved much.
paulpma
Posts: 18
Joined: 08 Sep 2018, 22:05

Re: FindText - Capture screen image into text and then find it

24 Oct 2018, 18:33

If you want to identify English words and match it with 100%,
you can use 0.00001 instead of 0 as the fault-tolerant value,
avoiding an automatic 10% second attempt.
Feiyue, thank you for pointing out 0.00001 ability of the function. I have done some research and it seems that is has worked...I'll do more testing the next day...Thank you very much.

@pedroabs , thank you for answer in detail... I have tried your solution. It does work for this given problem, but it creates issues with other letters when in use. Seems like modifying error factor to 0.0001 or something like that is the best solution now, but I'm pretty sure I'll use it in future for other purposes...thank you.
leosouza85
Posts: 17
Joined: 22 Jul 2016, 16:28

Re: FindText - Capture screen image into text and then find it

26 Oct 2018, 22:43

@paulpma, FindTextOCR() is mainly used to identify validation codes.
If you want to identify English words and match it with 100%,
you can use 0.00001 instead of 0 as the fault-tolerant value,
avoiding an automatic 10% second attempt.

@iseahound, Updated to the v5.5 version, in order to identify a variety of color verification code,
I improved the color model, now it can adapt to various colors.

When upgrading to the v5.5 version, the specific color mode is modified to color location mode.
When upgrading to the v6.1 version, The specific color mode has been restored, so there are three modes.

Welcome to improve my code. I'm lazy, so the script's interface and functionality haven't improved much.
HI Feiyue, I have a sugestion!

Add a third character to act as an wildcard, the pixels corresponding to that character in the image will be ignored on the search.

Thankyou!!
paulpma
Posts: 18
Joined: 08 Sep 2018, 22:05

Re: FindText - Capture screen image into text and then find it

28 Oct 2018, 06:47

Hi leosouza85,
Can you please be more specific. Because it seems like what are you trying to do is already built-in the function. The function ignores the characters it cannot find.
pk23
Posts: 58
Joined: 24 Apr 2015, 00:49

Re: FindText - Capture screen image into text and then find it

28 Oct 2018, 14:17

Sorry, my English is so poor that I can't guarantee to express myself clearly, so have to ask the author some questions in my native language ( and I know the author can read it ). If this is not allowed, please tell me and I will delete it.

Edit: Basically, I want to constantly check whether the chat box appears on screen, but the actual test shows that 1 of the average 10 tests fails to find out, that is, when the chat box is clearly on the screen, the script says not found. I don't know why. I have tried gray mode, color mode ( with color similarity ), all cannot fix this issue. I'm consulting the author about what may caused this false negative issue.

When I was testing, the screen remained stationary, so the possibility that caused by high-speed motion could be eliminated. The FPS of the game is also above 60. I really can't think of any reason for this issue.

---

请教feiyue兄,我用您这个工具一两年了,很好用。但最近遇到个问题,难住我了

我自用的一个游戏脚本,需要Loop死循环的判断当前屏幕上是否出现“聊天框”,也就是找图找字。根据是否出现聊天框,来决定快捷键的启禁用情况。但是,不管我怎么尝试,都有一定概率不能成功找出来。聊天框明明出现了,却返回“没找到”(打开聊天框,屏幕静止不动,10次里平均约9次成功,1次失败)
Image

我试过抓文字的方式(如图1),也试过抓聊天框的上下边框的方式(如图2);试过灰度模式,也试过偏色模式;err1和err0的容错,我试过调整过低的,也调整过0.7这种高的(但即使调整成高达0.7,也出现过找不到的情况,太奇怪了,按理说这俩参数调高以后,不应该是宁错杀一千,不放过一个么?)
Image
Image

但都有一定概率找不到。基本代码如下。请教:有什么办法解决这个问题吗(比如找100次能成功99次,这个成功率才可以接受吧)?
或者只有这些信息,不足以定位,那么有什么能帮助我进一步troubleshooting的建议吗?
PS:我测试时,屏幕是静止不动的,所以不存在画面高速运动,导致找字不准的问题。游戏的FPS也在60以上,我实在想不到什么原因造成的

Code: Select all

Loop {
        ;如果在游戏内,则检查是否有聊天框,从而判断是否正在聊天打字
        IfWinActive, ahk_group war3WindowGroup
        {
            Text:="|<>0xF8C322@0.80$25.zzzzzzzzk0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000Dzzzw"

            if (ok:=FindText(860-300//2, 582-160//2, 300, 160, 0.7, 0.7, Text)) {
                Suspend, On                                     ;暂停运行
                SoundPlay, %A_WorkingDir%\Sound\1.wav           ;播放声音,在游戏中判断是否成功找到聊天框
            }
            else {
                Suspend, Off                                    ;恢复运行
                SoundPlay, %A_WorkingDir%\Sound\5.wav
            }
        }
        Sleep, 1000                                              ;休息1000ms 防止不停循环 影响性能
    }
    return
Last edited by pk23 on 28 Oct 2018, 15:24, edited 1 time in total.
leosouza85
Posts: 17
Joined: 22 Jul 2016, 16:28

Re: FindText - Capture screen image into text and then find it

28 Oct 2018, 14:27

Hi leosouza85,
Can you please be more specific. Because it seems like what are you trying to do is already built-in the function. The function ignores the characters it cannot find.
Im sugesting, that via a third character, like _ ° *, you can manually select the pixels to ignore, instead the automatic error tolerance... Like if you know that a specific part of the image always change, or you want to avoid a false positive in a specific part of the image
leosouza85
Posts: 17
Joined: 22 Jul 2016, 16:28

Re: FindText - Capture screen image into text and then find it

28 Oct 2018, 14:30

Sorry, my English is so poor that I can't guarantee to express myself clearly, so have to ask the author some questions in my native language ( and I know the author can read it ). If this is not allowed, please tell me and I will delete it.

---

请教feiyue兄,我用您这个工具一两年了,很好用。但最近遇到个问题,难住我了

我自用的一个游戏脚本,需要Loop死循环的判断当前屏幕上是否出现“聊天框”,也就是找图找字。根据是否出现聊天框,来决定快捷键的启禁用情况。但是,不管我怎么尝试,都有一定概率不能成功找出来。聊天框明明出现了,却返回“没找到”(打开聊天框,屏幕静止不动,10次里平均约9次成功,1次失败)
Image

我试过抓文字的方式(如图1),也试过抓聊天框的上下边框的方式(如图2);试过灰度模式,也试过偏色模式;err1和err0的容错,我试过调整过低的,也调整过0.7这种高的(但即使调整成高达0.7,也出现过找不到的情况,太奇怪了,按理说这俩参数调高以后,不应该是宁错杀一千,不放过一个么?)
Image
Image

但都有一定概率找不到。基本代码如下。请教:有什么办法解决这个问题吗(比如找100次能成功99次,这个成功率才可以接受吧)?
或者只有这些信息,不足以定位,那么有什么能帮助我进一步troubleshooting的建议吗?
PS:我测试时,屏幕是静止不动的,所以不存在画面高速运动,导致找字不准的问题。游戏的FPS也在60以上,我实在想不到什么原因造成的

Code: Select all

Loop {
        ;如果在游戏内,则检查是否有聊天框,从而判断是否正在聊天打字
        IfWinActive, ahk_group war3WindowGroup
        {
            Text:="|<>0xF8C322@0.80$25.zzzzzzzzk0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000Dzzzw"

            if (ok:=FindText(860-300//2, 582-160//2, 300, 160, 0.7, 0.7, Text)) {
                Suspend, On                                     ;暂停运行
                SoundPlay, %A_WorkingDir%\Sound\1.wav           ;播放声音,在游戏中判断是否成功找到聊天框
            }
            else {
                Suspend, Off                                    ;恢复运行
                SoundPlay, %A_WorkingDir%\Sound\5.wav
            }
        }
        Sleep, 1000                                              ;休息1000ms 防止不停循环 影响性能
    }
    return
I dont understand your language, but by looking at your code, your error tolerance seems too high

0.7 means that 70% of the pixels can be wrong, so generating false positives...
pk23
Posts: 58
Joined: 24 Apr 2015, 00:49

Re: FindText - Capture screen image into text and then find it

28 Oct 2018, 15:21

I dont understand your language, but by looking at your code, your error tolerance seems too high

0.7 means that 70% of the pixels can be wrong, so generating false positives...
Thank you. The high error tolerance is on purpose, because the issue I encountered is not false positive, but false negative. During the test, I intentionally raised the error tolerance to see if it can eliminate false negatives, but no use, false negatives are still there. I'm consulting the author about what may caused this.

Basically, I want to constantly check whether the chat box appears on screen, but the actual test shows that 1 of the average 10 tests fails to find out, that is, when the chat box is clearly on the screen, the script says not found. I don't know why. I have tried gray mode, color mode ( with color similarity ), all cannot fix this issue.

When I was testing, the screen remained stationary, so the possibility that caused by high-speed motion could be eliminated. The FPS of the game is also above 60. I really can't think of any reason for this issue.
leosouza85
Posts: 17
Joined: 22 Jul 2016, 16:28

Re: FindText - Capture screen image into text and then find it

28 Oct 2018, 16:21

I dont understand your language, but by looking at your code, your error tolerance seems too high

0.7 means that 70% of the pixels can be wrong, so generating false positives...
Thank you. The high error tolerance is on purpose, because the issue I encountered is not false positive, but false negative. During the test, I intentionally raised the error tolerance to see if it can eliminate false negatives, but no use, false negatives are still there. I'm consulting the author about what may caused this.

Basically, I want to constantly check whether the chat box appears on screen, but the actual test shows that 1 of the average 10 tests fails to find out, that is, when the chat box is clearly on the screen, the script says not found. I don't know why. I have tried gray mode, color mode ( with color similarity ), all cannot fix this issue.

When I was testing, the screen remained stationary, so the possibility that caused by high-speed motion could be eliminated. The FPS of the game is also above 60. I really can't think of any reason for this issue.
the chatbox always appear on the same coordinates?
could you send me a full screen print of the game to me so I can do some test and some code to send to you



also try change this:

if (ok:=FindText(860-300//2, 582-160//2, 300, 160, 0.7, 0.7, Text))


to this:

if (ok:=FindText(468, 481, 300, 160, 0.7, 0.7, Text))


I don't understand the need to use of suspend too
pk23
Posts: 58
Joined: 24 Apr 2015, 00:49

Re: FindText - Capture screen image into text and then find it

28 Oct 2018, 17:59

the chatbox always appear on the same coordinates?
could you send me a full screen print of the game to me so I can do some test and some code to send to you
also try change this:

if (ok:=FindText(860-300//2, 582-160//2, 300, 160, 0.7, 0.7, Text))


to this:

if (ok:=FindText(468, 481, 300, 160, 0.7, 0.7, Text))


I don't understand the need to use of suspend too
The position the chat bar appears on the screen is constant, although I'm not sure whether there are any 1px floating here. When I modify the parameter to "468, 481" like what you said, it still succeeds most of the time and fail a few times ( about 10% false negative )

And here is the whole screenshot, which captured at exactly the moment when the false negative occurred using the code below, hoping to figure out what the script saw at that instant and is it really cannot found, but still no use....
https://i.loli.net/2018/10/29/5bd63d5ebf6f1.png

[code]Loop {
IfWinActive, ahk_group war3WindowGroup
{
Text:="|<>0xF8C322@0.80$25.zzzzzzzzk0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000Dzzzw"

if (ok:=FindText(468, 481, 300, 160, 0.1, 0.1, Text)) {
SoundPlay, %A_WorkingDir%\Sound\1.wav
}
else {
SendInput {PrintScreen} ;if cannot find the chat bar, capture the screenshot
SoundPlay, %A_WorkingDir%\Sound\5.wav ;play sound to remind me the chat bar cannot be found
SendInput, #d ;go to the desktop, to prevent this dead loop still going
}
}
Sleep, 1000
}
return[/code] <---I don't know why the bbcode has not been parsed

About the suspend, the whole purpose of this script is to disable the hotkeys the script provided when a chat bar is detected, and enable the hotkeys when it is detected that the chat bar disappears. so that's what suspend do. we don't need it here for testing.
feiyue
Posts: 125
Joined: 08 Aug 2014, 04:08

Re: FindText - Capture screen image into text and then find it

28 Oct 2018, 18:57

@pk23, When search text in the game,
1. Because the background is transparent, it is best to use color mode, and the background character (_) fault tolerance set to 1 (100%).
2. when making a text template, you can check the Modify box to adjust all the text to be black and all the background to be white.
3. Because you ignore the background and only look for text colors, it's best to narrow the search scope to avoid large areas of this color.
4. It's best to use ok:= NearestOK(ok, x, y) to return the sorting results closest to the specified coordinates.

if ( ok:=NearestOK(FindText(860-300//2, 582-160//2, 300, 160, 0.3, 1, Text), 860, 582) ) )
pk23
Posts: 58
Joined: 24 Apr 2015, 00:49

Re: FindText - Capture screen image into text and then find it

28 Oct 2018, 20:29

@pk23, When search text in the game,
1. Because the background is transparent, it is best to use color mode, and the background character (_) fault tolerance set to 1 (100%).
2. when making a text template, you can check the Modify box to adjust all the text to be black and all the background to be white.
3. Because you ignore the background and only look for text colors, it's best to narrow the search scope to avoid large areas of this color.
4. It's best to use ok:= NearestOK(ok, x, y) to return the sorting results closest to the specified coordinates.

if ( ok:=NearestOK(FindText(860-300//2, 582-160//2, 300, 160, 0.3, 1, Text), 860, 582) ) )
Thank you for your help, I just retested it using what you recommended, and still no use. Fortunately, I wanted to record a GIF to clearly illustrate my testing process to you, and this GIF revealed the reason.

This gif flickers very badly, some frames have surprisingly no chat bar, which totally different from what my human eye see. This makes me wonder if the screen image captured by the FindText does not have chat bar too (although completely imperceptible to the human eye), so the "false negative" is reasonable. Turns out it is true. Once I turn the game into window mode, the script works like a charm and 100% can be found.

This reminds me some games, in which players can clearly see the screen with their eyes, but the screenshot software can only capture the black screen. Maybe these are the same reason. Anyway, thank you for your help again.
paulpma
Posts: 18
Joined: 08 Sep 2018, 22:05

Re: FindText - Capture screen image into text and then find it

29 Oct 2018, 09:33

Hi leosouza85,
[/quote]
Im sugesting, that via a third character, like _ ° *, you can manually select the pixels to ignore, instead the automatic error tolerance... Like if you know that a specific part of the image always change, or you want to avoid a false positive in a specific part of the image
[/quote]

Thank you for your clarification. Yes, to me it seems that this ability of function can be very useful. However, this type of coding is way above my abilities and I think it would require a major rewrite of the most of the code, since we are introducing another character. I think its up to feiyue on this one. I am sorry I can't help you here.
paulpma
Posts: 18
Joined: 08 Sep 2018, 22:05

Re: FindText - Capture screen image into text and then find it

29 Oct 2018, 10:17

I am progressing with my FindTextOCR function and have hit a rock that I can't pass by.... Please any help would be appreciated and you guys have been awesome....

It happens the font that is being used by system has same pixel image for letter "l" and "i", when letter "i" is capital. So "II" looks the same to script, when letter i is capital. Bottom line the script is confused between capital i's and l's.

For example system can present name in two formats all caps BIELL, ILLI or Biel, Illi. All caps reading is no issue and works great, but first letter capitalized mode gives a lot of issues. NOTE: it is not a % error issue, because I have checked and in this font "i" capital and lower case "l" look exactly the same. I wish I could change the font.

I think I may have found a solution, but need help with it. Via script I can find out if the name is ALL capitals or NOT. If Not all capitals, then first letter of first and last name will be capital. Example: Biel, Illi. Therefore, at first letter position "B" and after comma position "I" is always capital. Is there a way to get position of of next character? Position of next character after first letter of last name "Bxxxx" and first name Ixxxx (IN BOLD). Why because then we know the following characters are all in lower case and we can then do proper FindTextOCR on lower case letters.

Maybe, there is a better solution to this issue, or not. Any input is highly appropriated.

Thank you all.

YOU have been great.

Return to “Scripts and Functions”

Who is online

Users browsing this forum: Gh0sTG0, Google [Bot], jesusarell, shipaddicted and 38 guests