Easy OCR

[Descolada]

@Carrottie there are two possible problems with your code:
1. Run doesn't wait for the website to load, so FromDesktop might not capture the page you want to capture
2. Click uses coordinates relative to the window by default, whereas FromDesktop returns coordinates relative to screen.

You might want to take a look at Example 5 which does pretty much what you are trying to do.

[pho7271]

@DescoladaThe search string function FindStrings() can only search for one Chinese character and cannot find multiple Chinese characters. How can this be resolved? thanks

[Carrottie]

@Descolada
Thanks for your prompt reply. Shall try again.

[Descolada]

@pho7271 it appears that one Chinese character is considered a "word" by the Windows OCR engine. This means that the OCR engine reports for example two words "我" and "不", but they won't be matched if the needle is "我不" (one word). Currently the way around this is to separate the characters in the needle as well by spaces, for example

Code: Select all

#Requires AutoHotkey v2
#include <OCR>

needle := "我 不 会 说 中 文"
result := OCR.FromDesktop("zh-CN")
try found := result.FindString(needle)
if !IsSet(found) {
    MsgBox '"' needle '" was not found!'
    ExitApp
}
result.Highlight(found)

Unfortunately I don't immediately see a good way to remedy this. I could try detecting for Chinese characters and remove spaces between them, but I don't speak Chinese and don't know its linguistics, so I'm not sure what consequences it would bring. It would also not be a general solution because other languages (eg Japanese?) might have the same problem.

[Carrottie]

@Carrottie there are two possible problems with your code:
1. Run doesn't wait for the website to load, so FromDesktop might not capture the page you want to capture
2. Click uses coordinates relative to the window by default, whereas FromDesktop returns coordinates relative to screen.

You might want to take a look at Example 5 which does pretty much what you are trying to do.

@Descolada I have tried to run Example5 and the computer return the follow error

[Descolada]

@Carrottie, make sure you have the latest version of OCR.ahk and the latest Example files. If the issue still persists, please PM me your AHK version number, Windows version, and the library folder packaged into a zip file.

[Carrottie]

@Descolada I see, maybe I’m using the older version of OCR.ahk

[pho7271]

@DescoladaWhen I am using the following code now, I found that there are strings that cannot be found, such as the ones highlighted in red in the image. I am not sure if it is because I used the wrong method or if there are special requirements for finding strings. In addition, it was found that there are multiple identical strings on the screen, and the search results may not be found. I am not sure what the reason is?

Code: Select all

    CoordMode "Mouse", "Screen"
    Loop {
        ib := InputBox("Insert RegEx search phrase to find all matches from Desktop: ", "OCR")
        Sleep 500 ; Small delay to wait for the InputBox to close
        if ib.Result != "OK"
            ExitApp
        result := OCR.FromDesktop(,2)
        ;result := OCR.FromDesktop("zh-CN")

        found := result.FindStrings(ib.Value,,RegExMatch)
        if !found.Length {
            MsgBox 'Phrase "' ib.Value '" not found!'
            continue
        }
        for match in found {
            ; MouseMove is set to CoordMode Screen, so no coordinate conversion necessary
            MouseMove match.x, match.y
            result.Highlight(match)
        }
        break
    }

[Mod edit: Added [code][/code] tags. Please use them yourself when posting code!]

[Descolada]

@pho7271 I've noticed the UWP OCR engine has trouble detecting white text on bright colors, so that word might not be detected. Check the output of result.Text whether it contains it altogether or not, and if it doesn't then of course FindString can't find it either.

[Carrottie]

@Carrottie, make sure you have the latest version of OCR.ahk and the latest Example files. If the issue still persists, please PM me your AHK version number, Windows version, and the library folder packaged into a zip file.

I have updated the files. When I run example 5, the OCR cannot recognize Yourself, but when I change to other works with black color like select, it works. I try to adapt that into my own application as follows:

Code: Select all

#Requires AutoHotkey v2
#include ..\Lib\OCR.ahk

Run "https:\\contacts.google.com"

WinWaitActive "HTML input type",,10
if !WinActive("HTML input type") {
    MsgBox "Failed to find test window!"
    ExitApp
}

; Wait for text "Create" to appear, case-insensitive search, indefinite wait. Search only the active window.
result := OCR.WaitText("Create",, OCR.FromWindow.Bind(OCR, "A"))
; Find the Word for "select" in the result, and click it.
result.Click(result.FindString("Create"))

The program end up saying "Failed to find test window!". Is there any error in the coding?

[Descolada]

@Carrottie probably because the window opened by Run "https:\\contacts.google.com" isn't titled "HTML input type". You need to change the WinWaitActive and WinActive arguments as well to match your target window title (or remove the check and replace that part with a small sleep).

[Carrottie]

@Descolada
Thanks a lot! I replaced it with sleep 300 and it works

[Starkom]

[Mod edit: Added translation:]

Hello, how can I make OCR recognize an example in a certain area on the screen and solve it?

a := OCR([406, 261, 87, 36], "eng")

It reads for example: 74 + 24

The pullover copes with the string:

b := (%a%)

But when I export the script, I get an error.

I am not very strong in the ahk language, I sort of realized that the result of reading is not numbers, but a string, but how to translate it into numbers and calculate nowhere found

Original post (in russian):

Spoiler

Здравствуйте, как сделать что бы OCR распознал пример в определенной области на экране и решил его?

a := OCR([406, 261, 87, 36], "eng")

Считывается например: 74 + 24

В пуловере справляется строка:

b := (%a%)

Но когда экспортирую скрипт, вылетает ошибка.

Я не особо силен в языке ahk, вроде как понял что результат считывания не числа, а строка, но как ее перевести в числа и посчитать нигде не нашел

[Descolada]

@Starkom perhaps something like this:

Code: Select all

#Requires AutoHotkey v2
#include <OCR>

result := OCR.FromRect(406, 261, 87, 36, "en-us")
MsgBox eval(RegExReplace(result.Text, "[^\d-+*\\]"))

; Source: https://github.com/TheArkive/eval_ahk2/blob/master/_eval.ahk
; ======================================================================================
; Usage:
;
;   result := eval(math_expr, test := false)
;
; Returns the evaluated string expression as an integer or float.  If the number is
; too large and AHK returns "inf", then an error is thrown.
;
; If you want to test and see if an expression is valid, then set [ test := TRUE ]:
;
;   is_valid := eval(math_expr, true)
;
; If you don't want eval() to throw when a number ends up being too large, then:
;
;   result := eval(math_expr, "nothrow")
;
; Be aware that in the case of using "nothrow", then the string "inf" will be
; returned, and it is up to the coder to decide how to handle that.
; ======================================================================================

eval(e,test:=false) {
    
    nothrow := false
    If test="nothrow"
        nothrow := true, test:=false
    
    e := RegExReplace(e,"(!|~)[ \t]+","$1")
    
    If (test And Trim(e,"`t ") = "")
        return false
    Else If (test) {
        t1 := !RegExMatch(Trim(e),"i)(^[^\d!~\-\x28 ]|! |~ |[g-wyz]+|['\" . '"\$@#%\{\}\[\]\\,;\``_])') ; only return true/false testing "e" as expression
        t2 := ( !InStr(e,"++") && !InStr(e,"--"))
        t3 := !RegExMatch(e,"i)(?<![a-f\dx])[a-f]")
        t4a := !RegExMatch(e,"i)(?<!0)x")
        t4b := !RegExMatch(e,"i)x(?![a-f\d])")
        t4 := (t4a || t4b)
        
        StrReplace(e,"?","?",,&q) ; count question marks
        StrReplace(e,":",":",,&c) ; count colons
        t5 := (q=c)
        
        return (t1 && t2 && t3 && t4 && t5)
    }
    
    If RegExMatch(e,"i)(! |~ |[g-wyz]+|['\" . '"\$@#%\{\}\[\]\\,;\``_])',&m) ; check for invalid characters, non-numbers, invalid punctuation, etc.
        throw Error("Syntax error.`r`n     Reason: " Chr(34) m[1] Chr(34) "`r`n`r`nExpression: " e,-1,"Not a math expression.")
    
    If ( InStr(e,"++") || InStr(e,"--") )
        throw Error("Syntax error.`r`n     Reason: -- and ++ are not valid.",-1,"Not a math expression.")
    
    StrReplace(e,"?","?",,&q) ; count question marks
    StrReplace(e,":",":",,&c) ; count colons
    If (q!=c)
        throw Error("Syntax error.`r`n     Reason: ternary statement must be complete with question mark (?) and colon (:).",-1)
    
    StrReplace(e,"(","(",,&LP), StrReplace(e,")",")",,&RP)
    If (LP != RP)
        throw Error("Invalid grouping with parenthesis.  You must ensure the same number of ( and ) exist in the expression.`r`n`r`nExpression:`r`n    " e,-1)
    
    e := RegExReplace(e,'(?<!\d)\.','0.')                               ; fix instances of decimal without leading integer
    
    While RegExMatch(e, "i)(\x28[^\x28\x29]+\x29)", &m) {               ; match phrase surrounded by parenthesis, inner-most first
        ans := _eval(match := m[0])                                     ; match and calculate result
        ans := (SubStr(ans,1,1) = "-") ? " " ans : ans                  ; resolved sub-expr value, add space for legit negative sign, ie. " -3"
        e := RegExReplace(StrReplace(e,match,ans,,,1),"(!|~) +","$1")   ; perform substitution, remove resulting spaces between !/~ and resolved value
        
        If e="inf"
            Break
    }
    
    If e!="inf"
        e := _eval(e)
    
    If IsInteger(e)
        return Integer(e)
    Else if (e="inf") && nothrow
        return e
    Else if (e="inf")
        throw Error("Number too large.",-1)
    Else
        return Float(e)
}

_eval(e) { ; support function for pure math expression without parenthesis
    If IsNumber(e)
        return e
    
    If RegExMatch(e,"i)(^[^\d!~\-\x28 ]|! |~ |[g-wyz]+|['\" . '"\$@#%\{\}\[\]\\,;\``_])',&m) ; check for invalid characters, non-numbers, invalid punctuation, etc.
        throw Error("Syntax error.`r`n     Reason: " Chr(34) m[1] Chr(34),-1,"Not a math expression.")
    
    Static _n   := "(?:\d+\.\d+(?:e\+\d+|e\-\d+|e\d+)?|0x[\dA-F]+|\d+)"  ; Regex to identify float/scientific notation, then hex, then base-10 numbers.  Only positive.
    Static _num := "([!~\-]*" _n ")"                                   ; Expand number definition to include - / ~ / !
    Static _ops := "(?:\*\*|\*|//|/|\+|\-|>>>|<<|>>|&&|&|\^|"            ; Define list of operators, in order of prescedence.
                 . "\|\|" . "|" . "\|" . "|" . ">=|<=|>|<|!=|==|=|\?|:)"
    
    new_e := "", p := 1, prev_m := ""
    typ := "number", expr := _num           ; Start looking for a number first.
    
    While RegExMatch(e,"i)" expr,&_m,p) {   ; Separate numbers and operators (except !/~ operators) with spaces.
        mat := _m[0]                        ; Capture match pattern.  Pattern starts with "number" / alternates with "oper".
        If (typ="number") {                 ; Alternate the RegEx search between numbers and operators to improve grouping/spacing of the expression.
            mat := RegExReplace(_m[1],"\-(\d+)","#$1") ; find "negative" values, replace "-" with "#"
            typ   := "oper"
            expr  := _ops
        } Else {
            typ  := "number"
            expr := _num
        }
        
        new_e .= ((new_e!="")?" ":"") mat
        p := _m.Pos(0) + _m.Len(0)
    }
    
    e := RegExReplace(new_e," {2,}"," ")                        ; Replace e with spaced-out/grouped expression, and replace multiple spaces with single space.
    old_e := e
    
    Static order := "** !~ */ +- <> &^| >= == && ?:"            ; Order of operations with appropriate grouping.
    Static opers := StrSplit(order," ")
    Static n := "#?" _n                                         ; Basic number defiintion with # in place - for negative numbers.
    
    For i, op in opers {                                        ; Loop through operators in order of prescedence.
        
        If e="inf"
            return e
        
        Switch op {
            
            Case "**":
                
                val2 := "", i_count := 0 ; just in case...
                sub_e := "", new_sub_e := ""                    ; Initialize temp vars for the building of sub-expressions.
                p := 1                                          ; Position tracking.
                fail_count := 0                                 ; These expressions can be broken up in different sections in the main expression, so track search fails.
                Static rg_ex1 := "([#!~\-]*" _n ")( *\*\* *)"   ; RegEx for number and exponent (**).  For 1st iteration in next WHILE loop.
                Static rg_ex2 := "([#!~\-]*" _n ")( *\*\* *)?"  ; RegEx for number and maybe exponent (**) for 2nd+ iteration in next WHILE loop.
                rg_ex := rg_ex1
                
                While (r := RegExMatch(e,"i)" rg_ex,&z,p)) {    ; Extract expr before resolving, because this needs to be right-to-left.
                    (z[2] = "") ? fail_count++ : ""             ; Increment fail count when "**" not found.  fail_count = 1 means the end of a sub-expr, but there may be more.
                    p := z.Pos(0) + z.Len(0)                    ; Adjust search position.
                    sub_e .= z[0]                               ; Append valid match to sub_e.
                    
                    If (fail_count = 1 And InStr(sub_e,"**")) { ; ****** End of exponenent expression, so evalute and replace. ******
                        new_sub_e := sub_e
                        While RegExMatch(new_sub_e,"i)([#!~]*)?(" _n ") *(\*\*) *([#!~\-]*" _n ")$",&y) { ; Get last 2 operands and operator with any unary - ! or ~.
                            mat := y[0]                          ; Capture full match.
                            o_op := y[1]                         ; Outside operators must be solved last, ie. -2 ** 3 is -(2 ** 3).
                            v1 := y[2]                           ; First operand.
                            v2 := StrReplace(y[4],"#","-")       ; Switch "#" to "-"
                            v2 := _eval(v2)                      ; Evaluate the exponent (2nd operand), resolve all ! and ~ first.  This behavior is undocumented in AHK v2.
                            val2 := v1 ** v2                     ; Resolve sub-sub-expression.
                            new_sub_e := RegExReplace(new_sub_e,"\Q" mat "\E$",o_op val2) ; Ensure this substitution only happens at the end of the sub-expression.
                        }
                        
                        RegExMatch(val2,"([\-!~]*)?(" _n ")",&y) ; Check for "-" to convert to "#".
                        If (IsObject(y) And InStr(y[1],"-"))
                            val2 := StrReplace(y[1],"-","#") y[2]
                        e := StrReplace(e,sub_e,new_sub_e,,,1)  ; Replace only the first instance of the match.  Maintain "#" sub for "-".
                        sub_e := "", new_sub_e := ""            ; Reset temp vars / sub-expressions.  
                        p := 1, fail_count := 0                 ; Reset postion tracking and fail_count.  Continue looping for another exponent sub-expression.
                        
                    } Else If (fail_count > 1)                  ; No more exponent expressions to evaluate, so Break.
                        Break
                    
                    (A_Index = 1) ? rg_ex := rg_ex2 : ""        ; Switch to new search right before 2nd iteration.
                }
                
            Case "!~":
                
                While (r := RegExMatch(e,"i)(!|\~)" n,&z)) {    ; Find "inner most" expression and solve first.
                    _op  := z[1]
                    _mat := z[0]
                    v1 := StrReplace(SubStr(_mat,2),"#","-")    ; Omit regex stored operator (! or ~) and convert "#" to "-".
                    
                    If (_op = "!")
                        val2 := !v1
                    Else If (_op = "~") {
                        If !IsInteger(v1)
                            throw Error("Bitwise NOT (~) operator against non-integer value.`r`n     Invalid operation: ~" v1,-1,"Bitwise operation with non-integer.")
                        v1 := Integer(v1)
                        val2 := ~v1
                    } Else
                        throw Error("Unexpected error in NOT (! / ~) expression.",-1,"First char is not ! or ~.`r`n`r`n     Sub-Expression: " _mat)
                    
                    e := StrReplace(e,_mat,StrReplace(val2,"-","#"),,,1) ; Substitute resolved value in main expression.
                    e := RegExReplace(e,"\-(\d+)","#$1")
                    e := RegExReplace(e,"\-#","")                ; The only time a double negative "--" won't throw an error, so "##" will cancel itself out.
                }
                
            Default: ; basic left-to-right operations that need to be grouped together
                    Switch op {
                        Case "*/":  op_reg := "\*|//|/"
                        Case "+-":  op_reg := "\+|\-"
                        Case "<>":  op_reg := ">>>|<<|>>"
                        Case "&^|": op_reg := "\&|\^|\|"
                        Case ">=":  op_reg := ">\=|<\=|>|<"
                        Case "==":  op_reg := "!=|==|="
                        Case "&&":  op_reg := "&&" . "|" . "\|\|" ; && then ||
                        Case "?:":
                            If !(q := InStr(e,"?"))
                                Continue
                            c := InStr(e,":")
                            expr := StrReplace(SubStr(e,1,q-1),"#","-")
                            expr := _eval(expr)
                            res_A := SubStr(e,q+1,c-q-1)
                            res_B := SubStr(e,c+1)
                            e := (expr) ? Trim(res_A) : Trim(res_B)
                            Continue
                    }
                    
                    While (r := RegExMatch(e,"i)(" n ") +(" op_reg ") +(" n ")",&z)) {
                        o := z[2]
                        v1 := StrReplace(z[1],"#","-"), v2 := StrReplace(z[3],"#","-")
                        
                        ; =========================================================
                        ; capture operator-specific errors
                        ; =========================================================
                        If (o = "<<" Or o = ">>") And (!IsInteger(v1) Or !IsInteger(v2) Or v2<0) ; check for invalid expressions
                            throw Error("Invalid expression.`r`n     Expr: " v1 " " o " " v2,-1,"Bit shift with non-integers.")
                        If (o = "/" Or o = "//") And v2=0
                            throw Error("Invalid expression.`r`n     Expr: " v1 " " o " " v2,-1,"Divide by zero.")
                        If (o = "//") And (!IsInteger(v1) Or !IsInteger(v2))
                            throw Error("Invalid expression.`r`n     Expr: " v1 " " o " " v2,-1,"Floor division with non-integer divisor.")
                        If (o = "&" Or o = "^" Or o = "|") And (!IsInteger(v1) Or !IsInteger(v2))
                            throw Error("Invalid expression.`r`n     Expr: " v1 " " o " " v2,-1,"Bitwise operation with non-integers.")
                        
                        (IsFloat(v1)) ? v1 := Float(v1) : (IsInteger(v1)) ? v1 := Integer(v1) : ""
                        (IsFloat(v2)) ? v2 := Float(v2) : (IsInteger(v2)) ? v2 := Integer(v2) : ""
                        
                        Switch o {
                            Case "*":   val2 := v1  *  v2
                            Case "//":  val2 := v1 //  v2
                            Case "/":   val2 := v1  /  v2
                            Case "+":   val2 := v1  +  v2
                            Case "-":   val2 := v1  -  v2
                            Case ">>>": val2 := v1 >>> v2
                            Case "<<":  val2 := v1 <<  v2
                            Case ">>":  val2 := v1 >>  v2
                            Case "&":   val2 := v1  &  v2
                            Case "^":   val2 := v1  ^  v2
                            Case "|":   val2 := v1  |  v2
                            Case ">=":  val2 := v1 >=  v2
                            Case "<=":  val2 := v1 <=  v2
                            Case ">":   val2 := v1  >  v2
                            Case "<":   val2 := v1  <  v2
                            Case "!=":  val2 := v1 !=  v2
                            Case "==":  val2 := v1 ==  v2
                            Case "=":   val2 := v1  =  v2
                            Case "&&":  val2 := v1 &&  v2
                            Case "||":  val2 := v1 ||  v2
                        }
                        
                        e := StrReplace(e,z[0],StrReplace(val2,"-","#"),,,1)
                    }
                    r := 0 ; disable substitution before next iteration in FOR loop, because these subs were already done
        }
        
        If IsNumber(StrReplace(e,"#","-"))
            Break
    }
    
    e := StrReplace(e,"#","-")
    If IsNumber(StrReplace(e,"#","-")) {
        final := StrReplace(e,"#","-")
        If IsInteger(final)
            return Integer(final)
        Else If IsFloat(final)
            return Float(final)
        Else
            throw Error("fix this type: " Type(final),-1) ; this isn't supposed to be here, but just in case there's some weird type conflict, please tell me and post example.
    } Else {
        return e
    }
}