windows capture, image search and colors search

Post your working scripts, libraries and tools.
User avatar
thqby
Posts: 560
Joined: 16 Apr 2021, 11:18
Contact:

windows capture, image search and colors search

02 Jan 2022, 09:03

This library contains 3 types of windows capture, as well as waiting for screen changes, bitmap grayscale, binarization, colors search and image search.

Windows Capture
DXGI Screen capture by DXGI desktop duplication, support for use in multiple threads and only one instance in the process.

DWM Window capture by DwmGetDxSharedSurface, background windows(excluding minimization) are supported, but some windows are not supported.

WGC Window and Monitor capture by winrt Windows.Graphics.Capture, background windows(excluding minimization) are supported, and only win10 1903 or above is supported.

Image Search
Image search supports RGB three channel color difference, treats the specified color of target image as transparent color (the source image ignores the alpha channel, and the color with alpha < 255 is also regarded as transparent color), similarity (it is allowed to match some pixels) and multiple matching results.
Source image and target image can be loaded by file, Base64 string, memory, capture, etc.

Colors Search
Colors search supports ARGB channel color difference(target colors without alpha will ignore the alpha of the source image color), multiple colors with relative position, similarity (it is allowed to match some pixels) and multiple matching results.

Grayscale and binarization
Generate grayscale or black-and-white images.

Download on GitHub

Dll Source on GitHub

example

Code: Select all

dxcp := wincapture.DXGI()

F7:: {
	t := A_TickCount, i := 0
	loop n := 100000
		; capture full screen
		try {
			bb := dxcp.captureAndSave()
			i++
			continue
		} catch TimeoutError	; DXGI_ERROR_WAIT_TIMEOUT
			continue
	t := A_TickCount - t
	MsgBox (t / i) 'ms (valid frame)`n' (t / n) 'ms (total)'
}

box := Buffer(16,0)
; capture range
x := Random(0, A_ScreenWidth - 200), y := Random(0, A_ScreenHeight - 200)
NumPut("uint", x, "uint", y, "uint", x + 200, "uint", y + 200, box)

if dxcp.waitScreenChange(15000, box)
	MsgBox "screen0[" x "," y "," (x + 200) "," (y + 200) "] has changed"
else MsgBox("wait timeout")

; save to bmp
try dxcp.captureAndSave(box).save('1.bmp')
dxcp.freeBuffer()	; It's not necessary

cb := CallbackCreate(revice)
; revice by callback
dxcp.capture(cb)
dxcp.release()	; It's not necessary

revice(pdata, pitch, sw, sh, tick) {
	if tick && pdata {
		bb := BitmapBuffer(pdata, pitch, sw, sh)
		
		; find picture
		t3 := A_TickCount
		if bb.findPic(&x,&y, t := BitmapBuffer.loadPicture("1.bmp"))
			t3 := A_TickCount - t3, fillcolor(x, y, 0xff0000ff, 40)

		; get pixel color
		color := bb[Random(0, A_ScreenWidth - 1), Random(0, A_ScreenHeight - 1)]
		; search pixel
		t1 := A_TickCount
		if bb.findColor(&x, &y, color)
			t1 := A_TickCount - t1, fillcolor(x, y, 0xff00ff00)

		; search multi pixel combination
		arr := [], x := Random(0, A_ScreenWidth), y := Random(0, A_ScreenHeight)
		loop 9 {
			xx := Random(-x, A_ScreenWidth - x - 1), yy := Random(-y, A_ScreenHeight - y - 1)
			arr.Push([bb[x + xx, y + yy], xx, yy])
		}
		t2 := A_TickCount
		if bb.findMultiColors(&x, &y, arr) {
			t2 := A_TickCount - t2
			for a in arr
				fillcolor(a[2] + x, a[3] + y, 0xffff0000)
		}

		; preview
		bb.show("src")
		t.show("dst")
		MsgBox "findColor: " t1 "ms`nfindMultiColors: " t2 "ms`nfindPic: " t3 "ms"
	}
	fillcolor(x, y, color, r := 20) {
		loop r {
			i := A_Index - 1
			loop r
				bb[x + i, y + A_Index - 1] := color
		}
	}
}
Last edited by thqby on 23 Jan 2022, 03:07, edited 6 times in total.
iseahound
Posts: 1582
Joined: 13 Aug 2016, 21:04
Contact:

Re: windows capture, colors search and picture search.

02 Jan 2022, 10:22

Congrats! I was thinking the same thing but it would have taken me much longer to build it. You will have to teach me how to compile dlls sometime. 🎉
guest3456
Posts: 3477
Joined: 09 Oct 2013, 10:31

Re: windows capture, colors search and picture search.

02 Jan 2022, 12:50

dll source?

why compile to dll instead of writing in pure ahk?

User avatar
thqby
Posts: 560
Joined: 16 Apr 2021, 11:18
Contact:

Re: windows capture, colors search and picture search.

02 Jan 2022, 19:02

Finding a picture requires a lot of operations, and using pure AHK will be very slow.

The capture work of WGC runs in free threads, while AHK is single threaded.
User avatar
thqby
Posts: 560
Joined: 16 Apr 2021, 11:18
Contact:

Re: windows capture, colors search and picture search.

02 Jan 2022, 19:38

@iseahound
It is very easy to get the dll with the exported items according to the vs new project wizard.
iseahound
Posts: 1582
Joined: 13 Aug 2016, 21:04
Contact:

Re: windows capture, colors search and picture search.

03 Jan 2022, 17:26

Can you share the source so I can learn? Thanks!

+1 guest3456
tuzi
Posts: 223
Joined: 27 Apr 2016, 23:40

Re: windows capture, image search and colors search

17 Jan 2022, 07:28

I created two simpler examples to help users understand how to use them.

example 1
capture full screen and show.

Code: Select all

dxcp := wincapture.DXGI()

F7:: {
	bb := dxcp.captureAndSave()
	bb.show()
}

#include wincapture.ahk
example 2
capture area and save to file.

Code: Select all

dxcp := wincapture.DXGI()

F7:: {
	; capture range
	x1 := 100, y1 := 100
	x2 := 300, y2 := 400
	box := Buffer(16,0)
	NumPut("uint", x1, "uint", y1, "uint", x2, "uint", y2, box)
	
	; save to 1.bmp
	dxcp.captureAndSave(box).save('1.bmp')
	msgbox 'Screenshots have been saved as 1.bmp'
	run '1.bmp'
}

#include wincapture.ahk
swizzlefoshizzle
Posts: 7
Joined: 18 Dec 2021, 02:59

Re: windows capture, image search and colors search

09 Feb 2022, 03:59

I thought this was too good to be true -- I've been hoping for this exact library in v2 for a long time. Thank you! This is incredible stuff.

One thing -- I'm struggling to get BitmapBuffer.findPic() working when the needle bitmap has transparent pixels. I'm using paint.net to make unwanted pixels completely transparent, and saving it as a 24bpp .bmp file (it's either 8bpp or 24bpp with paint.net). Any suggestions on what I could try to make this work?
User avatar
thqby
Posts: 560
Joined: 16 Apr 2021, 11:18
Contact:

Re: windows capture, image search and colors search

09 Feb 2022, 06:18

@swizzlefoshizzle

Except saved as 32bpp, you can specify a color as transparent, as same as ImageSearch's option *Trans.
The param variation of findPic and findAllPic is equal to the transparent_color << 32 | variation.
For example

Code: Select all

variation := 0xffffffff << 32 | 0 ; it will ignore the white in the image

variation := 0xff000000 << 32 | 0x050505 ; it will ignore the black in the image, and allowed number of shades of variation in either direction for the intensity of the red, green, and blue components of each pixel's color
swizzlefoshizzle
Posts: 7
Joined: 18 Dec 2021, 02:59

Re: windows capture, image search and colors search

17 Feb 2022, 11:40

EDIT: I was struggling with the variation parameter in BitmapBuffer.findAllPic() -- you can see how I got it working here.

In short, transparent_color is only necessary when handling .bmp files with a bit depth below 32 (not sure about 64 bit .bmps). Make your pixels transparent, save it as a .bmp with 32 bit depth, and then specify the R/G/B channel variations in a single hex value:

Code: Select all

variationR := variationG := variationB := 10
variation := Format("0x{1:02X}{2:02X}{3:02X}", variationR, variationG, variationB)

coords := screen.findAllPic(needle, , , variation)

It helped that I was able to save .bmps at 32bpp after updating paint.net :)

Here's my code in case it's helpful to some:

Code: Select all

needleFile := A_ScriptDir "\needle.bmp"

; Results are not correct when the file doesn't exist
if !FileExist(needleFile) {
	MsgBox("Needle file not found: " needleFile)
	return
}

wgc := wincapture.WGC()

needle := BitmapBuffer.loadPicture(needleFile)

; Note: regardless of capture method, make sure it's working with screen.show()
screen := wgc.capture([0, 0, A_ScreenWidth, A_ScreenHeight])

variationR := variationG := variationB := 10
variation := Format("0x{1:02X}{2:02X}{3:02X}", variationR, variationG, variationB)

coords := screen.findAllPic(needle, , , variation)
if coords {
	MsgBox(coords.Length " matches found")
	for i, c in coords {
		Sleep 250
		MouseMove(c.x, c.y)
		MsgBox(c.x ", " c.y)
	}
} else {
	MsgBox("No matches found")
}



My original reply:

I'm getting a 0xc0000005 error from a DLL call when variation is specified like so:

Code: Select all

coords := screen.findAllPic(needle, 1.0, 10, (0xff000000 << 32 | 0x050505), 0)
Here's the error:

akh-wincap-dll-error.png
akh-wincap-dll-error.png (16.13 KiB) Viewed 8179 times
This doesn't happen when variation := 0


The same needle .bmp works with LoadPicture() and ImageSearch with some options:

Code: Select all

needle := LoadPicture(needleBmpFile)

Sleep 100
if ImageSearch(&x, &y, 1200, 0, A_ScreenWidth, A_ScreenHeight, "*TransBlack *10 HBITMAP:*" needle) {
	MouseMove(x, y)
}
User avatar
thqby
Posts: 560
Joined: 16 Apr 2021, 11:18
Contact:

Re: windows capture, image search and colors search

17 Feb 2022, 20:49

@swizzlefoshizzle
I've fixed it. Download dlls again.
swizzlefoshizzle
Posts: 7
Joined: 18 Dec 2021, 02:59

Re: windows capture, image search and colors search

18 Feb 2022, 19:05

Great stuff!

I'm not 100% sure but I think the default regions for DWM are off? With dwm.capture(hwnd, 0) I get a white border around the left, right, and bottom edges, and dwm.capture(hwnd, 1) shows a border on the left edge. Pixels along some of these edges are lost in the capture.

This code replicates the problem on my computer. I included adjustments which shift the region to exactly match the client area.

Code: Select all

#Include <wincapture/wincapture>

MsgBox("Press OK to run capture test")
Sleep 100

winTitle := "A"
DIR := A_ScriptDir "\"

dwm := wincapture.DWM()
hwnd := WinGetId(winTitle)

dwm.capture(hwnd, 0).save(DIR "dwm-window-capture-0.bmp")
dwm.capture(hwnd, 1).save(DIR "dwm-client-capture-1.bmp")

WinGetClientPos(&clientX, &clientY, &clientWidth, &clientHeight, winTitle)
WinGetPos(&winX, &winY, &winWidth, &winHeight, winTitle)

shiftX := clientX - winX
shiftY := clientY - winY

dwm.capture(hwnd, [shiftX, shiftY, clientWidth + shiftX, clientHeight + shiftY])
			 .save(DIR "dwm-client-corrected.bmp")
User avatar
thqby
Posts: 560
Joined: 16 Apr 2021, 11:18
Contact:

Re: windows capture, image search and colors search

19 Feb 2022, 06:53

With dwm.capture(hwnd, 0), captured image has a white border around the left, right, and bottom edges, but the width and height of the image are same with the result of WinGetPos.
However, wgc will not have the border, it should be better. With dwm.capture(hwnd, 1), the image is not exactly match the client area, because it uses the same calculation as the WGC and not suitable for this.
Finallf
Posts: 18
Joined: 30 Oct 2020, 13:52

Re: windows capture, image search and colors search

01 Mar 2022, 07:47

I'm using gdip_all to take a screenshot of a part of the screen and save it.
Then I go to leptonica >tesseract>text.txt
load text.txt to a variable.

Something like:
Spoiler
.
With this lib I could do the same, and already transform the saved file into grayscale, or even black and white?

That way I wouldn't need to use gdip_all, correct?

I saw that there are 3 methods to take the screeshot, which would be the fastest of the 3 options?

Could you post an example of use?

Thanks.
User avatar
thqby
Posts: 560
Joined: 16 Apr 2021, 11:18
Contact:

Re: windows capture, image search and colors search

01 Mar 2022, 09:05

Yes, but it only supports saving as 32bpp or 24bpp BMP.

There should be no obvious difference in the effective screen capture speed of the three. DXGI will return overtime because there is no new frame on the screen. DWM is suitable for window capture, but some windows may not support this. WGC has no advantage if it is only an occasional screen/window capture.
doubledave22
Posts: 346
Joined: 08 Jun 2019, 17:36

Re: windows capture, image search and colors search

15 Mar 2022, 11:31

Does DXGI capture work on background windows? Like ones fully covered by other windows or is it just the desktop?

If it does work on background windows any chance of a v1 version?? :pray:
malcev
Posts: 1769
Joined: 12 Aug 2014, 12:37

Re: windows capture, image search and colors search

15 Mar 2022, 12:32

Does DXGI capture work on background windows?
No.
frifix
Posts: 3
Joined: 07 May 2022, 12:09

Re: windows capture, image search and colors search

07 May 2022, 12:22

Hi,

How use this library in order to find a picture on the screen with a threshold? Recently I have struggled with this problem. Here is my code:

Code: Select all

dxcp := wincapture.DXGI()

screen := dxcp.captureAndSave()
screen := screen.threshold(64)

image := BitmapBuffer.loadPicture("picture.bmp")
image := image.threshold(64)
if screen.findPic(&FoundX, &FoundY, image)
    MsgBox("Picture found at: " FoundX ", " FoundY)
else
    MsgBox("No matches found")
Without using a threshold it works. Nevertheless sometimes I have to match focused elements on the screen. I have also found that saved the screen with threshold is completely different than when we show it:

Code: Select all

dxcp := wincapture.DXGI()
    
screen := dxcp.captureAndSave()
screen := screen.threshold(64)
screen.show()
screen.save('screen.bmp')
run 'screen.bmp'
User avatar
thqby
Posts: 560
Joined: 16 Apr 2021, 11:18
Contact:

Re: windows capture, image search and colors search

10 May 2022, 01:21

@frifix
When using local adaptive threshold, the common parts of different regions may be slightly different. This is not the case with global thresholds.
frifix
Posts: 3
Joined: 07 May 2022, 12:09

Re: windows capture, image search and colors search

10 May 2022, 03:42

@thqby
Ok, but is there any way to use this library like we can use ImageSearch where we can specify number between 0 and 255 (inclusive) to indicate the allowed number of shades of variation in either direction? For example:

Code: Select all

if ImageSearch(&FoundX, &FoundY, 0, 0, A_ScreenWidth, A_ScreenHeight, "*150 picture.bmp")
    MsgBox("Picture found at: " FoundX ", " FoundY)
else
    MsgBox("No matches found")

Return to “Scripts and Functions (v2)”

Who is online

Users browsing this forum: No registered users and 28 guests