windows capture, image search and colors search

[thqby]

This library contains 3 types of windows capture, as well as waiting for screen changes, bitmap grayscale, binarization, colors search and image search.

Windows Capture
DXGI Screen capture by DXGI desktop duplication, support for use in multiple threads and only one instance in the process.

DWM Window capture by DwmGetDxSharedSurface, background windows(excluding minimization) are supported, but some windows are not supported.

WGC Window and Monitor capture by winrt Windows.Graphics.Capture, background windows(excluding minimization) are supported, and only win10 1903 or above is supported.

Image Search
Image search supports RGB three channel color difference, treats the specified color of target image as transparent color (the source image ignores the alpha channel, and the color with alpha < 255 is also regarded as transparent color), similarity (it is allowed to match some pixels) and multiple matching results.
Source image and target image can be loaded by file, Base64 string, memory, capture, etc.

Colors Search
Colors search supports ARGB channel color difference(target colors without alpha will ignore the alpha of the source image color), multiple colors with relative position, similarity (it is allowed to match some pixels) and multiple matching results.

Grayscale and binarization
Generate grayscale or black-and-white images.

Download on GitHub

Dll Source on GitHub

example

Code: Select all

dxcp := wincapture.DXGI()

F7:: {
	t := A_TickCount, i := 0
	loop n := 100000
		; capture full screen
		try {
			bb := dxcp.captureAndSave()
			i++
			continue
		} catch TimeoutError	; DXGI_ERROR_WAIT_TIMEOUT
			continue
	t := A_TickCount - t
	MsgBox (t / i) 'ms (valid frame)`n' (t / n) 'ms (total)'
}

box := Buffer(16,0)
; capture range
x := Random(0, A_ScreenWidth - 200), y := Random(0, A_ScreenHeight - 200)
NumPut("uint", x, "uint", y, "uint", x + 200, "uint", y + 200, box)

if dxcp.waitScreenChange(15000, box)
	MsgBox "screen0[" x "," y "," (x + 200) "," (y + 200) "] has changed"
else MsgBox("wait timeout")

; save to bmp
try dxcp.captureAndSave(box).save('1.bmp')
dxcp.freeBuffer()	; It's not necessary

cb := CallbackCreate(revice)
; revice by callback
dxcp.capture(cb)
dxcp.release()	; It's not necessary

revice(pdata, pitch, sw, sh, tick) {
	if tick && pdata {
		bb := BitmapBuffer(pdata, pitch, sw, sh)
		
		; find picture
		t3 := A_TickCount
		if bb.findPic(&x,&y, t := BitmapBuffer.loadPicture("1.bmp"))
			t3 := A_TickCount - t3, fillcolor(x, y, 0xff0000ff, 40)

		; get pixel color
		color := bb[Random(0, A_ScreenWidth - 1), Random(0, A_ScreenHeight - 1)]
		; search pixel
		t1 := A_TickCount
		if bb.findColor(&x, &y, color)
			t1 := A_TickCount - t1, fillcolor(x, y, 0xff00ff00)

		; search multi pixel combination
		arr := [], x := Random(0, A_ScreenWidth), y := Random(0, A_ScreenHeight)
		loop 9 {
			xx := Random(-x, A_ScreenWidth - x - 1), yy := Random(-y, A_ScreenHeight - y - 1)
			arr.Push([bb[x + xx, y + yy], xx, yy])
		}
		t2 := A_TickCount
		if bb.findMultiColors(&x, &y, arr) {
			t2 := A_TickCount - t2
			for a in arr
				fillcolor(a[2] + x, a[3] + y, 0xffff0000)
		}

		; preview
		bb.show("src")
		t.show("dst")
		MsgBox "findColor: " t1 "ms`nfindMultiColors: " t2 "ms`nfindPic: " t3 "ms"
	}
	fillcolor(x, y, color, r := 20) {
		loop r {
			i := A_Index - 1
			loop r
				bb[x + i, y + A_Index - 1] := color
		}
	}
}

[iseahound]

Congrats! I was thinking the same thing but it would have taken me much longer to build it. You will have to teach me how to compile dlls sometime.

[guest3456]

dll source?

why compile to dll instead of writing in pure ahk?

[thqby]

Finding a picture requires a lot of operations, and using pure AHK will be very slow.

The capture work of WGC runs in free threads, while AHK is single threaded.

[thqby]

@iseahound
It is very easy to get the dll with the exported items according to the vs new project wizard.

[iseahound]

Can you share the source so I can learn? Thanks!

+1 guest3456

[tuzi]

I created two simpler examples to help users understand how to use them.

example 1
capture full screen and show.

Code: Select all

dxcp := wincapture.DXGI()

F7:: {
	bb := dxcp.captureAndSave()
	bb.show()
}

#include wincapture.ahk

example 2
capture area and save to file.

Code: Select all

dxcp := wincapture.DXGI()

F7:: {
	; capture range
	x1 := 100, y1 := 100
	x2 := 300, y2 := 400
	box := Buffer(16,0)
	NumPut("uint", x1, "uint", y1, "uint", x2, "uint", y2, box)
	
	; save to 1.bmp
	dxcp.captureAndSave(box).save('1.bmp')
	msgbox 'Screenshots have been saved as 1.bmp'
	run '1.bmp'
}

#include wincapture.ahk

[swizzlefoshizzle]

I thought this was too good to be true -- I've been hoping for this exact library in v2 for a long time. Thank you! This is incredible stuff.

One thing -- I'm struggling to get BitmapBuffer.findPic() working when the needle bitmap has transparent pixels. I'm using paint.net to make unwanted pixels completely transparent, and saving it as a 24bpp .bmp file (it's either 8bpp or 24bpp with paint.net). Any suggestions on what I could try to make this work?

[thqby]

@swizzlefoshizzle

Except saved as 32bpp, you can specify a color as transparent, as same as ImageSearch's option *Trans.
The param variation of findPic and findAllPic is equal to the transparent_color << 32 | variation.
For example

Code: Select all

variation := 0xffffffff << 32 | 0 ; it will ignore the white in the image

variation := 0xff000000 << 32 | 0x050505 ; it will ignore the black in the image, and allowed number of shades of variation in either direction for the intensity of the red, green, and blue components of each pixel's color

[swizzlefoshizzle]

EDIT: I was struggling with the variation parameter in BitmapBuffer.findAllPic() -- you can see how I got it working here.

In short, transparent_color is only necessary when handling .bmp files with a bit depth below 32 (not sure about 64 bit .bmps). Make your pixels transparent, save it as a .bmp with 32 bit depth, and then specify the R/G/B channel variations in a single hex value:

Code: Select all

variationR := variationG := variationB := 10
variation := Format("0x{1:02X}{2:02X}{3:02X}", variationR, variationG, variationB)

coords := screen.findAllPic(needle, , , variation)

It helped that I was able to save .bmps at 32bpp after updating paint.net

Here's my code in case it's helpful to some:

Code: Select all

needleFile := A_ScriptDir "\needle.bmp"

; Results are not correct when the file doesn't exist
if !FileExist(needleFile) {
	MsgBox("Needle file not found: " needleFile)
	return
}

wgc := wincapture.WGC()

needle := BitmapBuffer.loadPicture(needleFile)

; Note: regardless of capture method, make sure it's working with screen.show()
screen := wgc.capture([0, 0, A_ScreenWidth, A_ScreenHeight])

variationR := variationG := variationB := 10
variation := Format("0x{1:02X}{2:02X}{3:02X}", variationR, variationG, variationB)

coords := screen.findAllPic(needle, , , variation)
if coords {
	MsgBox(coords.Length " matches found")
	for i, c in coords {
		Sleep 250
		MouseMove(c.x, c.y)
		MsgBox(c.x ", " c.y)
	}
} else {
	MsgBox("No matches found")
}

My original reply:

I'm getting a 0xc0000005 error from a DLL call when variation is specified like so:

Code: Select all

coords := screen.findAllPic(needle, 1.0, 10, (0xff000000 << 32 | 0x050505), 0)

Here's the error:

akh-wincap-dll-error.png (16.13 KiB) Viewed 6493 times

This doesn't happen when variation := 0


The same needle .bmp works with LoadPicture() and ImageSearch with some options:

Code: Select all

needle := LoadPicture(needleBmpFile)

Sleep 100
if ImageSearch(&x, &y, 1200, 0, A_ScreenWidth, A_ScreenHeight, "*TransBlack *10 HBITMAP:*" needle) {
	MouseMove(x, y)
}

[thqby]

@swizzlefoshizzle
I've fixed it. Download dlls again.

[swizzlefoshizzle]

Great stuff!

I'm not 100% sure but I think the default regions for DWM are off? With dwm.capture(hwnd, 0) I get a white border around the left, right, and bottom edges, and dwm.capture(hwnd, 1) shows a border on the left edge. Pixels along some of these edges are lost in the capture.

This code replicates the problem on my computer. I included adjustments which shift the region to exactly match the client area.

Code: Select all

#Include <wincapture/wincapture>

MsgBox("Press OK to run capture test")
Sleep 100

winTitle := "A"
DIR := A_ScriptDir "\"

dwm := wincapture.DWM()
hwnd := WinGetId(winTitle)

dwm.capture(hwnd, 0).save(DIR "dwm-window-capture-0.bmp")
dwm.capture(hwnd, 1).save(DIR "dwm-client-capture-1.bmp")

WinGetClientPos(&clientX, &clientY, &clientWidth, &clientHeight, winTitle)
WinGetPos(&winX, &winY, &winWidth, &winHeight, winTitle)

shiftX := clientX - winX
shiftY := clientY - winY

dwm.capture(hwnd, [shiftX, shiftY, clientWidth + shiftX, clientHeight + shiftY])
			 .save(DIR "dwm-client-corrected.bmp")

[thqby]

With dwm.capture(hwnd, 0), captured image has a white border around the left, right, and bottom edges, but the width and height of the image are same with the result of WinGetPos.
However, wgc will not have the border, it should be better. With dwm.capture(hwnd, 1), the image is not exactly match the client area, because it uses the same calculation as the WGC and not suitable for this.

[Finallf]

I'm using gdip_all to take a screenshot of a part of the screen and save it.
Then I go to leptonica >tesseract>text.txt
load text.txt to a variable.

Something like:

Spoiler

Code: Select all

	pToken := Gdip_Startup()

	snap := Gdip_BitmapFromScreen("44|814|150|22")

	Gdip_SaveBitmapToFile(snap, "./imgs/ss.tif")

	Gdip_DisposeImage(snap)

.
With this lib I could do the same, and already transform the saved file into grayscale, or even black and white?

That way I wouldn't need to use gdip_all, correct?

I saw that there are 3 methods to take the screeshot, which would be the fastest of the 3 options?

Could you post an example of use?

Thanks.

[thqby]

Yes, but it only supports saving as 32bpp or 24bpp BMP.

There should be no obvious difference in the effective screen capture speed of the three. DXGI will return overtime because there is no new frame on the screen. DWM is suitable for window capture, but some windows may not support this. WGC has no advantage if it is only an occasional screen/window capture.

[doubledave22]

Does DXGI capture work on background windows? Like ones fully covered by other windows or is it just the desktop?

If it does work on background windows any chance of a v1 version??

[malcev]

Does DXGI capture work on background windows?

No.

[frifix]

Hi,

How use this library in order to find a picture on the screen with a threshold? Recently I have struggled with this problem. Here is my code:

Code: Select all

dxcp := wincapture.DXGI()

screen := dxcp.captureAndSave()
screen := screen.threshold(64)

image := BitmapBuffer.loadPicture("picture.bmp")
image := image.threshold(64)
if screen.findPic(&FoundX, &FoundY, image)
    MsgBox("Picture found at: " FoundX ", " FoundY)
else
    MsgBox("No matches found")

Without using a threshold it works. Nevertheless sometimes I have to match focused elements on the screen. I have also found that saved the screen with threshold is completely different than when we show it:

Code: Select all

dxcp := wincapture.DXGI()
    
screen := dxcp.captureAndSave()
screen := screen.threshold(64)
screen.show()
screen.save('screen.bmp')
run 'screen.bmp'

[thqby]

@frifix
When using local adaptive threshold, the common parts of different regions may be slightly different. This is not the case with global thresholds.

[frifix]

@thqby
Ok, but is there any way to use this library like we can use ImageSearch where we can specify number between 0 and 255 (inclusive) to indicate the allowed number of shades of variation in either direction? For example:

Code: Select all

if ImageSearch(&FoundX, &FoundY, 0, 0, A_ScreenWidth, A_ScreenHeight, "*150 picture.bmp")
    MsgBox("Picture found at: " FoundX ", " FoundY)
else
    MsgBox("No matches found")