Jump to content

Sky Slate Blueberry Blackcurrant Watermelon Strawberry Orange Banana Apple Emerald Chocolate
Photo

[Solved] ImageSearch Failure w/DirectX Fullscreen Game(Win7)


  • Please log in to reply
10 replies to this topic
NoobSawce
  • Members
  • 103 posts
  • Last active: Mar 19 2011 05:11 AM
  • Joined: 15 Feb 2011
*** Final Edit - Had to juggle the subject again to mark my issue as solved. Read further down to get my solution (may not fix all cases). ***

Hi all, first time poster, long time fan. :D

I tried searching, but I only found a few posts that appeared related to my problem, and all of those had answers that didn't apply to my situation. So, here I am.

Basically, my problem is this. I am writing a script for Hitman: Blood Money that will detect whether or not I am in the menu or not (i.e. paused). Currently, my methodology is to search for an instance of a HUD image, which only displays when a game is loaded and not paused.

The way I prepared the sample image was to take a screen shot in full screen mode using Fraps, and then crop the image from that. I then removed part of the background of the image and replaced it with a transparency color (0x00FF00 or Lime), and then saved the file as a 24 bit BMP.

I then wrote the following function to test for the menu (some of the code is for troubleshooting, and not directly related to the function's job, like the three lines starting with PixelGetColor):

HitmanBM_IsGameMenuOpen()
{
	CoordMode, Pixel, Screen
	
	vImageFile := A_MyDocuments . "\AutoHotkey\Img\HitmanBM_HUD.bmp"
	ImageSearch, vImgPosX, vImgPosY, 0, 0, A_ScreenWidth, A_ScreenHeight, *20 *TransLime %vImageFile%
	vErrorLevel := ErrorLevel
	
	PixelGetColor, vPixelColor, 65. 969, RGB
	_LogInfo(A_ThisFunc, "PixelGetColor returned {1}", vPixelColor)
	SoundBeep

	If (vErrorLevel >= 1) {
		_LogInfo(A_ThisFunc, "HUD was not detected - {1}", vErrorLevel)
		;Return True
	} Else {
		_LogInfo(A_ThisFunc, "HUD detected on screen - {1} [{2}, {3}]", vErrorLevel, vImgPosX, vImgPosY)
		;Return False
	}
	
	PixelSearch, vImgPosX, vImgPosY, 0, 0, A_ScreenWidth, A_ScreenHeight, 0xF20000, 20, Fast RGB
	If ErrorLevel {
		_LogInfo(A_ThisFunc, "PixelSearch Failure! [{1}]", ErrorLevel)
	} Else {
		_LogInfo(A_ThisFunc, "PixelSearch Success! [{1}, {2}]", vImgPosX, vImgPosY)
	}
}

I then bound this function to the home key, and fired up the game to test it out. Unfortunately, no matter how many variations I tried, ImageSearch always failed with 1. Basically, this is what my log output said:

[2011-02-15 01:42:09] HitmanBM_IsGameMenuOpen <INF> PixelGetColor returned 0x343434
[2011-02-15 01:42:09] HitmanBM_IsGameMenuOpen <INF> HUD was not detected - 1
[2011-02-15 01:42:09] HitmanBM_IsGameMenuOpen <INF> PixelSearch Failure! [1]

So, I decided to switch the game to windowed mode, and low and behold, the function started working (same image, same location in game, same code), and this was my new log output:

[2011-02-15 01:42:05] HitmanBM_IsGameMenuOpen <INF> PixelGetColor returned 0x121918
[2011-02-15 01:42:05] HitmanBM_IsGameMenuOpen <INF> HUD detected on screen - 0 [61, 968]
[2011-02-15 01:42:05] HitmanBM_IsGameMenuOpen <INF> PixelSearch Success! [65, 969]

There were a few oddities I noticed during troubleshooting, namely with PixelGetColor. Basically, I randomly sampled several different pixels in full screen mode, and I got 0x343434 every time. Also, when the game was in windowed mode, PixelGetColor repeatedly returned a different value for the same pixel, even though the scene in the game didn't change (and was a pixel in the image it already found). So, I guess my question is this, does ImageSearch not work in full screen mode with DirectX, or am I missing something obvious? If not, is there a workaround? (hopefully without Win32 API calls, although I am not above doing that if necessary)

Oh, and before I forget, some environment information. This game is using DirectX (9, I believe), and I am running Win7 Ultimate x64 (although the game is still 32 bit). I am also running AHK_L version 1.0.92.2.

My game resolution is the same as my desktop resolution (I always do that to minimize the nasty flickering you get when Alt-Tabbing), so I don't suspect image scaling issues at the moment. Plus, the ImageSearch call works flawlessly in windowed mode (same image mask, same res), even though the source image mask was culled from a screen shot taken in full screen mode. I also tried PixelSearch in both full screen mode and windowed mode (using a unique color from the HUD), and I got pretty much the same results, namely unequivocal failure in full screen, and total success in windowed mode (in the correct pixel position).

Any help anyone could offer would be greatly appreciated, even if it's telling me I'm SOL. :lol:

EDIT: Quick update, just for craps and giggles, I decided to try the function under different resolutions (windowed mode only), and I can verify that image scaling is not an issue. It turns out the HUD element I am searching for is that same size in all resolutions (verified against screen shots).

Log results in 1440x900:

[2011-02-15 01:57:27] HitmanBM_IsGameMenuOpen <INF> PixelGetColor returned 0x616B66
[2011-02-15 01:57:28] HitmanBM_IsGameMenuOpen <INF> HUD detected on screen - 0 [61, 848]
[2011-02-15 01:57:28] HitmanBM_IsGameMenuOpen <INF> PixelSearch Success! [65, 849]

And in 800x600:

[2011-02-15 01:59:20] HitmanBM_IsGameMenuOpen <INF> PixelGetColor returned 0x060809
[2011-02-15 01:59:21] HitmanBM_IsGameMenuOpen <INF> HUD detected on screen - 0 [61, 548]
[2011-02-15 01:59:21] HitmanBM_IsGameMenuOpen <INF> PixelSearch Success! [65, 549]

Just for reference, the logs from my initial post where done in 1680x1050 (full screen and windowed).

EDIT2: Oh, and yes, I have tried turning off Windows Aero, with no effect. I also tried setting "Disable desktop composition" in the compatibility settings tab, with no effect, either.

EDIT3: I updated the subject to be a bit more descriptive.

MasterFocus
  • Moderators
  • 4323 posts
  • Last active: Jan 28 2016 01:38 AM
  • Joined: 08 Apr 2009
1) Just to make sure: are you running it as admin?
2) You could try something related to GDI+

-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Antonio França -- git.io -- github.com -- ahk4.net -- sites.google.com -- ahkscript.org

Member of the AHK community since 08/Apr/2009. Moderator since mid-2012.


NoobSawce
  • Members
  • 103 posts
  • Last active: Mar 19 2011 05:11 AM
  • Joined: 15 Feb 2011
1) Yes, I am running the game as admin (as well as AutoHotkey). Specifically, the game is currently running in compatibility mode for WinXP SP2 (which seems to default to running as admin; I get the escalation prompt at least).
2) I was afraid you might say that. That will require Win32 API calls, won't it?

I suppose I could try wrapping the GDI+ calls in a C++ Dll and exporting a function for AHK (seems simpler than dealing with a bunch of DllCall's, in my twisted world, at least :lol:), but I still can't escape the feeling that I'd be killing a fly with a shotgun while the ImageSearch function points at me and laughs. :sad:

EDIT: Sorry for not saying it initially, but thank you for the quick reply. :grin:

MasterFocus
  • Moderators
  • 4323 posts
  • Last active: Jan 28 2016 01:38 AM
  • Joined: 08 Apr 2009
GDI+ by tic :!: :)

-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Antonio França -- git.io -- github.com -- ahk4.net -- sites.google.com -- ahkscript.org

Member of the AHK community since 08/Apr/2009. Moderator since mid-2012.


NoobSawce
  • Members
  • 103 posts
  • Last active: Mar 19 2011 05:11 AM
  • Joined: 15 Feb 2011
Ok, I did some more digging, and it doesn't appear the GDI will help me, at least not directly. Apparently, there is a recurring question on the net about how to get a screen capture of a FS DirectX application programmatically (in C++, not AutoIt or AHK), all of which seems to point to a thinly veiled conclusion. The gist of what I gathered is that DirectX on Windows 7 (and in some respects, Vista) is either bypassing the GDI stack completely, or at the very least is operating at an abstraction layer below it (at least in FS mode). In other words, using GDI (and the classic GetDC(NULL) method) to do screen captures won't work reliably to capture a FS DirectX application, and desktop composition and DWM seems to make the problem worse (or at least, more complicated).

The programming solutions I was able to find all pretty much said the same thing; it's possible, quite complicated, and hinges on hooking the application (using something like detour), using IDirect3DDevice9::GetFrontBufferData to get a pointer to a surface that contains the screen data, and then calling IDirect3DSurface9::GetDC to get a blitable device context. However, I really hate dealing with hooking for many reasons I won't go into here, so that's not an option for me anyway (well, I suppose it is, I'm just dismissing it).

So, the problem seems to be squarely with Windows, and AHK is getting screwed because of it. Regardless, I'm going to try a few generic tests in C++, but if none of those pan out, I suppose I can try sending a PrintScreen key press and then doing the comparison against the clipboard image (instead of relying on GetDC). It might be alright for my needs (since resolution doesn't appear to be a factor at the moment), but I guess it's not really a good catch all solution for everyone else. :sad: Either way, if I make any headway, I'll be sure to post back. :)

My kingdom for a piece of Fraps source code right about now. :lol:

MasterFocus
  • Moderators
  • 4323 posts
  • Last active: Jan 28 2016 01:38 AM
  • Joined: 08 Apr 2009
Here is some related stuff:
- Capture Screenshot to JPG
- Save Screen (HDC) as BMP or PNG w/o external tools, GDIPlus
- http://www.autohotke...pic.php?t=11860
If you manage to get something working, you'd "just" need to check pixels from the image file (via GDI+).

-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Antonio França -- git.io -- github.com -- ahk4.net -- sites.google.com -- ahkscript.org

Member of the AHK community since 08/Apr/2009. Moderator since mid-2012.


NoobSawce
  • Members
  • 103 posts
  • Last active: Mar 19 2011 05:11 AM
  • Joined: 15 Feb 2011
So, I just confirmed my suspicions. I decided to write a simple windows app in C++, whose sole purpose was to capture the current screen and then exit. I then wrote a function in AHK to launch the utility and bound it to PrintScreen. For reference, here's the AHK function:

_RunCaptureUtil()
{
	SoundBeep
	
	vScnCapPath := A_MyDocuments . "\AutoHotkey\Bin\ScnCap32.exe"
	RunWait, %vScnCapPath%, _GetBinFolderPath(), UseErrorLevel
	
	If ((ErrorLevel == "ERROR") || (ErrorLevel != 0)) {
		_LogError(A_ThisFunc, """{1}"" failed with code {2}", vScnCapPath, ErrorLevel)
		Return False		
	} Else If (ErrorLevel == 0) {
		_LogInfo(A_ThisFunc, "screen capture utility executed successfully")
		Return True
	}
}

On a somewhat humorous note, my application set off Norton (apparently screen captures are a no-no), so I had to add an exception for my own app; go figure. Anyway, after I got around that, I launched Hitman: BM and hit my PrintScreen key, and this is the screen capture I got (vastly different than what was actually on the screen):


Posted Image

Since AHK is using the same GetDC(NULL) method as my app, it's no wonder poor ImageSearch can't find anything. :sad: And before anyone asks, I did try using CreateDC("DISPLAY", NULL, NULL, NULL) instead of GetDC(NULL), with the same results. It seems the two calls really are one and the same. :(

So, back to the print screen idea and the clipboard. Good news is that I don't have to completely start from scratch, since I can use the AHK_L source code as a reference. I will likely put it in a C Dll and expose a function to call from AHK, along with an AHK wrapper to mimic ImageSearch (maybe _ImageSeachClip?). In any case, if I have any success (which I'm sure I probably will), I'll be sure to post back with a solution, source code, and pre-compiled binaries for everyone. :)

EDIT: Forgot to mention that I got the same screen shot with desktop composition disabled.

tic
  • Members
  • 1934 posts
  • Last active: May 30 2018 08:13 PM
  • Joined: 22 Apr 2007
Does print screen actually work?? if the GetDC doesnt, I find it hard to believe that printscreen will...

NoobSawce
  • Members
  • 103 posts
  • Last active: Mar 19 2011 05:11 AM
  • Joined: 15 Feb 2011
PrintScreen is handled by windows as a pre-registered hotkey, and is sent to the top-level window (via WM_HOTKEY) like any other message. Like a majority of window messages, it is likely not being intercepted and is being passed to the default handler, which is in the context of the operating system. So, it's not inconceivable that the default system handler has ways to access the raw video buffer that a user mode handler wouldn't (or can't). Regardless of how it works, I'm almost positive it's not resorting to user level GDI API's, since it knows how to access the data with less overhead.

In any case, yes, PrintScreen is working in my case (that's how I got my initial screen shots prior to switching to Fraps). Plus, I have already modified the code I mentioned in my last post to send the Alt+PrintScreen key stroke and successfully pulled the data from the clipboard and saved it to a test file.

At the moment I am weeding through the AHK source determining what I need to extract/modify to implement a custom search function that behaves in (roughly) the same way as the built-in function, which I have tenatively dubbed ImageSearchClip. Once that is done and I have a working Dll, I'll be sure to share it with everyone (along with the modified source). :)

NoobSawce
  • Members
  • 103 posts
  • Last active: Mar 19 2011 05:11 AM
  • Joined: 15 Feb 2011
For anyone who may be watching this topic, I just wanted to provide a quick status update on my progress. I have finished extracting and modifying the required code from the AutoHotkey source, and I have finished my first round of testing in Visual Studio (but not from AutoHotkey yet).

So, this is what I have left:

[*:2l8itzo2]Move the code into a Dll (atm it's in a EXE for easier debugging).
[*:2l8itzo2]Create a wrapper function in AHK to connect the library.
[*:2l8itzo2]Run a couple of basic tests within AHK.
[*:2l8itzo2]Package up the source and binaries for distribution.Once I've finished all that, I'll upload and post in the Script & Functions forum, and link to that post here. If all goes well, I'm hoping to have something ready by Tomorrow or Friday at the latest. :)

NoobSawce
  • Members
  • 103 posts
  • Last active: Mar 19 2011 05:11 AM
  • Joined: 15 Feb 2011
Ok here it is: Extended Image Search Library

And this is how I'm using it:
HitmanBM_IsGameMenuOpen()
{
	vImageFile := "*20 *TransLime " . A_MyDocuments . "\AutoHotkey\Img\HitmanBM_HUD.bmp"
	_LogInfo(A_ThisFunc, "aImageFile is ""{1}""", vImageFile)
	
	Clipboard := ""
	Send, {Alt Down}{PrintScreen}{Alt Up}
	ClipWait, , 1
	
	;ImageSearch, vImgPosX, vImgPosY, 0, 0, A_ScreenWidth, A_ScreenHeight, *20 *TransLime %vImageFile%
	_ImageSearchClip(vImgPosX, vImgPosY, 50, A_ScreenHeight - 100, 150, A_ScreenHeight, vImageFile)
	vErrorLevel := ErrorLevel
	
	If (vErrorLevel >= 1) {
		_LogInfo(A_ThisFunc, "HUD was not detected - {1}", vErrorLevel)
		Return True
	} Else {
		_LogInfo(A_ThisFunc, "HUD detected on screen - {1} [{2}, {3}]", vErrorLevel, vImgPosX, vImgPosY)
		Return False
	}
}
And the function now works like a charm. I love it when a plan comes together. :)