Easy OCR

[Avasilev]

@Avasilev, narrowing FindStrings area shouldn't make the search too much faster, because most of the slow operations are capturing the screenshot and the OCR itself, whereas the FindStrings narrowing simply ignores results which coordinates are located in that area. I added a way to partially OCR a window as you asked, where now it's possible to specify an object with coordinates for argument onlyClientArea, Example7 demonstrates the usage. So the FindStrings equivalent should be OCR.FromWindow("Untitled - Notepad", , {x:100, y:100, w:900, h:900}). Note that you aren't using "mode" 1 with FromWindow, it's actually the "scale" factor (a bigger scale factor scales up the image and the OCR will take longer, yet be more accurate).

Wow, thanks, the function became much resource-efficient, if a large number of parallel searches are done. As for the "mode 1" yeah that was my mistake - should have been more accurate. Great job!

[Felix Siano]

What an incredible script!
As in example 3, when clicking with the left mouse, paste the text to the clipboard and close the script? It's possible?

[Saear]

Great script.

I'm trying to pull sets of numbers using the variable := OCR.FromRect(,,2). If I output the variable in a message box I get the expected numbers. If I try to include it in a gui or if statement I get (Error: Expected a Number but got an OCR.). What am I doing wrong? This is my first timer using the version 2 code.

Code: Select all

	UTS1 := OCR.FromRect(1720, 1186, 40, 30,,2)	
		if (UTS1 < 90 && UTS1 > 70) {

[Descolada]

@Saear, OCR.FromRect doesn't return text/number but instead an OCR results object which contains all the text, lines, words, and the location of all those. If UTS1 is expected to contain text for just one number then you could use if (UTS1.Text < 90 && UTS1.Text > 70), otherwise loop through the results until you find a line/word that is only one number (eg use IsNumber to check for it).

[Serj-Lewa]

Hello! Have some weird results with OCR.FromFile().

I have a set of files (attached):
screen.png - Original image.
screen.jpg - Original image converted to JPG.
screen_big.jpg - Original image with white margins on the bottom and right and converted to JPG.

I have a script that searches for the word "WATCH" and outputs coordinates:

Code: Select all

#include "OCR.ahk"
files:=["screen.png","screen.jpg","screen_big.jpg"]
for m,name in files {
  result := OCR.FromFile(name,"en")
  w:=result.FindString("WATCH",,1)
  ;MsgBox "x=" w.x  " y="  w.y
  OutputDebug name
  OutputDebug "x=" w.x " y=" w.y
}

When I run my script I get this:

screen.png
x=1261 y=669
screen.jpg
x=1239 y=714
screen_big.jpg
x=1222 y=744

Correct coordinstes are for screen_big.jpg only!

Could somebody help me figure out why this is happening? I'm writing a bot based on OCR and such behavior worries me a little.

[Descolada]

@Serj-Lewa, this is bizarre behavior for which I can't find any explanation from the OCR.ahk library. I think this might be a bug on Microsofts part, perhaps in how the image is converted into IRandomAccessStream, or how certain IRandomAccessStream sizes are handled on the OcrEngine part?

The problem seems to be in the size 1600x900. For example, if I open screen.jpg in Paint and manually resize the width to 1597 pixels or 1603 pixels, and then save, the problem is fixed. However if I change the width back to 1600, then the problem reappears. For screen.png I had to resize the width to exactly 1750 pixels to make the problem disappear. This width 1750 pixels seems to also work for resizing screen_big.jpg (1750x900 works).

Have you tried whether it matters how you take the screenshot? That is, do you use some kind of in-game method, normal Print Screen + paste, or screen snipper tool? Are you playing it in window-mode (not in fullscreen) and is the offset related to the window position? Do you have multiple screens with differing DPIs?

Pinging @iseahound as well, since I don't have much experience with problems related to images. I tried OCR(ImagePutRandomAccessStream(fileName)) which didn't seem to solve the problem, nor did converting it to HBITMAP as an intermediate step. Have you encountered such a problem before?

[Serj-Lewa]

Have you tried whether it matters how you take the screenshot? That is, do you use some kind of in-game method, normal Print Screen + paste, or screen snipper tool?

Are you playing it in window-mode (not in fullscreen) and is the offset related to the window position? Do you have multiple screens with differing DPIs?

I run game in Android emulator BlueStacks 5 in window-mode.
I tried BlueStacks build-in screenshot tool and Android adb tool screencapture. Result are slightly different, but both are incorrect. I also tried making screenshot of whole screen with Print Screen and then cropping it in Paint to 1600x900. Got incorrect result also.

My laptop have a single screen with 1920x1080 resolution, and no additional monitors.

[Descolada]

@Serj-Lewa, I tried cropping a screenshot to 1600x900 as well and it worked properly...

If you are running the game in an emulator, why not use OCR.FromWindow (with onlyClientArea:=1) instead? Then you can use Click/ControlClick/MouseMove etc relative to the client and all the coordinates should be correct. Using FromWindow would avoid a lot of problems for you:
1) If you OCR an image, then all results will be relative to the top left corner of the image, not the screen or emulator window. This means you would have to calculate all offsets yourself, whereas with FromWindow you could either use the correct CoordMode (Window or Client), or one of the OCR.ahk built-in methods (e.g. Result.Click).
2) If you use an emulators built-in screenshot tool then it's hard to determine what scale the image is at (100%? 200%?), what the resolution is, etc. This makes it so much harder to debug.

[Serj-Lewa]

If you are running the game in an emulator, why not use OCR.FromWindow (with onlyClientArea:=1) instead? Then you can use Click/ControlClick/MouseMove etc relative to the client and all the coordinates should be correct.

I tried, and coords are really correct!

I there a way to extract text from minimized or hidden window?

[Descolada]

@Serj-Lewa, glad you got it working! Minimized, no. Hidden, perhaps. FromWindow has two modes of capture, try if one of those works. Otherwise you could also WinMove the window offscreen and capture from there.

[danman]

Since I already included Easy OCR into my scripts, it would be more preferable to utilize it directly instead of using Snipper, which I currently use to run when needed. But I will ask FG anyway.

If you are just wanting to drag a rectangle and then OCR on that area, it is really not that hard.

Here is a simple SelectScreeenRegion function that feeds the area selected into OCR.

Code: Select all

#Requires AutoHotkey v2

; Select Screen Region with Mouse
^#LButton:: ; Control+Win+Left Mouse to Select
{
	Area := SelectScreenRegion("LButton")
	Result := OCR.FromRect(Area.X, Area.Y, Area.W, Area.H)
	MsgBox(Result.Text)
}

Esc:: ExitApp

SelectScreenRegion(Key, Color := "Lime", Transparent:= 80)
{
	CoordMode("Mouse", "Screen")
	MouseGetPos(&sX, &sY)
	ssrGui := Gui("+AlwaysOnTop -caption +Border +ToolWindow +LastFound -DPIScale")
	WinSetTransparent(Transparent)
	ssrGui.BackColor := Color
	Loop 
	{
		Sleep 10
		MouseGetPos(&eX, &eY)
		W := Abs(sX - eX), H := Abs(sY - eY)
		X := Min(sX, eX), Y := Min(sY, eY)
		ssrGui.Show("x" X " y" Y " w" W " h" H)
	} Until !GetKeyState(Key, "p")
	ssrGui.Destroy()
	Return { X: X, Y: Y, W: W, H: H, X2: X + W, Y2: Y + H }
}

FG

Hi all,
This is my first post here and I joined because I have a terrible time running V2 scripts, always values not assigned errors, etc. I have quite a few V1 scripts but this switch has been a nightmare for me. The 'Easy OCR' code is exactly what I needed but it will not run correctly. Others seem to run it ok but not me, what am I missing? This has pretty much been my experience with V2 on two different computers....
I have V2 and the Scite4Autohotkey is the latest. Notice that the SelectScreenRegion etc formatting isn't colored correctly. Thanks for looking.

Screenshot (12).png (142.46 KiB) Viewed 1910 times

[boiler]

what am I missing?

Code: Select all

#Include "OCR.ahk"

[danman]

what am I missing?

Code: Select all

#Include "OCR.ahk"

Thank you! I just assumed if code is pasted it would have everything needed. Now to see if I can bend it to do my will.

[boiler]

If you did paste it into the same script, that would have what it needs. The error message seems to suggest that you didn’t do that.

[nick__]

Hi, @Descolada. Thank you for your work!

Is there a way to force the library to explicitly treat the source as numbers? Something like "numbers only" mode. I'm working with sheets filled with numerical data and in most cases lib shows itself great, but from time to time some sequence of numbers make OCR see it as letters, so I have reliability problem.

[Descolada]

@nick__, welcome to the AHK forums!
Unfortunately the UWP OCR engine doesn't have a way to achieve that. You can do some post-processing yourself though, accounting for some of the more common mistakes. Eg replace all occurrences of "S" with "5" or "I" with "1".

[nick__]

@Descolada, got it, thanks. Will try that.

[hotdogee]

Hi, @Descolada. Thank you for this great library!
I've opened a pull request on GitHub with fixes for GetAvailableLanguages(): https://github.com/Descolada/OCR/pull/2

The current GetAvailableLanguages() implementation gets the list of `Languages` from the `Windows.System.UserProfile.GlobalizationPreferences` class, then checks if each language has OCR support, by calling `IsLanguageSupported` from the `Windows.Media.Ocr.OcrEngine` class. This returns an incomplete list of supported OCR languages since the user may directly install additional OCR languages without adding a system language.

The new implementation fetches the list of `AvailableRecognizerLanguages` directly from the `Windows.Media.Ocr.OcrEngine` class, instead of fetching the list of languages from the `Windows.System.UserProfile.GlobalizationPreferences` class and then checking for OCR support.

How to install an OCR language pack

The following commands install the OCR pack for "ja-JP":

Code: Select all

$Capability = Get-WindowsCapability -Online | Where-Object { $_.Name -Like 'Language.OCR~~~ja-JP~0.0.1.0' }
$Capability | Add-WindowsCapability -Online

[HarryPotter69]

is it working on visual studio? by the wat amazing library to use

[Descolada]

@hotdogee, thanks for the contibution, the request has been merged!

@HarryPotter69, I'm not sure what you mean by "is it working". OCR should work in most programs, including Visual Studio. If the program has used SetWindowDisplayAffinity with WDA_EXCLUDEFROMCAPTURE then OCR (and any other image-based method) won't work out of the box, however even that can be worked around.