Easy OCR

[Spitzi]

Hi @Descolada .

I use ocr on a text like

L= 20
M= 40
S= 60

The text I get is L= M= S= 20 40 60

Is there a way to get the text line by line, e.g L=20 M=40 S=60?

Thanks. Great library.

[Descolada]

@Spitzi

Code: Select all

result := OCR.FromDesktop()
text := ""
for line in result.Lines
    text .= line.Text "`n"
MsgBox RTrim(text)

You can separate word by word as well in a similar fashion.

[Spitzi]

Thanks @Descolada

I have screen text formatted like this:

mean: -105.76HU
max: -83.00HU



The ocr'd lines with your code are:

mean:
max :
-105.76 HIJ
-83.00 HU

Obviously the OCR treats the image as two areas of text, which are ocrd one after the other. But I only need the first line and it's corresponding value

[Jscottj]

Love this! Thank you! I ALMOST have it working for what I need. I am NOT a developer. But, I can often take existing code and tweak it a bit

I need to go to a specific coordinate on the screen. I have text that pops up with mouse over, and that is what I am trying to grab. That is working great, and I can see the text in the tooltip. I just need to copy the text to the clipboard (I can put it into excel easily)

I don't actually need the red box or tooltip, but they don't hurt anything. I'm sure it's easy, but I am struggling! This part just need to be copied to clipboard instead of tooltip "OCR.FromRect(X+55, Y-55, 120, 40,,2).Text, , Y+40)" Can anyone help?

Code: Select all

#Requires AutoHotkey v2.0
#include .\Lib\OCR.ahk


CoordMode "Mouse", "Screen"
CoordMode "ToolTip", "Screen"
X_Search_Box:="1100,135"
Y_Search_Box:="1200,135"
Jump_To:="1260,150"

^9:: ;Run
{
	Click X_Search_Box, 2
	Send 555
	Click Y_Search_Box, 2
	Send 555
	Click Jump_To
	Click 1300,900
	Click 1290,850
}

Loop {
    MouseGetPos(&X, &Y)
    Highlight(x+55, y-55, 120, 40)
    ToolTip(OCR.FromRect(X+55, Y-55, 120, 40,,2).Text, , Y+40)
}

Highlight(x?, y?, w?, h?, showTime:=0, color:="Red", d:=2) {
	static guis := []

	if !IsSet(x) {
        for _, r in guis
            r.Destroy()
        guis := []
		return
    }
    if !guis.Length {
        Loop 4
            guis.Push(Gui("+AlwaysOnTop -Caption +ToolWindow -DPIScale +E0x08000000"))
    }
	Loop 4 {
		i:=A_Index
		, x1:=(i=2 ? x+w : x-d)
		, y1:=(i=3 ? y+h : y-d)
		, w1:=(i=1 or i=3 ? w+2*d : d)
		, h1:=(i=2 or i=4 ? h+2*d : d)
		guis[i].BackColor := color
		guis[i].Show("NA x" . x1 . " y" . y1 . " w" . w1 . " h" . h1)
	}
	if showTime > 0 {
		Sleep(showTime)
		Highlight()
	} else if showTime < 0
		SetTimer(Highlight, -Abs(showTime))
}

[boiler]

This part just need to be copied to clipboard instead of tooltip "OCR.FromRect(X+55, Y-55, 120, 40,,2).Text, , Y+40)"

It should just be this:

Code: Select all

A_Clipboard := OCR.FromRect(X+55, Y-55, 120, 40,,2).Text

…not the , , Y+40) that you included on the end, which are parameters and close parenthesis for the ToolTip call. You only removed the front part of the ToolTip call that is wrapped around the OCR.FromRect call, leaving the end part.

[Jscottj]

@boiler That did it. Thank you!

[rommmcek]

@Descolada: Thanks for this in other work of yours!

Obviously the OCR treats the image as ... areas of text ...

This is known issue of the UWP.
I made an attempt to sort the lines/words (with minimal formatting which could improve the readability of the output).

Variables:

• diff - (UInt) tolerance of the lines/words. If 0 auto setting will occur (medium height (in pixels) of the text).
• wr - (boolean) word, if false lines (as outputed by EasyOCR) will be processed, otherwise words.
• ht - (UInt) horizontal iteration for minimal formatting. If 0 no horizontal formatting will occure.
• vt - (UInt) vertical iteration for minimal formatting. If 0 no vertical formatting will occure.
• hd - (UInt) horizontal distance, a factor of how many diff units should trigger horizontal formating.
• vd - (UInt) vertical distance, a factor of how many diff units should trigger vertical formating.
• ds - (UInt) diff scaling factor to manually correct auto setting.
• arr - (internal) array to sort lines/words.
• hm - (internal) used to get medium height of the text.
• txt - (internal) to store lines/words.
• bb1 - (internal) below bottom 1, to approx. correct base bottom line of the text (add more characters if needed).
• bb2 - (internal) below bottom 2, to assess correction factor (add more characters if needed).

P.s.: For basic idea see here (v1.1 lines 160-190).

[Spitzi]

@rommmcek
Thank you. That looks promising. Can you post the actual code and not just a picture of it?
Also, i did not know about the extended parameters. Nice.

[boiler]

Can you post the actual code and not just a picture of it?

See the zip file attachment in his post.

[Spitzi]

@boiler Thanks for the hint, I did not see the link

[Spitzi]

@rommmcek Your code works nicely, thank you. Very dense and hard to understand... but it does what it should! Great!

I rearranged the code into a function to incorporate it into my app. Maybe this is easier for somebody else reading this thread later:

Code: Select all

#Requires AutoHotKey v2.0
#SingleInstance Force

#Include "OCR.ahk"			; Interface to windows UWP OCR by 'descolada'  from https://github.com/Descolada/OCR/tree/main/Lib

;MsgBox OCR.GetAvailableLanguages()
Esc:: ExitApp
Sleep 2000
;result := OCR.FromFile("MinimalFormat.jpg")
;result := OCR.FromFile("Kid_slice.jpg")
;result := OCR.FromFile("Weather_London.jpg")
; result := OCR.FromDesktop()
;result := OCR.FromWindow("A",,, 1)
result := OCR.FromWindow("A")
text := rearrangeOCRresult(result, diff:=0)
MsgBox("diff:" diff "`n`n" RTrim(text))
ExitApp

;text := ""
;for line in result.Lines
    ;text .= line.Text "`n"
;MsgBox RTrim(text)
;ExitApp

; Courtesy of rommmcek https://www.autohotkey.com/boards/viewtopic.php?f=83&t=116406&p=556071#p556071
; UPW OCR groups recognized text into areas, not lines: a known issue
; this function takes an OCR result and rearranges the recognized text into the lines where they appear
; - result: an OCR result
; - diff: tolerance of the lines/words. If 0 auto setting will occur (medium height (in pixels) of the text).
; returns a string with the recognized lines, separated by linefeed
rearrangeOCRresult(result, diff:=0) {
  /*diff - (UInt) tolerance of the lines/words. If 0 auto setting will occur (medium height (in pixels) of the text).
    wr - (boolean) word, if false lines (as outputed by EasyOCR) will be processed, otherwise words.
    ht - (UInt) horizontal iteration for minimal formatting. If 0 no horizontal formatting will occure.
    vt - (UInt) vertical iteration for minimal formatting. If 0 no vertical formatting will occure.
    hd - (UInt) horizontal distance, a factor of how many diff units should trigger horizontal formating.
    vd - (UInt) vertical distance, a factor of how many diff units should trigger vertical formating.
    ds - (UInt) diff scaling factor to manually correct auto setting.
    arr - (internal) array to sort lines/words.
    hm - (internal) used to get medium height of the text.
    txt - (internal) to store lines/words.
    bb1 - (internal) below bottom 1, to approx. correct base bottom line of the text (add more characters if needed).
    bb2 - (internal) below bottom 2, to assess correction factor (add more characters if needed).*/
    loop (arr:=Map(), hm:=0, txt:="", wr:= 0, ht:= 2, vt:= 1, hd:= 2, vd:= 3, ds:= .75, diff:= 0, 2) {
        for lw in (aInd:= A_Index, bb1:= "[,;gjpqyQ]", bb2:= "[A-Z0-9%bdfhkltij]", wr? result.words: result.lines) {
             if (lb:= wr? lw: OCR.WordsBoundingRect(lw.Words*), aInd=1 && !diff) {
                hm+= lb.h
             } else {
                 While (diff? "": diff:= Round(ds*hm/result.lines.Length), x:= lb.x, y:= lb.y, w:= lb.w, h:= lb.h, txt:= lw.text, A_Index<=(ma:=2*diff-1)) {
                    if arr.Has(yh:= (hh:=Round(y+h-(RegExMatch(txt, bb1)? RegExMatch(txt, bb2)? h/5: h/3: 0)))-(A_Index-diff) )
                        Break
                    else A_Index=ma? yh:=hh: ""
                 }  arr.Has(yh)? "": arr[yh]:= Map(), arr[yh][x]:= [x, yh, w, h, txt]
             }
        }
    }
    mf(ks, ch, it) {
        Loop (gp:="", ks*it)
            gp.= ch
        Return gp
    }
    for i, j in , (text:="", oy:=0, arr) {
        for k, l in (vp:= (ii:=i-oy)>vd*diff? mf(Round(ii/vd/diff), "`n", vt): "", text.= vp, oi:= i, ok:= 0, j)
            sp:= (kk:=k-ok)>hd*diff? mf(Round(kk/hd/diff), "`s", ht): "", text.= sp l[5] " ", ok:= k+l[3], oy:= l[2]
        text.= "`n"
    }
    return text
}

[Descolada]

@rommmcek, looks really nice! I haven't so far had the need for prettier formatting, but the rearranging feature is bound to be useful. I attempted a solution myself as well using the DBSCAN algorithm, but it lacks the formatting feature:

Code: Select all

#Requires AutoHotkey v2
#include OCR.ahk

result := OCR.FromFile("MinimalFormat.jpg")
;result := OCR.FromFile("Kid_slice.jpg")
;result := OCR.FromFile("Weather_London.jpg")

clusters := OCR.Cluster(result.Words), out := ""
for cluster in clusters
    out .= cluster.Text "`n"
MsgBox out

Outputs:
MinimalFormat.jpg

Code: Select all

32 -loop (arr:=Map(), txt : = ht:= 2 vt:= 1, hd:= 2, vd: = 3, ds:= .75 diff:=
33 for lw in (alnd:= A Index, bbl:= ' [ , ;gjpqyQ] " , bb2:= " [A-ZO-9%bdfhk1tij ] " , result . words : result. lines)
34 if (lb:= wr? lw: OCR. WordsBoundingRect(1w.Words*), alnd=l && !diff) {
35 hm+= 1b. h
36 } else {
While (diff? diff:= Length), x:= 1b. x, Y 1b. y, w:= 1b. w, h:= lb.h, txt:= lw
38 if arr.Has(yh:= (hh:=Round(Y+h-(RegExMatch(txt, bbl)? RegExMatch (txt, bb2)? h/5 e) ) (A Index-diffl
_
39 Break
else A Index=ma? yh:=hh:
41 arr.Has(yh)? arr[yh]:= Map(), [x, yh, w, h, txt]
42
43
45 -mf(ks, ch, it) {
46 Loop (gp:= ks*it)
47 gp.= ch
48 Return gp
49
50 - for i, j in (text : = arr) {
51 for k, 1 in (vp:= (ii:=i-oy)>vd*diff? mf(Round(ii/vd/diff), n' vt): text . = vp, 1 0k
52 sp:= (kk:=k-ok)>hd*diff? mf(Round (kk/hd/diff) , s' ht): text . = sp 1[5] ok:=
53 text . = n'
54

Kid_slice.jpg

Code: Select all

90 3250
Kid receive slice stoss Roses
90, master2 stoss, master, Roses 3250
master3

Weather_london.jpg

Code: Select all

Results for London, UK • Choose area
OF Precipitation: 5% Weather
Humidity: 89%
Sunday 05:00
Wind: 34 km/h
Windy
Temperature Precipitation Wind
11 12 11 12 12 12
9
6
06:00 09:00 12:00 15:00 18:00 21 00:00 03:00
Sun Mon Tue Wed Thu Fri Sat Sun
120 60 100 50 140 120 60 120 60 90 20 90 40 100 60

@Spitzi, for some reason I didn't get a notification for your first post so I didn't see it Now you have a few solutions available though!

[Spitzi]

@Descolada Hi. Wow, yeah. Thank you!! Will test and let you know.

[Spitzi]

Hi @Descolada.

Your cluster function works nicely, but I got this DivideByZero error:

Error.png (13.44 KiB) Viewed 1030 times

I think it happens when the OCR result is empty.

Greets Simon

[Descolada]

@Spitzi you are correct, it happens when the array is empty. I've updated the library to fix this.

[Spitzi]

@Descolada thank you. I like your cluster function a lot, by the way. In some cases I still use @rommmcek 's function because of it's formating, which I like a lot.
Thank you both!!

[Spydor]

Forgive my ignorance, fairly new to AHK, trying to follow the solution above.

I'm running Snipper (Version 2024 01 09) and have an Include for "Snipper - Extension - OCR" (Version: 2023 05 05) in the lib folder. When I OCR a clip, it's all concatenated on one line.

I changed the OCR2Clipboard method in OCR extension based on post by Descolada, (untested). That made an improvement, but line splits were still incorrect, breaking on words.

I'm trying to follow the post above. @Descolada you said

I've updated the library to fix this.

Which library is it?

I also downloaded EasyOcr_pseudo_4_v2.zip from @rommmcek, but wasn't sure how to implement it.

I'm trying to determine what I need need to modify where, to get multi-line OCR to output in multi-line. Thank you.

[Descolada]

@Spydor try the following modification to OCR2Clipboard (note that the Snippers OCR extension might be out of date and not contain OCR.Cluster method, in which case you need to update the OCR library manually from my Github repo):

Code: Select all

OCR2Clipboard(Borders := false)
{
	; Get Hwnd of Active Snip
	Hwnd := WinGetID('A')
	; Get hBitMap of Snip
	hBitMap := SendMessage(0x173, 0, 0, guiSnips[Hwnd].GuiObj.Pic)
	; OCR of hBitMap
	Result := OCR.FromBitmap(hBitMap)
	clusters := OCR.Cluster(Result.Words), Text := ""
	for cluster in clusters
	    Text .= cluster.Text "`n"
	; Put Text in Clipboard
	A_Clipboard := Trim(Text)
}

Alternatively you can try rommmcek's implementation from here, in which case the OCR2Clipboard should look like this:

Code: Select all

OCR2Clipboard(Borders := false)
{
	; Get Hwnd of Active Snip
	Hwnd := WinGetID('A')
	; Get hBitMap of Snip
	hBitMap := SendMessage(0x173, 0, 0, guiSnips[Hwnd].GuiObj.Pic)
	; OCR of hBitMap
	Result := OCR.FromBitmap(hBitMap)
	text := rearrangeOCRresult(result, diff:=0)
	; Put Text in Clipboard
	A_Clipboard := Trim(Text)
}

Which library is it?

There is only one library: OCR.ahk

[Spydor]

@Descolada , thank you! That was the piece I was missing, didn't realize the "class OCR" in the Snippers OCR extension was the library. I had your OCR.ahk in the lib folder, but hadn't included it, wasn't sure what it was supposed to replace.

I gutted the extension, just left the following, working now.

Code: Select all

; Snipper - Extension - OCR
; Version: 2023 05 05

; OCR.ahk from https://github.com/Descolada/OCR
#Include OCR.ahk

Extensions.Push({ OCR: { Text: 'COPY:  OCR to Clipboard', Func: OCR2Clipboard } })

OCR2Clipboard(Borders := false)
{
	; Get Hwnd of Active Snip
	Hwnd := WinGetID('A')
	; Get hBitMap of Snip
	hBitMap := SendMessage(0x173, 0, 0, guiSnips[Hwnd].GuiObj.Pic)
	; OCR of hBitMap
	Result := OCR.FromBitmap(hBitMap)
	clusters := OCR.Cluster(Result.Words), Text := ""
	for cluster in clusters
	    Text .= cluster.Text "`n"
	; Put Text in Clipboard
	A_Clipboard := Trim(Text)
}

[Carrottie]

@Descolada, thank you very much for sharing. I have tried to write a very primitive hotkey using this OCR library but it does not work, can you see how to modify it?

#Requires AutoHotkey v2
#include ..\AutoHotkey\OCR-main\Lib\OCR.ahk

Code: Select all

!g::
	{
	Run "https:\\contacts.google.com"
	result := OCR.FromDesktop()
        try found := result.FindString('Create contact')
        Click found.x, found.y
	}

[Mod edit: Added [code][/code] tags. Please use them yourself when posting code.]