Extract data from screen

Get help with using AutoHotkey (v1.1 and older) and its commands and hotkeys
user1824
Posts: 29
Joined: 10 Sep 2021, 10:40

Extract data from screen

23 Sep 2021, 10:21

I wanted to collect data from the table shown in the window(see the attached picture for reference), to be precise, I want to get the text written under the
Items
column, how can I extract it?
Attachments
snip.jpg
snip.jpg (66.46 KiB) Viewed 3269 times
User avatar
mikeyww
Posts: 26591
Joined: 09 Sep 2014, 18:38

Re: Extract data from screen

23 Sep 2021, 11:22

It's easier if you could post the table instead of a screenshot of it.
user1824
Posts: 29
Joined: 10 Sep 2021, 10:40

Re: Extract data from screen

23 Sep 2021, 12:21

@mikeyww
Thanks for getting back, It's an application(3Shape) that displays the list of items added to it, so I don't think I can post the table. I just called it table because it had rows & columns.
Any direction where I should go?
Will ControlGetText be of any help?
User avatar
mikeyww
Posts: 26591
Joined: 09 Sep 2014, 18:38

Re: Extract data from screen

23 Sep 2021, 12:38

Yes, it may, or WinGetText. Instr, SubStr, or RegExMatch would often be used at that point.
AHKStudent
Posts: 1472
Joined: 05 May 2018, 12:23

Re: Extract data from screen

23 Sep 2021, 14:02

user1824 wrote:
23 Sep 2021, 12:21
@mikeyww
Thanks for getting back, It's an application(3Shape) that displays the list of items added to it, so I don't think I can post the table. I just called it table because it had rows & columns.
Any direction where I should go?
Will ControlGetText be of any help?
in your autohotkey installation folder there is winspy, run the program, put the mouse over that table and paste to the forum what you get in the top box (class info etc) this will give info how to access it
user1824
Posts: 29
Joined: 10 Sep 2021, 10:40

Re: Extract data from screen

27 Sep 2021, 10:06

I already tried window spy but window spy doesn't show any unique id that I could point at for the different rows in that columns, it just shows column title for the whole list.
Ahk_fan
Posts: 237
Joined: 31 Aug 2018, 14:34
Contact:

Re: Extract data from screen

27 Sep 2021, 14:25

perhaps this solution:

Scan for picture "Items" with ImageSearch, OutputVarX, OutputVarY, X1, Y1, X2, Y2, ImageFile
ImageFileOfItems.png
ImageFileOfItems.png (3.38 KiB) Viewed 3087 times
then OCR with (from here: viewtopic.php?f=6&t=72674&hilit=ocr+uwp&start=20 by @malcev )

Code: Select all

SetBatchLines, -1
; here part with ImageSearch
ImageSearch, OutputVarX, OutputVarY, X1, Y1, X2, Y2, *100 %A_ScriptDir%\ImageFileOfItems.png
hBitmap := HBitmapFromScreen(OutputVarX, OutputVarY, OutputVarX+200, OutputVarY+500)
pIRandomAccessStream := HBitmapToRandomAccessStream(hBitmap)
text := ocr(pIRandomAccessStream, "en")
MsgBox, %text%
Return


HBitmapFromScreen(X, Y, W, H) {
   HDC := DllCall("GetDC", "Ptr", 0, "UPtr")
   HBM := DllCall("CreateCompatibleBitmap", "Ptr", HDC, "Int", W, "Int", H, "UPtr")
   PDC := DllCall("CreateCompatibleDC", "Ptr", HDC, "UPtr")
   DllCall("SelectObject", "Ptr", PDC, "Ptr", HBM)
   DllCall("BitBlt", "Ptr", PDC, "Int", 0, "Int", 0, "Int", W, "Int", H
                   , "Ptr", HDC, "Int", X, "Int", Y, "UInt", 0x00CC0020)
   DllCall("DeleteDC", "Ptr", PDC)
   DllCall("ReleaseDC", "Ptr", 0, "Ptr", HDC)
   Return HBM
}

HBitmapToRandomAccessStream(hBitmap) {
   static IID_IRandomAccessStream := "{905A0FE1-BC53-11DF-8C49-001E4FC686DA}"
        , IID_IPicture            := "{7BF80980-BF32-101A-8BBB-00AA00300CAB}"
        , PICTYPE_BITMAP := 1
        , BSOS_DEFAULT   := 0
        
   DllCall("Ole32\CreateStreamOnHGlobal", "Ptr", 0, "UInt", true, "PtrP", pIStream, "UInt")
   
   VarSetCapacity(PICTDESC, sz := 8 + A_PtrSize*2, 0)
   NumPut(sz, PICTDESC)
   NumPut(PICTYPE_BITMAP, PICTDESC, 4)
   NumPut(hBitmap, PICTDESC, 8)
   riid := CLSIDFromString(IID_IPicture, GUID1)
   DllCall("OleAut32\OleCreatePictureIndirect", "Ptr", &PICTDESC, "Ptr", riid, "UInt", false, "PtrP", pIPicture, "UInt")
   ; IPicture::SaveAsFile
   DllCall(NumGet(NumGet(pIPicture+0) + A_PtrSize*15), "Ptr", pIPicture, "Ptr", pIStream, "UInt", true, "UIntP", size, "UInt")
   riid := CLSIDFromString(IID_IRandomAccessStream, GUID2)
   DllCall("ShCore\CreateRandomAccessStreamOverStream", "Ptr", pIStream, "UInt", BSOS_DEFAULT, "Ptr", riid, "PtrP", pIRandomAccessStream, "UInt")
   ObjRelease(pIPicture)
   ObjRelease(pIStream)
   Return pIRandomAccessStream
}

CLSIDFromString(IID, ByRef CLSID) {
   VarSetCapacity(CLSID, 16, 0)
   if res := DllCall("ole32\CLSIDFromString", "WStr", IID, "Ptr", &CLSID, "UInt")
      throw Exception("CLSIDFromString failed. Error: " . Format("{:#x}", res))
   Return &CLSID
}


ocr(file, lang := "FirstFromAvailableLanguages")
{
   static OcrEngineStatics, OcrEngine, MaxDimension, LanguageFactory, Language, CurrentLanguage, BitmapDecoderStatics, GlobalizationPreferencesStatics
   if (OcrEngineStatics = "")
   {
      CreateClass("Windows.Globalization.Language", ILanguageFactory := "{9B0252AC-0C27-44F8-B792-9793FB66C63E}", LanguageFactory)
      CreateClass("Windows.Graphics.Imaging.BitmapDecoder", IBitmapDecoderStatics := "{438CCB26-BCEF-4E95-BAD6-23A822E58D01}", BitmapDecoderStatics)
      CreateClass("Windows.Media.Ocr.OcrEngine", IOcrEngineStatics := "{5BFFA85A-3384-3540-9940-699120D428A8}", OcrEngineStatics)
      DllCall(NumGet(NumGet(OcrEngineStatics+0)+6*A_PtrSize), "ptr", OcrEngineStatics, "uint*", MaxDimension)   ; MaxImageDimension
   }
   if (file = "ShowAvailableLanguages")
   {
      if (GlobalizationPreferencesStatics = "")
         CreateClass("Windows.System.UserProfile.GlobalizationPreferences", IGlobalizationPreferencesStatics := "{01BF4326-ED37-4E96-B0E9-C1340D1EA158}", GlobalizationPreferencesStatics)
      DllCall(NumGet(NumGet(GlobalizationPreferencesStatics+0)+9*A_PtrSize), "ptr", GlobalizationPreferencesStatics, "ptr*", LanguageList)   ; get_Languages
      DllCall(NumGet(NumGet(LanguageList+0)+7*A_PtrSize), "ptr", LanguageList, "int*", count)   ; count
      loop % count
      {
         DllCall(NumGet(NumGet(LanguageList+0)+6*A_PtrSize), "ptr", LanguageList, "int", A_Index-1, "ptr*", hString)   ; get_Item
         DllCall(NumGet(NumGet(LanguageFactory+0)+6*A_PtrSize), "ptr", LanguageFactory, "ptr", hString, "ptr*", LanguageTest)   ; CreateLanguage
         DllCall(NumGet(NumGet(OcrEngineStatics+0)+8*A_PtrSize), "ptr", OcrEngineStatics, "ptr", LanguageTest, "int*", bool)   ; IsLanguageSupported
         if (bool = 1)
         {
            DllCall(NumGet(NumGet(LanguageTest+0)+6*A_PtrSize), "ptr", LanguageTest, "ptr*", hText)
            buffer := DllCall("Combase.dll\WindowsGetStringRawBuffer", "ptr", hText, "uint*", length, "ptr")
            text .= StrGet(buffer, "UTF-16") "`n"
         }
         ObjRelease(LanguageTest)
      }
      ObjRelease(LanguageList)
      return text
   }
   if (lang != CurrentLanguage) or (lang = "FirstFromAvailableLanguages")
   {
      if (OcrEngine != "")
      {
         ObjRelease(OcrEngine)
         if (CurrentLanguage != "FirstFromAvailableLanguages")
            ObjRelease(Language)
      }
      if (lang = "FirstFromAvailableLanguages")
         DllCall(NumGet(NumGet(OcrEngineStatics+0)+10*A_PtrSize), "ptr", OcrEngineStatics, "ptr*", OcrEngine)   ; TryCreateFromUserProfileLanguages
      else
      {
         CreateHString(lang, hString)
         DllCall(NumGet(NumGet(LanguageFactory+0)+6*A_PtrSize), "ptr", LanguageFactory, "ptr", hString, "ptr*", Language)   ; CreateLanguage
         DeleteHString(hString)
         DllCall(NumGet(NumGet(OcrEngineStatics+0)+9*A_PtrSize), "ptr", OcrEngineStatics, ptr, Language, "ptr*", OcrEngine)   ; TryCreateFromLanguage
      }
      if (OcrEngine = 0)
      {
         msgbox Can not use language "%lang%" for OCR, please install language pack.
         ExitApp
      }
      CurrentLanguage := lang
   }
   IRandomAccessStream := file
   DllCall(NumGet(NumGet(BitmapDecoderStatics+0)+14*A_PtrSize), "ptr", BitmapDecoderStatics, "ptr", IRandomAccessStream, "ptr*", BitmapDecoder)   ; CreateAsync
   WaitForAsync(BitmapDecoder)
   BitmapFrame := ComObjQuery(BitmapDecoder, IBitmapFrame := "{72A49A1C-8081-438D-91BC-94ECFC8185C6}")
   DllCall(NumGet(NumGet(BitmapFrame+0)+12*A_PtrSize), "ptr", BitmapFrame, "uint*", width)   ; get_PixelWidth
   DllCall(NumGet(NumGet(BitmapFrame+0)+13*A_PtrSize), "ptr", BitmapFrame, "uint*", height)   ; get_PixelHeight
   if (width > MaxDimension) or (height > MaxDimension)
   {
      msgbox Image is to big - %width%x%height%.`nIt should be maximum - %MaxDimension% pixels
      ExitApp
   }
   BitmapFrameWithSoftwareBitmap := ComObjQuery(BitmapDecoder, IBitmapFrameWithSoftwareBitmap := "{FE287C9A-420C-4963-87AD-691436E08383}")
   DllCall(NumGet(NumGet(BitmapFrameWithSoftwareBitmap+0)+6*A_PtrSize), "ptr", BitmapFrameWithSoftwareBitmap, "ptr*", SoftwareBitmap)   ; GetSoftwareBitmapAsync
   WaitForAsync(SoftwareBitmap)
   DllCall(NumGet(NumGet(OcrEngine+0)+6*A_PtrSize), "ptr", OcrEngine, ptr, SoftwareBitmap, "ptr*", OcrResult)   ; RecognizeAsync
   WaitForAsync(OcrResult)
   DllCall(NumGet(NumGet(OcrResult+0)+6*A_PtrSize), "ptr", OcrResult, "ptr*", LinesList)   ; get_Lines
   DllCall(NumGet(NumGet(LinesList+0)+7*A_PtrSize), "ptr", LinesList, "int*", count)   ; count
   loop % count
   {
      DllCall(NumGet(NumGet(LinesList+0)+6*A_PtrSize), "ptr", LinesList, "int", A_Index-1, "ptr*", OcrLine)
      DllCall(NumGet(NumGet(OcrLine+0)+6*A_PtrSize), "ptr", OcrLine, "ptr*", WordsList)   ; get_Words
      DllCall(NumGet(NumGet(WordsList+0)+7*A_PtrSize), "ptr", WordsList, "int*", WordsCount)   ; Words count
      loop % WordsCount
      {
         DllCall(NumGet(NumGet(WordsList+0)+6*A_PtrSize), "ptr", WordsList, "int", A_Index-1, "ptr*", OcrWord)
         VarSetCapacity(RECT, 16, 0)
         DllCall(NumGet(NumGet(OcrWord+0)+6*A_PtrSize), "ptr", OcrWord, "ptr", &RECT)   ; get_BoundingRect
         DllCall(NumGet(NumGet(OcrWord+0)+7*A_PtrSize), "ptr", OcrWord, "ptr*", hText)   ; get_Text
         buffer := DllCall("Combase.dll\WindowsGetStringRawBuffer", "ptr", hText, "uint*", length, "ptr")
         x := NumGet(&RECT, 0, "float")
         y := NumGet(&RECT, 4, "float")
         w := NumGet(&RECT, 8, "float")
         h := NumGet(&RECT, 12, "float")
         text := StrGet(buffer, "UTF-16")
         msgbox %text%`nx: %x%, y: %y%, w: %w%, h: %h%
         ObjRelease(OcrWord)
      }
      ObjRelease(WordsList)
      ObjRelease(OcrLine)
   }
   Close := ComObjQuery(IRandomAccessStream, IClosable := "{30D5A829-7FA4-4026-83BB-D75BAE4EA99E}")
   DllCall(NumGet(NumGet(Close+0)+6*A_PtrSize), "ptr", Close)   ; Close
   ObjRelease(Close)
   Close := ComObjQuery(SoftwareBitmap, IClosable := "{30D5A829-7FA4-4026-83BB-D75BAE4EA99E}")
   DllCall(NumGet(NumGet(Close+0)+6*A_PtrSize), "ptr", Close)   ; Close
   ObjRelease(Close)
   ObjRelease(IRandomAccessStream)
   ObjRelease(BitmapDecoder)
   ObjRelease(BitmapFrame)
   ObjRelease(BitmapFrameWithSoftwareBitmap)
   ObjRelease(SoftwareBitmap)
   ObjRelease(OcrResult)
   ObjRelease(LinesList)
   return text
}



CreateClass(string, interface, ByRef Class)
{
   CreateHString(string, hString)
   VarSetCapacity(GUID, 16)
   DllCall("ole32\CLSIDFromString", "wstr", interface, "ptr", &GUID)
   result := DllCall("Combase.dll\RoGetActivationFactory", "ptr", hString, "ptr", &GUID, "ptr*", Class, "uint")
   if (result != 0)
   {
      if (result = 0x80004002)
         msgbox No such interface supported
      else if (result = 0x80040154)
         msgbox Class not registered
      else
         msgbox error: %result%
      ExitApp
   }
   DeleteHString(hString)
}

CreateHString(string, ByRef hString)
{
    DllCall("Combase.dll\WindowsCreateString", "wstr", string, "uint", StrLen(string), "ptr*", hString)
}

DeleteHString(hString)
{
   DllCall("Combase.dll\WindowsDeleteString", "ptr", hString)
}

WaitForAsync(ByRef Object)
{
   AsyncInfo := ComObjQuery(Object, IAsyncInfo := "{00000036-0000-0000-C000-000000000046}")
   loop
   {
      DllCall(NumGet(NumGet(AsyncInfo+0)+7*A_PtrSize), "ptr", AsyncInfo, "uint*", status)   ; IAsyncInfo.Status
      if (status != 0)
      {
         if (status != 1)
         {
            DllCall(NumGet(NumGet(AsyncInfo+0)+8*A_PtrSize), "ptr", AsyncInfo, "uint*", ErrorCode)   ; IAsyncInfo.ErrorCode
            msgbox AsyncInfo status error: %ErrorCode%
            ExitApp
         }
         ObjRelease(AsyncInfo)
         break
      }
      sleep 10
   }
   DllCall(NumGet(NumGet(Object+0)+8*A_PtrSize), "ptr", Object, "ptr*", ObjectResult)   ; GetResults
   ObjRelease(Object)
   Object := ObjectResult
}
regards,
AHK_fan :)
https://hr-anwendungen.de
user1824
Posts: 29
Joined: 10 Sep 2021, 10:40

Re: Extract data from screen

28 Sep 2021, 09:26

Thanks for posting your reply @Ahk_fan
But I don't have to read data from an image, rather I've to read data from screen.
Ahk_fan
Posts: 237
Joined: 31 Aug 2018, 14:34
Contact:

Re: Extract data from screen

28 Sep 2021, 12:01

Hi,
you read from screen with OCR with my solution.
Alternatively you could test ACC.ahk.
viewtopic.php?f=6&t=26201
regards,
AHK_fan :)
https://hr-anwendungen.de
user1824
Posts: 29
Joined: 10 Sep 2021, 10:40

Re: Extract data from screen

29 Sep 2021, 09:58

Okay will try it out
User avatar
flyingDman
Posts: 2791
Joined: 29 Sep 2013, 19:01

Re: Extract data from screen

29 Sep 2021, 11:31

You should post a screenshot of Windows Spy. We do not know what type of control it is. If you cannot retrieve the text from the control, the screen clipping / OCR is the way to go.
14.3 & 1.3.7
user1824
Posts: 29
Joined: 10 Sep 2021, 10:40

Re: Extract data from screen

30 Sep 2021, 14:40

Hi @flyingDman,
Does this screenshot give any more context?
I want to read the text under items in the list.
Attachments
SharedScreenshot.jpg
SharedScreenshot.jpg (253.91 KiB) Viewed 2826 times
User avatar
flyingDman
Posts: 2791
Joined: 29 Sep 2013, 19:01

Re: Extract data from screen

30 Sep 2021, 16:36

Well it shows what type of control it is. And, unfortunately, TBrowserListView2 is not one that I am familiar with and a quick search on the forum did not reveal anything either. Add it been a SysListView321 for instance, you might have lifted the content with little effort. Maybe someone else knows more about TBrowserListView2?
14.3 & 1.3.7
user1824
Posts: 29
Joined: 10 Sep 2021, 10:40

Re: Extract data from screen

01 Oct 2021, 07:58

Still thanks for your help @flyingDman!
user1824
Posts: 29
Joined: 10 Sep 2021, 10:40

Re: Extract data from screen

01 Oct 2021, 09:05

Hey @mikeyww ,

None of the methods given by you worked, do you have any other suggestion for me?
SOTE
Posts: 1426
Joined: 15 Jun 2015, 06:21

Re: Extract data from screen

01 Oct 2021, 16:41

@user1824

Have you tried Ahk_fan's ImageSearch and OCR suggested solution? If you can't get text from the control directly, then it looks like that is the direction you must go.
User avatar
mikeyww
Posts: 26591
Joined: 09 Sep 2014, 18:38

Re: Extract data from screen

01 Oct 2021, 16:52

Can always give it a go:

Code: Select all

F3::
ControlGet, text, List,, TBrowserListView2, ahk_exe DentalManager.exe
MsgBox, 64, Text, %text%
Return

Return to “Ask for Help (v1)”

Who is online

Users browsing this forum: garry, Google [Bot], marypoppins_1, Rohwedder, RussF and 140 guests