How to make AHK utilize resources and run faster?

Get help with using AutoHotkey (v1.1 and older) and its commands and hotkeys
User avatar
zvit
Posts: 224
Joined: 07 Nov 2017, 06:15

How to make AHK utilize resources and run faster?

Post by zvit » 27 May 2022, 05:47

My script splits a list of words into sub-words that must have a minimum of three characters and cannot be the original word.

For example, the word "hello" would be split into:
hel
hell
ell
ello

I'm wondering why, despite AHK using internal arrays, the script takes an extremely long time (see attached gif below) and, as you can see in the gif below, it only consumes 3% of my CPU and 5.5MB of RAM. Since I have a pretty powerful system, why can't AHK utilize more resources and complete this task 1000x faster?

My system:
Operating System
Windows 10 Pro 64-bit

CPU
Intel Core i9 10900K @ 3.70GHz, 3696 Mhz, 10 Core(s), 20 Logical Processor(s)

RAM
Corsair Vengeance RGB Pro 64 GB (2 x 32 GB) DDR4-3200 CL16

Motherboard
Gigabyte Z590 AORUS MASTER (U3E1)

Graphics
LG ULTRAWIDE (3840x1600@60Hz)
Intel UHD Graphics 630 (Gigabyte)
2047MB NVIDIA GeForce RTX 3080

Here is the gif:
Image

Here is the code:

Code: Select all

#NoEnv ; Recommended for performance and compatibility with future AutoHotkey releases.
; #Warn  ; Enable warnings to assist with detecting common errors.
SendMode Input ; Recommended for new scripts due to its superior speed and reliability.
SetWorkingDir %A_ScriptDir% ; Ensures a consistent starting directory.

#Include <GlobalsKDP>
xlApp := XL_Handle(1)
MyPath := GlobalsKDP_GetGridsWorkingPath()

XL_Speedup(xlApp,0)
XL_Select_Sheet(xlApp,"MAIN-WORDS")
if xlApp.Range(rang_Word_tblMainWords).Find("*",,,,,2)
    xlApp.Range(rang_Word_tblMainWords).Sort(xlApp.Range(rang_Word_tblMainWords),,,,,,,1) ; with Header

_RangeSize := xlApp.Range(rang_Word_tblMainWords).rows.Count()
xlApp.Range(rang_Word_tblMainWords).Copy
ClipWait, 1
global MainWords := Clipboard
xlApp.Application.CutCopyMode := False

SplitArray := 0

;=========================
;// NOTE FOR PROGRESS BAR
;=========================
Gui, Add, Text, xm w360 vProgressWord, Checking Word...`nPlease Wait
Gui, Add, Progress, xm w300 h20 vProgressBar -Smooth
Gui, Add, Button, yp-1 hp+2 xp+305 w60 gButtonCancel, Cancel
Gui, Add, Text, xm w360 vProgressText
Gui, Show,, Splitting Words...

Loop, parse, MainWords, `n, `r
{
    String := A_LoopField
    StrLen := StrLen(String)
    CurrOrgWordPos := A_Index

    Loop, % StrLen
    {
        CurrStr := SubStr(String, A_Index)
        CurrStrLen := StrLen(CurrStr)

        Loop, % CurrStrLen
        {
            ;=========================
            ;// NOTE FOR PROGRESS BAR
            ;=========================
            GuiControl,, ProgressWord, Splitting word: %CurrOrgWordPos% out of %_RangeSize%: %String%`nSplit Word: %CurrStr%
            Percent := Floor( (CurrOrgWordPos / _RangeSize) * 100 )
            GuiControl,, ProgressBar, %Percent%
            GuiControl,, ProgressText, %Percent%`% Complete 

            Loop, % CurrStrLen
            {
                UseStr:=SubStr(CurrStr, 1, A_Index)
                if ((StrLen(UseStr) >= 3) and (IsInMainWords(UseStr)=false))
                    FinalString .= UseStr "`n"
            }
        }
    }
}
NewString:=EX_RemoveDuplicates(FinalString, SplitArray)
msgbox % "Total split words created after removing dups: " SplitArray "."
StartTime := A_TickCount
XL_Select_Sheet(xlApp,"SPLIT-WORDS")
Clipboard := NewString
ClipWait, 1
xlApp.Range(rang_Word_tblSplitWords).ClearContents
xlApp.Range("A2").PasteSpecial()
xlApp.Range(rang_Word_tblSplitWords).Sort(xlApp.Range("A2"),,,,,,,1)

ElapsedTime := ConvertMSToMinSec(A_TickCount - StartTime)
MsgBox, %ElapsedTime% have elapsed.

XL_Speedup(xlApp,1)
msgbox % "Finished spliting words." 

ConvertMSToMinSec(ms)
{
    return, floor((ms / 1000) / 60) " minutes " 
    . floor(mod((ms / 1000), 60)) " seconds "
    . floor((mod((ms / 1000), 60) - sec) * 1000) " mseconds "
}
IsInMainWords(str)
{
    Loop, Parse, MainWords, `n, `r
    {
        if (str = A_LoopField)
            return true
    }
    return false
}

GetOutOfHere:
GuiClose:
GuiEscape:
ButtonCancel:
    CancelCopy := true
    Gui, Destroy 
ExitMe:
ExitApp

Rohwedder
Posts: 7551
Joined: 04 Jun 2014, 08:33
Location: Germany

Re: How to make AHK utilize resources and run faster?

Post by Rohwedder » 27 May 2022, 06:31

Hallo,
A little faster it should run with:

Code: Select all

SetBatchLines, -1
Try this with and without Sleep:

Code: Select all

SetBatchLines, -1
Text = My script splits a list of words into sub-words that must have a minimum of three characters and cannot be the original word
q::
Loop, Parse, Text, %A_Space%
	Loop,% Len:=StrLen(A_LoopField)-2		
		Loop,% Len-(1=Index:=A_Index)
		{
			Part := SubStr(A_LoopField,Index,A_Index+2)
			ToolTip,% Part
			Sleep, 100
		}
Return

User avatar
zvit
Posts: 224
Joined: 07 Nov 2017, 06:15

Re: How to make AHK utilize resources and run faster?

Post by zvit » 27 May 2022, 08:51

Thank you! After some more testing, I found that removing a function from my script enables both of our scripts to do a list of 4000 words in a little over a minute. My second rule wasn't explained. I meant that split words must be != to ANY word in the original list of words.

My script has two functions to take care of this. EX_RemoveDuplicates() removes split duplicates (If you noticed, your results have duplicate split words) and IsInMainWords() to not add splits that exist in the original list of words.

This is the function that removes split dups after the loops - only adding 4 seconds to the total time:

Code: Select all

EX_RemoveDuplicates(String, byref SplitArray){
    For i, value in StrSplit(String , "`n"), uniques := {}, StringNew := ""
    {
        if !uniques.HasKey(value)
        {
            StringNew .= value . "`n" , uniques[value] := true
            SplitArray+=1
        }
    }
    return StringNew
}
This function happens DURING the loop, and is the reason for the long time - and it's understandable: For EVERY split word, it loops the main list - 4000 words!

Code: Select all

IsInMainWords(str)
{
    Loop, Parse, MainWords, `n, `r
    {
        if (str = A_LoopField)
            return true
    }
    return false
}
I think I can get rid of the IsInMainWords() and do that check AFTER the loops by using the same EX_RemoveDuplicates() but I'm not sure how. The function uses if !uniques.HasKey(value) to compare dups in the same list. I'm not sure how to convert it so that it compares the split words to the main words.

Rohwedder
Posts: 7551
Joined: 04 Jun 2014, 08:33
Location: Germany

Re: How to make AHK utilize resources and run faster?

Post by Rohwedder » 27 May 2022, 09:46

Faster IsInMainWords():

Code: Select all

Text = My script splits a list of words into sub-words that must have a minimum of three characters and cannot be the original word
Global MainWords := []
Loop, Parse, Text, %A_Space%
	MainWords[A_LoopField] := True
q::MsgBox,% IsInMainWords("three")
w::MsgBox,% IsInMainWords("hello")

IsInMainWords(str)
{
    Return, MainWords[str]?"True":"False"
}

User avatar
Smile_
Posts: 857
Joined: 03 May 2020, 00:51

Re: How to make AHK utilize resources and run faster?

Post by Smile_ » 27 May 2022, 14:35

Hallo, how about file object

Code: Select all

Gui, Add, Progress, xm w300 h20 vProgressBar -Smooth
Gui, Show,, Splitting Words...

StartTime   := A_TickCount

FileAppend, % "`n" StrReplace(MainWords, " ", "`n") "`n", Input.txt

Output          := ""
InputObj        := FileOpen("Input.txt", "r")
MainWords       := InputObj.Read()
EOF             := InputObj.Pos
InputObj.Pos    := 0

While !InputObj.AtEOF() {
    Word    := Trim(InputObj.ReadLine(), "`r`n")
    Output  .= Word ":`n"

    ;Extract sub-word and check if it is a main word
    WLen    := 2
    While, (++WLen < StrLen(Word)) {
        Loop, % StrLen(Word) - WLen + 1 {
            SubWord := SubStr(Word, A_Index, WLen)
            If !InStr(MainWords, "`n" SubWord "`n") {
                Output .= SubWord "`n"
            }
        }
    }
    ;-------------------------------------------------
    
    Output .= "`n"
    GuiControl,, ProgressBar, % Percent := (InputObj.Pos / EOF) * 100
}

InputObj.Close()
FileAppend, % Output, Output.txt

Msgbox, % "Complete in " A_TickCount - StartTime " ms"
Took about 7500 ms to generate a file with 5 k words about 46 k lines (including their sub-words).

Edit:
Replaced Word := Trim(InputObj.ReadLine(), "`n") with Word := Trim(InputObj.ReadLine(), "`r`n")
Last edited by Smile_ on 27 May 2022, 15:32, edited 2 times in total.

User avatar
zvit
Posts: 224
Joined: 07 Nov 2017, 06:15

Re: How to make AHK utilize resources and run faster?

Post by zvit » 27 May 2022, 15:07

@Rohwedder The IsInMainWords didn't work but apparently, your code didn't need it, :-) thanks for all the help.

User avatar
zvit
Posts: 224
Joined: 07 Nov 2017, 06:15

Re: How to make AHK utilize resources and run faster?

Post by zvit » 27 May 2022, 15:18

Smile_ wrote:
27 May 2022, 14:35
Hallo, how about file object

Code: Select all

Gui, Add, Progress, xm w300 h20 vProgressBar -Smooth
Gui, Show,, Splitting Words...

StartTime   := A_TickCount

FileAppend, % "`n" StrReplace(MainWords, " ", "`n") "`n", Input.txt

Output          := ""
InputObj        := FileOpen("Input.txt", "r")
MainWords       := InputObj.Read()
EOF             := InputObj.Pos
InputObj.Pos    := 0

While !InputObj.AtEOF() {
    Word    := Trim(InputObj.ReadLine(), "`n")
    Output  .= Word ":`n"

    ;Extract sub-word and check if it is a main word
    WLen    := 2
    While, (++WLen < StrLen(Word)) {
        Loop, % StrLen(Word) - WLen + 1 {
            SubWord := SubStr(Word, A_Index, WLen)
            If !InStr(MainWords, "`n" SubWord "`n") {
                Output .= SubWord "`n"
            }
        }
    }
    ;-------------------------------------------------
    
    Output .= "`n"
    GuiControl,, ProgressBar, % Percent := (InputObj.Pos / EOF) * 100
}

InputObj.Close()
FileAppend, % Output, Output.txt

Msgbox, % "Complete in " A_TickCount - StartTime " ms"
Took about 7500 ms to generate a file with 5 k words about 46 k lines (including their sub-words).
OMG!! It adds the main word and then a new line, so needs fixing up, but OMG!

Image
Last edited by zvit on 27 May 2022, 15:32, edited 2 times in total.

User avatar
Smile_
Posts: 857
Joined: 03 May 2020, 00:51

Re: How to make AHK utilize resources and run faster?

Post by Smile_ » 27 May 2022, 15:28

I added it in purpose, to make sure to take the sub-words that are not in main words
I edited the code, there was a little bug in it, now it should be OK.

User avatar
zvit
Posts: 224
Joined: 07 Nov 2017, 06:15

Re: How to make AHK utilize resources and run faster?

Post by zvit » 27 May 2022, 15:32

There was a reason I didn't use InStr but your way: If !InStr(str_MainWords, "`n" SubWord "`n") is genius!!

Thanks for the edit.

User avatar
zvit
Posts: 224
Joined: 07 Nov 2017, 06:15

Re: How to make AHK utilize resources and run faster?

Post by zvit » 27 May 2022, 15:36

Thanks. I will use yours.

Image

User avatar
Smile_
Posts: 857
Joined: 03 May 2020, 00:51

Re: How to make AHK utilize resources and run faster?

Post by Smile_ » 27 May 2022, 15:39

Glad to see it works, I tested on my own and the results seems to be fine, but no harm to re-check your side, maybe there is something not working as expected (Output.txt).
Last edited by Smile_ on 27 May 2022, 15:40, edited 1 time in total.

User avatar
zvit
Posts: 224
Joined: 07 Nov 2017, 06:15

Re: How to make AHK utilize resources and run faster?

Post by zvit » 27 May 2022, 15:40

Why is this so much faster? Is it because objects are faster than arrays?

User avatar
Smile_
Posts: 857
Joined: 03 May 2020, 00:51

Re: How to make AHK utilize resources and run faster?

Post by Smile_ » 27 May 2022, 16:01

Arrays are also objects, when you deal with big text files it is better to use the File-Object method (Faster than parsing with loop).
Also the way you write your code, lesser loops and cleaner code will certainly make it faster.

SetBatchLines, -1 removes the sleep between commands execution of the script, as the doc say
Use SetBatchLines -1 to never sleep (i.e. have the script run at maximum speed).

Also these texts updating on the GUI, it can slow your script speed, try avoiding them as much as possible.
I tried to add one text control to display progress percentage and got about 2 seconds delay.

User avatar
Chunjee
Posts: 1400
Joined: 18 Apr 2014, 19:05
Contact:

Re: How to make AHK utilize resources and run faster?

Post by Chunjee » 27 May 2022, 18:39

it would probably be slightly slower than that string search but https://chunjee.github.io/array.ahk/#/docs?id=includes will tell you if your word is in the array

Code: Select all

myArr := ["Bill", "Ted", "Socrates"]
myArr.includes("Socrates")
; => true

myArr.includes("Lincoln")
; => false

Alternatively just add them all and https://biga-ahk.github.io/biga.ahk/#/?id=uniq can remove duplicates

Code: Select all

A := new biga() ; requires https://www.npmjs.com/package/biga.ahk

myArr := ["Bill", "Ted", "Socrates", "Ted", "Ted", "Socrates"]
noDupesArr := A.uniq(myArr)
; => ["Bill", "Ted", "Socrates"]

Rohwedder
Posts: 7551
Joined: 04 Jun 2014, 08:33
Location: Germany

Re: How to make AHK utilize resources and run faster?

Post by Rohwedder » 28 May 2022, 04:23

I think now it is faster:

Code: Select all

SetWinDelay, 0
SetBatchLines, -1
Gui, Add, Progress, xm w300 h20 vProgressBar -Smooth
Gui, Show,, Splitting Words...
StartTime := A_TickCount, Output := "", MainWords := []
Loop, Read, Input.txt
	MainWords[A_LoopReadLine] := ""
No := 100/MainWords.Count(), N := 0
For Word in MainWords
{
	Output .= Word ":`n", WLen := 2, WoLen := StrLen(Word)
	;Extract sub-word and check if it is a main word
	While, (++WLen < WoLen)
		Loop, % WoLen - WLen + 1
			If !MainWords.HasKey(SubWord:=SubStr(Word, A_Index, WLen))
				Output .= SubWord "`n"
	;-------------------------------------------------
	Output .= "`n"
	GuiControl,, ProgressBar,% N += No	
}
FileAppend, % Output, Output.txt
Msgbox, % "Complete in " A_TickCount - StartTime " ms"

User avatar
zvit
Posts: 224
Joined: 07 Nov 2017, 06:15

Re: How to make AHK utilize resources and run faster?

Post by zvit » 29 May 2022, 05:03

OMG, how are you guys doing this?? lol

Image

User avatar
zvit
Posts: 224
Joined: 07 Nov 2017, 06:15

Re: How to make AHK utilize resources and run faster?

Post by zvit » 29 May 2022, 05:07

@Rohwedder by the way, I had a question about your line of code: Loop,% Len-(1=Index:=A_Index) I see that Index is getting the value of A_Index but I'm curious what 1= means.

Rohwedder
Posts: 7551
Joined: 04 Jun 2014, 08:33
Location: Germany

Re: How to make AHK utilize resources and run faster?

Post by Rohwedder » 29 May 2022, 07:33

In Loop,% Len-(1=Index:=A_Index)
first Index gets the value of A_Index, so remains with the updated index:
Loop,% Len-(1=Index)
(1=Index) has the logical value True (i.e. 1) if Index contains the value 1 otherwise False (i.e. 0).
The loop count Len is reduced by this logical value i.e. Loop,% Len-1 or Loop,% Len
Oh dear, Autohotkey is often easier than English (or German)!

User avatar
zvit
Posts: 224
Joined: 07 Nov 2017, 06:15

Re: How to make AHK utilize resources and run faster?

Post by zvit » 29 May 2022, 09:20

Rohwedder wrote:
29 May 2022, 07:33
Oh dear, Autohotkey is often easier than English (or German)!
That's my problem; I am terrible at compacting script. That's why my scripts are 5 times longer than they should be and take longer to execute.
I would have done the long way cause I'm used the values being on the right side of the operands. :D : Loop,% Len-(Index:=A_Index = 1 ? 1 : 0)

I learn something new everyday.

Rohwedder
Posts: 7551
Joined: 04 Jun 2014, 08:33
Location: Germany

Re: How to make AHK utilize resources and run faster?

Post by Rohwedder » 29 May 2022, 09:51

Long and wrong! The loop count would have been correct, but Index would not have taken the value of A_Index

Code: Select all

Len = 2
Loop, 3
	MsgBox,% Len-(Index:=A_Index = 1 ? 1 : 0) "`n" 
	. A_Index "`n"
	. Index
I don't think "compacting" saves much time. I do it for fun. My sport is short.
Last edited by Rohwedder on 29 May 2022, 09:58, edited 1 time in total.

Post Reply

Return to “Ask for Help (v1)”