MapFile() : Maps a file and returns a memory pointer.

Post your working scripts, libraries and tools for AHK v1.1 and older
User avatar
SKAN
Posts: 1551
Joined: 29 Sep 2013, 16:58

MapFile() : Maps a file and returns a memory pointer.

Post by SKAN » 10 May 2021, 09:21

MapFile(File object, Out Bytes)
The function is best used with x64 version of AutoHotkey. My system has 8 GiB RAM and there is competition for sub 4 GIB memory space.
Owing to this, I can hardly map a 700 MiB file in x86 version of AutoHotkey, though theoretically I should be able to map upto 2 GIB.
x64 is solid.. but anyways, there is not much use mapping a file greater than 4 GIB. Only a few functions would accept size_t bytes.
 
Here is a simple example:

Code: Select all

#NoEnv
#Warn
#SingleInstance, Force

File := FileOpen(A_AhkPath, "r")               ; Open file
pHaystack := MapFile(File, HaystackBytes:=0)   ; Create file mapping

nCrc := DllCall("ntdll.dll\RtlComputeCrc32", "Int",0, "Ptr",pHaystack, "Int",HaystackBytes, "UInt")

pHaystack := MapFile(File)                     ; Clear file mapping
File.Close()                                   ; Close file

MsgBox % Format("0x{:X}", nCrc)
 
The following 2 examples require InBin() for binary search.

Example 1:
AutoHotkey's license.txt has the word "Cambridge" occurring 4 times.
The file is ansi text and would require double the memory to load it into Unicode AHK and search for "Cambridge" using inStr().
The following example will search for the second occurrence of the word "Cambridge" (from bottom) and extract that particular line.
 

Code: Select all

#NoEnv
#Warn
#SingleInstance, Force

File := FileOpen(A_AhkPath . "\..\license.txt", "r")    ; Open file
pHaystack := MapFile(File, HaystackBytes:=0)            ; Create file mapping

; find the word "Cambridge", second occurence from bottom.
FoundPtr := InBin(pHaystack, HaystackBytes, "Cambridge", File.Encoding,, 0, 2)

; find the CRLF preceding the word.
FoundPtr := InBin(pHaystack, FoundPtr-pHaystack, 0x0A0D, -2,, 0) ; CRLF := 0x0A0D

File.Pos := FoundPtr-pHaystack +2 ; +2 to move past CRLF

MsgBox % File.ReadLine()

pHaystack := MapFile(File)                              ; Clear file mapping
File.Close()                                            ; Close file
 
Example 2:
Save the following code as text.ahk and run.

Code: Select all

#NoEnv
#Warn
#SingleInstance, Force
Gui, Show, w320 h180
Sleep 500
WinGetClass, Class, A
Gui, Show,, %Class%
Return

GuiClose:
  ExitApp
You will see in the title-bar, the ahk_class for GUI, which is AutoHotkeyGUI
 
image.png
image.png (3.44 KiB) Viewed 1935 times
 
Compile above test.ahk to test.exe and use the following code to change the ahk_class to SKAN-UI.
This is just a simple search and replace and will work on both Ansi/Unicode exes.

Code: Select all

#NoEnv
#Warn
#SingleInstance, Force

File := FileOpen("test.exe", "rw")             ; Open file
pHaystack := MapFile(File, HaystackBytes:=0)   ; Create file mapping

If ! FoundPtr := InBin(pHaystack, HaystackBytes, "AutoHotkeyGUI", Encoding := "utf-16")
     FoundPtr := InBin(pHaystack, HaystackBytes, "AutoHotkeyGUI", Encoding := "cp0")

If ( FoundPtr )
{
    VarSetCapacity(Var, 28, 0)
    StrPut("SKAN-UI", &Var, Encoding)

    File.Pos := FoundPtr-pHaystack
    File.RawWrite(&Var, Encoding="utf-16" ? 26 : 13)
}

pHaystack := MapFile(File)                     ; Clear file mapping
File.Close()                                   ; Close file
Run the exe again to see the change. :)
 
 
The function:
 

Code: Select all

MapFile(File, ByRef Bytes:=0) {     ; MapFile V0.33 by SKAN on D459/D45A @ tiny.cc/mapfile
Local
Static db := {}
  If ! ( h := File.__handle )
       Return (0, "Invalid File object.")

  If ( IsByRef(Bytes) )
  If ( DB.HasKey(h) )
       Return (db[h].FileView, ErrorLevel := "")

  If ( !IsByRef(Bytes) )
  {
       If ( DB.HasKey(h) )
          , DllCall("UnmapViewOfFile", "Ptr",db[h].FileView)
          , DllCall("CloseHandle", "Ptr",db[h].MapFile)
          , db.Delete(h)
       Return (0, ErrorLevel := "")
  }

  VarSetCapacity(ioStatus, 12, 0)
  DllCall("ntdll.dll\NtQueryInformationFile", "Ptr",File.__handle, "Ptr",&ioStatus
        , "UIntP",ACCESS_MASK:=0, "Int",4, "Int",FileAccessInformation:=8)

  READ_ACCESS := ACCESS_MASK & 1,   WRITE_ACCESS := (ACCESS_MASK >> 1) & 1
  If ! ( READ_ACCESS )
         Return (0, ErrorLevel := "File object doesn't have read access.")

  If ! ( Bytes := File.Length )
         Return (0, ErrorLevel := "MapFile cannot map a zero byte file.")
  
  PAGE_READWRITE := 0x4,   PAGE_READONLY := 0x2
  hMapFile := DllCall("CreateFileMapping", "Ptr",h, "Ptr",0
                    , "Int",WRITE_ACCESS ? PAGE_READWRITE : PAGE_READONLY
                    , "Int",0, "Int",0, "Ptr",0, "Ptr")
  If ! ( hMapFile )
         Return (Bytes:=0, ErrorLevel := "MapFile failed.`nLastError: " . A_LastError)

  FILE_MAP_ALL_ACCESS := 0x000F001F,   FILE_MAP_READ := 0x00000004
  pFileView := DllCall("MapViewOfFile", "Ptr",hMapFile
                     , "Int",WRITE_ACCESS ? FILE_MAP_ALL_ACCESS : FILE_MAP_READ
                     , "Int",0, "Int",0, "Ptr",0, "Ptr")
  If ! ( pFileView )
  {
         Err := A_LastError
         DllCall("CloseHandle", "Ptr",hMapFile)
         Return (Bytes:=0, ErrorLevel := "MapFile failed.`nLastError: " . Err)
  }
  
Return ( (db[h]:={"MapFile":hMapFile, "FileView":pFileView}).FileView,  ErrorLevel := "" )
}
My Scripts and Functions: V1  V2

hasantr
Posts: 933
Joined: 05 Apr 2016, 14:18
Location: İstanbul

Re: MapFile() : Bir dosyayı eşler ve bir bellek işaretçisi döndürür.

Post by hasantr » 10 May 2021, 19:12

I'm reviewing this now. I need some time to understand. Thank you so much.
I learn a lot from their functions. :)

User avatar
joedf
Posts: 8940
Joined: 29 Sep 2013, 17:08
Location: Canada
Contact:

Re: MapFile() : Maps a file and returns a memory pointer.

Post by joedf » 10 May 2021, 19:54

Never thought I'd see file mapping being used in ahk, neat! :+1:
Image Image Image Image Image
Windows 10 x64 Professional, Intel i5-8500, NVIDIA GTX 1060 6GB, 2x16GB Kingston FURY Beast - DDR4 3200 MHz | [About Me] | [About the AHK Foundation] | [Courses on AutoHotkey]
[ASPDM - StdLib Distribution] | [Qonsole - Quake-like console emulator] | [LibCon - Autohotkey Console Library]

User avatar
SKAN
Posts: 1551
Joined: 29 Sep 2013, 16:58

Re: MapFile() : Maps a file and returns a memory pointer.

Post by SKAN » 11 May 2021, 02:49

joedf wrote:
10 May 2021, 19:54
Never thought I'd see file mapping being used in ahk, neat! :+1:
Thanks @joedf
User wOxxOm had used file mapping in InFile()
https://autohotkey.com/board/topic/23627-machine-code-binary-buffer-searching-regardless-of-null/

Mine is standalone/simple and creates file mapping based on the access mode of the file object.
Figuring out the file access-mode was hardest part for me.
It now looks easy though:

Code: Select all

#NoEnv
#Warn
#SingleInstance, Force

filepath := A_Temp . "\skan.tmp"

File := FileOpen(filepath, "w")
MsgBox % accessmode(File)
File.Close()

File := FileOpen(filepath, "rw")
MsgBox % accessmode(File)
File.Close()

File := FileOpen(filepath, "r")
MsgBox % accessmode(File)
File.Close()


accessmode(File) {
Local
  VarSetCapacity(ioStatus, 12, 0)
  DllCall("ntdll.dll\NtQueryInformationFile", "Ptr",File.__handle, "Ptr",&ioStatus
        , "UIntP",ACCESS_MASK:=0, "Int",4, "Int",FileAccessInformation:=8)
Return Format("{:}{:}", ACCESS_MASK & 1 ? "r" : "", ACCESS_MASK>>1 & 1 ? "w" : "")
}

User avatar
SKAN
Posts: 1551
Joined: 29 Sep 2013, 16:58

Re: MapFile()

Post by SKAN » 11 May 2021, 02:52

hasantr wrote:
10 May 2021, 19:12
I'm reviewing this now. I need some time to understand. Thank you so much.
I learn a lot from their functions. :)
:thumbup:

whynotregister
Posts: 147
Joined: 05 Nov 2016, 22:42

Re: MapFile() : Maps a file and returns a memory pointer.

Post by whynotregister » 14 Oct 2021, 14:53

I don't know what this functions means. Can someone explain this function?

Tre4shunter
Posts: 139
Joined: 26 Jan 2016, 16:05

Re: MapFile() : Maps a file and returns a memory pointer.

Post by Tre4shunter » 14 Oct 2021, 21:57

Very cool functions - as always!

How would you use this to find All instances of a string within a file?

I did it by simply looping using your first script as an example - and that worked, but am curious how you would do it -- as you are the master!
Seemed like i could have been doing it better, as each time it was searching the whole mapped file, where i think it could have 'stripped out' portions where nothing was found? If that makes sense...

Thanks!

Post Reply

Return to “Scripts and Functions (v1)”