binary data: copy ('move'), compare, search, replace nulls

Get help with using AutoHotkey and its commands and hotkeys
User avatar
jeeswg
Posts: 6902
Joined: 19 Dec 2016, 01:58
Location: UK

binary data: copy ('move'), compare, search, replace nulls

21 Apr 2017, 23:51

What are the current best methods to achieve binary copy ('move'), compare, search, replace nulls? Cheers.

==================================================

COPY BINARY DATA ('MOVE' BINARY DATA)

- RtlMoveMemory?

- memmove?
Is memmove faster?

- StrPut?
Is StrPut faster? Can AHK's StrPut function be used to move binary data from one location to another, unchanged?

[Note: 'copy' v. 'move'.][RtlCopyMemory v. RtlMoveMemory][memcpy v. memmove]
RtlMoveMemory routine (Windows Drivers)
https://msdn.microsoft.com/en-us/librar ... s.85).aspx
The RtlCopyMemory routine runs faster than RtlMoveMemory, but RtlCopyMemory requires that the source and destination memory blocks do not overlap.
==================================================

COMPARE BINARY DATA

- RtlCompareMemory?

- memcmp?
Is memcmp faster?

==================================================

SEARCH BINARY DATA / REPLACE NULLS

E.g. how Notepad opens non-text files: replace null bytes with space bytes, and treat the resulting text as ANSI.
E.g. when creating an hDrop (CF_HDROP), to cut/copy files to the clipboard, or drag-and-drop files to a window: replace LFs with null characters in a LF-delimited list of files.

- InBuf function?
Machine code binary buffer searching regardless of NULL - Scripts and Functions - AutoHotkey Community
https://autohotkey.com/board/topic/2362 ... s-of-null/
InBuf function currently 32-bit only (machine code binary buffer searching) - AutoHotkey Community
https://autohotkey.com/boards/viewtopic.php?f=5&t=28393

- repeated NumGet?

- create a RegExMatch 'ANSI' version for AHK Unicode versions?
RegExMatch in AHK Unicode versions can only search for pairs of bytes at even offsets.

- File object?
see: https://autohotkey.com/docs/objects/File.htm
Num := File.ReadNumType()
i.e. Num := File.ReadXXX()
e.g. Num := File.ReadUChar()

==================================================

Note: what if the files you want to binary compare/binary search, are too big to be stored in a variable?

[For a list of RTL functions, go to the link below, and press the down arrow (triangle) at the top.]
Run-Time Library (RTL) Routines (Windows Drivers)
https://msdn.microsoft.com/en-us/librar ... s.85).aspx

[Interesting link. I would read the 'show _ more comment(s)' also.]
windows - Why is RtlFillMemory/RtlCopyMemory defined as macro - Stack Overflow
http://stackoverflow.com/questions/1600 ... d-as-macro

Thanks for reading.
homepage | tutorials | wish list | fun threads | donate
WARNING: copy your posts/messages before hitting Submit as you may lose them due to CAPTCHA
User avatar
SKAN
Posts: 737
Joined: 29 Sep 2013, 16:58

Re: binary data: copy ('move'), compare, search, replace nulls

12 Sep 2017, 09:27

jeeswg wrote:COMPARE BINARY DATA

- RtlCompareMemory?

- memcmp?
Is memcmp faster?
For a function I'm writing, I need to know if the bits of a bitmap had changed.
Currently I'm using my old technique ( I used in PixelCheckSum() ) of calling CRC32 twice and check if something has changed.
This uses the whole buffer which is redundant when a bitmap turns from complete white to black.. since comparing just the first pixel is enough.
So I site:googled for memcpy and landed on this first result.

Well, I did a test and RtlCompareMemory() seems to be the winner.
Speed gain at the cost of maintaining two buffers, but its too faster/efficient than CRC hash check.

Here is the test script.. and a caution to anybody wanting to test it:
The following script uses over 200 MiB of memory!

Code: Select all

#NoEnv
#Warn
#SingleInstance, Force
SetWorkingDir %A_ScriptDir% 



Len := 100*1024*1024 ; 100 MiB

VarSetCapacity( Var1,Len,0)
NumPut( 0xFFFFFFFF, Var1, Len-4, "UInt" )

VarSetCapacity( Var2,Len,0)
NumPut( 0xEEEEEEEE, Var2, Len-4, "UInt" )

Qpc(1)
R := DllCall("msvcrt\memcmp", "Ptr",&Var1, "Ptr",&Var2, "UInt",Len, "CDecl Int")
t1 := Qpc(0)

Qpc(1)
R := DllCall("ntdll\RtlCompareMemory", "Ptr", &Var1, "Ptr", &Var2, "Ptr",Len)
t2 := Qpc(0)

Qpc(1)
nCRC1 := DllCall( "NTDLL\RtlComputeCrc32", "UInt",0, "Ptr",&Var1, "UInt",Len, "UInt")
nCRC2 := DllCall( "NTDLL\RtlComputeCrc32", "UInt",0, "Ptr",&Var2, "UInt",Len, "UInt")
t3 := Qpc(0)


MsgbOx % T1 "`n" T2 "`n" T3


QPC( R := 0 ) {    ; By SKAN,  http://goo.gl/nf7O4G,  CD:01/Sep/2014 | MD:01/Sep/2014
  Static P := 0,  F := 0,     Q := DllCall( "QueryPerformanceFrequency", "Int64P",F )
Return ! DllCall( "QueryPerformanceCounter","Int64P",Q ) + ( R ? (P:=Q)/F : (Q-P)/F ) 
}
Helgef
Posts: 4459
Joined: 17 Jul 2016, 01:02
Contact:

Re: binary data: copy ('move'), compare, search, replace nulls

12 Sep 2017, 16:43

Well, I did a test and RtlCompareMemory() seems to be the winner.
memcmp is about 2-3 times faster here. There is a small mistake in the RtlComputeCrc32 version, it doesn't matter though.
User avatar
SKAN
Posts: 737
Joined: 29 Sep 2013, 16:58

Re: binary data: copy ('move'), compare, search, replace nulls

12 Sep 2017, 16:50

Helgef wrote:There is a small mistake in the RtlComputeCrc32 version, it doesn't matter though.
Please tell, I will correct it.
Helgef
Posts: 4459
Joined: 17 Jul 2016, 01:02
Contact:

Re: binary data: copy ('move'), compare, search, replace nulls

12 Sep 2017, 16:59

Code: Select all

nCRC1 := DllCall( "NTDLL\RtlComputeCrc32", "UInt",0, "Ptr",&Var1, "UInt",Len, "UInt")
nCRC2 := DllCall( "NTDLL\RtlComputeCrc32", "UInt",0, "Ptr",&Var1, "UInt",Len, "UInt")
You use &Var1 for both ;). But it doesn't matter for the speed test ofc.
User avatar
SKAN
Posts: 737
Joined: 29 Sep 2013, 16:58

Re: binary data: copy ('move'), compare, search, replace nulls

12 Sep 2017, 17:04

Helgef wrote:You use &Var1 for both ;).
Thanks.. Corrected!. :)
User avatar
SKAN
Posts: 737
Joined: 29 Sep 2013, 16:58

Re: binary data: copy ('move'), compare, search, replace nulls

12 Sep 2017, 18:16

Helgef wrote:memcmp is about 2-3 times faster here.
I googled and did not find much
Ref: http://www.osronline.com/showThread.cfm?link=1866
Just skimmed the topic.

Seemingly, RtlCompareMemory() is a wrapper for MemCmp() and is the recommended method.
I just tested after reading the topic, the compare commands run almost twice the speed in x64 than in x86. :shock:

PS:
AHK 1.1 Unicode 32bit has been my preferred flavor
I have a x64 windows' for the sake of Adobe Photoshop (I am a hobbyist nature photographer)
BoBo
Posts: 3636
Joined: 13 May 2014, 17:15

Re: binary data: copy ('move'), compare, search, replace nulls

13 Sep 2017, 00:54

I have a x64 windows' for the sake of Adobe Photoshop (I am a hobbyist nature photographer)
So, where we can see those photos (instagram/fb/pinterest/500px/...)?? :)

PS. do you have any photoshop.ahk's in stock :?: 8-) Let's guess theres's a huge fan-base for those out there :!: :idea:
User avatar
SKAN
Posts: 737
Joined: 29 Sep 2013, 16:58

Re: binary data: copy ('move'), compare, search, replace nulls

13 Sep 2017, 01:40

Hi BoBo! :)
BoBo wrote:So, where we can see those photos (instagram/fb/pinterest/500px/...)?? :)
https://www.flickr.com/photos/ansk/
BoBo wrote:PS. do you have any photoshop.ahk's in stock :?: 8-) Let's guess theres's a huge fan-base for those out there :!: :idea:
Not much.. Just single click: Saving to JPG/PSD, Resizing and few other thing I never use.
User avatar
jeeswg
Posts: 6902
Joined: 19 Dec 2016, 01:58
Location: UK

Re: binary data: copy ('move'), compare, search, replace nulls

24 Aug 2019, 15:58

I found an answer to the 'search binary data / replace nulls' query.

See READ BINARY DATA AND REPLACE NULLS, here:
jeeswg's File object mini-tutorial - AutoHotkey Community
https://autohotkey.com/boards/viewtopic.php?f=74&t=66674

Also, here, for an introduction to the main idea:
InBuf function currently 32-bit only (machine code binary buffer searching) - Page 2 - AutoHotkey Community
https://autohotkey.com/boards/viewtopic.php?f=5&t=28393&p=289604#p289604
homepage | tutorials | wish list | fun threads | donate
WARNING: copy your posts/messages before hitting Submit as you may lose them due to CAPTCHA

Return to “Ask For Help”

Who is online

Users browsing this forum: Bing [Bot], malcev, mikeyww and 46 guests