binary data: copy ('move'), compare, search, replace nulls

[jeeswg]

What are the current best methods to achieve binary copy ('move'), compare, search, replace nulls? Cheers.

==================================================

COPY BINARY DATA ('MOVE' BINARY DATA)

- RtlMoveMemory?

- memmove?
Is memmove faster?

- StrPut?
Is StrPut faster? Can AHK's StrPut function be used to move binary data from one location to another, unchanged?

[Note: 'copy' v. 'move'.][RtlCopyMemory v. RtlMoveMemory][memcpy v. memmove]
RtlMoveMemory routine (Windows Drivers)
https://msdn.microsoft.com/en-us/librar ... s.85).aspx

The RtlCopyMemory routine runs faster than RtlMoveMemory, but RtlCopyMemory requires that the source and destination memory blocks do not overlap.

==================================================

COMPARE BINARY DATA

- RtlCompareMemory?

- memcmp?
Is memcmp faster?

==================================================

SEARCH BINARY DATA / REPLACE NULLS

E.g. how Notepad opens non-text files: replace null bytes with space bytes, and treat the resulting text as ANSI.
E.g. when creating an hDrop (CF_HDROP), to cut/copy files to the clipboard, or drag-and-drop files to a window: replace LFs with null characters in a LF-delimited list of files.

- InBuf function?
Machine code binary buffer searching regardless of NULL - Scripts and Functions - AutoHotkey Community
https://autohotkey.com/board/topic/2362 ... s-of-null/
InBuf function currently 32-bit only (machine code binary buffer searching) - AutoHotkey Community
https://autohotkey.com/boards/viewtopic.php?f=5&t=28393

- repeated NumGet?

- create a RegExMatch 'ANSI' version for AHK Unicode versions?
RegExMatch in AHK Unicode versions can only search for pairs of bytes at even offsets.

- File object?
see: https://autohotkey.com/docs/objects/File.htm
Num := File.ReadNumType()
i.e. Num := File.ReadXXX()
e.g. Num := File.ReadUChar()

==================================================

Note: what if the files you want to binary compare/binary search, are too big to be stored in a variable?

[For a list of RTL functions, go to the link below, and press the down arrow (triangle) at the top.]
Run-Time Library (RTL) Routines (Windows Drivers)
https://msdn.microsoft.com/en-us/librar ... s.85).aspx

[Interesting link. I would read the 'show _ more comment(s)' also.]
windows - Why is RtlFillMemory/RtlCopyMemory defined as macro - Stack Overflow
http://stackoverflow.com/questions/1600 ... d-as-macro

Thanks for reading.

[SKAN]

COMPARE BINARY DATA

- RtlCompareMemory?

- memcmp?
Is memcmp faster?

For a function I'm writing, I need to know if the bits of a bitmap had changed.
Currently I'm using my old technique ( I used in PixelCheckSum() ) of calling CRC32 twice and check if something has changed.
This uses the whole buffer which is redundant when a bitmap turns from complete white to black.. since comparing just the first pixel is enough.
So I site:googled for memcpy and landed on this first result.

Well, I did a test and RtlCompareMemory() seems to be the winner.
Speed gain at the cost of maintaining two buffers, but its too faster/efficient than CRC hash check.

Here is the test script.. and a caution to anybody wanting to test it:
The following script uses over 200 MiB of memory!

Code: Select all

#NoEnv
#Warn
#SingleInstance, Force
SetWorkingDir %A_ScriptDir% 



Len := 100*1024*1024 ; 100 MiB

VarSetCapacity( Var1,Len,0)
NumPut( 0xFFFFFFFF, Var1, Len-4, "UInt" )

VarSetCapacity( Var2,Len,0)
NumPut( 0xEEEEEEEE, Var2, Len-4, "UInt" )

Qpc(1)
R := DllCall("msvcrt\memcmp", "Ptr",&Var1, "Ptr",&Var2, "UInt",Len, "CDecl Int")
t1 := Qpc(0)

Qpc(1)
R := DllCall("ntdll\RtlCompareMemory", "Ptr", &Var1, "Ptr", &Var2, "Ptr",Len)
t2 := Qpc(0)

Qpc(1)
nCRC1 := DllCall( "NTDLL\RtlComputeCrc32", "UInt",0, "Ptr",&Var1, "UInt",Len, "UInt")
nCRC2 := DllCall( "NTDLL\RtlComputeCrc32", "UInt",0, "Ptr",&Var2, "UInt",Len, "UInt")
t3 := Qpc(0)


MsgbOx % T1 "`n" T2 "`n" T3


QPC( R := 0 ) {    ; By SKAN,  http://goo.gl/nf7O4G,  CD:01/Sep/2014 | MD:01/Sep/2014
  Static P := 0,  F := 0,     Q := DllCall( "QueryPerformanceFrequency", "Int64P",F )
Return ! DllCall( "QueryPerformanceCounter","Int64P",Q ) + ( R ? (P:=Q)/F : (Q-P)/F ) 
}

[TLM]

Nice work SKAN

[SKAN]

Nice work SKAN

Hi TLM

[Helgef]

Well, I did a test and RtlCompareMemory() seems to be the winner.

memcmp is about 2-3 times faster here. There is a small mistake in the RtlComputeCrc32 version, it doesn't matter though.

[SKAN]

There is a small mistake in the RtlComputeCrc32 version, it doesn't matter though.

Please tell, I will correct it.

[Helgef]

Code: Select all

nCRC1 := DllCall( "NTDLL\RtlComputeCrc32", "UInt",0, "Ptr",&Var1, "UInt",Len, "UInt")
nCRC2 := DllCall( "NTDLL\RtlComputeCrc32", "UInt",0, "Ptr",&Var1, "UInt",Len, "UInt")

You use &Var1 for both . But it doesn't matter for the speed test ofc.

[SKAN]

You use &Var1 for both .

Thanks.. Corrected!.

[SKAN]

memcmp is about 2-3 times faster here.

I googled and did not find much
Ref: http://www.osronline.com/showThread.cfm?link=1866
Just skimmed the topic.

Seemingly, RtlCompareMemory() is a wrapper for MemCmp() and is the recommended method.
I just tested after reading the topic, the compare commands run almost twice the speed in x64 than in x86.

PS:
AHK 1.1 Unicode 32bit has been my preferred flavor
I have a x64 windows' for the sake of Adobe Photoshop (I am a hobbyist nature photographer)

[BoBo]

I have a x64 windows' for the sake of Adobe Photoshop (I am a hobbyist nature photographer)

So, where we can see those photos (instagram/fb/pinterest/500px/...)??

PS. do you have any photoshop.ahk's in stock Let's guess theres's a huge fan-base for those out there

[SKAN]

Hi BoBo!

So, where we can see those photos (instagram/fb/pinterest/500px/...)??

https://www.flickr.com/photos/ansk/

PS. do you have any photoshop.ahk's in stock Let's guess theres's a huge fan-base for those out there

Not much.. Just single click: Saving to JPG/PSD, Resizing and few other thing I never use.

[jeeswg]

I found an answer to the 'search binary data / replace nulls' query.

See READ BINARY DATA AND REPLACE NULLS, here:
jeeswg's File object mini-tutorial - AutoHotkey Community
https://autohotkey.com/boards/viewtopic.php?f=74&t=66674

Also, here, for an introduction to the main idea:
InBuf function currently 32-bit only (machine code binary buffer searching) - Page 2 - AutoHotkey Community
https://autohotkey.com/boards/viewtopic.php?f=5&t=28393&p=289604#p289604