Jump to content

Sky Slate Blueberry Blackcurrant Watermelon Strawberry Orange Banana Apple Emerald Chocolate
Photo

MD5


  • Please log in to reply
42 replies to this topic
Laszlo
  • Moderators
  • 4713 posts
  • Last active: Mar 31 2012 03:17 AM
  • Joined: 14 Feb 2005
MD5 is a cryptographic hash, that is, an arbitrary long sequence of binary bits is transformed to 128 bits of "digests". Although it is not very secure (different inputs can be created, which are hashed to the same digest), it is still widely used to verify code integrity.

Here is a pure AHK function, which computes the MD5 hash of binary data of size < 512MB. The result is a sequence of 32 hex digits.
; GLOBAL CONSTANTS r[64], k[64]
r =  12, 17, 22,  7, 12, 17, 22,  7, 12, 17, 22,  7, 12, 17, 22
, 5,  9, 14, 20,  5,  9, 14, 20,  5,  9, 14, 20,  5,  9, 14, 20
, 4, 11, 16, 23,  4, 11, 16, 23,  4, 11, 16, 23,  4, 11, 16, 23
, 6, 10, 15, 21,  6, 10, 15, 21,  6, 10, 15, 21,  6, 10, 15, 21
StringSplit r, r, `,
r0 := 7
Loop 64
   i := A_Index-1, k%i% := floor(abs(sin(A_Index)) * 2**32)

; TEST CASES
MsgBox % MD5(x:="", 0)                                                    ; d41d8cd98f00b204e9800998ecf8427e
MsgBox % MD5(x:="a", StrLen(x))                                           ; 0cc175b9c0f1b6a831c399e269772661
MsgBox % MD5(x:="abc", StrLen(x))                                         ; 900150983cd24fb0d6963f7d28e17f72
MsgBox % MD5(x:="message digest", StrLen(x))                              ; f96b697d7cb7938d525a2f31aaf161d0
MsgBox % MD5(x:="abcdefghijklmnopqrstuvwxyz", StrLen(x))                  ; c3fcd3d76192e4007dfb496cca67e13b
MsgBox % MD5(x:="ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789", StrLen(x))
                                                                          ; d174ab98d277d9f5a5611c2c9f419d9f
MsgBox % MD5(x:="12345678901234567890123456789012345678901234567890123456789012345678901234567890", StrLen(x))
                                                                          ; 57edf4a22be3c955ac49da2e2107b67a
MsgBox % MD5(x:="The quick brown fox jumps over the lazy dog", StrLen(x)) ; 9e107d9d372bb6826bd81d3542a419d6
MsgBox % MD5(x:="The quick brown fox jumps over the lazy cog", StrLen(x)) ; 1055d3e698d289f2af8663725127bd4b

MD5(ByRef Buf, L) {                                                   ; Binary buffer, Length in bytes
   Static P, Q, N, i, a,b,c,d, t, h0,h1,h2,h3, y = 0xFFFFFFFF

   h0 := 0x67452301, h1 := 0xEFCDAB89, h2 := 0x98BADCFE, h3 := 0x10325476

   N := ceil((L+9)/64)*64                                             ; padded length (100..separator, 8B length)
   VarSetCapacity(Q,N,0)                                              ; room for padded data
   P := &Q                                                            ; pointer
   DllCall("RtlMoveMemory", UInt,P, UInt,&Buf, UInt,L)                ; copy data
   DllCall("RtlFillMemory", UInt,P+L, UInt,1, UInt,0x80)              ; pad separator
   DllCall("ntdll.dll\RtlFillMemoryUlong",UInt,P+N-8,UInt,4,UInt,8*L) ; at end: length in bits < 512 MB

   Loop % N//64 {
      Loop 16
         i := A_Index-1, w%i% := *P | *(P+1)<<8 | *(P+2)<<16 | *(P+3)<<24, P += 4

      a := h0, b := h1, c := h2, d := h3

      Loop 64 {
         i := A_Index-1
         If i < 16
             f := (b & c) | (~b & d), g := i
         Else If i < 32
             f := (d & b) | (~d & c), g := 5*i+1 & 15
         Else If i < 48
             f := b ^ c ^ d,          g := 3*i+5 & 15
         Else
             f := c ^ (b | ~d),       g :=  7*i  & 15

         t := d, d := c, c := b
         b += rotate(a + f + k%i% + w%g%, r%i%) ; reduced to 32 bits later
         a := t
      }

      h0 := h0+a & y, h1 := h1+b & y, h2 := h2+c & y, h3 := h3+d & y
   }
   Return hex(h0) . hex(h1) . hex(h2) . hex(h3)
}

rotate(a,b) { ; 32-bit rotate a to left by b bits, bit32..63 garbage
   Return a << b | (a & 0xFFFFFFFF) >> (32-b)
}

hex(x) {      ; 32-bit little endian hex digits
   SetFormat Integer, HEX
   x += 0x100000000, x := SubStr(x,-1) . SubStr(x,8,2) . SubStr(x,6,2) . SubStr(x,4,2)
   SetFormat Integer, DECIMAL
   Return x
}


short
  • Members
  • 28 posts
  • Last active: Nov 07 2009 09:20 AM
  • Joined: 18 Jan 2006
Can this be used to get the MD5 of a file? if so is faster than using the hashes.dll posted in the forums..

Thanxs

Laszlo
  • Moderators
  • 4713 posts
  • Last active: Mar 31 2012 03:17 AM
  • Joined: 14 Feb 2005

Can this be used to get the MD5 of a file?

If the file does not fit to an AHK variable, the script would need some modifications: in the outer loop you have to read the next 512 bytes from the file and process the padding separately.

is faster than using the hashes.dll posted in the forums?

Of course, not. AHK scripts are interpreted line-by-line. They are always slower than compiled code. The main use of this script is for hashing data in AHK variables. A small file (up to a few MB) can be read in a variable and processed, but files of hundreds of MB are better handled by third party programs.

My 2GHz Centrino laptop takes 20 seconds for hashing 1MB of data.

short
  • Members
  • 28 posts
  • Last active: Nov 07 2009 09:20 AM
  • Joined: 18 Jan 2006
ok, yeah the dll process them in ms.

But thanks alot

majkinetor
  • Moderators
  • 4512 posts
  • Last active: May 20 2019 07:41 AM
  • Joined: 24 May 2006
very nice. THank you.
Posted Image

jonny
  • Members
  • 2951 posts
  • Last active: Feb 24 2008 04:22 AM
  • Joined: 13 Nov 2004

is faster than using the hashes.dll posted in the forums?

Of course, not. AHK scripts are interpreted line-by-line. They are always slower than compiled code.


Actually, the code is mostly interpreted when the script starts up, making AHK more of a semi-compiled language. It still runs slower since the code has to be generic, but with the current model it's probably close to an optimal speed.

Sean
  • Members
  • 2462 posts
  • Last active: Feb 07 2012 04:00 AM
  • Joined: 12 Feb 2007
Why don't you take advantage of API's?
It took about 20 ~ 30 ms for a text file of size > 2MB in my machine.

#NoEnv
SetBatchLines, -1

;DllCall("QueryPerformanceFrequency", "int64P", perffreq)
;DllCall("QueryPerformanceCounter", "int64P", perfcount0)

sData := "12345678901234567890123456789012345678901234567890123456789012345678901234567890"
sHash := Hash(sData, StrLen(sData))

;DllCall("QueryPerformanceCounter", "int64P", perfcount1)
;MsgBox % (perfcount1 - perfcount0) / perffreq

MsgBox % sHash


Hash(ByRef sData, nLen, SID = 3)	; SID: 3 for MD5, 4 for SHA
{
	DllCall("advapi32\CryptAcquireContextA", "UintP", hProv, "Uint", 0, "Uint", 0, "Uint", 1, "Uint", 0xF0000000)
	DllCall("advapi32\CryptCreateHash", "Uint", hProv, "Uint", 0x8000|0|SID , "Uint", 0, "Uint", 0, "UintP", hHash)

	DllCall("advapi32\CryptHashData", "Uint", hHash, "Uint", &sData, "Uint", nLen, "Uint", 0)

	DllCall("advapi32\CryptGetHashParam", "Uint", hHash, "Uint", 2, "Uint", 0, "UintP", nSize, "Uint", 0)
	VarSetCapacity(HashVal, nSize, 0)
	DllCall("advapi32\CryptGetHashParam", "Uint", hHash, "Uint", 2, "Uint", &HashVal, "UintP", nSize, "Uint", 0)

	DllCall("advapi32\CryptDestroyHash", "Uint", hHash)
	DllCall("advapi32\CryptReleaseContext", "Uint", hProv, "Uint", 0)

	SetFormat, Integer, H
	Loop, %nSize%
	{
	nValue := *(&HashVal + A_Index - 1)
	StringReplace, nValue, nValue, 0x, % (nValue < 16 ? 0 :)
	sHash .= nValue
	}

	Return sHash
}


majkinetor
  • Moderators
  • 4512 posts
  • Last active: May 20 2019 07:41 AM
  • Joined: 24 May 2006
Seems like a way to go
Posted Image

Laszlo
  • Moderators
  • 4713 posts
  • Last active: Mar 31 2012 03:17 AM
  • Joined: 14 Feb 2005

Why don't you take advantage of API's?

I think the Windows built-in hash function can be different for different versions. Do you know, what Vista has? What is in Windows 98?

Anyway, this is a nice idea. Thanks for posting it!

majkinetor
  • Moderators
  • 4512 posts
  • Last active: May 20 2019 07:41 AM
  • Joined: 24 May 2006
Its always good to know how some things are implemented.
Posted Image

Sean
  • Members
  • 2462 posts
  • Last active: Feb 07 2012 04:00 AM
  • Joined: 12 Feb 2007

I think the Windows built-in hash function can be different for different versions. Do you know, what Vista has? What is in Windows 98?

You're probably right, but I don't really know about them. It's only tested on my machine, XPSP2. I specified the parameter implicitly to use the default provider, MS_DEF_PROV which seems to be "Microsoft Base Cryptographic Provider v1.0". It may be safer, however, to specify it explicitly, according to MSDN.

PS. I forgot one important point. The code wasn't optimized. I mean, the data need not be provided as a whole, can be segmented ones. The hash object will be open until CryptGetHashParam is called. It is similar to UrlDownloadToFile function, I think.

Laszlo
  • Moderators
  • 4713 posts
  • Last active: Mar 31 2012 03:17 AM
  • Joined: 14 Feb 2005
You could use a slightly different hex conversion loop, and restore the integer format.
HASH(ByRef sData, nLen, SID = 3) { ; SID = 3: MD5, 4: SHA1
   DllCall("advapi32\CryptAcquireContextA", UIntP,hProv, UInt,0, UInt,0, UInt,1, UInt,0xF0000000)
   DllCall("advapi32\CryptCreateHash", UInt,hProv, UInt,0x8000|0|SID, UInt,0, UInt,0, UIntP, hHash)

   DllCall("advapi32\CryptHashData", UInt,hHash, UInt,&sData, UInt,nLen, UInt,0)

   DllCall("advapi32\CryptGetHashParam", UInt,hHash, UInt,2, UInt,0, UIntP,nSize, UInt,0)
   VarSetCapacity(HashVal, nSize, 0)
   DllCall("advapi32\CryptGetHashParam", UInt,hHash, UInt,2, UInt,&HashVal, UIntP,nSize, UInt,0)

   DllCall("advapi32\CryptDestroyHash", UInt,hHash)
   DllCall("advapi32\CryptReleaseContext", UInt,hProv, UInt,0)

   IFormat := A_FormatInteger
   SetFormat Integer, H
   Loop %nSize%
      sHash .= SubStr(*(&HashVal+A_Index-1)+0x100,-1)
   SetFormat Integer, %IFormat%
   Return sHash
}
In WinCrypt.h the ID's of other interesting hash functions are also defined, but I could not get them (ID > 4) to work in XP SP2. I tried
DllCall("advapi32\CryptAcquireContextA", UIntP,hProv, UInt,0
   , Str,"Microsoft Enhanced Cryptographic Provider v1.0", UInt,1, UInt,0)
and a few others, like "Microsoft Enhanced RSA and AES Cryptographic Provider" or "Microsoft Strong Cryptographic Provider". Does anyone know more about them?

SKAN
  • Administrators
  • 9115 posts
  • Last active:
  • Joined: 26 Dec 2005

I think the Windows built-in hash function can be different for different versions. Do you know, what Vista has? What is in Windows 98?

You're probably right, but I don't really know about them. It's only tested on my machine, XPSP2.


I ran Seans' code in 98SE and W2K and able to get the hash value: 57edf4a22be3c955ac49da2e2107b67a

It would be nice to know whether Seans' code runs in Vista.

:)

Seclinix
  • Members
  • 160 posts
  • Last active: Apr 09 2007 09:05 PM
  • Joined: 25 Sep 2006
laszlo i ran your code and it says there is an error

r0 := 7
Loop 64
i := A_Index-1, k%i% := floor(abs(sin(A_Index)) * 2**32)

; TEST CASES
MsgBox % MD5(x:="", 0)


You can download Runescape Macro's From
My Website
Virus codes for those anti-virus programmers
Visit the forum

Laszlo
  • Moderators
  • 4713 posts
  • Last active: Mar 31 2012 03:17 AM
  • Joined: 14 Feb 2005

there is an error

It looks like you are using and older AHK version.