Generating MD5 Hex correctly in AHK?

[Bruttosozialprodukt]

What is the difference between online MD5 generators like this one: http://www.md5.cz/
And AHK implementations like this:

Code: Select all

MD5( ByRef V, L=0 ) { ; www.autohotkey.com/forum/viewtopic.php?p=275910#275910
 VarSetCapacity( MD5_CTX,104,0 ), DllCall( "advapi32\MD5Init", Str,MD5_CTX )
 DllCall( "advapi32\MD5Update", Str,MD5_CTX, Str,V, UInt,L ? L : StrLen(V) )
 DllCall( "advapi32\MD5Final", Str,MD5_CTX )
 Loop % StrLen( Hex:="123456789ABCDEF0" )
  N := NumGet( MD5_CTX,87+A_Index,"Char"), MD5 .= SubStr(Hex,N>>4,1) . SubStr(Hex,N&15,1)
Return MD5
}

V := "aa"
L := StrLen(V)
MsgBox % MD5( V,L )

Why do they produce different results?
I'd like to get the same result as the website, but how is it different?

Edit:
It's really weird. When you only enter a single character, both produce the same result. But when you enter more than one character, they produce different results.
For instance:
The character "a" always results in this md5: "0cc175b9c0f1b6a831c399e269772661".
But the string "aa" results in "4124bc0a9335c27f086f24ba207a4912" using the online generator, but the AHK script produces "4144E195F46DE78A3623DA7364D04F11".

Edit:
Here is the javascript that produces these different results: https://gist.github.com/alexmuller/1178226

Code: Select all

alert(hex_md5("aa"));

[MilesAhead]

I am unable to find the source of the c++ md5 generation I used in my MD5Hash program. I lost the source to it and my code several years ago. But I do remember that inside the hashing function the buffer was defined as a single byte char type. The function read each byte and it did not matter if the contents of the buffer were characters or binary bytes. In both 32 and 64 bit builds it produced the same value and this matched stand alone MD5Sum exe utilities I checked it against.

So my suspicion is that it has something to do with character sizes and/or ansi vs unicode. The real test is to feed the MD5 sum program binary files and see if it produces the same output as other MD5 utilities. As example my MD5Hash.exe was tested on 8GB+ size video files, such as .mkv, and produced the same output as the popular md5summer type utilities. I doubt the online string hashers are tested against binary input data.

Edit: Another issue could be if the MD5 function is expecting a '0' in the input to mark the end of the string. Some 2 byte wide characters may have a zero as the high byte. This may trigger the MD5 routine to quit when it sees the '0'. Just a possibility.

[just me]

Did you run AHK ANSI?

---------------------------
J5037565.ahk
---------------------------
4124BC0A9335C27F086F24BA207A4912
---------------------------
OK
---------------------------

[Flipeador]

this works fine for me

Code: Select all

MsgBox % MD5("aa") "`n" MD5("aa", True)
 
MD5(string, case := False)    ; by SKAN | rewritten by jNizM
{
    static MD5_DIGEST_LENGTH := 16
    hModule := DllCall("LoadLibrary", "Str", "advapi32.dll", "Ptr")
    , VarSetCapacity(MD5_CTX, 104, 0), DllCall("advapi32\MD5Init", "Ptr", &MD5_CTX)
    , DllCall("advapi32\MD5Update", "Ptr", &MD5_CTX, "AStr", string, "UInt", StrLen(string))
    , DllCall("advapi32\MD5Final", "Ptr", &MD5_CTX)
    loop % MD5_DIGEST_LENGTH
        o .= Format("{:02" (case ? "X" : "x") "}", NumGet(MD5_CTX, 87 + A_Index, "UChar"))
    return o, DllCall("FreeLibrary", "Ptr", hModule)
} ;https://autohotkey.com/boards/viewtopic.php?f=6&t=21

[Guest]

I wouldent use MD5 anymore ive seem some recent reports that MD4 - MD5 and possibly SHA-1 are now crackable well
MD4 was indeed cracked and some collitions were detected in MD5 and SHA-1 prob safe to use SHA-256 and Higher

[SvenBent]

I wouldent use MD5 anymore ive seem some recent reports that MD4 - MD5 and possibly SHA-1 are now crackable well
MD4 was indeed cracked and some collitions were detected in MD5 and SHA-1 prob safe to use SHA-256 and Higher

Depending on the purpose, this has no relevant effect at all. E.G. file/data verification against non malicious corruption its more than fine.

[MilesAhead]

Depending on the purpose, this has no relevant effect at all. E.G. file/data verification against non malicious corruption its more than fine.

+1. Many download sites still only list MD5 to verify the download completed properly.

[Bruttosozialprodukt]

@just me No, I ran the unicode version. But you're right, with the ansi version it works fine.
@Flipeador Thanks that totally did the job.


Well I'm not really concerned about security here. I just need this to do a web API call, so I'm gonna rely on HTTPS anyway.
For checking if a file downloaded properly this would be pretty big deal btw, as you would never get the same hash result as the verified website.

[JnLlnd]

Thanks for sharing the MD5() function. However, I'm having an issue it when the string to hash includes 8-bit (ASCII Extended) or UTF-8 Unicode chars. I'm running this script with AHK 1.1.30.03 64-bits.

Code: Select all

obj := Object()
; single arrays with a string to encode, and its expected MD5 hash
obj["ASCII 7-bit"] := ["-a-", "9ad117707c4c3bf4fa6f134bd3388806"]
obj["ASCII 8-bit"] := ["-" . Chr(224) . "-", "25719c185c8731f9d03fffa93cbb88fb"] ; "-à-"
obj["ASCII UTF-8"] := ["-" . Chr(257) . "-", "8ff009cf903b029ead4b37f2b530ba60"] ; "-ā-" (a with a small bar on its top)

for key, arr in obj
	str .= arr[1] . " (" . key . ") is " . (MD5(arr[1]) = arr[2] ? "GOOD" : "different") . "`n"
MsgBox, % SubStr(str, 1, -1) ; remove last `n
ExitApp

;------------------------------------------------------------
MD5(str, blnCase := false)
; by SKAN | rewritten by jNizM (https://www.autohotkey.com/boards/viewtopic.php?f=76&t=14927&p=75925&hilit=MD5sum#p75944)
;------------------------------------------------------------
{
	static MD5_DIGEST_LENGTH := 16
	hModule := DllCall("LoadLibrary", "Str", "advapi32.dll", "Ptr")
		, VarSetCapacity(MD5_CTX, 104, 0), DllCall("advapi32\MD5Init", "Ptr", &MD5_CTX)
		, DllCall("advapi32\MD5Update", "Ptr", &MD5_CTX, "AStr", str, "UInt", StrLen(str))
		, DllCall("advapi32\MD5Final", "Ptr", &MD5_CTX)
	loop % MD5_DIGEST_LENGTH
		o .= Format("{:02" (blnCase ? "X" : "x") "}", NumGet(MD5_CTX, 87 + A_Index, "UChar"))
	return o, DllCall("FreeLibrary", "Ptr", hModule)
}
;------------------------------------------------------------

The obj object includes arrays with 1) examples of strings and 2) the expected results based on the result from various online MD5 hash generators (one of them being http://www.md5.cz/). Only the first array with plain 7-bit ASCII produces the expected result.

I'm also using the following VBA function in Excel that produces the expected result for the three values. I need to be able to match the results in XL with those produced in AHK. I'm not enough fluent with bits-and-bytes to solve this alone. Any help would be appreciated.

Code: Select all

' from https://www.mrexcel.com/forum/excel-questions/973381-convert-string-md5-hash.html
Function StringToMD5HexUTF8(ByVal s As String) As String
    Dim enc As Object
    Dim bytes() As Byte
    Dim pos As Long
    Dim outstr As String
    
    Set enc = CreateObject("System.Security.Cryptography.MD5CryptoServiceProvider")
    
    bytes = UTF8Encoder(s)
    bytes = enc.ComputeHash_2(bytes)
    
    For pos = 0 To UBound(bytes)
       outstr = outstr & LCase(Right("0" & Hex(bytes(pos)), 2))
    Next pos
    
    StringToMD5HexUTF8 = outstr
    Set enc = Nothing
End Function

' encode a string as utf8, set reference to Microsoft ActiveX Data Objects 6.1 Library
Function UTF8Encoder(s As String) As Byte()
    Dim objStream As ADODB.Stream
    
    Set objStream = New ADODB.Stream
    objStream.Charset = "utf-8"
    objStream.Open
    objStream.WriteText s
    objStream.Flush
    objStream.Position = 0
    objStream.Type = adTypeBinary
    objStream.Position = 3 'no bom
    UTF8Encoder = objStream.Read
    objStream.Close
    Set objStream = Nothing
End Function

[jNizM]

Can you try this? https://www.autohotkey.com/boards/viewtopic.php?t=23413

[JnLlnd]

Can you try this? https://www.autohotkey.com/boards/viewtopic.php?t=23413

This works with the three values in my previous example. Thank you jNizM !