Hash from string, hex or file, HMAC, Salt

[jNizM]

[Funktion] Hash from string, hex or file by Bentschi & SKAN
Calculate cheksum from string / hex / file
Source:
- CRC32
- MD2
- MD4
- MD5
- SHA-1
- SHA-256
- SHA-384
- SHA-512
Screenshot:

Spoiler

=========================================================================================

[Funktion] HMAC by just me
Source:
- HMAC
Screenshot:

Spoiler

=========================================================================================

[Funktion] Secure Salted Hash by IsNull
Source:
- SecureSalted
Screenshot:

Spoiler

=========================================================================================

Example:
Gui with Hash Calculator: HashCalc

Others:
- SHA-3 (Keccak) Calculator

Contributing
- thanks to Bentschi, SKAN, just me, atnbueno & IsNull
- thanks to AutoHotkey Community

[jNizM]

MD4

Code: Select all

MsgBox % MD4("Hello World") "`n" MD4("Hello World", True)

MD4(string, case := False)    ; by SKAN | rewritten by jNizM
{
    static MD4_DIGEST_LENGTH := 16
    hModule := DllCall("LoadLibrary", "Str", "advapi32.dll", "Ptr")
    , VarSetCapacity(MD4_CTX, 104, 0), DllCall("advapi32\MD4Init", "Ptr", &MD4_CTX)
    , DllCall("advapi32\MD4Update", "Ptr", &MD4_CTX, "AStr", string, "UInt", StrLen(string))
    , DllCall("advapi32\MD4Final", "Ptr", &MD4_CTX)
    loop % MD4_DIGEST_LENGTH
        o .= Format("{:02" (case ? "X" : "x") "}", NumGet(MD4_CTX, 87 + A_Index, "UChar"))
    return o, DllCall("FreeLibrary", "Ptr", hModule)
}

---

MD5

Code: Select all

MsgBox % MD5("Hello World") "`n" MD5("Hello World", True)

MD5(string, case := False)    ; by SKAN | rewritten by jNizM
{
    static MD5_DIGEST_LENGTH := 16
    hModule := DllCall("LoadLibrary", "Str", "advapi32.dll", "Ptr")
    , VarSetCapacity(MD5_CTX, 104, 0), DllCall("advapi32\MD5Init", "Ptr", &MD5_CTX)
    , DllCall("advapi32\MD5Update", "Ptr", &MD5_CTX, "AStr", string, "UInt", StrLen(string))
    , DllCall("advapi32\MD5Final", "Ptr", &MD5_CTX)
    loop % MD5_DIGEST_LENGTH
        o .= Format("{:02" (case ? "X" : "x") "}", NumGet(MD5_CTX, 87 + A_Index, "UChar"))
    return o, DllCall("FreeLibrary", "Ptr", hModule)
}

---

SHA

Code: Select all

MsgBox % SHA("Hello World") "`n" SHA("Hello World", True)

SHA(string, case := False)    ; by SKAN | rewritten by jNizM
{
    static SHA_DIGEST_LENGTH := 20
    hModule := DllCall("LoadLibrary", "Str", "advapi32.dll", "Ptr")
    , VarSetCapacity(SHA_CTX, 136, 0), DllCall("advapi32\A_SHAInit", "Ptr", &SHA_CTX)
    , DllCall("advapi32\A_SHAUpdate", "Ptr", &SHA_CTX, "AStr", string, "UInt", StrLen(string))
    , DllCall("advapi32\A_SHAFinal", "Ptr", &SHA_CTX, "UInt", &SHA_CTX + 116)
    loop % SHA_DIGEST_LENGTH
        o .= Format("{:02" (case ? "X" : "x") "}", NumGet(SHA_CTX, 115 + A_Index, "UChar"))
    return o, DllCall("FreeLibrary", "Ptr", hModule)
}

[fischgeek]

Awesome! Thank you!

[jigga]

Can you add the ability to get the sha1 hash of a hex string? As it works now, the SHA hash it returns for A10000009296F2 is 4cf522d69e6334472377eed5959dbfaa422005dc. The actual hash should be 2d0e6efea8f8082c9703cf6636c4a2195c75b7ed. I have been trying to add this functionality myself but its a bit over my head

The script works great for files and utf strings

[atnbueno]

Very nice work, jNizM.

One question though: what's the purpose of the byref hash = 0, byref hashlength = 0 parameters? Aren't they overwritten when advapi32\CryptSetHashParam is used?

jigga, one way of hashing hex strings would be to modify each hash function like this:

Code: Select all

SHA(string, encoding = "UTF-8")
{
    If (encoding != "HEX")
        Return CalcStringHash(string, 0x8004, encoding)
    Else
        Return CalcHexHash(string, 0x8004)
}

where an "HEX" encoding would switch from the UTF8-aware CalcStringHash to this:

Code: Select all

CalcHexHash(hexstring, algid)
{
    length := StrLen(hexstring) // 2
    VarSetCapacity(data, length)
    Loop % length
        NumPut("0x" SubStr(hexstring, 2 * A_Index - 1, 2), data, A_Index - 1, "Char")
    Return CalcAddrHash(&data, length, algid)
}

Regards,
Antonio

[jigga]

Wow that was an easy fix, thanks! I spent hours learning about every aspect related to hashes and bitwise operations etc. My brain melted a bit.

[jNizM]

thx atnbueno
CalcHexHash() is added in top post and in my example

[nahusa]

Would you consider adding CRC32 support?

[jNizM]

in the next few days

[jNizM]

Update:
CRC32 from String, Hex & File

[nahusa]

Fantastic. Thank you very much.

[nahusa]

Hi jNizM. I've been doing some tests with your libs using large files (1GB, 2GB, 3GB, 4GB, 5GB, 8GB, 10GB, & 15GB) and I'm running into some issues. Here are my findings:

CRC32: Everything works perfectly. No problems at all with any of the test files.
MD4/5 & SHA-1/256/384/512: The 1GB file hashes fine. 2GB and above cause the following error: Error: Out of memory. VarSetCapacity(data, hashlength + readlength, 0)

If you have time, is this something you could take a look at? If you need anything further, please ask. Thank you.

[atnbueno]

nahusa, try it with the 64-bit version of AutoHotkey (AutoHotkeyU64.exe) and it should work.

Whatever the problem is it's probably related to the fact that 32-bit processes have a 2GB RAM limit.


Regards,
Antonio

[just me]

Hmmm, I'm not that sure. As far as I understand the MSDN, you have to add data to an existing hash with consecutive calls of CryptHashData(), so memory shouldn't be a problem. But that's not what CalcFileHash() actually does:

Code: Select all

    hashlength := 0
    while (f.pos < f.length)
    {
        readlength := (f.length - fpos > continue) ? continue : f.length - f.pos
        VarSetCapacity(data, hashlength + readlength, 0) ; <<<<<<<<<<<<<<<<<<<<<
        DllCall("RtlMoveMemory", "Ptr", &data, "Ptr", &hash, "Ptr", hashlength)
        f.rawRead(&data + hashlength, readlength)
        h := CalcAddrHash(&data, hashlength + readlength, algid, hash, hashlength)
    }

[atnbueno]

I'm sorry. I should have been more specific. I tried this code:

Code: Select all

file := "C:\Users\atnbueno\Copy\Microsoft\SW_DVD5_SA_Win_Ent_8.1_64BIT_Spanish_MLF_X18-96760.ISO" ; 3,897,587,712 bytes long file
MsgBox, % "File:`n" (file) "`n`nMD5:`n" FileMD5(file)

With AutoHotkeyU32.exe it gives an "Out of memory" error (because readlength is equal to 3897587712). With AutoHotkeyU64.exe it works as expected (after using almost 4 GiB of RAM).

[just me]

Interesting, FileMD5()

Code: Select all

FileMD5(filename)
{
    return CalcFileHash(filename, 0x8003, 64 * 1024)
}

is setting the parameter continue of CalcFileHash() to 64 * 1024 when called this way, so readlength should never exceed this value.

But there's a little bug here:
readlength := (f.length - fpos > continue) ? continue : f.length - f.pos

Without this bug I assume that the hash wouldn't be valid when calculated as is.

[atnbueno]

Indeed. Changing fpos by an f.pos makes the script work without errors with AutoHotkeyU32.exe and it never uses more than 2MiB of RAM. The line fpos := "" is also unnecessary (it was probably in answer to #Warn complaining about an uninitialized variable).

Unfortunately now the output is not the correct hash (I've tested it with my code above and MD5). It worked before because the fpos bug prevented the script from going inside the while (f.pos < f.length) loop.

I'm not sure how CalcAddrHash works but I suspect the problem is in successive calls to it. I've tested changing continue from 64 KiB to 128 KiB and the partial hashes in h don't match when f.pos is the same (every two iterations).

[just me]

Maybe something like this will do it:

Code: Select all

#NoEnv
MD51 := CalcFileHash(A_ScriptFullPath, 0x8003)
MD52 := CalcFileHash(A_ScriptFullPath, 0x8003, 80)
MsgBox, % MD51 . "`n" . MD52
ExitApp
; CalcFileHash ======================================================================
CalcFileHash(filename, algid, continue = 0, byref hash = 0, byref hashlength = 0)
{
    if (!(f := FileOpen(filename, "r")))
    {
        return
    }
    f.pos := 0
    if (!continue && f.length > 0x7fffffff)
    {
        return
    }
    if (!continue)
    {
        f.rawRead(data, f.length)
        h := CalcAddrHash(&data, f.length, algid, hash, hashlength)
    }
    else
        h := CalcHashFromFile(f, continue, algid, hash, hashlength)
    f.close()
    return h
}
; CalcHashFromFile ======================================================================
CalcHashFromFile(hFile, length, algid, byref hash = 0, byref hashlength = 0)
{
    static h := [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, "a", "b", "c", "d", "e", "f"]
    static b := h.minIndex()
    hProv := hHash := o := ""
    if (DllCall("advapi32\CryptAcquireContext", "Ptr*", hProv, "Ptr", 0, "Ptr", 0, "UInt", 24, "UInt", 0xf0000000))
    {
        if (DllCall("advapi32\CryptCreateHash", "Ptr", hProv, "UInt", algid, "UInt", 0, "UInt", 0, "Ptr*", hHash))
        {
            While !hFile.AtEOF && (DataRead := hFile.rawRead(Data, length))
                DllCall("advapi32\CryptHashData", "Ptr", hHash, "Ptr", &Data, "UInt", DataRead, "UInt", 0)
            if (DllCall("advapi32\CryptGetHashParam", "Ptr", hHash, "UInt", 2, "Ptr", 0, "UInt*", hashlength, "UInt", 0))
            {
                VarSetCapacity(hash, hashlength, 0)
                if (DllCall("advapi32\CryptGetHashParam", "Ptr", hHash, "UInt", 2, "Ptr", &hash, "UInt*", hashlength, "UInt", 0))
                {
                    loop % hashlength
                    {
                        v := NumGet(hash, A_Index - 1, "UChar")
                        o .= h[(v >> 4) + b] h[(v & 0xf) + b]
                    }
                }
            }
            DllCall("advapi32\CryptDestroyHash", "Ptr", hHash)
        }
        DllCall("advapi32\CryptReleaseContext", "Ptr", hProv, "UInt", 0)
    }
    return o
}
; CalcAddrHash ======================================================================
CalcAddrHash(addr, length, algid, byref hash = 0, byref hashlength = 0)
{
    static h := [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, "a", "b", "c", "d", "e", "f"]
    static b := h.minIndex()
    hProv := hHash := o := ""
    if (DllCall("advapi32\CryptAcquireContext", "Ptr*", hProv, "Ptr", 0, "Ptr", 0, "UInt", 24, "UInt", 0xf0000000))
    {
        if (DllCall("advapi32\CryptCreateHash", "Ptr", hProv, "UInt", algid, "UInt", 0, "UInt", 0, "Ptr*", hHash))
        {
            if (DllCall("advapi32\CryptHashData", "Ptr", hHash, "Ptr", addr, "UInt", length, "UInt", 0))
            {
                if (DllCall("advapi32\CryptGetHashParam", "Ptr", hHash, "UInt", 2, "Ptr", 0, "UInt*", hashlength, "UInt", 0))
                {
                    VarSetCapacity(hash, hashlength, 0)
                    if (DllCall("advapi32\CryptGetHashParam", "Ptr", hHash, "UInt", 2, "Ptr", &hash, "UInt*", hashlength, "UInt", 0))
                    {
                        loop % hashlength
                        {
                            v := NumGet(hash, A_Index - 1, "UChar")
                            o .= h[(v >> 4) + b] h[(v & 0xf) + b]
                        }
                    }
                }
            }
            DllCall("advapi32\CryptDestroyHash", "Ptr", hHash)
        }
        DllCall("advapi32\CryptReleaseContext", "Ptr", hProv, "UInt", 0)
    }
    return o
}

[nahusa]

Maybe something like this will do it

Thank you for your efforts and replies. Your code does work and it calculates the proper hash. I tested it using a 10 GB file and it only consumed a minimal amount of memory. However, it's very very slow compared to some other methods/apps. That's not a knock on you, just an honest observation.

[atnbueno]

nahusa, that's testing code. If you look at it carefully you'll see it's reading the file 80 bytes at a time. Change it to something larger (I'm guessing something in the order of 256 * 1024) to make it faster. For comparison, jNizM's code works with 64 * 1024 chunks.

And just me, good job