Hash from string, hex or file, HMAC, Salt

Post your working scripts, libraries and tools
User avatar
jNizM
Posts: 2595
Joined: 30 Sep 2013, 01:33
GitHub: jNizM
Contact:

Hash from string, hex or file, HMAC, Salt

30 Sep 2013, 08:13

[Funktion] Hash from string, hex or file by Bentschi & SKAN
Calculate cheksum from string / hex / file
Source:
- CRC32
- MD2
- MD4
- MD5
- SHA-1
- SHA-256
- SHA-384
- SHA-512
Screenshot:
Spoiler
=========================================================================================

[Funktion] HMAC by just me
Source:
- HMAC
Screenshot:
Spoiler
=========================================================================================

[Funktion] Secure Salted Hash by IsNull
Source:
- SecureSalted
Screenshot:
Spoiler
=========================================================================================

Example:
Gui with Hash Calculator: HashCalc

Others:
- SHA-3 (Keccak) Calculator

Contributing
- thanks to Bentschi, SKAN, just me, atnbueno & IsNull
- thanks to AutoHotkey Community
Last edited by jNizM on 16 May 2014, 01:58, edited 15 times in total.
[AHK] 1.1.30.03 x64 Unicode | [WIN] 10 Pro (Version 1909) x64 | [GitHub] Profile
Donations are appreciated if I could help you
User avatar
jNizM
Posts: 2595
Joined: 30 Sep 2013, 01:33
GitHub: jNizM
Contact:

Re: Hash from string or file, HMAC, Salt

04 Feb 2014, 06:10

MD4

Code: Select all

MsgBox % MD4("Hello World") "`n" MD4("Hello World", True)

MD4(string, case := False)    ; by SKAN | rewritten by jNizM
{
    static MD4_DIGEST_LENGTH := 16
    hModule := DllCall("LoadLibrary", "Str", "advapi32.dll", "Ptr")
    , VarSetCapacity(MD4_CTX, 104, 0), DllCall("advapi32\MD4Init", "Ptr", &MD4_CTX)
    , DllCall("advapi32\MD4Update", "Ptr", &MD4_CTX, "AStr", string, "UInt", StrLen(string))
    , DllCall("advapi32\MD4Final", "Ptr", &MD4_CTX)
    loop % MD4_DIGEST_LENGTH
        o .= Format("{:02" (case ? "X" : "x") "}", NumGet(MD4_CTX, 87 + A_Index, "UChar"))
    return o, DllCall("FreeLibrary", "Ptr", hModule)
}

MD5

Code: Select all

MsgBox % MD5("Hello World") "`n" MD5("Hello World", True)

MD5(string, case := False)    ; by SKAN | rewritten by jNizM
{
    static MD5_DIGEST_LENGTH := 16
    hModule := DllCall("LoadLibrary", "Str", "advapi32.dll", "Ptr")
    , VarSetCapacity(MD5_CTX, 104, 0), DllCall("advapi32\MD5Init", "Ptr", &MD5_CTX)
    , DllCall("advapi32\MD5Update", "Ptr", &MD5_CTX, "AStr", string, "UInt", StrLen(string))
    , DllCall("advapi32\MD5Final", "Ptr", &MD5_CTX)
    loop % MD5_DIGEST_LENGTH
        o .= Format("{:02" (case ? "X" : "x") "}", NumGet(MD5_CTX, 87 + A_Index, "UChar"))
    return o, DllCall("FreeLibrary", "Ptr", hModule)
}

SHA

Code: Select all

MsgBox % SHA("Hello World") "`n" SHA("Hello World", True)

SHA(string, case := False)    ; by SKAN | rewritten by jNizM
{
    static SHA_DIGEST_LENGTH := 20
    hModule := DllCall("LoadLibrary", "Str", "advapi32.dll", "Ptr")
    , VarSetCapacity(SHA_CTX, 136, 0), DllCall("advapi32\A_SHAInit", "Ptr", &SHA_CTX)
    , DllCall("advapi32\A_SHAUpdate", "Ptr", &SHA_CTX, "AStr", string, "UInt", StrLen(string))
    , DllCall("advapi32\A_SHAFinal", "Ptr", &SHA_CTX, "UInt", &SHA_CTX + 116)
    loop % SHA_DIGEST_LENGTH
        o .= Format("{:02" (case ? "X" : "x") "}", NumGet(SHA_CTX, 115 + A_Index, "UChar"))
    return o, DllCall("FreeLibrary", "Ptr", hModule)
}
[AHK] 1.1.30.03 x64 Unicode | [WIN] 10 Pro (Version 1909) x64 | [GitHub] Profile
Donations are appreciated if I could help you
User avatar
fischgeek
Posts: 432
Joined: 29 Jan 2014, 21:39

Re: Hash from string or file, HMAC, Salt

04 Feb 2014, 10:54

Awesome! Thank you!
User avatar
jigga
Posts: 93
Joined: 24 Jan 2014, 00:31
Facebook: joel.fazio
Google: +JoelFazio

Re: Hash from string or file, HMAC, Salt

04 Feb 2014, 11:06

Can you add the ability to get the sha1 hash of a hex string? As it works now, the SHA hash it returns for A10000009296F2 is 4cf522d69e6334472377eed5959dbfaa422005dc. The actual hash should be 2d0e6efea8f8082c9703cf6636c4a2195c75b7ed. I have been trying to add this functionality myself but its a bit over my head :(

The script works great for files and utf strings :)
User avatar
atnbueno
Posts: 85
Joined: 12 Oct 2013, 04:45

Re: Hash from string or file, HMAC, Salt

04 Feb 2014, 15:43

Very nice work, jNizM.

One question though: what's the purpose of the byref hash = 0, byref hashlength = 0 parameters? Aren't they overwritten when advapi32\CryptSetHashParam is used?

jigga, one way of hashing hex strings would be to modify each hash function like this:

Code: Select all

SHA(string, encoding = "UTF-8")
{
    If (encoding != "HEX")
        Return CalcStringHash(string, 0x8004, encoding)
    Else
        Return CalcHexHash(string, 0x8004)
}
where an "HEX" encoding would switch from the UTF8-aware CalcStringHash to this:

Code: Select all

CalcHexHash(hexstring, algid)
{
    length := StrLen(hexstring) // 2
    VarSetCapacity(data, length)
    Loop % length
        NumPut("0x" SubStr(hexstring, 2 * A_Index - 1, 2), data, A_Index - 1, "Char")
    Return CalcAddrHash(&data, length, algid)
}
Regards,
Antonio
User avatar
jigga
Posts: 93
Joined: 24 Jan 2014, 00:31
Facebook: joel.fazio
Google: +JoelFazio

Re: Hash from string or file, HMAC, Salt

04 Feb 2014, 16:28

Wow that was an easy fix, thanks! I spent hours learning about every aspect related to hashes and bitwise operations etc. My brain melted a bit.
User avatar
jNizM
Posts: 2595
Joined: 30 Sep 2013, 01:33
GitHub: jNizM
Contact:

Re: Hash from string or file, HMAC, Salt

05 Feb 2014, 06:08

thx atnbueno
CalcHexHash() is added in top post and in my example
[AHK] 1.1.30.03 x64 Unicode | [WIN] 10 Pro (Version 1909) x64 | [GitHub] Profile
Donations are appreciated if I could help you
nahusa
Posts: 6
Joined: 26 Feb 2014, 05:45

Re: Hash from string, hex or file, HMAC, Salt

26 Feb 2014, 05:50

Would you consider adding CRC32 support?
User avatar
jNizM
Posts: 2595
Joined: 30 Sep 2013, 01:33
GitHub: jNizM
Contact:

Re: Hash from string, hex or file, HMAC, Salt

26 Feb 2014, 12:32

in the next few days
[AHK] 1.1.30.03 x64 Unicode | [WIN] 10 Pro (Version 1909) x64 | [GitHub] Profile
Donations are appreciated if I could help you
User avatar
jNizM
Posts: 2595
Joined: 30 Sep 2013, 01:33
GitHub: jNizM
Contact:

Re: Hash from string, hex or file, HMAC, Salt

27 Feb 2014, 06:49

Update:
CRC32 from String, Hex & File
[AHK] 1.1.30.03 x64 Unicode | [WIN] 10 Pro (Version 1909) x64 | [GitHub] Profile
Donations are appreciated if I could help you
nahusa
Posts: 6
Joined: 26 Feb 2014, 05:45

Re: Hash from string, hex or file, HMAC, Salt

27 Feb 2014, 17:05

Fantastic. Thank you very much.
nahusa
Posts: 6
Joined: 26 Feb 2014, 05:45

Re: Hash from string, hex or file, HMAC, Salt

25 Apr 2014, 23:57

Hi jNizM. I've been doing some tests with your libs using large files (1GB, 2GB, 3GB, 4GB, 5GB, 8GB, 10GB, & 15GB) and I'm running into some issues. Here are my findings:

CRC32: Everything works perfectly. No problems at all with any of the test files.
MD4/5 & SHA-1/256/384/512: The 1GB file hashes fine. 2GB and above cause the following error: Error: Out of memory. VarSetCapacity(data, hashlength + readlength, 0)

If you have time, is this something you could take a look at? If you need anything further, please ask. Thank you.
User avatar
atnbueno
Posts: 85
Joined: 12 Oct 2013, 04:45

Re: Hash from string, hex or file, HMAC, Salt

26 Apr 2014, 07:49

nahusa, try it with the 64-bit version of AutoHotkey (AutoHotkeyU64.exe) and it should work.

Whatever the problem is it's probably related to the fact that 32-bit processes have a 2GB RAM limit.


Regards,
Antonio
just me
Posts: 7022
Joined: 02 Oct 2013, 08:51
Location: Germany

Re: Hash from string, hex or file, HMAC, Salt

26 Apr 2014, 10:40

Hmmm, I'm not that sure. As far as I understand the MSDN, you have to add data to an existing hash with consecutive calls of CryptHashData(), so memory shouldn't be a problem. But that's not what CalcFileHash() actually does:

Code: Select all

    hashlength := 0
    while (f.pos < f.length)
    {
        readlength := (f.length - fpos > continue) ? continue : f.length - f.pos
        VarSetCapacity(data, hashlength + readlength, 0) ; <<<<<<<<<<<<<<<<<<<<<
        DllCall("RtlMoveMemory", "Ptr", &data, "Ptr", &hash, "Ptr", hashlength)
        f.rawRead(&data + hashlength, readlength)
        h := CalcAddrHash(&data, hashlength + readlength, algid, hash, hashlength)
    }
User avatar
atnbueno
Posts: 85
Joined: 12 Oct 2013, 04:45

Re: Hash from string, hex or file, HMAC, Salt

26 Apr 2014, 11:40

I'm sorry. I should have been more specific. I tried this code:

Code: Select all

file := "C:\Users\atnbueno\Copy\Microsoft\SW_DVD5_SA_Win_Ent_8.1_64BIT_Spanish_MLF_X18-96760.ISO" ; 3,897,587,712 bytes long file
MsgBox, % "File:`n" (file) "`n`nMD5:`n" FileMD5(file)
With AutoHotkeyU32.exe it gives an "Out of memory" error (because readlength is equal to 3897587712). With AutoHotkeyU64.exe it works as expected (after using almost 4 GiB of RAM).
just me
Posts: 7022
Joined: 02 Oct 2013, 08:51
Location: Germany

Re: Hash from string, hex or file, HMAC, Salt

27 Apr 2014, 02:06

Interesting, FileMD5()

Code: Select all

FileMD5(filename)
{
    return CalcFileHash(filename, 0x8003, 64 * 1024)
}
is setting the parameter continue of CalcFileHash() to 64 * 1024 when called this way, so readlength should never exceed this value.

But there's a little bug here:
readlength := (f.length - fpos > continue) ? continue : f.length - f.pos

Without this bug I assume that the hash wouldn't be valid when calculated as is.
User avatar
atnbueno
Posts: 85
Joined: 12 Oct 2013, 04:45

Re: Hash from string, hex or file, HMAC, Salt

27 Apr 2014, 09:46

Indeed. Changing fpos by an f.pos makes the script work without errors with AutoHotkeyU32.exe and it never uses more than 2MiB of RAM. The line fpos := "" is also unnecessary (it was probably in answer to #Warn complaining about an uninitialized variable).

Unfortunately now the output is not the correct hash (I've tested it with my code above and MD5). It worked before because the fpos bug prevented the script from going inside the while (f.pos < f.length) loop.

I'm not sure how CalcAddrHash works but I suspect the problem is in successive calls to it. I've tested changing continue from 64 KiB to 128 KiB and the partial hashes in h don't match when f.pos is the same (every two iterations).
just me
Posts: 7022
Joined: 02 Oct 2013, 08:51
Location: Germany

Re: Hash from string, hex or file, HMAC, Salt

27 Apr 2014, 11:25

Maybe something like this will do it:

Code: Select all

#NoEnv
MD51 := CalcFileHash(A_ScriptFullPath, 0x8003)
MD52 := CalcFileHash(A_ScriptFullPath, 0x8003, 80)
MsgBox, % MD51 . "`n" . MD52
ExitApp
; CalcFileHash ======================================================================
CalcFileHash(filename, algid, continue = 0, byref hash = 0, byref hashlength = 0)
{
    if (!(f := FileOpen(filename, "r")))
    {
        return
    }
    f.pos := 0
    if (!continue && f.length > 0x7fffffff)
    {
        return
    }
    if (!continue)
    {
        f.rawRead(data, f.length)
        h := CalcAddrHash(&data, f.length, algid, hash, hashlength)
    }
    else
        h := CalcHashFromFile(f, continue, algid, hash, hashlength)
    f.close()
    return h
}
; CalcHashFromFile ======================================================================
CalcHashFromFile(hFile, length, algid, byref hash = 0, byref hashlength = 0)
{
    static h := [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, "a", "b", "c", "d", "e", "f"]
    static b := h.minIndex()
    hProv := hHash := o := ""
    if (DllCall("advapi32\CryptAcquireContext", "Ptr*", hProv, "Ptr", 0, "Ptr", 0, "UInt", 24, "UInt", 0xf0000000))
    {
        if (DllCall("advapi32\CryptCreateHash", "Ptr", hProv, "UInt", algid, "UInt", 0, "UInt", 0, "Ptr*", hHash))
        {
            While !hFile.AtEOF && (DataRead := hFile.rawRead(Data, length))
                DllCall("advapi32\CryptHashData", "Ptr", hHash, "Ptr", &Data, "UInt", DataRead, "UInt", 0)
            if (DllCall("advapi32\CryptGetHashParam", "Ptr", hHash, "UInt", 2, "Ptr", 0, "UInt*", hashlength, "UInt", 0))
            {
                VarSetCapacity(hash, hashlength, 0)
                if (DllCall("advapi32\CryptGetHashParam", "Ptr", hHash, "UInt", 2, "Ptr", &hash, "UInt*", hashlength, "UInt", 0))
                {
                    loop % hashlength
                    {
                        v := NumGet(hash, A_Index - 1, "UChar")
                        o .= h[(v >> 4) + b] h[(v & 0xf) + b]
                    }
                }
            }
            DllCall("advapi32\CryptDestroyHash", "Ptr", hHash)
        }
        DllCall("advapi32\CryptReleaseContext", "Ptr", hProv, "UInt", 0)
    }
    return o
}
; CalcAddrHash ======================================================================
CalcAddrHash(addr, length, algid, byref hash = 0, byref hashlength = 0)
{
    static h := [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, "a", "b", "c", "d", "e", "f"]
    static b := h.minIndex()
    hProv := hHash := o := ""
    if (DllCall("advapi32\CryptAcquireContext", "Ptr*", hProv, "Ptr", 0, "Ptr", 0, "UInt", 24, "UInt", 0xf0000000))
    {
        if (DllCall("advapi32\CryptCreateHash", "Ptr", hProv, "UInt", algid, "UInt", 0, "UInt", 0, "Ptr*", hHash))
        {
            if (DllCall("advapi32\CryptHashData", "Ptr", hHash, "Ptr", addr, "UInt", length, "UInt", 0))
            {
                if (DllCall("advapi32\CryptGetHashParam", "Ptr", hHash, "UInt", 2, "Ptr", 0, "UInt*", hashlength, "UInt", 0))
                {
                    VarSetCapacity(hash, hashlength, 0)
                    if (DllCall("advapi32\CryptGetHashParam", "Ptr", hHash, "UInt", 2, "Ptr", &hash, "UInt*", hashlength, "UInt", 0))
                    {
                        loop % hashlength
                        {
                            v := NumGet(hash, A_Index - 1, "UChar")
                            o .= h[(v >> 4) + b] h[(v & 0xf) + b]
                        }
                    }
                }
            }
            DllCall("advapi32\CryptDestroyHash", "Ptr", hHash)
        }
        DllCall("advapi32\CryptReleaseContext", "Ptr", hProv, "UInt", 0)
    }
    return o
}
nahusa
Posts: 6
Joined: 26 Feb 2014, 05:45

Re: Hash from string, hex or file, HMAC, Salt

28 Apr 2014, 12:48

just me wrote:Maybe something like this will do it
Thank you for your efforts and replies. Your code does work and it calculates the proper hash. I tested it using a 10 GB file and it only consumed a minimal amount of memory. However, it's very very slow compared to some other methods/apps. That's not a knock on you, just an honest observation.
Last edited by nahusa on 28 Apr 2014, 14:19, edited 1 time in total.
User avatar
atnbueno
Posts: 85
Joined: 12 Oct 2013, 04:45

Re: Hash from string, hex or file, HMAC, Salt

28 Apr 2014, 14:19

nahusa, that's testing code. If you look at it carefully you'll see it's reading the file 80 bytes at a time. Change it to something larger (I'm guessing something in the order of 256 * 1024) to make it faster. For comparison, jNizM's code works with 64 * 1024 chunks.

And just me, good job 8-)

Return to “Scripts and Functions”

Who is online

Users browsing this forum: Frosti, MTRNord, sazd and 50 guests