 |
AutoHotkey Community Let's help each other out
|
| View previous topic :: View next topic |
| Author |
Message |
SKAN
Joined: 26 Dec 2005 Posts: 8688
|
Posted: Thu May 31, 2007 4:16 pm Post subject: |
|
|
@Laszlo
Dear Sir,
I am experimenting with your Hash().. and full of surprises
I did a test: Hash() Vs Hashes.DLL
File: MPG
Size: 76MB
System: 1.4Ghz Sempron 256 MB Ram
AHK: 1.0.46.16 ( Latest )
OS: Win2k SP4
Result: I am happy to discontinue from using Hashes.DLL
For Hashes.DLL, I timed only the DllCall()
Hashes.DLL
MD5: 2580ms
SHA1: 4100ms
For Hash(). I timed for FileRead + FileGetSize + Hash() call
Hash()
MD5: 2050ms
SHA1: 2420
However I need some clarifications:
1) Can you explain why Hashes.DLL take phenomenally more time for SHA1 ?
2) Would we gain significant performance if Hash() is built-in as a function ?
PS: Hashes_DLL.ZIP (45KB) |
|
| Back to top |
|
 |
Laszlo
Joined: 14 Feb 2005 Posts: 4710 Location: Boulder, CO
|
Posted: Thu May 31, 2007 5:13 pm Post subject: |
|
|
1) Probably bad code. The work is only 25% more for SHA-1 than for MD5, so anything slower is not justified.
2) The dllcall overhead is just a couple of ms (except loading it in the first place), so no much speedup is expected with built in code. |
|
| Back to top |
|
 |
SKAN
Joined: 26 Dec 2005 Posts: 8688
|
Posted: Thu May 31, 2007 7:50 pm Post subject: |
|
|
Thanks for the clarification!.
| Laszlo wrote: | | 2) The dllcall overhead is just a couple of ms (except loading it in the first place), so no much speedup is expected with built in code. |
Oh! I was thinking about wishing this in Wish List.
I will be content if this is available in the forthcoming standard include.
I thank Sean and you for the wonderful code.
Regards,  |
|
| Back to top |
|
 |
SKAN
Joined: 26 Dec 2005 Posts: 8688
|
Posted: Mon Jun 18, 2007 10:09 pm Post subject: |
|
|
How can a compiled script hash self in order to determine its integrity?  |
|
| Back to top |
|
 |
engunneer
Joined: 30 Aug 2005 Posts: 8255 Location: Maywood, IL
|
Posted: Mon Jun 18, 2007 10:42 pm Post subject: |
|
|
not sure how to do that skan, but it reminded me of this:
Previous IOCCC winners:
omoikane - 2004 in particular
http://www.ioccc.org/2004/omoikane.hint
I think you have to store the hash in a separate file, or unpack the UPX, and mess with it. _________________
(Common Answers) |
|
| Back to top |
|
 |
SKAN
Joined: 26 Dec 2005 Posts: 8688
|
Posted: Mon Jun 18, 2007 10:48 pm Post subject: |
|
|
| engunneer wrote: | | I think you have to store the hash in a separate file, or unpack the UPX, and mess with it. |
I had guessed so, but would like to confirm it from Laszlo.  |
|
| Back to top |
|
 |
Laszlo
Joined: 14 Feb 2005 Posts: 4710 Location: Boulder, CO
|
Posted: Tue Jun 19, 2007 5:27 am Post subject: |
|
|
What do you need this MD5 signature checking for?
- If it is detecting random memory-, disk- or transfer errors of the program file, you can store the signature separately (maybe appended to the file). This is not very secure, because many random errors could modify the verification code, such that the program would believe it passed the test, which has not even been performed. Because of these and other weaknesses, a CRC is almost as good.
- If you have a trusted (intact) piece of SW, you are better off with some error correction code attached to the file.
- If it is intended as protection from malicious alterations, you would need trusted SW with the signature also coming from a trusted source (or use public keys). This cannot be done by the program to be checked, because rogue software could also lie about the result of the signature verification.
Consequently, self verification is not very good with MD5. |
|
| Back to top |
|
 |
SKAN
Joined: 26 Dec 2005 Posts: 8688
|
Posted: Tue Jun 19, 2007 6:32 am Post subject: |
|
|
Thanks for the valuable info.  |
|
| Back to top |
|
 |
tic
Joined: 22 Apr 2007 Posts: 1786
|
Posted: Mon Jun 25, 2007 6:31 pm Post subject: |
|
|
ok ive tried this hash() method and the hash dll, but then also tried this console based program. it can calculate the md5 and many other forms of cryptography in no time
http://www.dominik-reichl.de/opensource.shtml#rehash
I use at the terminal:
rehash.exe -none -md5 filename
and it managed to output the md5 of a 700mb avi in under 10 secs and it was the same as the one found by hash()
I used:
| Code: | FileGetSize, DataLength, filename
FileRead , BinaryData, filename
MD5Hash := HASH( BinaryData, DataLength , 3 )
MsgBox, %MD5Hash%
HASH(ByRef sData, nLen, SID = 3) { ; SID = 3: MD5, 4: SHA1
; Laszlo: http://www.autohotkey.com/forum/viewtopic.php?p=113252#113252
DllCall("advapi32\CryptAcquireContextA", UIntP,hProv, UInt,0, UInt,0, UInt,1, UInt,0xF0000000)
DllCall("advapi32\CryptCreateHash", UInt,hProv, UInt,0x8000|0|SID, UInt,0, UInt,0, UIntP, hHash)
DllCall("advapi32\CryptHashData", UInt,hHash, UInt,&sData, UInt,nLen, UInt,0)
DllCall("advapi32\CryptGetHashParam", UInt,hHash, UInt,2, UInt,0, UIntP,nSize, UInt,0)
VarSetCapacity(HashVal, nSize, 0)
DllCall("advapi32\CryptGetHashParam", UInt,hHash, UInt,2, UInt,&HashVal, UIntP,nSize, UInt,0)
DllCall("advapi32\CryptDestroyHash", UInt,hHash)
DllCall("advapi32\CryptReleaseContext", UInt,hProv, UInt,0)
IFormat := A_FormatInteger
SetFormat Integer, H
Loop %nSize%
sHash .= SubStr(*(&HashVal+A_Index-1)+0x100,-1)
SetFormat Integer, %IFormat%
Return sHash
} |
and it uses the amount of ram equal to the size of the file for obvious reasons, but using the program rehash it only used a couple mb and cpu use was under 5% and still worked it out in seconds! a fraction of the time!
the source code is available for rehash, would anyone mind taking a look at it that is adept at cryptography and C++ and mind finding out how its working and perhaps create a dll from it that could be used in ahk?? |
|
| Back to top |
|
 |
computerspazzz
Joined: 25 May 2010 Posts: 22
|
Posted: Thu Jun 03, 2010 7:45 am Post subject: Is this old and no longer working? |
|
|
I am running windows and I tried it and it gives wrong MD5's and sometimes just gives all the same code no matter what. If anyone care to look at my code its below:
| Code: |
FileRead , BinaryData, C:\test\2.jpg
DataLength := StrLen(BinaryData)
;FileGetSize, DataLength, C:\test\2.jpg ; Also Tried This instead of the line above
MD5Hash := HASH( BinaryData, DataLength , 3 )
MsgBox, %MD5Hash%
HASH(ByRef sData, nLen, SID = 3) { ; SID = 3: MD5, 4: SHA1
; Laszlo: http://www.autohotkey.com/forum/viewtopic.php?p=113252#113252
DllCall("advapi32\CryptAcquireContextA", UIntP,hProv, UInt,0, UInt,0, UInt,1, UInt,0xF0000000)
DllCall("advapi32\CryptCreateHash", UInt,hProv, UInt,0x8000|0|SID, UInt,0, UInt,0, UIntP, hHash)
DllCall("advapi32\CryptHashData", UInt,hHash, UInt,&sData, UInt,nLen, UInt,0)
DllCall("advapi32\CryptGetHashParam", UInt,hHash, UInt,2, UInt,0, UIntP,nSize, UInt,0)
VarSetCapacity(HashVal, nSize, 0)
DllCall("advapi32\CryptGetHashParam", UInt,hHash, UInt,2, UInt,&HashVal, UIntP,nSize, UInt,0)
DllCall("advapi32\CryptDestroyHash", UInt,hHash)
DllCall("advapi32\CryptReleaseContext", UInt,hProv, UInt,0)
IFormat := A_FormatInteger
SetFormat Integer, H
Loop %nSize%
sHash .= SubStr(*(&HashVal+A_Index-1)+0x100,-1)
SetFormat Integer, %IFormat%
Return sHash
}
|
I am looking to build a patcher program that will get a manifest from a ftp server with a file list and MD5's listed, check the files it already has against the manifest for any MD5 discrepancies and then download the files that don't match up. I have everything else, just not the MD5 (or any other good solid method will work). Would appreciate a point in the right direction if anyone happens upon this that knows. Spent the last 4 hours searching the forums and nothing I found works. _________________ Computerspazzz The Technowizard
The Wizard Is In! |
|
| Back to top |
|
 |
SKAN
Joined: 26 Dec 2005 Posts: 8688
|
Posted: Thu Jun 03, 2010 8:26 am Post subject: |
|
|
| Code: | FileRead , BinaryData, C:\test\2.jpg
; DataLength := StrLen(BinaryData)
;FileGetSize, DataLength, C:\test\2.jpg ; Also Tried This instead of the line above
MD5Hash := HASH( BinaryData, DataLength , 3 )
MsgBox, %MD5Hash% |
StrLen() would not work correctly with Binary Data
Use FileGetSize command like:
| Code: | | FileGetSize, DataLength, C:\test\2.jpg |
On a related note, you could try FileMD5() |
|
| Back to top |
|
 |
Laszlo
Joined: 14 Feb 2005 Posts: 4710 Location: Boulder, CO
|
Posted: Thu Jun 03, 2010 9:06 am Post subject: |
|
|
When you read a file into a variable with no already assigned value or a small one, the capacity of the variable is set to the size of the file, so you can also use | Code: | | DataLength := VarSetCapacity(BinaryData) | The hash function above otherwise works for me.
You can even combine SKAN's MD5 function with the hex conversion function of the crypto API, to get shorter, faster code, which gets the length, if not given: | Code: | MD5(ByRef V, L=0) { ; Compute MD5 on binary data in V of length L [all allocated] bytes
VarSetCapacity(M,104) ; Use the MD5 Windows functions directly
DllCall("advapi32\MD5Init", Str,M)
DllCall("advapi32\MD5Update",Str,M, Str,V, UInt,L ? L : VarSetCapacity(V))
DllCall("advapi32\MD5Final", Str,M)
VarSetCapacity(T,L:=64) ; Convert to hex with the crypto API
DllCall("Crypt32.dll\CryptBinaryToStringA", UInt,&M+88, UInt,16, UInt,4, Str,T, UIntP,L, "CDECL UInt")
Return RegExReplace(T,"\W") ; Remove white space
} |
|
|
| Back to top |
|
 |
computerspazzz
Joined: 25 May 2010 Posts: 22
|
Posted: Sun Jun 06, 2010 9:05 pm Post subject: FileMD5() Folder or File Hasher |
|
|
Thanks for the replies. Yeah i was messing around with the FileGetSize and DataLength and tried both. I also tried the FileMD5()... and the two scripts agree with one another! So i Had previously downloaded a MD5 checker to compare the results against... IT WAS WRONG! I didn't think to flip it around and download several programs as a test, another thing Ive learned. The FileMD5() is very neatly setup and easy to use as an #Include so going to stick with that, but this script also worked once i modified as you said. Thank you all very much.
On another note, I wrote a loop to hash a folder and output the data, its 100% ready to run as is, but has some options that can be configured or made into a GUI which I might do later cause I like GUI's on my stuff.
| Code: | ; AutoHotkey Version: 1.x
; Language: English
; Platform: Win9x/NT
; Author: Computerspazzz the Technowizard <computerspazzz@gmail.com>
; Additional Credits: Credit goes to the author of FileMD5(), who I beleive to be "SKAN" from the AutoHotKey Forums. http://www.autohotkey.com/forum/topic8728-105.html
; Script Function:
; Hash MD5's for a selected folder and log them for future comparison.
;
#NoEnv ; Recommended for performance and compatibility with future AutoHotkey releases.
SendMode Input ; Recommended for new scripts due to its superior speed and reliability.
SetWorkingDir %A_ScriptDir% ; Ensures a consistent starting directory.
#Include FileMD5.ahk ;FileMD5()
; Setup these options if you want to setup a GUI, MakeINI makes it as an INI, MakeFILE just makes a text file with filename on 1 line, and MD5 on next line. These options would be useful to set in a GUI.
MakeINI = 1
ININame = hash.ini
MakeFILE = 1
FileName = hash.txt
;Prior to Loops, select folder
FileSelectFolder, folder, %A_Desktop%,, Select Folder to Get Hashes
If ErrorLevel = 1 ; Exit App if User Cancels Select Folder Box
ExitApp
; Count number of files, this allows theo progress indicator to work properly.
Loop, %folder%\*, 0, 1
{
FileCounter := A_Index
}
ProgressUp := (100 / FileCounter)
ProgressCurrent = 0
Progress, b w400, Hashing Files, Hashing File, Hashing Files
; Do MD5's and output to specified output files.
Loop, %folder%\*, 0, 1
{
MD5Hash := FileMD5(A_LoopFileFullPath)
If MakeINI = 1
IniWrite, %MD5Hash%, %ININame%, MD5s, %A_LoopFileFullPath%
If MakeFILE = 1
FileAppend, %A_LoopFileFullPath%`n%MD5Hash%`n, %filename%
FileCount := A_Index
ProgressCurrent := (ProgressCurrent + ProgressUp)
Progress, %ProgressCurrent%
;sleep 20 ;Only use for slowing down for var and interface checking
}
Progress, Off
MsgBox, 4, Done Hashing MD5's, Hashing MD5's Done:`n%FileCounter% Files Found`n%FileCount% Files Processed.`n`nWould you like to see the results now?
IfMsgBox Yes
{
If MakeINI = 1
Run notepad.exe %A_ScriptDir%\%ININame%
If MakeFILE = 1
Run notepad.exe %A_ScriptDir%\%FileName%
}
ExitApp
|
Dont forget it requires FileMD5.ahk
| Code: | FileMD5( sFile="", cSz=4 )
{
cSz := (cSz<0||cSz>8) ? 2**22 : 2**(18+cSz), VarSetCapacity( Buffer,cSz,0 )
hFil := DllCall( "CreateFile", Str,sFile,UInt,0x80000000, Int,1,Int,0,Int,3,Int,0,Int,0 )
IfLess,hFil,1, Return,hFil
DllCall( "GetFileSizeEx", UInt,hFil, Str,Buffer ), fSz := NumGet( Buffer,0,"Int64" )
VarSetCapacity( MD5_CTX,104,0 ), DllCall( "advapi32\MD5Init", Str,MD5_CTX )
Loop % ( fSz//cSz+!!Mod(fSz,cSz) )
DllCall( "ReadFile", UInt,hFil, Str,Buffer, UInt,cSz, UIntP,bytesRead, UInt,0 )
, DllCall( "advapi32\MD5Update", Str,MD5_CTX, Str,Buffer, UInt,bytesRead )
DllCall( "advapi32\MD5Final", Str,MD5_CTX ), DllCall( "CloseHandle", UInt,hFil )
Loop % StrLen( Hex:="123456789ABCDEF0" )
N := NumGet( MD5_CTX,87+A_Index,"Char"), MD5 .= SubStr(Hex,N>>4,1) . SubStr(Hex,N&15,1)
Return MD5
}
|
Or You can do the basic striped down single file version:
| Code: | FileSelectFile, filefullpath, 3, %A_Desktop%, Chose a file to MD5 Hash, All Files (*.*)
Hash := FileMD5(filefullpath)
FileAppend, %filefullpath%`n, MD5Hash.log
FileAppend, %Hash%`n, MD5Hash.log
ExitApp
FileMD5( sFile="", cSz=4 )
{
cSz := (cSz<0||cSz>8) ? 2**22 : 2**(18+cSz), VarSetCapacity( Buffer,cSz,0 )
hFil := DllCall( "CreateFile", Str,sFile,UInt,0x80000000, Int,1,Int,0,Int,3,Int,0,Int,0 )
IfLess,hFil,1, Return,hFil
DllCall( "GetFileSizeEx", UInt,hFil, Str,Buffer ), fSz := NumGet( Buffer,0,"Int64" )
VarSetCapacity( MD5_CTX,104,0 ), DllCall( "advapi32\MD5Init", Str,MD5_CTX )
Loop % ( fSz//cSz+!!Mod(fSz,cSz) )
DllCall( "ReadFile", UInt,hFil, Str,Buffer, UInt,cSz, UIntP,bytesRead, UInt,0 )
, DllCall( "advapi32\MD5Update", Str,MD5_CTX, Str,Buffer, UInt,bytesRead )
DllCall( "advapi32\MD5Final", Str,MD5_CTX ), DllCall( "CloseHandle", UInt,hFil )
Loop % StrLen( Hex:="123456789ABCDEF0" )
N := NumGet( MD5_CTX,87+A_Index,"Char"), MD5 .= SubStr(Hex,N>>4,1) . SubStr(Hex,N&15,1)
Return MD5
}
|
P.S. I wrote all this as I plan to integrate it into a program "patcher" system where it checks MD5's and compares them to a downloaded manifest to see which files have changed and need to be re-downloaded (in case of corruption or updates available). _________________ Computerspazzz The Technowizard
The Wizard Is In! |
|
| Back to top |
|
 |
|
|
You can post new topics in this forum You can reply to topics in this forum
|
Powered by phpBB © 2001, 2005 phpBB Group
|