AutoHotkey Homepage AutoHotkey Community
Let's help each other out
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

MD5
Goto page Previous  1, 2, 3
 
Reply to topic    AutoHotkey Community Forum Index -> Scripts & Functions
View previous topic :: View next topic  
Author Message
SKAN



Joined: 26 Dec 2005
Posts: 8688

PostPosted: Thu May 31, 2007 4:16 pm    Post subject: Reply with quote

@Laszlo

Dear Sir,

I am experimenting with your Hash().. and full of surprises Surprised

I did a test: Hash() Vs Hashes.DLL

File: MPG
Size: 76MB
System: 1.4Ghz Sempron 256 MB Ram
AHK: 1.0.46.16 ( Latest )
OS: Win2k SP4

Result: I am happy to discontinue from using Hashes.DLL

For Hashes.DLL, I timed only the DllCall()

Hashes.DLL
MD5: 2580ms
SHA1: 4100ms

For Hash(). I timed for FileRead + FileGetSize + Hash() call

Hash()
MD5: 2050ms
SHA1: 2420

However I need some clarifications:

1) Can you explain why Hashes.DLL take phenomenally more time for SHA1 ?
2) Would we gain significant performance if Hash() is built-in as a function ?

Smile

PS: Hashes_DLL.ZIP (45KB)
Back to top
View user's profile Send private message Send e-mail
Laszlo



Joined: 14 Feb 2005
Posts: 4710
Location: Boulder, CO

PostPosted: Thu May 31, 2007 5:13 pm    Post subject: Reply with quote

1) Probably bad code. The work is only 25% more for SHA-1 than for MD5, so anything slower is not justified.

2) The dllcall overhead is just a couple of ms (except loading it in the first place), so no much speedup is expected with built in code.
Back to top
View user's profile Send private message
SKAN



Joined: 26 Dec 2005
Posts: 8688

PostPosted: Thu May 31, 2007 7:50 pm    Post subject: Reply with quote

Thanks for the clarification!.

Laszlo wrote:
2) The dllcall overhead is just a couple of ms (except loading it in the first place), so no much speedup is expected with built in code.


Oh! I was thinking about wishing this in Wish List.
I will be content if this is available in the forthcoming standard include.
I thank Sean and you for the wonderful code.

Regards, Smile
Back to top
View user's profile Send private message Send e-mail
SKAN



Joined: 26 Dec 2005
Posts: 8688

PostPosted: Mon Jun 18, 2007 10:09 pm    Post subject: Reply with quote

How can a compiled script hash self in order to determine its integrity? Rolling Eyes
Back to top
View user's profile Send private message Send e-mail
engunneer



Joined: 30 Aug 2005
Posts: 8255
Location: Maywood, IL

PostPosted: Mon Jun 18, 2007 10:42 pm    Post subject: Reply with quote

not sure how to do that skan, but it reminded me of this:
Previous IOCCC winners:
omoikane - 2004 in particular

http://www.ioccc.org/2004/omoikane.hint


I think you have to store the hash in a separate file, or unpack the UPX, and mess with it.
_________________

(Common Answers)
Back to top
View user's profile Send private message Visit poster's website
SKAN



Joined: 26 Dec 2005
Posts: 8688

PostPosted: Mon Jun 18, 2007 10:48 pm    Post subject: Reply with quote

engunneer wrote:
I think you have to store the hash in a separate file, or unpack the UPX, and mess with it.


I had guessed so, but would like to confirm it from Laszlo. Smile
Back to top
View user's profile Send private message Send e-mail
Laszlo



Joined: 14 Feb 2005
Posts: 4710
Location: Boulder, CO

PostPosted: Tue Jun 19, 2007 5:27 am    Post subject: Reply with quote

What do you need this MD5 signature checking for?

- If it is detecting random memory-, disk- or transfer errors of the program file, you can store the signature separately (maybe appended to the file). This is not very secure, because many random errors could modify the verification code, such that the program would believe it passed the test, which has not even been performed. Because of these and other weaknesses, a CRC is almost as good.
- If you have a trusted (intact) piece of SW, you are better off with some error correction code attached to the file.
- If it is intended as protection from malicious alterations, you would need trusted SW with the signature also coming from a trusted source (or use public keys). This cannot be done by the program to be checked, because rogue software could also lie about the result of the signature verification.

Consequently, self verification is not very good with MD5.
Back to top
View user's profile Send private message
SKAN



Joined: 26 Dec 2005
Posts: 8688

PostPosted: Tue Jun 19, 2007 6:32 am    Post subject: Reply with quote

Thanks for the valuable info. Smile
Back to top
View user's profile Send private message Send e-mail
tic



Joined: 22 Apr 2007
Posts: 1786

PostPosted: Mon Jun 25, 2007 6:31 pm    Post subject: Reply with quote

ok ive tried this hash() method and the hash dll, but then also tried this console based program. it can calculate the md5 and many other forms of cryptography in no time

http://www.dominik-reichl.de/opensource.shtml#rehash

I use at the terminal:
rehash.exe -none -md5 filename

and it managed to output the md5 of a 700mb avi in under 10 secs and it was the same as the one found by hash()

I used:

Code:
FileGetSize, DataLength, filename
FileRead   , BinaryData, filename
MD5Hash := HASH( BinaryData, DataLength , 3 )

MsgBox, %MD5Hash%

HASH(ByRef sData, nLen, SID = 3) { ; SID = 3: MD5, 4: SHA1
;  Laszlo: http://www.autohotkey.com/forum/viewtopic.php?p=113252#113252
   DllCall("advapi32\CryptAcquireContextA", UIntP,hProv, UInt,0, UInt,0, UInt,1, UInt,0xF0000000)
   DllCall("advapi32\CryptCreateHash", UInt,hProv, UInt,0x8000|0|SID, UInt,0, UInt,0, UIntP, hHash)

   DllCall("advapi32\CryptHashData", UInt,hHash, UInt,&sData, UInt,nLen, UInt,0)

   DllCall("advapi32\CryptGetHashParam", UInt,hHash, UInt,2, UInt,0, UIntP,nSize, UInt,0)
   VarSetCapacity(HashVal, nSize, 0)
   DllCall("advapi32\CryptGetHashParam", UInt,hHash, UInt,2, UInt,&HashVal, UIntP,nSize, UInt,0)

   DllCall("advapi32\CryptDestroyHash", UInt,hHash)
   DllCall("advapi32\CryptReleaseContext", UInt,hProv, UInt,0)

   IFormat := A_FormatInteger
   SetFormat Integer, H
   Loop %nSize%
      sHash .= SubStr(*(&HashVal+A_Index-1)+0x100,-1)
   SetFormat Integer, %IFormat%
   Return sHash
}


and it uses the amount of ram equal to the size of the file for obvious reasons, but using the program rehash it only used a couple mb and cpu use was under 5% and still worked it out in seconds! a fraction of the time!

the source code is available for rehash, would anyone mind taking a look at it that is adept at cryptography and C++ and mind finding out how its working and perhaps create a dll from it that could be used in ahk??
Back to top
View user's profile Send private message
computerspazzz



Joined: 25 May 2010
Posts: 22

PostPosted: Thu Jun 03, 2010 7:45 am    Post subject: Is this old and no longer working? Reply with quote

I am running windows and I tried it and it gives wrong MD5's and sometimes just gives all the same code no matter what. If anyone care to look at my code its below:

Code:


FileRead   , BinaryData, C:\test\2.jpg
DataLength := StrLen(BinaryData)
;FileGetSize, DataLength, C:\test\2.jpg   ; Also Tried This instead of the line above
MD5Hash := HASH( BinaryData, DataLength , 3 )
MsgBox, %MD5Hash%

HASH(ByRef sData, nLen, SID = 3) { ; SID = 3: MD5, 4: SHA1
;  Laszlo: http://www.autohotkey.com/forum/viewtopic.php?p=113252#113252
   DllCall("advapi32\CryptAcquireContextA", UIntP,hProv, UInt,0, UInt,0, UInt,1, UInt,0xF0000000)
   DllCall("advapi32\CryptCreateHash", UInt,hProv, UInt,0x8000|0|SID, UInt,0, UInt,0, UIntP, hHash)

   DllCall("advapi32\CryptHashData", UInt,hHash, UInt,&sData, UInt,nLen, UInt,0)

   DllCall("advapi32\CryptGetHashParam", UInt,hHash, UInt,2, UInt,0, UIntP,nSize, UInt,0)
   VarSetCapacity(HashVal, nSize, 0)
   DllCall("advapi32\CryptGetHashParam", UInt,hHash, UInt,2, UInt,&HashVal, UIntP,nSize, UInt,0)

   DllCall("advapi32\CryptDestroyHash", UInt,hHash)
   DllCall("advapi32\CryptReleaseContext", UInt,hProv, UInt,0)

   IFormat := A_FormatInteger
   SetFormat Integer, H
   Loop %nSize%
      sHash .= SubStr(*(&HashVal+A_Index-1)+0x100,-1)
   SetFormat Integer, %IFormat%
   Return sHash
}


I am looking to build a patcher program that will get a manifest from a ftp server with a file list and MD5's listed, check the files it already has against the manifest for any MD5 discrepancies and then download the files that don't match up. I have everything else, just not the MD5 (or any other good solid method will work). Would appreciate a point in the right direction if anyone happens upon this that knows. Spent the last 4 hours searching the forums and nothing I found works.
_________________
Computerspazzz The Technowizard
The Wizard Is In!
Back to top
View user's profile Send private message
SKAN



Joined: 26 Dec 2005
Posts: 8688

PostPosted: Thu Jun 03, 2010 8:26 am    Post subject: Reply with quote

Code:
FileRead   , BinaryData, C:\test\2.jpg
; DataLength := StrLen(BinaryData)
;FileGetSize, DataLength, C:\test\2.jpg   ; Also Tried This instead of the line above
MD5Hash := HASH( BinaryData, DataLength , 3 )
MsgBox, %MD5Hash%


StrLen() would not work correctly with Binary Data
Use FileGetSize command like:
Code:
FileGetSize, DataLength, C:\test\2.jpg


On a related note, you could try FileMD5()
Back to top
View user's profile Send private message Send e-mail
Laszlo



Joined: 14 Feb 2005
Posts: 4710
Location: Boulder, CO

PostPosted: Thu Jun 03, 2010 9:06 am    Post subject: Reply with quote

When you read a file into a variable with no already assigned value or a small one, the capacity of the variable is set to the size of the file, so you can also use
Code:
DataLength := VarSetCapacity(BinaryData)
The hash function above otherwise works for me.

You can even combine SKAN's MD5 function with the hex conversion function of the crypto API, to get shorter, faster code, which gets the length, if not given:
Code:
MD5(ByRef V, L=0) { ; Compute MD5 on binary data in V of length L [all allocated] bytes
   VarSetCapacity(M,104) ; Use the MD5 Windows functions directly
   DllCall("advapi32\MD5Init",  Str,M)
   DllCall("advapi32\MD5Update",Str,M, Str,V, UInt,L ? L : VarSetCapacity(V))
   DllCall("advapi32\MD5Final", Str,M)

   VarSetCapacity(T,L:=64)     ; Convert to hex with the crypto API
   DllCall("Crypt32.dll\CryptBinaryToStringA", UInt,&M+88, UInt,16, UInt,4, Str,T, UIntP,L, "CDECL UInt")
   Return RegExReplace(T,"\W") ; Remove white space
}
Back to top
View user's profile Send private message
computerspazzz



Joined: 25 May 2010
Posts: 22

PostPosted: Sun Jun 06, 2010 9:05 pm    Post subject: FileMD5() Folder or File Hasher Reply with quote

Thanks for the replies. Yeah i was messing around with the FileGetSize and DataLength and tried both. I also tried the FileMD5()... and the two scripts agree with one another! So i Had previously downloaded a MD5 checker to compare the results against... IT WAS WRONG! I didn't think to flip it around and download several programs as a test, another thing Ive learned. The FileMD5() is very neatly setup and easy to use as an #Include so going to stick with that, but this script also worked once i modified as you said. Thank you all very much.

On another note, I wrote a loop to hash a folder and output the data, its 100% ready to run as is, but has some options that can be configured or made into a GUI which I might do later cause I like GUI's on my stuff.

Code:
; AutoHotkey Version: 1.x
; Language:       English
; Platform:       Win9x/NT
; Author:         Computerspazzz the Technowizard <computerspazzz@gmail.com>
; Additional Credits: Credit goes to the author of FileMD5(), who I beleive to be "SKAN" from the AutoHotKey Forums. http://www.autohotkey.com/forum/topic8728-105.html
; Script Function:
;   Hash MD5's for a selected folder and log them for future comparison.
;

#NoEnv  ; Recommended for performance and compatibility with future AutoHotkey releases.
SendMode Input  ; Recommended for new scripts due to its superior speed and reliability.

SetWorkingDir %A_ScriptDir%  ; Ensures a consistent starting directory.
#Include FileMD5.ahk ;FileMD5()
; Setup these options if you want to setup a GUI, MakeINI makes it as an INI, MakeFILE just makes a text file with filename on 1 line, and MD5 on next line.  These options would be useful to set in a GUI.
MakeINI = 1
ININame = hash.ini
MakeFILE = 1
FileName = hash.txt
;Prior to Loops, select folder
FileSelectFolder, folder, %A_Desktop%,, Select Folder to Get Hashes
If ErrorLevel = 1   ; Exit App if User Cancels Select Folder Box
   ExitApp
; Count number of files, this allows theo progress indicator to work properly.
Loop, %folder%\*, 0, 1
   {
   FileCounter := A_Index
   }
ProgressUp := (100 / FileCounter)
ProgressCurrent = 0
Progress, b w400, Hashing Files, Hashing File, Hashing Files
; Do MD5's and output to specified output files.
Loop, %folder%\*, 0, 1
   {
   MD5Hash := FileMD5(A_LoopFileFullPath)
   If MakeINI = 1
      IniWrite, %MD5Hash%, %ININame%, MD5s, %A_LoopFileFullPath%
   If MakeFILE = 1
      FileAppend, %A_LoopFileFullPath%`n%MD5Hash%`n, %filename%
   FileCount := A_Index
   ProgressCurrent := (ProgressCurrent + ProgressUp)
   Progress, %ProgressCurrent%
   ;sleep 20 ;Only use for slowing down for var and interface checking
}
Progress, Off
MsgBox, 4, Done Hashing MD5's, Hashing MD5's Done:`n%FileCounter% Files Found`n%FileCount% Files Processed.`n`nWould you like to see the results now?
IfMsgBox Yes
   {
   If MakeINI = 1
      Run notepad.exe %A_ScriptDir%\%ININame%
   If MakeFILE = 1
      Run notepad.exe %A_ScriptDir%\%FileName%
   }
ExitApp


Dont forget it requires FileMD5.ahk
Code:
FileMD5( sFile="", cSz=4 )
{
 cSz  := (cSz<0||cSz>8) ? 2**22 : 2**(18+cSz), VarSetCapacity( Buffer,cSz,0 )
 hFil := DllCall( "CreateFile", Str,sFile,UInt,0x80000000, Int,1,Int,0,Int,3,Int,0,Int,0 )
 IfLess,hFil,1, Return,hFil
 DllCall( "GetFileSizeEx", UInt,hFil, Str,Buffer ),   fSz := NumGet( Buffer,0,"Int64" )
 VarSetCapacity( MD5_CTX,104,0 ),    DllCall( "advapi32\MD5Init", Str,MD5_CTX )
 Loop % ( fSz//cSz+!!Mod(fSz,cSz) )
   DllCall( "ReadFile", UInt,hFil, Str,Buffer, UInt,cSz, UIntP,bytesRead, UInt,0 )
 , DllCall( "advapi32\MD5Update", Str,MD5_CTX, Str,Buffer, UInt,bytesRead )
 DllCall( "advapi32\MD5Final", Str,MD5_CTX ), DllCall( "CloseHandle", UInt,hFil )
 Loop % StrLen( Hex:="123456789ABCDEF0" )
  N := NumGet( MD5_CTX,87+A_Index,"Char"), MD5 .= SubStr(Hex,N>>4,1) . SubStr(Hex,N&15,1)
Return MD5
}


Or You can do the basic striped down single file version:

Code:
FileSelectFile, filefullpath, 3, %A_Desktop%, Chose a file to MD5 Hash, All Files (*.*)
Hash := FileMD5(filefullpath)
FileAppend, %filefullpath%`n, MD5Hash.log
FileAppend, %Hash%`n, MD5Hash.log
ExitApp

FileMD5( sFile="", cSz=4 )
{
 cSz  := (cSz<0||cSz>8) ? 2**22 : 2**(18+cSz), VarSetCapacity( Buffer,cSz,0 )
 hFil := DllCall( "CreateFile", Str,sFile,UInt,0x80000000, Int,1,Int,0,Int,3,Int,0,Int,0 )
 IfLess,hFil,1, Return,hFil
 DllCall( "GetFileSizeEx", UInt,hFil, Str,Buffer ),   fSz := NumGet( Buffer,0,"Int64" )
 VarSetCapacity( MD5_CTX,104,0 ),    DllCall( "advapi32\MD5Init", Str,MD5_CTX )
 Loop % ( fSz//cSz+!!Mod(fSz,cSz) )
   DllCall( "ReadFile", UInt,hFil, Str,Buffer, UInt,cSz, UIntP,bytesRead, UInt,0 )
 , DllCall( "advapi32\MD5Update", Str,MD5_CTX, Str,Buffer, UInt,bytesRead )
 DllCall( "advapi32\MD5Final", Str,MD5_CTX ), DllCall( "CloseHandle", UInt,hFil )
 Loop % StrLen( Hex:="123456789ABCDEF0" )
  N := NumGet( MD5_CTX,87+A_Index,"Char"), MD5 .= SubStr(Hex,N>>4,1) . SubStr(Hex,N&15,1)
Return MD5
}


P.S. I wrote all this as I plan to integrate it into a program "patcher" system where it checks MD5's and compares them to a downloaded manifest to see which files have changed and need to be re-downloaded (in case of corruption or updates available).
_________________
Computerspazzz The Technowizard
The Wizard Is In!
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    AutoHotkey Community Forum Index -> Scripts & Functions All times are GMT
Goto page Previous  1, 2, 3
Page 3 of 3

 
Jump to:  
You can post new topics in this forum
You can reply to topics in this forum


Powered by phpBB © 2001, 2005 phpBB Group