AutoHotkey Community

It is currently May 25th, 2012, 8:52 am

All times are UTC [ DST ]




Post new topic Reply to topic  [ 43 posts ]  Go to page Previous  1, 2, 3
Author Message
 Post subject:
PostPosted: May 31st, 2007, 5:16 pm 
Online
User avatar

Joined: December 26th, 2005, 4:40 pm
Posts: 8775
@Laszlo

Dear Sir,

I am experimenting with your Hash().. and full of surprises :o

I did a test: Hash() Vs Hashes.DLL

File: MPG
Size: 76MB
System: 1.4Ghz Sempron 256 MB Ram
AHK: 1.0.46.16 ( Latest )
OS: Win2k SP4

Result: I am happy to discontinue from using Hashes.DLL

For Hashes.DLL, I timed only the DllCall()

Hashes.DLL
MD5: 2580ms
SHA1: 4100ms

For Hash(). I timed for FileRead + FileGetSize + Hash() call

Hash()
MD5: 2050ms
SHA1: 2420

However I need some clarifications:

1) Can you explain why Hashes.DLL take phenomenally more time for SHA1 ?
2) Would we gain significant performance if Hash() is built-in as a function ?

:)

PS: Hashes_DLL.ZIP (45KB)


Report this post
Top
 Profile  
Reply with quote  
 Post subject:
PostPosted: May 31st, 2007, 6:13 pm 
Offline

Joined: February 14th, 2005, 4:05 pm
Posts: 4710
Location: Boulder, CO
1) Probably bad code. The work is only 25% more for SHA-1 than for MD5, so anything slower is not justified.

2) The dllcall overhead is just a couple of ms (except loading it in the first place), so no much speedup is expected with built in code.


Report this post
Top
 Profile  
Reply with quote  
 Post subject:
PostPosted: May 31st, 2007, 8:50 pm 
Online
User avatar

Joined: December 26th, 2005, 4:40 pm
Posts: 8775
Thanks for the clarification!.

Laszlo wrote:
2) The dllcall overhead is just a couple of ms (except loading it in the first place), so no much speedup is expected with built in code.


Oh! I was thinking about wishing this in Wish List.
I will be content if this is available in the forthcoming standard include.
I thank Sean and you for the wonderful code.

Regards, :)


Report this post
Top
 Profile  
Reply with quote  
 Post subject:
PostPosted: June 18th, 2007, 11:09 pm 
Online
User avatar

Joined: December 26th, 2005, 4:40 pm
Posts: 8775
How can a compiled script hash self in order to determine its integrity? :roll:


Report this post
Top
 Profile  
Reply with quote  
 Post subject:
PostPosted: June 18th, 2007, 11:42 pm 
Offline
User avatar

Joined: August 30th, 2005, 8:43 pm
Posts: 8647
Location: Salem, MA
not sure how to do that skan, but it reminded me of this:
Previous IOCCC winners:
omoikane - 2004 in particular

http://www.ioccc.org/2004/omoikane.hint


I think you have to store the hash in a separate file, or unpack the UPX, and mess with it.

_________________
Image
(Common Answers) - New Tutorials Forum - Humongous FAQ


Report this post
Top
 Profile  
Reply with quote  
 Post subject:
PostPosted: June 18th, 2007, 11:48 pm 
Online
User avatar

Joined: December 26th, 2005, 4:40 pm
Posts: 8775
engunneer wrote:
I think you have to store the hash in a separate file, or unpack the UPX, and mess with it.


I had guessed so, but would like to confirm it from Laszlo. :)


Report this post
Top
 Profile  
Reply with quote  
 Post subject:
PostPosted: June 19th, 2007, 6:27 am 
Offline

Joined: February 14th, 2005, 4:05 pm
Posts: 4710
Location: Boulder, CO
What do you need this MD5 signature checking for?

- If it is detecting random memory-, disk- or transfer errors of the program file, you can store the signature separately (maybe appended to the file). This is not very secure, because many random errors could modify the verification code, such that the program would believe it passed the test, which has not even been performed. Because of these and other weaknesses, a CRC is almost as good.
- If you have a trusted (intact) piece of SW, you are better off with some error correction code attached to the file.
- If it is intended as protection from malicious alterations, you would need trusted SW with the signature also coming from a trusted source (or use public keys). This cannot be done by the program to be checked, because rogue software could also lie about the result of the signature verification.

Consequently, self verification is not very good with MD5.


Report this post
Top
 Profile  
Reply with quote  
 Post subject:
PostPosted: June 19th, 2007, 7:32 am 
Online
User avatar

Joined: December 26th, 2005, 4:40 pm
Posts: 8775
Thanks for the valuable info. :)


Report this post
Top
 Profile  
Reply with quote  
 Post subject:
PostPosted: June 25th, 2007, 7:31 pm 
Online

Joined: April 22nd, 2007, 6:33 pm
Posts: 1832
ok ive tried this hash() method and the hash dll, but then also tried this console based program. it can calculate the md5 and many other forms of cryptography in no time

http://www.dominik-reichl.de/opensource.shtml#rehash

I use at the terminal:
rehash.exe -none -md5 filename

and it managed to output the md5 of a 700mb avi in under 10 secs and it was the same as the one found by hash()

I used:

Code:
FileGetSize, DataLength, filename
FileRead   , BinaryData, filename
MD5Hash := HASH( BinaryData, DataLength , 3 )

MsgBox, %MD5Hash%

HASH(ByRef sData, nLen, SID = 3) { ; SID = 3: MD5, 4: SHA1
;  Laszlo: http://www.autohotkey.com/forum/viewtopic.php?p=113252#113252
   DllCall("advapi32\CryptAcquireContextA", UIntP,hProv, UInt,0, UInt,0, UInt,1, UInt,0xF0000000)
   DllCall("advapi32\CryptCreateHash", UInt,hProv, UInt,0x8000|0|SID, UInt,0, UInt,0, UIntP, hHash)

   DllCall("advapi32\CryptHashData", UInt,hHash, UInt,&sData, UInt,nLen, UInt,0)

   DllCall("advapi32\CryptGetHashParam", UInt,hHash, UInt,2, UInt,0, UIntP,nSize, UInt,0)
   VarSetCapacity(HashVal, nSize, 0)
   DllCall("advapi32\CryptGetHashParam", UInt,hHash, UInt,2, UInt,&HashVal, UIntP,nSize, UInt,0)

   DllCall("advapi32\CryptDestroyHash", UInt,hHash)
   DllCall("advapi32\CryptReleaseContext", UInt,hProv, UInt,0)

   IFormat := A_FormatInteger
   SetFormat Integer, H
   Loop %nSize%
      sHash .= SubStr(*(&HashVal+A_Index-1)+0x100,-1)
   SetFormat Integer, %IFormat%
   Return sHash
}


and it uses the amount of ram equal to the size of the file for obvious reasons, but using the program rehash it only used a couple mb and cpu use was under 5% and still worked it out in seconds! a fraction of the time!

the source code is available for rehash, would anyone mind taking a look at it that is adept at cryptography and C++ and mind finding out how its working and perhaps create a dll from it that could be used in ahk??


Report this post
Top
 Profile  
Reply with quote  
PostPosted: June 3rd, 2010, 8:45 am 
Offline

Joined: May 25th, 2010, 8:26 am
Posts: 22
I am running windows and I tried it and it gives wrong MD5's and sometimes just gives all the same code no matter what. If anyone care to look at my code its below:

Code:

FileRead   , BinaryData, C:\test\2.jpg
DataLength := StrLen(BinaryData)
;FileGetSize, DataLength, C:\test\2.jpg   ; Also Tried This instead of the line above
MD5Hash := HASH( BinaryData, DataLength , 3 )
MsgBox, %MD5Hash%

HASH(ByRef sData, nLen, SID = 3) { ; SID = 3: MD5, 4: SHA1
;  Laszlo: http://www.autohotkey.com/forum/viewtopic.php?p=113252#113252
   DllCall("advapi32\CryptAcquireContextA", UIntP,hProv, UInt,0, UInt,0, UInt,1, UInt,0xF0000000)
   DllCall("advapi32\CryptCreateHash", UInt,hProv, UInt,0x8000|0|SID, UInt,0, UInt,0, UIntP, hHash)

   DllCall("advapi32\CryptHashData", UInt,hHash, UInt,&sData, UInt,nLen, UInt,0)

   DllCall("advapi32\CryptGetHashParam", UInt,hHash, UInt,2, UInt,0, UIntP,nSize, UInt,0)
   VarSetCapacity(HashVal, nSize, 0)
   DllCall("advapi32\CryptGetHashParam", UInt,hHash, UInt,2, UInt,&HashVal, UIntP,nSize, UInt,0)

   DllCall("advapi32\CryptDestroyHash", UInt,hHash)
   DllCall("advapi32\CryptReleaseContext", UInt,hProv, UInt,0)

   IFormat := A_FormatInteger
   SetFormat Integer, H
   Loop %nSize%
      sHash .= SubStr(*(&HashVal+A_Index-1)+0x100,-1)
   SetFormat Integer, %IFormat%
   Return sHash
}


I am looking to build a patcher program that will get a manifest from a ftp server with a file list and MD5's listed, check the files it already has against the manifest for any MD5 discrepancies and then download the files that don't match up. I have everything else, just not the MD5 (or any other good solid method will work). Would appreciate a point in the right direction if anyone happens upon this that knows. Spent the last 4 hours searching the forums and nothing I found works.

_________________
Computerspazzz The Technowizard
The Wizard Is In!


Report this post
Top
 Profile  
Reply with quote  
 Post subject:
PostPosted: June 3rd, 2010, 9:26 am 
Online
User avatar

Joined: December 26th, 2005, 4:40 pm
Posts: 8775
Code:
FileRead   , BinaryData, C:\test\2.jpg
; DataLength := StrLen(BinaryData)
;FileGetSize, DataLength, C:\test\2.jpg   ; Also Tried This instead of the line above
MD5Hash := HASH( BinaryData, DataLength , 3 )
MsgBox, %MD5Hash%


StrLen() would not work correctly with Binary Data
Use FileGetSize command like:
Code:
FileGetSize, DataLength, C:\test\2.jpg


On a related note, you could try FileMD5()


Report this post
Top
 Profile  
Reply with quote  
 Post subject:
PostPosted: June 3rd, 2010, 10:06 am 
Offline

Joined: February 14th, 2005, 4:05 pm
Posts: 4710
Location: Boulder, CO
When you read a file into a variable with no already assigned value or a small one, the capacity of the variable is set to the size of the file, so you can also use
Code:
DataLength := VarSetCapacity(BinaryData)
The hash function above otherwise works for me.

You can even combine SKAN's MD5 function with the hex conversion function of the crypto API, to get shorter, faster code, which gets the length, if not given:
Code:
MD5(ByRef V, L=0) { ; Compute MD5 on binary data in V of length L [all allocated] bytes
   VarSetCapacity(M,104) ; Use the MD5 Windows functions directly
   DllCall("advapi32\MD5Init",  Str,M)
   DllCall("advapi32\MD5Update",Str,M, Str,V, UInt,L ? L : VarSetCapacity(V))
   DllCall("advapi32\MD5Final", Str,M)

   VarSetCapacity(T,L:=64)     ; Convert to hex with the crypto API
   DllCall("Crypt32.dll\CryptBinaryToStringA", UInt,&M+88, UInt,16, UInt,4, Str,T, UIntP,L, "CDECL UInt")
   Return RegExReplace(T,"\W") ; Remove white space
}


Report this post
Top
 Profile  
Reply with quote  
PostPosted: June 6th, 2010, 10:05 pm 
Offline

Joined: May 25th, 2010, 8:26 am
Posts: 22
Thanks for the replies. Yeah i was messing around with the FileGetSize and DataLength and tried both. I also tried the FileMD5()... and the two scripts agree with one another! So i Had previously downloaded a MD5 checker to compare the results against... IT WAS WRONG! I didn't think to flip it around and download several programs as a test, another thing Ive learned. The FileMD5() is very neatly setup and easy to use as an #Include so going to stick with that, but this script also worked once i modified as you said. Thank you all very much.

On another note, I wrote a loop to hash a folder and output the data, its 100% ready to run as is, but has some options that can be configured or made into a GUI which I might do later cause I like GUI's on my stuff.

Code:
; AutoHotkey Version: 1.x
; Language:       English
; Platform:       Win9x/NT
; Author:         Computerspazzz the Technowizard <computerspazzz@gmail.com>
; Additional Credits: Credit goes to the author of FileMD5(), who I beleive to be "SKAN" from the AutoHotKey Forums. http://www.autohotkey.com/forum/topic8728-105.html
; Script Function:
;   Hash MD5's for a selected folder and log them for future comparison.
;

#NoEnv  ; Recommended for performance and compatibility with future AutoHotkey releases.
SendMode Input  ; Recommended for new scripts due to its superior speed and reliability.

SetWorkingDir %A_ScriptDir%  ; Ensures a consistent starting directory.
#Include FileMD5.ahk ;FileMD5()
; Setup these options if you want to setup a GUI, MakeINI makes it as an INI, MakeFILE just makes a text file with filename on 1 line, and MD5 on next line.  These options would be useful to set in a GUI.
MakeINI = 1
ININame = hash.ini
MakeFILE = 1
FileName = hash.txt
;Prior to Loops, select folder
FileSelectFolder, folder, %A_Desktop%,, Select Folder to Get Hashes
If ErrorLevel = 1   ; Exit App if User Cancels Select Folder Box
   ExitApp
; Count number of files, this allows theo progress indicator to work properly.
Loop, %folder%\*, 0, 1
   {
   FileCounter := A_Index
   }
ProgressUp := (100 / FileCounter)
ProgressCurrent = 0
Progress, b w400, Hashing Files, Hashing File, Hashing Files
; Do MD5's and output to specified output files.
Loop, %folder%\*, 0, 1
   {
   MD5Hash := FileMD5(A_LoopFileFullPath)
   If MakeINI = 1
      IniWrite, %MD5Hash%, %ININame%, MD5s, %A_LoopFileFullPath%
   If MakeFILE = 1
      FileAppend, %A_LoopFileFullPath%`n%MD5Hash%`n, %filename%
   FileCount := A_Index
   ProgressCurrent := (ProgressCurrent + ProgressUp)
   Progress, %ProgressCurrent%
   ;sleep 20 ;Only use for slowing down for var and interface checking
}
Progress, Off
MsgBox, 4, Done Hashing MD5's, Hashing MD5's Done:`n%FileCounter% Files Found`n%FileCount% Files Processed.`n`nWould you like to see the results now?
IfMsgBox Yes
   {
   If MakeINI = 1
      Run notepad.exe %A_ScriptDir%\%ININame%
   If MakeFILE = 1
      Run notepad.exe %A_ScriptDir%\%FileName%
   }
ExitApp


Dont forget it requires FileMD5.ahk
Code:
FileMD5( sFile="", cSz=4 )
{
 cSz  := (cSz<0||cSz>8) ? 2**22 : 2**(18+cSz), VarSetCapacity( Buffer,cSz,0 )
 hFil := DllCall( "CreateFile", Str,sFile,UInt,0x80000000, Int,1,Int,0,Int,3,Int,0,Int,0 )
 IfLess,hFil,1, Return,hFil
 DllCall( "GetFileSizeEx", UInt,hFil, Str,Buffer ),   fSz := NumGet( Buffer,0,"Int64" )
 VarSetCapacity( MD5_CTX,104,0 ),    DllCall( "advapi32\MD5Init", Str,MD5_CTX )
 Loop % ( fSz//cSz+!!Mod(fSz,cSz) )
   DllCall( "ReadFile", UInt,hFil, Str,Buffer, UInt,cSz, UIntP,bytesRead, UInt,0 )
 , DllCall( "advapi32\MD5Update", Str,MD5_CTX, Str,Buffer, UInt,bytesRead )
 DllCall( "advapi32\MD5Final", Str,MD5_CTX ), DllCall( "CloseHandle", UInt,hFil )
 Loop % StrLen( Hex:="123456789ABCDEF0" )
  N := NumGet( MD5_CTX,87+A_Index,"Char"), MD5 .= SubStr(Hex,N>>4,1) . SubStr(Hex,N&15,1)
Return MD5
}


Or You can do the basic striped down single file version:

Code:
FileSelectFile, filefullpath, 3, %A_Desktop%, Chose a file to MD5 Hash, All Files (*.*)
Hash := FileMD5(filefullpath)
FileAppend, %filefullpath%`n, MD5Hash.log
FileAppend, %Hash%`n, MD5Hash.log
ExitApp

FileMD5( sFile="", cSz=4 )
{
 cSz  := (cSz<0||cSz>8) ? 2**22 : 2**(18+cSz), VarSetCapacity( Buffer,cSz,0 )
 hFil := DllCall( "CreateFile", Str,sFile,UInt,0x80000000, Int,1,Int,0,Int,3,Int,0,Int,0 )
 IfLess,hFil,1, Return,hFil
 DllCall( "GetFileSizeEx", UInt,hFil, Str,Buffer ),   fSz := NumGet( Buffer,0,"Int64" )
 VarSetCapacity( MD5_CTX,104,0 ),    DllCall( "advapi32\MD5Init", Str,MD5_CTX )
 Loop % ( fSz//cSz+!!Mod(fSz,cSz) )
   DllCall( "ReadFile", UInt,hFil, Str,Buffer, UInt,cSz, UIntP,bytesRead, UInt,0 )
 , DllCall( "advapi32\MD5Update", Str,MD5_CTX, Str,Buffer, UInt,bytesRead )
 DllCall( "advapi32\MD5Final", Str,MD5_CTX ), DllCall( "CloseHandle", UInt,hFil )
 Loop % StrLen( Hex:="123456789ABCDEF0" )
  N := NumGet( MD5_CTX,87+A_Index,"Char"), MD5 .= SubStr(Hex,N>>4,1) . SubStr(Hex,N&15,1)
Return MD5
}


P.S. I wrote all this as I plan to integrate it into a program "patcher" system where it checks MD5's and compares them to a downloaded manifest to see which files have changed and need to be re-downloaded (in case of corruption or updates available).

_________________
Computerspazzz The Technowizard
The Wizard Is In!


Report this post
Top
 Profile  
Reply with quote  
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 43 posts ]  Go to page Previous  1, 2, 3

All times are UTC [ DST ]


Who is online

Users browsing this forum: Google [Bot], tic and 16 guests


You can post new topics in this forum
You can reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Powered by phpBB® Forum Software © phpBB Group