AutoHotkey Homepage AutoHotkey Community
Let's help each other out
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

Machine code functions: Bit Wizardry
Goto page Previous  1, 2, 3 ... 6, 7, 8 ... 18, 19, 20  Next
 
Reply to topic    AutoHotkey Community Forum Index -> Scripts & Functions
View previous topic :: View next topic  
Author Message
Laszlo



Joined: 14 Feb 2005
Posts: 4668
Location: Boulder, CO
PostPosted: Tue Sep 25, 2007 7:17 pm    Post subject: Reply with quote
olfen wrote:
Can you explain this…r1 := ~res → 0x292f0b53, r2 := res ^ 0xFFFFFFFF → -0xd6d0f4ad?
It is the difference between 32 or 64-bit signed number representation. Their least significant 32 bits are the same.
Back to top
View user's profile Send private message
tic



Joined: 22 Apr 2007
Posts: 1502
PostPosted: Wed Sep 26, 2007 1:13 am    Post subject: Reply with quote
ok. i was 1 version of ahk behind. dont know which updates have made it change, but now it always gives 434084816 no matter what i add to the file
Back to top
View user's profile Send private message
olfen



Joined: 04 Jun 2005
Posts: 100
Location: Stuttgart, Germany
PostPosted: Wed Sep 26, 2007 6:45 pm    Post subject: Reply with quote
Please try if this works:
Code:
#NoEnv

MCode(CRC32_Init, "5589E583EC10C745FC2083B8EDC745F400000000817DF4FF0000007F508B45F48945F8C745F008000000837DF0007E238B45F883"
. "E00185C0740D8B45F8D1E83345FC8945F8EB058D45F8D1288D45F0FF08EBD78B45F48D0C85000000008B55088B45F88904118D45F4FF00EBA7C9C3")
MCode(CRC32_Get, "5589E583EC08C745FCFFFFFFFFC745F8000000008B45F83B450C732E8B45080345F80FB6003345FC25FF0000008D0C85000000008"
. "B55108B45FCC1E8083304118945FC8D45F8FF00EBCA8B45FCC9C3")
 
VarSetCapacity(CRC32LookupTable, 256*4)
DllCall(&CRC32_Init, "uint",&CRC32LookupTable)

SetFormat, Integer, Hex
 
a = abcdef
res := DllCall(&CRC32_Get, "uint",&a, "uint",StrLen(a), "uint",&CRC32LookupTable)
If (res < 0)
  MsgBox % Abs(res) - 1
Else
  MsgBox % ~res

FileRead, a, %A_AhkPath%
FileGetSize, size, %A_AhkPath%
res := DllCall(&CRC32_Get, "uint",&a, "uint",size, "uint",&CRC32LookupTable)
If (res < 0)
  MsgBox % Abs(res) - 1
Else
  MsgBox % ~res
Back to top
View user's profile Send private message
tic



Joined: 22 Apr 2007
Posts: 1502
PostPosted: Wed Sep 26, 2007 7:46 pm    Post subject: Reply with quote
Nope Sad

I ran

Code:
#NoEnv

MCode(ByRef code, hex) { ; allocate memory and write Machine Code there
   VarSetCapacity(code,StrLen(hex)//2)
   Loop % StrLen(hex)//2
      NumPut("0x" . SubStr(hex,2*A_Index-1,2), code, A_Index-1, "Char")
}

MCode(CRC32_Init, "5589E583EC10C745FC2083B8EDC745F400000000817DF4FF0000007F508B45F48945F8C745F008000000837DF0007E238B45F883"
. "E00185C0740D8B45F8D1E83345FC8945F8EB058D45F8D1288D45F0FF08EBD78B45F48D0C85000000008B55088B45F88904118D45F4FF00EBA7C9C3")
MCode(CRC32_Get, "5589E583EC08C745FCFFFFFFFFC745F8000000008B45F83B450C732E8B45080345F80FB6003345FC25FF0000008D0C85000000008"
. "B55108B45FCC1E8083304118945FC8D45F8FF00EBCA8B45FCC9C3")
 
VarSetCapacity(CRC32LookupTable, 256*4)
DllCall(&CRC32_Init, "uint",&CRC32LookupTable)

FileRead, a, %A_AhkPath%
FileGetSize, size, %A_AhkPath%
res := DllCall(&CRC32_Get, "uint",&a, "uint",size, "uint",&CRC32LookupTable)
If (res < 0)
  MsgBox % Abs(res) - 1
Else
  MsgBox % ~res


and when modified it still will always give me 434084816
Back to top
View user's profile Send private message
olfen



Joined: 04 Jun 2005
Posts: 100
Location: Stuttgart, Germany
PostPosted: Wed Sep 26, 2007 7:53 pm    Post subject: Reply with quote
tic wrote:
and when modified it still will always give me 434084816
-> 0x19DF9BD0 in decimal form, which is correct. Try adding
Code:
SetFormat, Integer, Hex
before calling CRC32_Get.
Back to top
View user's profile Send private message
olfen



Joined: 04 Jun 2005
Posts: 100
Location: Stuttgart, Germany
PostPosted: Fri Sep 28, 2007 9:33 am    Post subject: Reply with quote
Here's an example demonstrating how to generate a list of CRC-32 - file pairs:
Code:
#NoEnv

MCode(ByRef code, hex) { ; allocate memory and write Machine Code there
   VarSetCapacity(code,StrLen(hex)//2)
   Loop % StrLen(hex)//2
      NumPut("0x" . SubStr(hex,2*A_Index-1,2), code, A_Index-1, "Char")
}

MCode(CRC32_Init, "5589E583EC10C745FC2083B8EDC745F400000000817DF4FF0000007F508B45F48945F8C745F008000000837DF0007E238B45F883"
. "E00185C0740D8B45F8D1E83345FC8945F8EB058D45F8D1288D45F0FF08EBD78B45F48D0C85000000008B55088B45F88904118D45F4FF00EBA7C9C3")
MCode(CRC32_Get, "5589E583EC08C745FCFFFFFFFFC745F8000000008B45F83B450C732E8B45080345F80FB6003345FC25FF0000008D0C85000000008"
. "B55108B45FCC1E8083304118945FC8D45F8FF00EBCA8B45FCC9C3")
 
VarSetCapacity(CRC32LookupTable, 256*4)
DllCall(&CRC32_Init, "uint",&CRC32LookupTable)

SetFormat, Integer, Hex
 
Loop, *.pdf
{
  FileRead, a, %a_loopfilename%
  FileGetSize, size, %a_loopfilename%
  res := DllCall(&CRC32_Get, "uint",&a, "uint",size, "uint",&CRC32LookupTable) 
  res := res < 0 ? Abs(res) - 1 : ~res

  StringUpper, res, res
  StringReplace, res, res, 0X, 0000000
  r .= SubStr(res, -7) . a_tab . a_loopfilename . "`n"
}
a =
Sort, r
FileAppend, %r%, res.txt
MsgBox, Done.
Back to top
View user's profile Send private message
Lexikos



Joined: 17 Oct 2006
Posts: 4726
Location: Australia
PostPosted: Fri Oct 26, 2007 9:42 am    Post subject: DllCallInWorkerThread Reply with quote
As an alternative to true multi-threading support in AutoHotkey, would it be possible to write a machine code function to wrap DllCall? Ideally, it would start a new thread, call the function (from the new thread), then send the return value back to the script (via window messages, I suppose.)
Back to top
View user's profile Send private message Visit poster's website
Laszlo



Joined: 14 Feb 2005
Posts: 4668
Location: Boulder, CO
PostPosted: Fri Oct 26, 2007 3:32 pm    Post subject: Reply with quote
These machine code blocks were meant for short, standalone functions. A complex task needs internal subroutines. The linker could generate relative call addresses, which you have to edit, after removing unnecessary code from the binary file. Interfacing the Windows dynamic library load and call mechanism needs some code, so I’d not be surprised to see several pages of hex data in the AHK function. It looks simpler to run a second copy of AHK via the RUN command.
Back to top
View user's profile Send private message
Leon



Joined: 27 Aug 2007
Posts: 179
PostPosted: Sat Oct 27, 2007 8:14 pm    Post subject: Reply with quote
Great work and info by everyone.
I'm really working at understanding how all this works.

How can i use this to have javascript in my ahk script / compiled exe?
This need only work in XP or later.

I found these hashing algorithms that seem to be great. Want to include one in a script.

In light of this topic, some of you may find these veruy useful.

DL / view source code:
MD4
MD5
SHA-1
Back to top
View user's profile Send private message
Laszlo



Joined: 14 Feb 2005
Posts: 4668
Location: Boulder, CO
PostPosted: Sat Oct 27, 2007 11:07 pm    Post subject: Reply with quote
There are many JavaScript compilers out there. Google it. However, they usually produce code with a lot of external function calls (like Java class libraries), so they have to be present at run time. Therefore, JavaScript does not seem to be the best source. I would stick to C, Delphi, VB, if I could.

For the hash algorithms you don’t need to compile anything. They are included in Windows, as discussed here. MD4 is not there, but it is not commonly used any more.
Back to top
View user's profile Send private message
tic



Joined: 22 Apr 2007
Posts: 1502
PostPosted: Tue Oct 30, 2007 10:55 pm    Post subject: Reply with quote
Ive been playing a bit with olfens crc32 but i would like to know how it could be rewritten to not need the entire string at once and then perform a crc, but actually do it piece by piece. Currently if it hashes a film, itll put 700mb into the memory which is not ideal


I posted on this a while ago and would like to know how it works and whther it can be used in ahk:

http://www.dominik-reichl.de/opensource.shtml#rehash

it took less than 2 seconds to get the crc32 of a 700mb file, never uses more than 25% cpu and always uses 1mb of ram.

In comparison the above machine code takes 11 seconds, sometimes uses 100% cpu and uses 700mb of ram

Please help!
Back to top
View user's profile Send private message
olfen



Joined: 04 Jun 2005
Posts: 100
Location: Stuttgart, Germany
PostPosted: Sat Nov 03, 2007 1:14 pm    Post subject: Reply with quote
The C function would have to be extended to accept a start value as an additional parameter, so it could be passed on between calls that operate on chunks of data.
But I'm not sure, if it can easily be done for binary files, as I haven't tested if it's possible to read chunks of data from a file, possibly using WinAPI (ReadFile in conjunction with SetFilePointer).

P.S.: I just looked at the source of the C++ Class: CSHA1 on the site you posted, and found, that it is using fopen et al from stdio.h for reading data. And I guess rehash is doing it the same way.
So for compareable performance, the C function would have to be rewritten, also implementing file i/o in machine code (using WinAPI would still require slow AHK loops).

P.P.S.: After a bit of testing I think it would require a lot of work to extract and integrate the file I/O functions from msvcrt.dll into machine code, that can be called from AHK. Maybe it isn't possible at all.
IMO the best option is to create a crc32.dll that is DllCall'ed the conventional way, when processing large files.
Back to top
View user's profile Send private message
tic



Joined: 22 Apr 2007
Posts: 1502
PostPosted: Sun Nov 04, 2007 6:48 pm    Post subject: Reply with quote
Thank you for having a look olfen.

I am very interested in this, not specificially the crc32, but perhaps more with md5. I think it would be really good to have some way in ahk that could compare files to check whether they are identical (but make it the fastest way possible and not use ridicuolously large memory + cpu to do it). Unfortunately I do not have nearly the knowledge to understand what is going on in that C code and steal it to a dll or any other way.
Back to top
View user's profile Send private message
Laszlo



Joined: 14 Feb 2005
Posts: 4668
Location: Boulder, CO
PostPosted: Mon Nov 12, 2007 1:06 am    Post subject: Reply with quote
Here is a version of Olfen’s CRC32 function, which is somewhat shorter, inverts the bits of the result and can be continued, when only sections of the data are available at a time (e.g. when computing the CRC of a large file, read piece-by-piece). The corresponding C code is
Code:
typedef unsigned long uint;

void CRC32_Init(uint* table) { // uint table[256]
  uint i, j, poly = 0xEDB88320, CRC;

  for(i = 0; i < 256; i++) {
    CRC = i;
    for(j = 0; j < 8; j++)
      if(CRC & 1)
        CRC = (CRC >> 1) ^ poly;
      else
        CRC >>= 1;
    table[i] = CRC;
  }
}

uint CRC32(unsigned char* buffer, uint len, uint crc32val, uint* table) { // init: crc32val = 0xFFFFFFFF
  uint i;
  for (i = 0; i < len; i++)
    crc32val = table[(crc32val ^ buffer[i]) & 255] ^ (crc32val >> 8);
  return ~crc32val;
}

Their machine code is converted to AHK functions by
Code:
MCode(CRC32_Init,"33c06a088bc85af6c101740ad1e981f12083b8edeb02d1e94a75ec8b542404890c82403d0001000072d8c3")
MCode(CRC32,"558bec33c039450c7627568b4d080fb60c08334d108b55108b751481e1ff000000c1ea0833148e403b450c89551072db5e8b4510f7d05dc3")


When the CRC of a string or a binary buffer is computed, crc32val=-1:
Code:
a = ABCDEFGHIJKLMNOPQRSTUVWXYZ ; CRC = 0xabf77822
MsgBox % DllCall(&CRC32, "uint",&a, "uint",StrLen(a), "int",-1, "uint",&CRC32LookupTable, "cdecl uint")


The call is the same, when the first section of long data is processed. Subsequent sections need modified calls: the third parameter be the bitwise-not of the previous CRC value:
Code:
a = clip
b = board ; CRC("clipboard") = 0x8fdbc496
c := DllCall(&CRC32, "uint",&a, "uint",StrLen(a), "int",-1, "uint",&CRC32LookupTable, "cdecl uint")
MsgBox % DllCall(&CRC32, "uint",&b, "uint",StrLen(b), "int",~c, "uint",&CRC32LookupTable, "cdecl uint")


This version is very fast, so it can be used to compare large files. First compare the size of the files. If they are equal, compare their CRC values. Because the CRC’s are only 32 bits long, there is a 50% chance that among 65,536 random files of the same size, two will be falsely labeled as equal. We don’t normally have that many files of the same size. If you do, you can compute more CRC values for the files, which possibly equal to another one. These CRC’s should use different starting values (like -1, 0, 0x55555555).
Back to top
View user's profile Send private message
Laszlo



Joined: 14 Feb 2005
Posts: 4668
Location: Boulder, CO
PostPosted: Mon Nov 12, 2007 2:49 am    Post subject: Reply with quote
Here is a wrapper for the CRC32 function, to make it easier to use, and some test cases.
Code:
SetFormat Integer, HEX
a = ABCDEFGHIJKLMNOPQRSTUVWXYZ
MsgBox % CRC32(a)               ; abf77822

a = clip
b = board
c:= CRC32(a)
MsgBox % CRC32(b,0,~c)          ; 8fdbc496

MsgBox % CRC32(a:="1234567890") ; 261daee5

FileRead a, %A_AhkPath%         ; Version 1.0.47.04
FileGetSize size, %A_AhkPath%
MsgBox % CRC32(a,size)          ; 19df9bd0

; ---Include lines below---
CRC32(ByRef Buffer, Bytes=0, Start=-1) {
   Static CRC32, CRC32_Init, CRC32LookupTable
   If (CRC32 = "") {
      MCode(CRC32_Init,"33c06a088bc85af6c101740ad1e981f12083b8edeb02d1e94a75ec8b542404890c82403d0001000072d8c3")
      MCode(CRC32,"558bec33c039450c7627568b4d080fb60c08334d108b55108b751481e1ff000000c1ea0833148e403b450c89551072db5e8b4510f7d05dc3")
      VarSetCapacity(CRC32LookupTable, 256*4)
      DllCall(&CRC32_Init, "uint",&CRC32LookupTable, "cdecl")
   }
   If Bytes <= 0
      Bytes := StrLen(Buffer)
   Return DllCall(&CRC32, "uint",&Buffer, "uint",Bytes, "int",Start, "uint",&CRC32LookupTable, "cdecl uint")
}

MCode(ByRef code, hex) { ; allocate memory and write Machine Code there
   VarSetCapacity(code,StrLen(hex)//2)
   Loop % StrLen(hex)//2
      NumPut("0x" . SubStr(hex,2*A_Index-1,2), code, A_Index-1, "Char")
}


The function CRC32 has three parameters.
- The first one is the name of a buffer, which can contain binary data.
- The second parameter is the length of the data in bytes. If omitted or not positive, Strlen(Buffer) is used internally.
- The 3rd parameter is used for continuing the CRC computation for second or later data sections. If omitted, -1 is used, the standard initial value for CRC32. If an earlier CRC operation is to be continued (which returned C), put here ~C. If a different CRC is needed than the standard CRC-32 (e.g. to resolve collisions), you can use any 32 bit integer for initialization.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    AutoHotkey Community Forum Index -> Scripts & Functions All times are GMT
Goto page Previous  1, 2, 3 ... 6, 7, 8 ... 18, 19, 20  Next
Page 7 of 20

 
Jump to:  
You can post new topics in this forum
You can reply to topics in this forum


Powered by phpBB © 2001, 2005 phpBB Group