BitGet/BitPut (NumGet/NumPut for bits)

Post your working scripts, libraries and tools
User avatar
jeeswg
Posts: 6902
Joined: 19 Dec 2016, 01:58
Location: UK

BitGet/BitPut (NumGet/NumPut for bits)

07 Dec 2018, 02:26

Some code for reading/writing bits from binary data. Do make any suggestions or notify of any issues or related scripts.

Code: Select all

q:: ;test bit put and bit get
vSizeBits := 3
oArray := [0, 2**vSizeBits-1]

VarSetCapacity(vData, 100000, 0)
vOffset := 0, vNext := 0
Loop, % 10*8
{
	vNext++
	if (vNext > oArray.Length())
		vNext := 1
	JEE_BitPut(oArray[vNext], &vData, vOffset, vSizeBits)
	vOffset += vSizeBits
}

vOutput := ""
Loop, % 10*8
	vOutput .= JEE_BitGet(&vData, A_Index-1, 1)
vOutput := RegExReplace(vOutput, ".{8}", "$0 ")
MsgBox, % vOutput

vOutput .= "`r`n`r`n"
vOffset := 0
Loop, % (10*8) / vSizeBits
	vOutput .= JEE_BitGet(&vData, vOffset, vSizeBits) " "
	, vOffset += vSizeBits
MsgBox, % vOutput

Clipboard := vOutput
return

;==================================================

;where vBit is zero/a positive integer and is from left-to-right
JEE_BitGetSimple(vAddr, vBit)
{
	local
	if (vBit > 7)
		vAddr += vBit // 8, vBit := Mod(vBit, 8)
	vNum := NumGet(vAddr+0, 0, "UChar")
	return !!(vNum & (1 << 7-vBit))
}

;==================================================

;where vNum is 0 or 1
;where vBit is zero/a positive integer and is from left-to-right
JEE_BitPutSimple(vNum, vAddr, vBit)
{
	local
	if (vBit > 7)
		vAddr += vBit // 8, vBit := Mod(vBit, 8)
	vNum2 := NumGet(vAddr+0, 0, "UChar")
	if vNum
		NumPut(vNum2 | (1 << 7-vBit), vAddr+0, 0, "UChar")
	else
		NumPut(vNum2 & ~(1 << 7-vBit), vAddr+0, 0, "UChar")
}

;==================================================

;where vBit is zero/a positive integer and is from left-to-right
;where vSizeBits is an integer between 0 and 8
;note: cannot handle overlaps into adjacent bytes
JEE_BitGet(vAddr, vBit, vSizeBits:=1)
{
	local
	if (vBit > 7)
		vAddr += vBit // 8, vBit := Mod(vBit, 8)
	if (vSizeBits < 0) || (vSizeBits > 8)
		return
	else if !vSizeBits
		return 0
	else if (vBit = 0) && (vSizeBits = 8)
		return NumGet(vAddr+0, 0, "UChar")

	;handle if bits split across 2 bytes
	vDiff := vSizeBits - (8-vBit) ;vDiff if positive is the number of bits after the byte divide
	if (vDiff > 0)
	{
		vExtra := JEE_BitGet(vAddr, vBit, vSizeBits-vDiff) << vDiff
		vAddr += 1, vBit := 0, vSizeBits := vDiff
	}
	else
		vExtra := 0

	vNum := NumGet(vAddr+0, 0, "UChar")
	vMask := (1 << vSizeBits) - 1
	return vExtra + ((vNum & (vMask << 8-vBit-vSizeBits)) >> 8-vBit-vSizeBits)
}

;==================================================

;where vBit is zero/a positive integer and is from left-to-right
;where vSizeBits is an integer between 0 and 8
JEE_BitPut(vNum, vAddr, vBit, vSizeBits:=1)
{
	local
	if (vBit > 7)
		vAddr += vBit // 8, vBit := Mod(vBit, 8)
	if (vSizeBits <= 0) || (vSizeBits > 8)
		return
	else if (vBit = 0) && (vSizeBits = 8)
		return NumPut(vNum, vAddr+0, 0, "UChar")

	;handle if bits split across 2 bytes
	vDiff := vSizeBits - (8-vBit) ;vDiff if positive is the number of bits after the byte divide
	if (vDiff > 0)
	{
		JEE_BitPut(vNum >> vDiff, vAddr, vBit, vSizeBits-vDiff)
		vNum &= (1 << vDiff) - 1
		, vAddr += 1, vBit := 0, vSizeBits := vDiff
	}

	vNum2 := NumGet(vAddr+0, 0, "UChar")
	vMask := (1 << vSizeBits) - 1
	vNum &= vMask
	NumPut((vNum2 & ~(vMask << 8-vBit-vSizeBits)) | (vNum << 8-vBit-vSizeBits), vAddr+0, 0, "UChar")
}

;==================================================
Last edited by jeeswg on 09 Dec 2018, 20:51, edited 1 time in total.
homepage | tutorials | wish list | fun threads | donate
WARNING: copy your posts/messages before hitting Submit as you may lose them due to CAPTCHA
Helgef
Posts: 4440
Joined: 17 Jul 2016, 01:02
Contact:

Re: BitGet/BitPut (NumGet/NumPut for bits)

08 Dec 2018, 14:32

Hello jeeswg.

I tried your simple functions, it seems reversed to me. I guessed that the vBit parameter would tell which bit to get, so I expect the first bit (that is the least significant one) to be extracted when vBit := 0, but consider,

Code: Select all

; v2
msgbox jee_bitgetsimple(&x:=1, 7) ; 1
msgbox jee_bitgetsimple(&x:= 1 << 7 , 0) ; 1
I wrote similar functions, which supports reading / writing from / to any bits of the usual numputget integer types.

Code: Select all

; v2
/*
	bitsput
	
	Params:
		adr, address to the number whose bits to manipulate
		n, the number to write
		w, the bit width of the number n
		o, bit offset to write to, 0 writes to the first bit and so on...
		tadr, the type of the number at adr
	
	Description:
		Writes the w first bits of n to the (o+1):th bit of the number of type tadr at address adr.
*/
bitsput(adr, n, w := 1, o := 0, tadr := 'uint'){
	local msk := ~(-1 << o) | (-1 << o + w - 1 << 1)
	return numput( (numget(adr, tadr) & msk) | ((n << o) & ~msk), adr, tadr )
}
/*
	bitsget
	
	Params:
		adr, address to the number whose bits read from
		w, the bit width of the number to get
		o, bit offset to read from, 0 reads to the first bit and so on...
		tadr, the type of the number at adr
		tval, the type of the number retrieving.
	
	Description:
		Reads the w first bits from the (o+1):th bit of the number of type tadr at address adr. 
		The read number is interpreted as being of type tval.
*/
bitsget(adr, w := 1, o := 0, tadr := 'uint', tval := 'uint'){
	return numget( &w := ( numget( adr, 'u' . ltrim(tadr,'uU') ) >> o ) & ~( -1 << w - 1 << 1 ) , tval)	
}
/*
	bitput
	
	Params:
		adr, address to the number whose bit to manipulate
		n, the number to write
		o, bit offset to write to, 0 writes to the first bit and so on...
		tadr, the type of the number at adr
	
	Description:
		Writes the first bit of n to the (o+1):th bit of the number of type tadr at address adr.
*/
bitput(adr, n, o := 0, tadr := 'uint'){
	return bitsput(adr, n, 1, o, tadr)
}
/*
	bitget
	
	Params:
		adr, address to the number whose bit to read from
		o, bit offset to read from, 0 reads to the first bit and so on...
		tadr, the type of the number at adr
	
	Description:
		Reads the bit from the (o+1):th bit of the number of type tadr at address adr. 
*/
bitget(adr, o := 0, tadr := 'uint'){
	return bitsget(adr, 1, o, tadr, 'uchar')
}
I leave it as an exercise for the interested reader to work out the details, eg, why this doesn't work in v1, and why the sign bit needs to be cleared.
Cheers.
Last edited by Helgef on 13 Dec 2018, 04:50, edited 3 times in total.
User avatar
jeeswg
Posts: 6902
Joined: 19 Dec 2016, 01:58
Location: UK

Re: BitGet/BitPut (NumGet/NumPut for bits)

09 Dec 2018, 20:52

- @Helgef: Thanks for testing and for the interesting code.
- I've updated the comments to say 'left-to-right'.
- One reason I posted is that there could be some elegant bit hacks that I might have missed.
- Had you already written functions like this? Did you have any example code from another language? (The functions are only semi-useful to me, but I thought on balance they would be worth writing.)
- Perhaps the AHK v1 issue you had in mind is: ~ (bitwise-not) works differently between AHK v1/v2 (re. 32-bit/64-bit integers), although ^ -1 (bitwise-exclusive-or) can be used for two-way compatibility.

Code: Select all

;AHK v1:
MsgBox, % ~0 ;4294967295 ;differs from AHK v2
MsgBox, % 0 ^ -1 ;-1
MsgBox, % Format("0x{:X}", ~0) ;0xFFFFFFFF ;differs from AHK v2
MsgBox, % Format("0x{:X}", 0 ^ -1) ;0xFFFFFFFFFFFFFFFF

;AHK v2:
MsgBox(~0) ;-1
MsgBox(0 ^ -1) ;-1
MsgBox(Format("0x{:X}", ~0)) ;0xFFFFFFFFFFFFFFFF
MsgBox(Format("0x{:X}", 0 ^ -1)) ;0xFFFFFFFFFFFFFFFF

;AHK v2:
MsgBox(Format("0x{:X}", 1<<63)) ;0x8000000000000000 ;the sign bit
MsgBox(Format("0x{:X}", ~(1<<63))) ;0x7FFFFFFFFFFFFFFF
- When you do bitshift right on a negative number, that's equivalent to shifting all the bits right by 1, but then setting the sign bit to 1. Perhaps that's what you had in mind re. the sign bit.
- It would be good if you could provide some answers to your questions, at least in a spoiler. Cheers.

[EDIT:] Is this the expected result for bitsput?

Code: Select all

;testing bitsput
n := 3, w := 2, o := 3
MsgBox(255 & ~(((((-1 << o ) & ~(1<<63) )>> 64 - w) | (1<<w-1)) << o )) | (n << o) ;231 = 0b11100111 (expected 255)

;testing bitsget
w := 2, o := 3
MsgBox(( (255 >> o) << 63 - w & ~(1 << 63) ) >> 63 - w) ;3
w := 4, o := 2
MsgBox(( (255 >> o) << 63 - w & ~(1 << 63) ) >> 63 - w) ;15
homepage | tutorials | wish list | fun threads | donate
WARNING: copy your posts/messages before hitting Submit as you may lose them due to CAPTCHA
Helgef
Posts: 4440
Joined: 17 Jul 2016, 01:02
Contact:

Re: BitGet/BitPut (NumGet/NumPut for bits)

10 Dec 2018, 02:55

Had you already written functions like this? Did you have any example code from another language? (The functions are only semi-useful to me, but I thought on balance they would be worth writing.)
I wrote them in response to this thread. Semi-useful, agree.
~ (bitwise-not) works differently between AHK v1/v2 (re. 32-bit/64-bit integers), although ^ -1 (bitwise-exclusive-or) can be used for two-way compatibility.
Yes. x ^ -1 vs ~x is forward compatible, and ~x & 0xffffffff is backwards compatible.

Is this the expected result...
Yes.
...for bitsput?
but you are not calling bitsput, and you are not using v2 either ;)
v2.0-a095-9f724c5 wrote:Changed math functions and operators to throw an exception instead of returning "" for errors.
Cheers.
User avatar
jeeswg
Posts: 6902
Joined: 19 Dec 2016, 01:58
Location: UK

Re: BitGet/BitPut (NumGet/NumPut for bits)

10 Dec 2018, 21:04

- Something may have gone wrong in the test based on bitsput. I was using AHK v2, it gave a result and then crashed!?
- Anyhow I've tried to look at it again below. It might be useful if you could write some simplified (split up) versions of your functions with some comments. They're very hard to follow. Thanks.

Code: Select all

n := 0, w := 2, o := 3
Y := 255
A := (-1 << o) & ~(1<<63)
B := A >> 64 - w
MsgBox(Format("0x{:X}", A) " " B " " (1<<w-1))
C := (B | (1<<w-1)) << o
Z := (Y & ~C) | (n << o)
homepage | tutorials | wish list | fun threads | donate
WARNING: copy your posts/messages before hitting Submit as you may lose them due to CAPTCHA
Helgef
Posts: 4440
Joined: 17 Jul 2016, 01:02
Contact:

Re: BitGet/BitPut (NumGet/NumPut for bits)

11 Dec 2018, 02:34

- Something may have gone wrong in the test based on bitsput. I was using AHK v2, it gave a result and then crashed!?
It doesn't crash, it throws an exception, due to type mismatch,

Code: Select all

('OK' | (n << o))
User avatar
jeeswg
Posts: 6902
Joined: 19 Dec 2016, 01:58
Location: UK

Re: BitGet/BitPut (NumGet/NumPut for bits)

11 Dec 2018, 06:32

Yes but if everything is numeric, what's causing the type mismatch!?
homepage | tutorials | wish list | fun threads | donate
WARNING: copy your posts/messages before hitting Submit as you may lose them due to CAPTCHA
Helgef
Posts: 4440
Joined: 17 Jul 2016, 01:02
Contact:

Re: BitGet/BitPut (NumGet/NumPut for bits)

11 Dec 2018, 06:42

Yes but if everything is numeric, what's causing the type mismatch!?
Everything isn't numeric,

Code: Select all

msgbox_result := msgbox()
msgbox msgbox_result
(msgbox_result | (n << o))
User avatar
jeeswg
Posts: 6902
Joined: 19 Dec 2016, 01:58
Location: UK

Re: BitGet/BitPut (NumGet/NumPut for bits)

11 Dec 2018, 06:51

What code written by you or me (above the 'OK' example) uses a string in a calculation?
[EDIT:] I'm not sure that any errors were type-related. There was a missing parentheses issue.

Code: Select all

n := 3, w := 2, o := 3
MsgBox((255 & ~(((((-1 << o ) & ~(1<<63) )>> 64 - w) | (1<<w-1)) << o )) | (n << o)) ;works
MsgBox(255 & ~(((((-1 << o ) & ~(1<<63) )>> 64 - w) | (1<<w-1)) << o )) | (n << o) ;error
homepage | tutorials | wish list | fun threads | donate
WARNING: copy your posts/messages before hitting Submit as you may lose them due to CAPTCHA
Helgef
Posts: 4440
Joined: 17 Jul 2016, 01:02
Contact:

Re: BitGet/BitPut (NumGet/NumPut for bits)

11 Dec 2018, 09:11

I'm not sure that any errors were type-related
:facepalm:
There was a missing parentheses issue.
Yeah the missing parentheses/space in your code causes the type mismatch. :morebeard:
User avatar
jeeswg
Posts: 6902
Joined: 19 Dec 2016, 01:58
Location: UK

Re: BitGet/BitPut (NumGet/NumPut for bits)

11 Dec 2018, 14:18

- @Helgef: I find this kind of thinking by you odd. I had a wrong number reported and then a script crash, the issue was parentheses, that was the *real* cause. If you tell me that the problem is something to do with types, well yes it is, sort of, but that's more of a consequence of the real cause, and not the real cause itself, it's less helpful for solving the problem. Ultimately it took me more time to fix the problem because you said it was to do with types, than if you'd said nothing.
- Ultimately all these posts since I've posted my original script are because you wrote some cryptic code with no explanation, and I was trying to reveal the logic of it to myself and others. That's fine, but I'm in a situation at the moment where I really don't want to spend any surplus time on anything. I'd really appreciate if you could attempt to explain your code more fully.
homepage | tutorials | wish list | fun threads | donate
WARNING: copy your posts/messages before hitting Submit as you may lose them due to CAPTCHA
Helgef
Posts: 4440
Joined: 17 Jul 2016, 01:02
Contact:

Re: BitGet/BitPut (NumGet/NumPut for bits)

12 Dec 2018, 06:10

I didn't say the root problem was a type isssue, I told you to use v2 because that would give you the error message, from there I thought it would be trivial to find the root cause, I am amazed that I was wrong and that it was such a puzzle for you.
Ultimately it took me more time to fix the problem because you said it was to do with types, than if you'd said nothing.
I have no idea why you even cut out the code, and modify it, and then report there is a problem with the original function without actually first checking if the original function presents the same issue. That is were you wasted your time, don't blame me for not debugging your code for you and feeding it to you with a spoon.
I really don't want to spend any surplus time on anything. I'd really appreciate if you could attempt to explain your code more fully.
I would really appreciate if anyone interested makes an effort for themselves, so I don't have to spend any surplus time on doing it for them.
you wrote some cryptic code with no explanation
Let's compare,
helgef:

Code: Select all

/*
	bitsput
	
	Params:
		adr, address to the number whose bits to manipulate
		n, the number to write
		w, the bit width of the number n
		o, bit offset to write to, 0 writes to the first bit and so on...
		tadr, the type of the number at adr
	
	Description:
		Writes the w first bits of n to the (o+1):th bit of the number of type tadr at address adr.
*/
bitsput(adr, n, w := 1, o := 0, tadr := 'uint'){
	return numput( (numget(adr, tadr) & ~(((((-1 << o ) & ~(1<<63) )>> 64 - w) | (1<<w-1)) << o )) | (n << o), adr, tadr )
}
jeeswg:

Code: Select all

;where vBit is zero/a positive integer and is from left-to-right
;where vSizeBits is an integer between 0 and 8
JEE_BitPut(vNum, vAddr, vBit, vSizeBits:=1)
{
	local
	if (vBit > 7)
		vAddr += vBit // 8, vBit := Mod(vBit, 8)
	if (vSizeBits <= 0) || (vSizeBits > 8)
		return
	else if (vBit = 0) && (vSizeBits = 8)
		return NumPut(vNum, vAddr+0, 0, "UChar")

	;handle if bits split across 2 bytes
	vDiff := vSizeBits - (8-vBit) ;vDiff if positive is the number of bits after the byte divide
	if (vDiff > 0)
	{
		JEE_BitPut(vNum >> vDiff, vAddr, vBit, vSizeBits-vDiff)
		vNum &= (1 << vDiff) - 1
		, vAddr += 1, vBit := 0, vSizeBits := vDiff
	}

	vNum2 := NumGet(vAddr+0, 0, "UChar")
	vMask := (1 << vSizeBits) - 1
	vNum &= vMask
	NumPut((vNum2 & ~(vMask << 8-vBit-vSizeBits)) | (vNum << 8-vBit-vSizeBits), vAddr+0, 0, "UChar")
}
All my parameters are well described. Your variables are not. Neither one of us describe the algorithm / logic of our code. I really thought it would be very easy for you to see that my code does new_num := (old_num & mask) | newBits, since you just had written functions doing the same thing, only in a more complicated and limited (and revered) way.
I can split the line into,

Code: Select all

bitsput(adr, n, w := 1, o := 0, tadr := 'uint'){
	vNum := numget(adr, tadr) 
	vMask := ~(((((-1 << o ) & ~(1<<63) )>> 64 - w) | (1<<w-1)) << o )
	return numput( (vNum & vMask) | (n << o), adr, tadr )
}
My mask is complicated due to the way ahk implements << >>.

Cheers.
User avatar
jeeswg
Posts: 6902
Joined: 19 Dec 2016, 01:58
Location: UK

Re: BitGet/BitPut (NumGet/NumPut for bits)

12 Dec 2018, 13:16

- What issues do you see with << or >>? Is AHK a bit nonstandard re. this?
- I'm surprised you didn't just point out the parentheses issue and answer my question!? And a facepalm emoticon ... really?
- Except I'm not surprised, I think you take the ideas of 'I must never spoonfeed' or 'give people the joy of self-discovery' too far sometimes, leading to protracted discussions. My main principles might be: 'I must save people time because they're busy', and 'I should give people the benefit of the doubt', and 'I should explain things to avoid misunderstandings'.
- The likelihood of bugs caused either by complex formulas or AHK v1/v2 differences, makes using clear explanations ('spoonfeeding' if you like) a wise approach. I would break something like this up to such simplicity that it would be demonstrably correct.
- Your split version is much clearer, but it's still pretty hard to follow the logic and to be certain that the algorithm is correct/succinct. I get the feeling that you're using an idea that can be described in one sentence, but that isn't immediately clear.

Code: Select all

vMask := ~(((((-1 << o ) & ~(1<<63) )>> 64 - w) | (1<<w-1)) << o )

o := 1, w := 6 ;middle 6 bits of a byte
vMask := ~(((((-1 << o ) & ~(1<<63) )>> 64 - w) | (1<<w-1)) << o )
vMask := ~((((0xFFFFFFFFFFFFFFC0 & 0x7FFFFFFFFFFFFFFF) >> 58) | 32) << 1)
vMask := ~(((0x7FFFFFFFFFFFFFC0 >> 58) | 32) << 1)
vMask := ~((31 | 32) << 1)
vMask := ~(63 << 1)
vMask := ~126
vMask := 0xFFFFFFFFFFFFFF81 ;0x81 = 0b10000001
homepage | tutorials | wish list | fun threads | donate
WARNING: copy your posts/messages before hitting Submit as you may lose them due to CAPTCHA
Helgef
Posts: 4440
Joined: 17 Jul 2016, 01:02
Contact:

Re: BitGet/BitPut (NumGet/NumPut for bits)

12 Dec 2018, 15:07

There is a bug in v2 I believe, it fooled me, my code might be incorrect. It's not really related to the code but to my testing methods. I will investigate further later and get back.

Jeeswg, I will consider your personal feedback.

Let's cheer up a bit :heart:.
User avatar
jeeswg
Posts: 6902
Joined: 19 Dec 2016, 01:58
Location: UK

Re: BitGet/BitPut (NumGet/NumPut for bits)

12 Dec 2018, 15:20

Haha no worries Helgef. :thumbup: I didn't think that there were necessarily any problems with your code. With tricky maths, even in my own code, I often break things down, simplifying it until I understand it 100%. (Although it's OK to squash everything together for performance in the real function.) Cheers.
homepage | tutorials | wish list | fun threads | donate
WARNING: copy your posts/messages before hitting Submit as you may lose them due to CAPTCHA

Return to “Scripts and Functions”

Who is online

Users browsing this forum: Xtra and 29 guests