Implement bit and byte for information and data #419

BrandonConder · 2021-01-31T20:20:00Z

Create information. Define Byte as basic unit for data storage. Include data rates. Modify barn (b -> bn) and Bel (B -> bel) to accommodate the new definitions.

Closes #173.
Mostly addresses #330.
Does not include e.g. methods for converting between bits and entropy, this is probably reasonable since most users will think of bits as data (e.g. storage capacity, transfer rates) rather than information.

Personal preferences:

I don't think this belongs in a separate units package because bit and byte are common and conflict with the existing definitions of barn and Bel.
Less common units are not included: nibble, 1024 multiples (kibibytes, mebibytes, etc.). A user could easily implement these.
Baud not defined because bits per symbol is not standardized.

Quirks:

I had to manually implement the SI prefixes for Bytes to avoid deci-bytes (dB), which is a meaningless unit and a conflict with Decibel.
ZettaByte and YottaByte were not included because the literal 8_000_000_000_000_000_000_000 undergoes type promotion to Float64 and causes a rounding error.
- It might be possible to implement this using Int128(8_000_000_000_000_000_000_000)
- If there were a way to use the Unitful prefix system excluding deciByte, Unitful handles these types greacefully (e.g. 1Zm/Em == 1000 returns true)
- Behavior with the 32-bit Julia binary needs review, to ensure no units promote to Float32

I've included tests for the manual SI prefixes.

Create information. Define Byte as basic unit for data storage. Include data rates. Modify barn (b -> bn) and Bel (B -> bel) to accommodate the new definitions.

Test that manually-added prefixes match expected SI prefixes.

BrandonConder · 2021-01-31T21:22:41Z

It looks like the Float32 problem is realized on x86. How to proceed? I can leave out SI prefixes and let users decide if they want to declare kB, MB, GB, etc.

I don't see any other units doing this, but can I declare e.g.

@unit      TB   "TB"     TeraByte          1_000GB    false

Alternatively I could try to rewrite the @Unit macro such that the input tf can accept a flag for "prefixes > 1 only", but that seems like a pretty big overhaul for this edge case.

giordano · 2021-02-01T00:32:50Z

src/pkgdefaults.jl

@@ -113,7 +115,7 @@ end
 # The hectare is used more frequently than any other power-of-ten of an are.
 @unit a      "a"        Are         100m^2                  false
 const ha = Unitful.FreeUnits{(Unitful.Unit{:Are, 𝐋^2}(2, 1//1),), 𝐋^2}()
-@unit b      "b"        Barn        100fm^2                 true
+@unit bn     "bn"       Barn        100fm^2                 true


This is breaking

giordano · 2021-02-01T00:33:20Z

src/pkgdefaults.jl

 #########
 # Logarithmic scales and units

 @logscale dB    "dB"       Decibel      10      10      false
-@logscale B     "B"        Bel          10      1       false
+@logscale bel   "Bel"      Bel          10      1       false


And this one as well

BrandonConder · 2021-02-01T05:35:55Z

Is backward-compatibility-breaking prohibited? In my field Bel is never used (only decibel), and I don't know how common or significant Barn is, so I assumed bit and Byte were more common definitions of those abbreviations.

giordano · 2021-02-01T22:50:49Z

We can have a breaking change, but need to do a breaking release (i.e., go to v2.0), which would cause some friction in the ecosystem of packages relying on Unitful, for little benefit

sostock · 2021-02-12T08:18:01Z

I would suggest to use Byte and bit instead of B and b, that way it isn’t breaking.

mcabbott · 2021-04-13T02:34:26Z

src/pkgdefaults.jl

@@ -9,6 +9,7 @@
 @dimension 𝚯 "𝚯" Temperature    # This one is \bfTheta
 @dimension 𝐉 "𝐉" Luminosity
 @dimension 𝐍 "𝐍" Amount
+@dimension 𝐛 "𝐛" Information


Should this be a dimension? I thought that nat is, like rad, a pure number -- it's just some p*log(p) where p is a probability. And bits, like degrees etc, are multiples of that.

Suggested change

@dimension 𝐛 "𝐛" Information

mcabbott · 2021-04-13T02:49:04Z

src/pkgdefaults.jl

+# Data
+@unit      B    "B"        Byte              8b          false


If the definitions were like this, would it still convert MB to GB as integers, without introducing the log(2)?

Suggested change

# Data

@unit B "B" Byte 8b false

# Information

@unit nat "nat" Nat 1 false

@unit bit "bit" Bit log(2)*u"nat" false

@unit B "B" Byte 8bit false

mcabbott · 2021-04-13T03:13:56Z

src/pkgdefaults.jl

+@unit      kB   "kB"       KiloByte          8_000b      false
+@unit      MB   "MB"       MegaByte          8_000_000b     false
+@unit      GB   "GB"       GigaByte          8_000_000_000b     false


Should the base 2 ones also be included?

Suggested change

@unit kB "kB" KiloByte 8_000b false

@unit MB "MB" MegaByte 8_000_000b false

@unit GB "GB" GigaByte 8_000_000_000b false

@unit kB "kB" KiloByte 8_000b false

@unit kiB "kiB" KibiByte 8_192b false

@unit MB "MB" MegaByte 8_000_000b false

@unit MiB "MiB" MebiByte 8_388_608b false

@unit GB "GiB" GigaByte 8_000_000_000b false

@unit GiB "GiB" GibiByte 8_589_934_592b false

"GiB" appears twice in the suggested change, looks like a typo.

rico-cl · 2021-11-21T09:46:03Z

Regarding the question of adding another dimension for the desired units, I propose to go with dimensionless units, since

Unitful seem to try and adhere to the SI which doesn't provide such a dimension
how to consistently handle units that are dimensionless in the framework of SI as of now is still an unsolved (or at least debated) problem in the respective research communities as far as I understand, so I wouldn't expect Unitful to come up with a solution itself
the units in question can be used to measure different things: amount of data, amount of storage, informational content of data (as denoted by Shannon entropy) - so even if one would like to assign a dimension to them, Information might not even be the most meaningful choice

mcabbott · 2022-04-01T12:18:37Z

I was going to say that these points all apply to angles, where radians are also pure numbers. But it turns out they have in fact been blessed by SI.

Nevertheless, it is not an accident that storage and information are measured in the same units. There are mistakes you can make which won't be caught by checking the units, but there are also such mistakes when measuring both height and distance in metres.

filchristou · 2022-04-09T09:23:29Z

I am in need of such a unit in order to handle data rates in networking problems.
I think it's worth having a specific unit for this case because there is a some confusion in my field when someone talks about Mbits/sec or MBytes/sec. A package that handles this transparently will be great.

singularitti · 2024-12-26T22:49:32Z

Hope it's not off-topic: there is a relevant package https://github.com/uriele/UnitfulData.jl

BrandonConder added 2 commits January 31, 2021 10:28

Implement bit as information, byte as data

a7d1679

Create information. Define Byte as basic unit for data storage. Include data rates. Modify barn (b -> bn) and Bel (B -> bel) to accommodate the new definitions.

Test sets for SI prefixed bytes

c7dd552

Test that manually-added prefixes match expected SI prefixes.

giordano reviewed Feb 1, 2021

View reviewed changes

mcabbott reviewed Apr 13, 2021

View reviewed changes

sostock mentioned this pull request Oct 6, 2021

Data sizes #486

Closed

sostock added the new units adding new units/dimensions/constants to this package label Dec 6, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement bit and byte for information and data #419

Implement bit and byte for information and data #419

BrandonConder commented Jan 31, 2021

BrandonConder commented Jan 31, 2021 •

edited

Loading

giordano Feb 1, 2021

giordano Feb 1, 2021

BrandonConder commented Feb 1, 2021

giordano commented Feb 1, 2021

sostock commented Feb 12, 2021

mcabbott Apr 13, 2021 •

edited

Loading

mcabbott Apr 13, 2021

mcabbott Apr 13, 2021 •

edited

Loading

adkabo Apr 1, 2022

rico-cl commented Nov 21, 2021

mcabbott commented Apr 1, 2022

filchristou commented Apr 9, 2022 •

edited

Loading

singularitti commented Dec 26, 2024

-# Data
-@unit      B    "B"        Byte              8b          false
+# Information
+@unit    nat    "nat"      Nat                1            false
+@unit    bit    "bit"      Bit          log(2)*u"nat"      false
+@unit      B    "B"        Byte              8bit          false

Implement bit and byte for information and data #419

Are you sure you want to change the base?

Implement bit and byte for information and data #419

Conversation

BrandonConder commented Jan 31, 2021

BrandonConder commented Jan 31, 2021 • edited Loading

giordano Feb 1, 2021

Choose a reason for hiding this comment

giordano Feb 1, 2021

Choose a reason for hiding this comment

BrandonConder commented Feb 1, 2021

giordano commented Feb 1, 2021

sostock commented Feb 12, 2021

mcabbott Apr 13, 2021 • edited Loading

Choose a reason for hiding this comment

mcabbott Apr 13, 2021

Choose a reason for hiding this comment

mcabbott Apr 13, 2021 • edited Loading

Choose a reason for hiding this comment

adkabo Apr 1, 2022

Choose a reason for hiding this comment

rico-cl commented Nov 21, 2021

mcabbott commented Apr 1, 2022

filchristou commented Apr 9, 2022 • edited Loading

singularitti commented Dec 26, 2024

BrandonConder commented Jan 31, 2021 •

edited

Loading

mcabbott Apr 13, 2021 •

edited

Loading

mcabbott Apr 13, 2021 •

edited

Loading

filchristou commented Apr 9, 2022 •

edited

Loading