Questions about details of EntropyBottleneck?

[JasonLSC]

Hi, thank you for your project. I'm trying to understand what is going on in the EntropyBottleneck but I failed to figure it out by myself. Specifically, I want to know what is the meaning of the variable "target" in the init() function and the method _logits_cumulative() ?

[YodaEmbedding]

The CDF (cumulative distribution function) is computed via

cdf(y) = sigmoid(cumulative_logits(y))

In order to ensure the sampled distribution for the lookup table used at runtime has finite support (i.e. doesn't have an infinite number of entries), we restrict the sampling region to where most of the probability mass is contained. The outer probability region has a target mass of tail_mass. Values outside the supported region are encoded using a bypass engine instead of the lookup table.

target = [
    inverse_sigmoid(tail_mass/2),
    0,
    inverse_sigmoid(1 - tail_mass/2),
]
≈ [-21, 0, 21] for tail_mass=1e-9

If we take the sigmoid of target, we get:

sigmoid(target) = [
    tail_mass/2,
    0.5,
    1 - tail_mass/2,
]
≈ [5e-10, 0.5, 1 - 5e-1] for tail_mass=1e-9

target are the desired values for cumulative_logits(quantiles). Applying sigmoid to both sides, we get that sigmoid(target) are the desired values for sigmoid(cumulative_logits(quantiles)) = cdf(quantiles). Thus, cdf(quantiles) approaches [5e-10, 0.5, 1 - 5e-10].

There are a few other earlier discussions which discuss EntropyBottleneck related things which explain more of the background details that may help. See the docs, too: https://interdigitalinc.github.io/CompressAI/models.html#compressai.models.CompressionModel.aux_loss . (I guess that description should be moved inside EntropyBottleneck directly for discoverability? I might look into documenting EntropyBottleneck more completely later.)

[JasonLSC]

Thanks! Your answer is very helpful!