Question on How EntropyBottleneck's Parameters are Updated

[Will0x6c5f]

Hi there,
I'm currently working on this paper : Variational image compression with a scale hyperprior and find this implementation (bmshj2018). It seems to me that this model use a MLP(or something like that) to approximate the 'rate' part of the rate-distortion loss according to appendix 6.1 and 6.2 in the original paper.
However, I didn't find the corresponding loss to update this EntropyBottleneck's parameter (which should exist in order to optimize the matrices and biases inside it), i. e. There is no measure on how well this surrogate network is approximating the true distribution.
So how does this module update itself ? If the only loss affecting EntropyBottleneck is the 'rate' loss, how can the network ensure it will converge to the true distribution instead of returning all zero ? Is this related to the math property of that particular kind of "MLP"? I wonder if my understanding is correct or I missed something in the paper.

Thank you for your time

[YodaEmbedding]

EntropyBottleneck.forward computes the likelihoods that are used for the rate loss:

CompressAI/compressai/entropy_models/entropy_models.py

Line 501 in 677506b

likelihood = self._likelihood(outputs)

These likelihoods are computed using an input tensor and the matrices/biases:

CompressAI/compressai/entropy_models/entropy_models.py

Lines 436 to 469 in 677506b

	def _logits_cumulative(self, inputs: Tensor, stop_gradient: bool) -> Tensor:
	# TorchScript not yet working (nn.Mmodule indexing not supported)
	logits = inputs
	for i in range(len(self.filters) + 1):
	matrix = getattr(self, f"_matrix{i:d}")
	if stop_gradient:
	matrix = matrix.detach()
	logits = torch.matmul(F.softplus(matrix), logits)
	bias = getattr(self, f"_bias{i:d}")
	if stop_gradient:
	bias = bias.detach()
	logits += bias
	if i < len(self.filters):
	factor = getattr(self, f"_factor{i:d}")
	if stop_gradient:
	factor = factor.detach()
	logits += torch.tanh(factor) * torch.tanh(logits)
	return logits
	@torch.jit.unused
	def _likelihood(self, inputs: Tensor) -> Tensor:
	half = float(0.5)
	v0 = inputs - half
	v1 = inputs + half
	lower = self._logits_cumulative(v0, stop_gradient=False)
	upper = self._logits_cumulative(v1, stop_gradient=False)
	sign = -torch.sign(lower + upper)
	sign = sign.detach()
	likelihood = torch.abs(
	torch.sigmoid(sign * upper) - torch.sigmoid(sign * lower)
	)
	return likelihood

The main "net" optimizer includes the matrices/biases in the list of parameters it optimizes. Therefore, these parameters are updated in such a way that they minimize the rate loss.

[Will0x6c5f]

Thank you for your reply !

But the problem remains... I'm aware that the matrices/biases are updated to minimize the rate loss. However, these matrices/biases are meant to approximate the actual distribution such that the _likelihood function take a value as input and return the approximate probability/likelihood of that value.
So the problem really is about how to ensure these matrices/biases will produce the approximately correct likelihoods only minimizing the rate loss ? After all, this rate loss can easily converge to zero as long as all the matrices/biases are set to zero, but that will be no where around the true likelihood that we expect to get.

Please correct me if I missed something.

[YodaEmbedding]

The distortion term in L = R + λD ensures that R does not go to 0. The EntropyBottleneck does not modify the input tensor data in any way, other than quantization noise. Thus, it has nearly no effect on D.

The rate R equals the cross-entropy H(p; q) between the true distribution p of the input tensors and the approximating distribution q. When p = q, the cross-entropy (and therefore, the rate) is minimized. If q is anything else, the rate loss is not at a minimum w.r.t. the target latent input distribution p. Since the optimizer minimizes the loss, it will do its best to ensure q tends towards p by appropriately adjusting the matrices/biases.

[Will0x6c5f]

I think I might get the rough idea of what's going on... but to ensure my understanding is correct, I'd like to confirm a few more things:
Generally R is an upper bound of the actual bit rate of that latent code... ?
And only at the best situation with optimal matrices/biases this R will give the actual entropy of one latent code... ?

[YodaEmbedding]

R is an upper bound on rate because it is always possible that the input may be further compressed via some other distribution q' that is closer to p. When p = q, the current entropy model cannot get any better by adjusting q (which is a deterministic function of the matrices/biases).

Note that the EntropyBottleneck entropy model assumes that all symbols within the ith channel are i.i.d. and are drawn from the same distribution q_i. In reality, this assumption is not accurate since there are clear visible relationships between these variables when we visualize the feature maps. So, there exists a better entropy model for our data that is capable of giving a tighter upper bound on R. The ideal entropy model would allow us to compress to exactly the true entropy R = H(y).

[Will0x6c5f]

Got it ! Thank you soooo much for answering my question ! !
Appreciate your sharing !