feat: add bytes_sent and bytes_received as metrics

[rhatgadkar-goog]

Port of CloudSQL's GoogleCloudPlatform/cloud-sql-go-connector#856

Fixes #607

[nancynh]

Did you check that the metrics were populated in monitoring? I'm not sure if you talked about this with Eno if that'd be too much of a hassle when OpenCensus is going away

[nancynh]

Should we be getting the LastValue instead of the Sum?

[rhatgadkar-goog]

I looked at the CloudSQL Go connector's code, and it used LastValue. But I'm thinking that we should use Sum instead, because we should be recording the total number of bytes sent/received.

With LastValue, the bytes_sent and bytes_received metrics would only show the most recent number of bytes that were sent/received. Not the accumulated sum. But I need to verify this in Cloud Monitoring.

[enocom]

cc @jackwotherspoon I can't remember if we talked about this or not. Sum is probably right, but I might have missed this in your PR review.

[enocom]

We also have count -- I forget what the difference between count and sum is here, but either way it should be a cumulative metric.

[jackwotherspoon]

Yes LastValue was probably an oversight. I wonder whether Count or Sum is better...

I'm leaning towards count that way the monitoring platform (Cloud Monitoring) can decide to sum if it wants or not.

[jackwotherspoon]

Let's use Sum for now then if its the most simple

[rhatgadkar-goog]

I tested this using both Sum() and Count(). I had a client program that does 2 operations: inserts a row into a table and then reads rows from the table. So as this program gets run more times, the number of bytes sent and received should increase over time.

I began using Sum() for both bytes_sent and bytes_received. And then at different times, I changed between using Count() and Sum(). It looked like this in Cloud Monitoring (this graph is for bytes_sent, but bytes_received had a similar shaped graph too):

I'm thinking that Sum() looks more accurate, because the number of bytes sent is increasing over time.

[rhatgadkar-goog]

On second thought, I'm thinking that as this program gets run more times, the number of bytes sent will be the same each time that it's run. But the number of bytes received will increase, because the table will grow larger.

The unaggregated graph in Cloud Monitoring might do a better job of showing this. I'll try testing this again

[rhatgadkar-goog]

With Sum(), I verified that bytes_sent is constant:
. And bytes_received increases over time:

Count() has a similar graph, but was showing bytes_sent and bytes_received as less than 1/s. So I'm guessing this metric is measuring number of requests per unit of time. And not number of bytes per unit time. Here is an example of its bytes_received graph:

LastValue() has units of bytes. Not bytes/sec, unlike Sum(). But I think it's only showing the last number of bytes received from a read query, because it was only showing ~6k bytes received in this example:

In the graph above for bytes received for Sum(), it was showing ~68k bytes received over a 3 minute period.

[nancynh]

Thank you very much for testing all of this!

[rhatgadkar-goog]

Did you check that the metrics were populated in monitoring? I'm not sure if you talked about this with Eno if that'd be too much of a hassle when OpenCensus is going away

Yes, I'm planning to to do this. I discussed with Eno that I'll use the Go connector in a client VM, send read/write requests to an AlloyDB cluster, and verify whether the metrics show up in Cloud Monitoring.

[nancynh]

Did you check that the metrics were populated in monitoring? I'm not sure if you talked about this with Eno if that'd be too much of a hassle when OpenCensus is going away

Yes, I'm planning to to do this. I discussed with Eno that I'll use the Go connector in a client VM, send read/write requests to an AlloyDB cluster, and verify whether the metrics show up in Cloud Monitoring.

Thanks! Do leave a comment on the PR when you verify this (and maybe some pictures would be nice :))

[nancynh]

LGTM, thanks!