caproto vs pyepics client speed

[vstadnytskyi]

I have been working on writing server and clients using caproto and pyepics libraries. I have a simple server written with caproto that has only one PV, '.image’. I can get the image with pyepics or caproto clients. It seems that pyepics client is 2 times slower than caproto. Does anyone know why caproto is faster than pyepics? Is it because the server is written in caproto?

The server code: https://github.com/vstadnytskyi/caproto-sandbox/blob/master/caproto_sandbox/io_camera_server.py
The client code: https://github.com/vstadnytskyi/caproto-sandbox/blob/master/caproto_sandbox/io_camera_client.py

Server is running on the same machine. The image size is 1,3960,3960

Results:
One request:
pyepics client: 2.068 seconds
caproto client: 0.856 seconds
Average of 10:
pyepics client: 1.203 seconds
caproto client: 0.628 seconds

[prjemian]

@anjohnson - you might be interested in this question

[dchabot]

Server is running on the same machine.

Can you please repeat your test with client and server on separate physical hardware?

[klauer]

It's likely we have an additional unnecessary copy operation in pyepics, which hits benchmarks with large arrays much harder.

If you're interested in digging into the details, you could use cProfile (perhaps along with a tool like snakeviz for some visualization help) to find the culprit. @newville and I would be happy for the help, in any case.

More details:
There's also a few things that are different under the hood: pyepics PV by default will request timestamps (i.e., DBR_TIME_*) along with the request whereas caproto will request just the native data type (i.e., DBR_*). Additionally, caproto .read() will perform the network request but will lazily deserialize the data buffer which you aren't accessing in the benchmark.

The server implementation should have no effect here on the results - it's pretty well the same once you get to the CA protocol level. You could certainly remove that from the equation and run it against a basic softIoc, though.

[vstadnytskyi]

@dchabot

Can you please repeat your test with client and server on separate physical hardware?
The difference becomes much smaller if I run the server and the client on separate machines.

Server: Linux Ubuntu 18
Client: MacOS, mac book pro

user:caproto-sandbox user-13$ python3 caproto_sandbox/io_camera_client.py
epics Client
2.081
caproto Client
1.834
pyepics client: 1.548
caproto client: 1.343

and I ran it one more time:

user:caproto-sandbox user-13$ python3 caproto_sandbox/io_camera_client.py
epics Client
2.022
caproto Client
1.967
pyepics client: 1.545
caproto client: 1.362

[danielballan]

Thanks. In the realm of benchmarking I would call that difference below the level of significance.

[newville]

@vstadnytskyi @dchabot A small performance hit with pyepics doesn't bother me, but if there is something that can be improved that would be fine too. Pyepics is doing some connection management and data translation from C data structures -- maybe that is done differently for caproto. For large images I can believe those translations could be not-completely-optimal, either within the C library or the python layer. But I have only passing knowledge of caproto so don't know any of the details of how the data transport differs.

The factor of 2 or more on the same machine would be more of a concern, but the given the results of different machines, the results seem odd (pyepics caget is faster to a different machine than to the same machine?).