Consider flatbuffers #23
Comments
|
Original comment by Jose Luis Rivero (Bitbucket: Jose Luis Rivero, GitHub: j-rivero). Interesting, thanks Andrew. Together with flatbuffers (by google) it exists something called Cap'n Proto (by protocolbuffer former developer) which seems to follow the same concept. I found useful the reading of: I get the impression that the good old protocol buffers have been deprecated by this 'zero copy serialization' projects which are also worried about one of the biggest bottlenecks, the memory access. |
|
Original comment by Nate Koenig (Bitbucket: Nathan Koenig). Definitely not too late to consider flatbuffers or capnproto. Both seem like good alternatives to protobufs. |
|
Original comment by Andrew Hundt (Bitbucket: ahundt). I've considered cap'n proto too. It was always tempting to me but having the backing of google is always a vote of confidence in my book that it will have more users and be ported to support more platforms longer term. Flatbuffers is also C++11 and fairly simple, plus a bit easier to sell to others as a serious tool as much as it pains me to complain about fun aesthetics. Practically I think flatbuffers also keeps things simpler with the key features. However ultimately that is just a recommendation and cap'n proto wouldn't be a bad choice either. flatbuffers has a comparison and benchmarks: |
|
Original comment by Nate Koenig (Bitbucket: Nathan Koenig). I agree with you assessment. We originally went with Protobuf because of its large user base and simplicity. Flatbuffers seem simple, and will likely have a large user base. I think we need to sit down give flatbuffers a try. |
|
Original comment by Carlos Agüero (Bitbucket: caguero, GitHub: caguero). Definitely an interesting library to consider but here are a few drawbacks:
|
|
Original comment by Andrew Hundt (Bitbucket: ahundt). To add to Carlos's point, it is worth noting that protobuf specifically recommends not serializing anything much larger than 1MB. Single images from modern cameras can easily exceed 1MB in size and it would be very reasonable to have a single message with stereo data. I don't think flatbuffers will have this problem but that would need to be tested. That said, I would probably expect data to not simply be one huge flatbuffer and instead be broken into chunks where reasonable. |
|
Original comment by Andrew Hundt (Bitbucket: ahundt). I've created a small test of azmq+flatbuffers which may be interesting to look at. The API isn't at the quality level I would recommend for this project, but it illustrates that the two seem to work together pretty easily. https://github.com/ahundt/robone/blob/develop/src/AzmqFlatbufferTest.cpp Another interesting thing about flatc and flatbuffers is there is an implementation to read/write directly to files in both binary and json formats. That could make it easy to use one tool/protocol for the "holy grail" data format combo of human and machine config files, network communication, IPC, and shared memory. Also of note is in-memory flatbuffers can be modified and accessed in place. |
|
Original comment by Andrew Hundt (Bitbucket: ahundt). Sorry for the double post, but I forgot to include the header: |
|
Original comment by Andrew Hundt (Bitbucket: ahundt). It seems facebook has reached the same conclusion about flatbuffers: |
|
Original comment by Nic Fischer (Bitbucket: pickledgator). This topic is fairly stale at this point, but just curious if there have there been any updates on the decision to consider to flatbuffers or capnp for serialization? |
|
Original comment by Andrew Hundt (Bitbucket: ahundt). almost certainly not implemented, protobufs are very deeply integrated at this point. Protobuf has also gotten faster, but perhaps not as fast as flatbuffers. https://github.com/google/protobuf/releases |
|
Original comment by Carlos Agüero (Bitbucket: caguero, GitHub: caguero). We didn't have time to move forward on flatbuffers but it's still a very interesting topic. @ahundt , would you like to collaborate and work together on this topic? No obligation but I can use some help to keep improving the library :) |
|
Original comment by Andrew Hundt (Bitbucket: ahundt). I do wish I could but I'm already over-committed on other open source projects. |
|
Original comment by Nate Koenig (Bitbucket: Nathan Koenig). I'm closing this topic. Please re-open if flatbuffers becomes relevant again. |
|
Original comment by Nate Koenig (Bitbucket: Nathan Koenig).
|
|
Original comment by Silvio Traversaro (Bitbucket: traversaro). If someone is reading this in 2018, I recently found several projects that implement a subset of the protobuf protocol but with a more lightweight implementation:
Another related topic is the usage of Arena Allocators to reduce the cost related to memory allocation during protobuf serialization/deserialization. Unfortunately as of late 2018, this technique is not really effective because the open source version of protobuf does not support allocating the memory related to the strings in the arena, see protocolbuffers/protobuf#4327 (comment) and protocolbuffers/protobuf#1896 . |
|
Original comment by Carlos Agüero (Bitbucket: caguero, GitHub: caguero). Thanks for the pointer @traversaro. I'm assuming that if the new protobuf message changes, everybody should use the new version of the library, right? If this is true, it might be a bit confusing for users that don't know about these new libraries and generate their messages with the old/current library. |
|
Original comment by Silvio Traversaro (Bitbucket: traversaro). I am not 100% sure, but I think that all those projects aim for on-wire compatibility with the Protobuf format, i.e. you could have a process serializing a message using the "Official" Protobuf, and the same message could be de-serialized by one of those libraries, and vice-versa. |
Original report (archived issue) by Andrew Hundt (Bitbucket: ahundt).
I know you guys have been developing for over a year and this is probably too little, too late, but I wanted to point out some problems with protocol buffers and the replacement that was created to solve some of the exact problems that are encountered in robotics.
One of the major issues with protocol buffers in high performance systems is the substantial CPU overhead involved in packing and unpacking protobufs. Google themselves ran into this exact problem in high performance situations such as games, which have many of the same real-time requirements as robotics, particularly for multiplayer games. For this reason they created their own alternative to protobuf called flatbuffers.
https://google.github.io/flatbuffers/
Much of flatbuffers is very similar to protobuf, so it might actually be less unreasonable to move to flatbuffers than it would be for any other alternative. However, I realize less than extremely unreasonable may still be unreasonable. :-)
I'd like to hear any thoughts on this or if there is anything else useful I could add here.
The text was updated successfully, but these errors were encountered: