by Arya Solhi
Rkyv is a data format and (de)serialization framework for Rust that “implements total zero-copy deserialization, which guarantees that no data is copied during deserialization and no work is done to deserialize data. It achieves this by structuring its encoded representation so that it is the same as the in-memory representation of the source type.” (Source: rkyv.org)
Zebra and zcashd currently provide JSON-RPC methods that accept/return JSON-encoded Zcash consensus-serialized data for historical and compatibility reasons. Where there isn’t a Zcash consensus serialization, RPCs serialize data in internal zcashd formats before it is JSON-encoded. JSON is widely-supported, flexible, and human-readable, but lacks type-checking and is relatively slow to encode/decode.
In PR #6313, a “rkyv-serialization” feature was added to Zebra with a new “rkyv_listen_addr” config field for starting a datacake RPC server that accepts/returns data in the rkyv format.
The datacake RPC methods route to the same RpcImpl and GetBlockTemplateRpcImpl methods provided by Zebra’s JSON-RPC server, except they use the rkyv data format for quicker transformations to and from Zebra’s internal in-memory representation of the source type. Clients calling RPC methods from a Rust environment can import the request/response types paired with message handlers from zebra-rpc and let the Rust compiler check correct usage.
Rkyv performs very well in benchmarks, (de)serializing over 100x faster than serde_json, and appears to be the most usable, versatile crate for zero-copy deserialization. Validating data with bytecheck can make a big difference, this could be optional if Zebra supports rkyv in the future.
Clients calling RPC methods from a Rust environment can reap the performance benefits from a data format that matches Zebra’s internal in-memory representation of source types. Clients in other languages can also use Zebra’s Rust libraries to deserialize rkyv RPCs.
There are also other excellent frameworks for serializing data (e.g. protobufs, flatbuffers) with better support for other programming languages. Ideally Zebra could expose its RPC methods via multiple encodings and protocols in the future by decoding/encoding request data according to ‘Content-Type’/‘Accept’ headers so clients can call methods from whatever environment they’re already using or which best meet their needs. A GraphQL server that exchanges JSON data may also be nice for making it easier to get started developing with Zebra.
Let us know what data formats and/or protocols you would like to see supported in Zebra on Discord or on the Zcash Community Forum!