capnp_conv

The capnp_conv macro facilitates conversions between Cap'n Proto structs/unions/unions/groups to their rust counterparts by implementing all the necessary Capn' Proto Rust generated builder/reader calls.

Inspired by the (seemingly abandoned?) existing PR by @realcr.

Usage

The following capnp schema file directly translates to the below rust file:

struct SomeStruct { }

enum CapnpEnum {
    val1 @0;
    val2 @1;
}

struct CapnpStruct {
  voidVal   @0   :Void;
  i32Val    @1   :Int32;
  textVal   @2   :Text;
  dataVal   @3   :Data;
  structVal @4   :SomeStruct;
  enumVal   @5   :CapnpEnum;
  listVal   @6   :List(SomeStruct);
}

#[capnp_conv(some_struct)]
pub struct SomeStruct {}

#[capnp_conv(capnp_struct)]
pub struct RustStruct {
  void_val: (),
  i32_val: i32,
  text_val: String,
  #[capnp_conv(type = "data")]
  data_val: Vec<u8>,
  struct_val: SomeStruct,
  #[capnp_conv(type = "enum")]
  enum_val: CapnpEnum,
  list_val: Vec<SomeStruct>,
}

The capnp_conv proc macro implements the capnp_conv::Readable, capnp_conv::Writable, and TryFrom<Reader> traits, which handle all of the reading/writing:

fn read(reader: capnp_struct::Reader) -> Result<RustStruct, capnp::Error> {
  RustStruct::read(reader)?
}
fn write(rust_struct: RustStruct, builder: capnp_struct::Builder) {
  rust_struct.write(builder)
}

Special Type Handling

Capnp group, enum, union, and data types require the field attribute with a type specifier.

Groups

Capnp groups are represented by separate rust structs.

struct CapnpStruct {
    groupVal :group {
        val @0 :Void;
    }
}

#[capnp_conv(capnp_struct)]
pub struct RustStruct {
  #[capnp_conv(type = "group")]
  group_val: RustStructGroup,
}
#[capnp_conv(capnp_struct::group_val)]
pub struct RustStructGroup {
  val: ()
}

Enums

The macro has two options when it comes to enums: enum and enum_remote. Because code generated capnp files already contain enum definitions, it is possible to use them directly with enum. This eliminates the need to write an extra enum definition, but there are some times when it is useful to define a separate enum, for example, if it is necessary to derive traits or use other macros on the enum. For these cases, use enum_remote with a separately defined rust enum. When used on a rust enum, the capnp_conv macro generates the From/Into trait implementations for its capnp counterpart.

enum CapnpEnum {
    val1 @0;
    val2 @1;
}
struct CapnpStruct {
    enumVal       @0 :CapnpEnum;
    enumValRemote @1 :CapnpEnum;
}

#[capnp_conv(CapnpEnum)]
pub enum RustEnum {
  Val1,
  Val2,
}
#[capnp_conv(capnp_struct)]
pub struct RustStruct {
  #[capnp_conv(type = "enum")]
  enum_val: CapnpEnum,
  #[capnp_conv(type = "enum_remote")]
  enum_val_remote: RustEnum,
}

Unions

Unions can be represented in two different ways. One is by using rust enums, but structs can also represent capnp unions with fields wrapped in Option<T>s and containing the #[capnp_conv(union_variant)] attribute macro. This eliminates the need to create a separate item for unnamed unions, but it can be more cumbersome to manipulate.

Rust enum variants have the same requirements for using the #[capnp_conv(type = xxx)] field attributes.

struct CapnpStruct {
    namedUnion :union {
        val1 @0 :Void;
        val2 @1 :Void;
    }
    union {
        val1 @1 :Void;
        val2 @2 :Void;
    }
}

#[capnp_conv(capnp_struct)]
pub struct RustStruct {
  #[capnp_conv(type = "union")]
  named_union: RustStructUnion,
  #[capnp_conv(type = "unnamed_union")]
  unnamed_union: RustStructUnamedUnion,
}
#[capnp_conv(capnp_struct::named_union)]
pub enum RustStructUnion {
  Val1(()),
  Val2(()).
}
#[capnp_conv(capnp_struct)]
pub enum RustStructUnamedUnion {
  Val1(()),
  Val2(()).
}

// or

#[capnp_conv(capnp_struct)]
pub struct RustStruct {
  #[capnp_conv(type = "union")]
  named_union: RustStructUnion,
  #[capnp_conv(union_variant)]
  val1: Option<()>,
  #[capnp_conv(union_variant)]
  val2: Option<()>,
}
#[capnp_conv(capnp_struct::named_union)]
pub struct RustStructUnion {
  #[capnp_conv(union_variant)]
  val1: Option<()>,
  #[capnp_conv(union_variant)]
  val2: Option<()>,
}

Non Exhaustive Flag

Generally capnp enums and unions must have the same variants as their rust counterparts, otherwise the macro will genreate a build error. A non_exhaustive flag can be specified on the capnp_conv macro to relax this requirement. This will allow additive changes to schema unions and enums, which, when encountered at runtime, will generate runtime exceptions. The From implementation fo the enum is also changed to a TryFrom implementation instead.

enum TestEnum {
  val1 @0;
  val2 @1;
  extra @2;
}

struct TestUnion {
  union {
    unionVal1 @0 :Void;
    unionVal2 @1 :Void;
    extra @2 :Void;
  }
}

#[capnp_conv(TestEnum, non_exhaustive)]           // <------------ flag
pub enum TestEnum {
  Val1,
  Val2,
}

#[capnp_conv(test_union, non_exhaustive)]          // <------------ flag
pub struct TestUnion {
  #[capnp_conv(union_variant)]
  pub union_val1: Option<()>,
  #[capnp_conv(union_variant)]
  pub union_val2: Option<()>,
}

Note that this only applies to additive changes. This will still result in a build error for example:

enum TestEnum {
  val1 @0;
}

#[capnp_conv(TestEnum, non_exhaustive)]
pub enum TestEnum {
  Val1,
  Val2,
}

Data

The capnp Data type is functionally identical to List(UInt8), both of which are represented with Vec<u8> in rust. However, capnpc generates two distinct structs that handle reading and writing of the two types. There is no way to specify which one a Vec<u8> is intended to represent, which necessitates the use of the field attribute in the case of Data types.

struct CapnpStruct {
    named_union :union {
        list @0 :List(UInt8);
        data @1 :Data;
    }
}

#[capnp_conv(capnp_struct)]
pub struct RustStruct {
  list: Vec<u8>,
  #[capnp_conv(type = "data")]
  data: Vec<u8>,
}

Extra Features

capnp_conv includes several other features that can be enabled through setting options in field attributes

Renaming fields

Rust field names must normally match the field names of their capnp counterparts (converted to the appropriate rust case). Using the name attribute, it is possible to disentangle them.

struct CapnpStruct {
    capVal @0 :Void;
    capnpUnion :union {
        capVal1 @1 : Void;
        capVal2 @2 : Void;
    }
}

#[capnp_conv(capnp_struct)]
pub struct RustStruct {
  #[capnp_conv(name = "capVal")] //or #[capnp_conv(name = "cap_val")]
  arbitrary_name: (),
  #[capnp_conv(name = "capnp_union")]
  #[capnp_conv(type = "union")]
  rust_union: RustUnnion,
}
#[capnp_conv(capnp_struct::capnp_union)]
pub enum RustUnion {
  #[capnp_conv(name = "capVal1")]
  ArbitraryName1(())
  #[capnp_conv(name = "cap_val2")]
  ArbitraryName2(())
}

Optional fields

Wrapping a field with Option<T> indicates that the field is optional. When a struct is written to a builder, optional fields will be skipped if they are None. When a struct is read from a reader, optional fields that are pointer types (List, Struct, Text, Data) will be set to None if the field has not been set in the capnp message. Primitive types and enums will always be read and set to Some.

• Unions cannot contain optional fields.
• Groups and unions cannot be optional.

struct CapnpStruct {
    capVal @0 :Void;
}

#[capnp_conv(capnp_struct)]
pub struct RustStruct {
  cap_val: Option<()>,
}

Skipped fields

• Adding #[capnp_conv(skip_write)] to a field's attributes will result in the field not being written. Similar to as if it were optional with None.

• Adding #[capnp_conv(skip_read)] to a field's attributes will result in the field never being read. During a read, it be set to the field type's default value (note the field type must implement the Default trait), or None if the field is optional. This can be futher configured by setting the field's default attribute (see below).

• #[capnp_conv(skip)] skips both reading and writing.

• Unions cannot contain skipped fields.

Default override fields

Setting the #[capnp_conv(default = "path_to_func_to_call")] attribute to a field that is configured as skip_read or skip will set the field to the output of calling the function specified rather than the default value during a read.

Unions cannot contain default overrides.

Generics

Generics are supported for both structs and enums.

struct CapnpStruct(T,Y) {
  tVal @0 :T;
  list @1 :List(Y);
  unionVal :union {
      tVal @2 :T;
      list @3 :List(Y);
  }
}

#[capnp_conv(capnp_struct)]
pub struct RustStruct<T,Y> {
  t_val: T,
  list: Vec<Y>,
  #[capnp_conv(type = "union")]
  union_val: RustStructUnionVal<T,Y>,
}
#[capnp_conv(capnp_struct::union_val)]
pub enum RustStructUnionVal<T,Y> {
  TVal(T),
  List(Vec<Y>),
}

Limitations:

Generics are limited to pointer types by capnp itself, and capnp_conv has some additional constraints in the following not being supported:

• List(UInt8)
• List(<struct>)
• List(<List>)

One feature of capnp schemas that is not easily reproduced in rust is nested struct definitions. This is not typically an issue as they can be implemented as flattened rust structs, but when combined with the fact that nested capnp structs/unions/groups have access to all the generic types of all of their ancestors, this can be problematic for rust models.

For example:

struct ParentStruct(T,Y,R) {
    tVal @0 :T;
    yVal @1 :Y;
    rVal @2 :R;
    unionVal :union {
        voidVal @3 :Void,
        tVal    @4 :T,
    }

    struct ChildStruct {
        tVal @0 :T
    }
}

When defining parent_struct::child_struct in rust, the struct must have T, Y, and R even though it only uses T. Rust does not allow structs to have generics that they do not use, but we can use PhantomData to overcome this:

pub struct ChildStruct<T,Y,R> {
 t_val: T,
 phantom: PhantomData<*const (Y,R)>,
}

When reading/writing, PhantomData types will be automatically skipped, even without the field attribute.

Unfortunately there is no "PhantomVariant" for rust enums, so for unions, instead make the second field of any variant a PhantomData:

pub enum UnionVal(T,Y,R) {
  VoidVal((), PhantomData<*const (Y,R)>),
  TVal(T),
}

Write/Read overrides

Fields can have custom reader/write implementations to allow arbitrary or derivative data to be injected when reading or writing a capnp_conv struct from/into capnp messages by using the read_with and write_with field attributes.

 struct CapnpStruct {
  integer @0 :Int16;
  data @1 :Data;
}

#[capnp_conv(capnp_struct)]
pub struct RustStruct {
  #[capnp_conv(read_with = "read_integer")]
  pub integer: i16,
  #[capnp_conv(write_with = "write_data")]
  #[capnp_conv(read_with = "read_data")]
  pub data: String,
}

fn read_integer(
  reader: <capnp_struct::Owned as capnp::traits::Owned>::Reader<'_>,
) -> Result<i16, capnp::Error> {
  // Increment the integer value from the message by 1 when reading
  Ok(reader.get_integer() + 1)
}

fn read_data(
  reader: <capnp_struct::Owned as capnp::traits::Owned>::Reader<'_>,
) -> Result<String, capnp::Error> {
  // Despite the schema type being "Data", we can add an arbitrary pair
  // of read/write functions to transform it into an entirely different type
  let data = reader.get_data()?;
  let string = std::str::from_utf8(data).unwrap();
  Ok(string.to_owned())
}

fn write_data(
  this: &RustStruct,
  builder: &mut <capnp_struct::Owned as capnp::traits::Owned>::Builder<'_>,
) {
  builder.set_data(this.data.as_bytes());
}

read_with and write_with are mutually exclusive with skip_read and skip_write respectively. Otherwise, the same limitations as the skip_read/write attributes apply.

Future work

Short term:

• Add support for top level Vec read and write
• Add support for anypointer type
• Add more validations to allow the compiler to provide more clues when the macro is not properly used.
  • unions must have at least 2 fields (both enums and struct unions apply)
  • cannot have more than 1 unnamed union
  • lists cannot have generic types as their type
  • enums must either all be unit types or complex. no mix & match
  • assert fields with type specifiers are not primitives, blobs, void, or lists (could be done in regular parsing)
  • fields with type = "data" attribute must be of type Vec<u8>
• Finish writing tests. Priority needs:
  • skipped and default fields
  • union struct representation
• Confirm if as_turbofish() function is sufficient for all possible cases (specifically, nested generic types? Type1<Type2<T>>)

Long term:

• Allow boxed fields for the rare case of a struct/enum containing a field of its own type (with different generics).
• Allow structs to contain lifetimes and generic constraints that carry over to the generated impls.
• Add a convenience clear_enum_fields function to struct represented capnp unions that sets all union fields to None.