Blog

This blog post reviews the evolution of the API for a FFI wrapper around one of my Rust libraries that I intend to use from multiple programming languages (e.g. Rust, C++, Python, etc.). I present and discuss the advantages and disadvantages of a few different API patterns for the C API of the FFI. I identify how the C API has minimal impact on the final API in other languages and I explore how code generation (e.g. flatbuffers) could be used instead to generate an ergonomic multi-language library.

Context

The context for this project is as follows: I have a Rust library to read/write a variety of custom file formats. Using this library as a dependency, I've built a few different tools that use these custom files in some way or other. Since all the tools I built were also written in Rust, using the library was trivial.

At some point I wanted to use these file formats in new codebases that were not written in Rust, instead they used C++ and Python respectively. Rather than having to rewrite all the read/write code in each of those languages, or for any other language I wanted/needed to use, I decided to try reusing the Rust library by wrapping it in a C-FFI. Ideally this would give me access to the read/write functionality in both those languages without having to duplicate (triplicate? n-plicate?) my efforts.

Planning

To start, I identified that there's two parts to the overall problem I needed to solve. Firstly, I needed to wrap the read/write functions so that they can be called through FFI and then I needed an easy way to access the data. What I needed to expose to FFI looked roughly something like this:

struct FileData {
    field1: u32, 
    field2: u32, 
    field3: u32
}

impl FileData {
    fn read(path: &Path) -> Result<FileData, Error> { ... }
    fn write(path: &Path) -> Result<(), Error> { ... }
}

Exposing those read/write functions was not a very difficult problem. It was simply a matter of creating some functions that operated on pointers and C-types that wrapped the inner Rust functions. I also wanted the library to handle memory allocation/deallocation so I created a couple of extra utility functions to help with that. The result looked roughly like so:

#[no_mangle]
pub unsafe extern "C" fn file_data_new() -> *mut FileData {
    // Heap allocate new `FileData` with default values
    Box::into_raw(Box::new(FileData::new()))
}

#[no_mangle]
pub unsafe extern "C" fn file_data_free(f: *mut FileData) {
    // Automatically drop `FileData`
    Box::from_raw(f);
}

#[no_mangle]
pub unsafe extrn "C" fn file_data_read(f: *mut FileData, path: *const libc::c_char) -> bool { 
    // Get pointer to heap allocated `FileData`
    let mut file_data = Box::from_raw(f);

    // Build path from c-string path argument
    let p = PathBuf::from(CStr::from_ptr(path).to_str().unwrap_or_default());

    // Read the data and provide some minimal error handling
    let res = file_data.read(&p).is_ok();

    // Forget the memory so Rust doesn't deallocate when `file_data` is dropped
    std::mem::forget(file_data);

    res 
} 

Using these functions from C was pretty straightforward, the only "gotcha" would be having to manually free the memory but that's standard practice in C.

FileData* file_data = file_data_new();
bool ret = file_data_read(file_data, "/path/to/file");
if(ret) { 
    // Do something 
} 
file_data_free(file_data);

At this stage I discovered that the bigger challenge would be to get meaningful data across the FFI-boundary. Invoking the opaque read/write processes was easy enough but I wanted to actually read the data and do things with it in the other code bases!

Method 1: #[repr(C)] on the existing struct

The first thing I explored was the idea of using #[repr(C)] on my existing structs so that they would be FFI friendly and I could easily access the member fields as needed. Something like so:

#[repr(C)]
struct FileData {
    field1: u32, 
    field2: u32, 
    field3: u32 
}

This is the recommended way for passing structs and similar types as it ensures that the struct uses the C-ABI for data alignment rather than the unstable Rust-ABI. While this solution is rather elegant since all it requires is a small macro decorating the struct, I found the ergonomics go out the window when using more complicated fields. Namely, using any collection or heap allocated Rust types such as Vec or String. Initially, to work around this, I experimented with creating helper functions for those fields which worked "okay" for String fields but quickly turned into a headache for Vec fields. I had to add lots of additional methods to handle the different access patterns for those fields so that I could safely handle those fields on the C-side.

For an example of the complexity, this is the solution I implemented for String fields. First, I created a new FfiString type so I could convert the Rust String data into a C-compatible type. The implementation looked something like this:

#[repr(C)]
#[derive(Debug)]
pub struct FfiString {
    chars: *mut libc::c_char,
    len: libc::size_t,
}

impl FfiString {
    pub fn new() -> FfiString {
        Self::default()
    }

    pub fn set_string(&mut self, v: &str) {
        let c_str = CString::new(v.as_bytes()).unwrap_or_default();
        self.len = c_str.as_bytes_with_nul().len();
        self.chars = c_str.into_raw() as *mut libc::c_char;
    }
}

impl Default for FfiString {
    fn default() -> FfiString {
        FfiString {
            chars: ptr::null::<libc::c_char>() as *mut libc::c_char,
            len: 0,
        }
    }
}

impl Drop for FfiString {
    fn drop(&mut self) {
        unsafe {
            CString::from_raw(self.chars); // Drop
        }
    }
}

impl fmt::Display for FfiString {
    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
        let s = unsafe { CStr::from_ptr(self.chars).to_string_lossy().to_string() };
        write!(f, "{}", s)
    }
}

Using the set_string() function, I could copy any String field data into a FfiString which I could then pass across the FFI-boundary. However, in order to actually do this I added methods to allocate/free this object and added a method for each String field in my struct. It looked something like this:

#[no_mangle]
pub unsafe extern "C" fn ffi_string_new() -> *mut FfiString {
    Box::into_raw(Box::new(FfiString::new()))
}

#[no_mangle]
pub unsafe extern "C" fn ffi_string_free(s: *mut FfiString) {
    Box::from_raw(s); // Drop
}

#[no_mangle]
pub unsafe extern "C" fn data_file_string_field(
    data_file: *mut DataFile, 
    _out: *mut FfiString
) -> bool {
    ...
   // Set _out string
    ...
}

With this implementation I could use my struct normally but then would have to use special functions to access the string data. Like so:

FileData* file_data = file_data_new();
bool ret = file_data_read(file_data, "/path/to/data");
if (ret) {
    int foo = file_data->int_field1;
    int bar = field_data->bool_field1;

    FfiString* baz = ffi_string_new();
    file_data_string_field(file_data, baz);

    // Do something with baz string!

    ffi_string_free(baz);
}
file_data_free(file_data);

For Vec it was a similar story but rather that implement a new type I just created a few methods to acces the internal data by index. It was a bit clunky but got the job done. The implementation for a Vec field looked like so:

#[no_mangle]
pub unsafe extern "C" fn data_file_field_count(data_file: *mut DataFile) -> u32 {
    let d = Box::from_raw(data_file);
    let res = d.field.len() as u32;
    std::mem::forget(d);
    res
}

#[no_mangle]
pub unsafe extern "C" fn data_file_field_item(
    data_file: *mut DataFile,
    idx: u32, 
    _out: *mut i16
) -> bool{
    let d = Box::from_raw(data_file);
    let idx = idx as usize;

    let res = if idx < d.field.len() {
        *_out = d.field[idx];
        true
    } else {
        false
    };

    std::mem::forget(d);
    res
}

Method 2: Opaque pointers and more functions

At this stage, while the #[repr(C)] method paired with the String/Vec helper functions worked well enough, it created a very inconsistent API. Some fields could be accessed directly while others needed to be accessed with their helper methods. I felt this was inelegant and I was keen to redesign the API to make it more consistent and predictable. Furthermore, it became clear to me that there was a big unsafe part of my code in that, while I provided accessor methods for the more complex fields, #[repr(C)] did not actually prevent direct access to the underlying Rust data that I had wrapped. Manipulating those pointers on the C side would be undefined behavior and would most likely crash my programs in unpredictable ways.

#[repr(C)]
struct FileData {
    field1: u32,
    field2: bool,
    field3: String,
    field4: Vec<u32>,
}

FileData* file_data = file_data_new();
FfiString* ffi_string = ffi_string_new();

// Access primitive fields directly, okay!
int foo = file_data->field1;
bool bar = file_data->field2;

// Access strings using helper function, okay!
file_data_field3(file_data, ffi_string);

// Access vec field using helper function, okay!
int baz;
file_data_field4_item(file_data, 0, &baz);  // Okay!

// Access string/vec field directly...uh oh!
void* uh_oh_ptr = (void*)file_data->field3;
void* oh_lawd_ptr = (void*)file_data->field4;

ffi_string_free(ffi_string);
file_data_free(file_data);

At this stage I had two problems to solve: firstly, I needed to make the API consistent and secondly I needed to hide the fields that I didn't want the user to access directly. Since C doesn't have any concept of public/private fields, I had to move away from simply applying #[repr(C)] to my structs and passing around pointers to them. While generating all the accessor methods for the Vec/String fields proved to be tedious, I felt the concept could be reused for all fields and instead of working with transparent pointers I would use opaque pointers. While my solution required more manually generated code, I gained a more ergonomic API and I could hide the struct implementation details (no pub) to prevent unwarranted access. My API evolved to look something like this:

struct FileData {
    field1: u32,
    field2: f32,
    field3: bool,
    field4: String,
}

pub unsafe extern "C" fn file_data_field1(fd: *mut FileData, _out: *mut u32) -> bool {}
pub unsafe extern "C" fn file_data_field2(fd: *mut FileData, _out: *mut f32) -> bool {}
pub unsafe extern "C" fn file_data_field1(fd: *mut FileData, _out: *mut bool) -> bool {}
pub unsafe extern "C" fn file_data_field1(fd: *mut FileData, _out: *mut FfiString) -> bool {}

On the C-side I used an opaque struct defined with no members, this way I could create a pointer to the data but would have to use the functions to access the fields. While this API design made both the Rust code and the C code more verbose, I was happy with the improved ergonomics and safety.

struct FileData;

void do_something(FileData* fd) {
    int i = 0;
    file_data_field1(fd, &i);

    float f = 0.0f;
    file_data_field2(fd, &f);

    bool b = false;
    file_data_field3(fd, &b);

    FfiString* s = ffi_string_new();
    file_data_field_4(fd, s);
    ffi_string_free();
}

Method 3: FFI-friendly intermediate struct

The opaque pointer + function for every field implementation was working well-enough initially as I was focusing on wrapping the simpler file types. However, once I started to wrap more complicated files with nested structs or vecs of structs it became clear to me that this API design was not practical in the long term. A struct with 9 fields (of which 5 were Vec, 1 was String and 1 was another struct) which only took 8 or so lines to define in Rust required about 500 lines of wrapping code for the FFI API...and that only included getter functions! As I started to plan for adding support for the write part of the API I foresaw a dizzying amount of work. Some of the more complex file types with lots of nested structs started to give me hand cramps just thinking about them... Here we can see an example of how the API footprint for a simple struct baloons up, even when excluding "setter" functions:

struct FileData {
    field1: String,
    field2: Vec<u32>,
    field3: Vec<f32>,
    field4: FileSub,
}

Struct FileSub {
    field1: u32,
    fiedl2: Vec<u32>
}

pub unsafe extern "C" fn file_data_new() -> *mut FileData {}
pub unsafe extern "C" fn file_data_read(f: *mut FileData, path: *const libc::c_char) -> bool { 
pub unsafe extern "C" fn file_data_field1(fd: *mut FileData, _out: *mut FfiString) -> bool {}
pub unsafe extern "C" fn file_data_field2_count(fd: *mut FileData) -> u32 {}
pub unsafe extern "C" fn file_data_field2_item(fd: *mut FileData, _out: *mut u32) -> bool {}
pub unsafe extern "C" fn file_data_field3_count(fd: *mut FileData) -> u32 {}
pub unsafe extern "C" fn file_data_field3_item(fd: *mut FileData, _out: *mut f32) -> bool {}
pub unsafe extern "C" fn file_data_field4(fd: *mut FileData, _out: *mut FileSub) -> bool {}
pub unsafe extern "C" fn file_data_free(fd: *mut FileData);

pub unsafe extern "C" fn file_sub_new() -> *mut FileSub {}
pub unsafe extern "C" fn file_sub_field1(fd: *mut FileData, _out: *mut u32) -> bool {}
pub unsafe extern "C" fn file_sub_field2_count(fd: *mut FileSub) -> u32 {}
pub unsafe extern "C" fn file_sub_field2_item(fd: *mut FileSub, _out: *mut f32) -> bool {}
pub unsafe extern "C" fn file_sub_free(fd: *mut FileSub);

Faced with this development it was clear to me that this problem arose from my choice of API and because I was implementing all the interfaces manually. In particular, having to wrap each field in a function was the root cause of the issue. I thought about an alternative implementation which used the previous #[repr(C)] implementation but also ensured that each field would be valid on the C side. The idea would be that I would copy the Rust struct into this C compatible struct and then I could drop all the field functions and deal with structs directly on both sides. It would look something like this:

struct FileData {
    field1: String,
    field2: Vec<u32>,
    field3: Vec<u64>,
}

#[repr(C)]
struct FfiFileData {
    field1: *mut libc::c_char,
    field1_len: u32,
    field2: *mut u32,
    field2_len: u32,
    field3: *mut u64,
    field3_len: u32,
}

impl FileData {
    fn to_ffi(&self) -> FfiFileData {}
}

I also thought about eliminating the copy step entirely and converting all my structs to the "C" versions and not using any Rust types. However, I quickly dismissed this idea because I was unwilling to abandon the great ergonomics when using this library as a dependency in a Rust application.

Ultimately, while this design does alleviate the need for wrapping each field in its own function it introduces a new issue. Namely that now there's special consideration to be given to how Rust allocates heap objects if I need to manipulate them on the C-side. For example, using the structs above, if I wanted to add another u32 to field2 I couldn't simply just allocate more memory on the C-side and increase field_len. Problems would arise on the Rust side when freeing that associated memory. While it may be possible to overcome this by enforcing the use of the system allocator on the Rust side, I've seen it recommended in several places to avoid "crossing-streams" like this when working with FFI code. As someone with minimal experience in custom memory allocation strategies, having to skirt around the nuances of memory allocation for this project was not ideal and I can forsee it being a troublesome issue.

Furthermore, with this strategy, for each struct I would need to generate a duplicate struct, but with slightly different fields. Barring some sort of macro magic (more on that later), I would have to keep the two structures in sync. Entirely possible and manageable but tedious and redundant, a programmer's worst nightmare!

Ergonomics

Having reviewed a few of the strategies for developing a C API around my Rust library, it's clear they each have their advantages and disadvantages. However, I'd like to return to the goal of this project: re-use my Rust library from C++ and Python applications.

While C is the lingua franca of the programming world and building a C-ABI compatible library is the tried-and-true method for cross-language interoperability, there's still an ergonomics problem. An ideal consumer of the C API in another language should wrap the library with language-specific logic to maximize the ergonomics and safety of using that library in said language. Thus, a final implementation from top to bottom from Rust -> C -> Python/C++ might look something like this:

// Rust
struct FileData {
    field1: String,
    field2: Vec<u32>,
    field3: Vec<u64>,
}

impl FileData {
    fn to_ffi(&self) -> FfiFileData {}
}

// Rust FFI
#[repr(C)]
struct FfiFileData {
    field1: *mut libc::c_char,
    field1_len: u32,
    field2: *mut u32,
    field2_len: u32,
    field3: *mut u64,
    field3_len: u32,
}

#[no_mangle]
pub unsafe extern "C" fn file_data_new() -> *mut FfiFileData;

#[no_mangle]
pub unsafe extern "C" fn file_data_free(f: *mut FfiFileData);

// C
struct FfiFileData {
    char* field1;
    uint32_t field1_len;
    uint32_t* field2;
    uint32_t field2_len;
    uint64_t* field3;
    uint64_t field3_len;
}

FfiFileData* file_data_new();
void file_data_free(foo);

# Python
class FfiFileData:
    def __init__(self):
        self._data = ffi_lib.file_data_new()

    def __del__(self):
        ffi_lib.file_data_free(self._data)

class FileData:
    def __init__(self):
        self.field1 = ""
        self.field2 = []
        self.field3 = []

    def from_ffi(self, ffi_data):
        # Copy from c data and buffers...

    def to_ffi(self, ffi_data):
        # Copy into c data and buffers...

// C++
class CFfiFileData {
public:
    FfiFileData*data;

    CFfiFileData() {
        this->data= file_data_new();
    }

    ~CFfiFileData() {
        file_data_free(this->data);
    }

};

class FileData {
public:
    std::string field1;
    std::vector<uint32_t> field2;
    std::vector<uint64_t> field3;

public:
    void from_ffi(const &CFfiFileData);
    void to_ffi(*CFfiFiledata);
};

It becomes clear that while changing the C API has various advantages and disadvantages depending on the implementation, it ultimately has little impact on the API in Python and C++. Ironically, for each language I would have to have make similar API decisions, regardless of what the API in C looks like. Namely, do I wrap the underlying unsafe data with functions that take native types as args and then manipulate the underlying unsafe types? Or do I copy the data back and forth as needed using the C types as basically an intermediate serialization/deserialization step (as exemplified above).

Going Forward: A little magic required

At this stage, I am firmly convinced that the only real solution here is to use code generation. We've already established that having a C-compatible API to run the "business logic" on the data is quite straightforward to implement. That bigger challenge is in communicating that data into various programming languages and maintaining an ergonomic API to interact with that data.

In the ideal solution there should be one "source of truth" that models the data. A code generation step to generate ergonomic structures for that model in the various target languages and finally a process to consistently deserialize/serialize to some common data type (bytes?) so that it can be marshalled across language boundaries.

As an aside, I find it amusing that I'm basically describing Windows COM or a similar technology. A technology I imagine as arcana only practiced by the wisest greybeards (or unfortunate maintenance programmers). I wonder how many times this exact problem has been solved in software history?

I looked around to see if anyone else had this issue and had maybe thrown up crate to help with this problem but I mostly only found ffi utilities. One interesting project was ffi-support which does mention support for marshalling data across the FFI boundary but it doesn't exactly fit the "code generation" criteria and seems specific to the internal needs of the mozilla application support team. Another young project that caught my eye was c_bridge which seems to also to target FFI data marshalling.

Since there's nothing really available out of the box, I'll have to "roll my own" solution to this problem but there are a lot of options here. The ffi-support crate makes use of JSON for marshalling data which could be an interesting solution. One could use syn to parse the existing Rust structs, write some generators for each of the target languages and then use JSON or a similar, language-agnostic interchange format to marshall the data between the objects. Alternatively, one could write their own "schema" to generate the structs for all languages (including Rust) and use the C-ABI as the "interchange format". One could even go as far as developing their own ABI in the spirit of COM. Opportunities be a plenty!

Foreign Function Flatbuffers

The solution I will be personally exploring is using flatbuffers because it solves both the code generation and data marshalling problem in one library. There are other similar libraries (protobuf, capnproto, etc.) but I settled on flatbuffers since I have familiarity with it. Furthermore, flatbuffers recently got Rust support and I've been meaning to try it out.

The plan is to leverage the flatbuffers compiler to generate the various structures in Rust, C++ and Python. Using the flatbuffer libraries and the generated classes makes the data marshalling trivial. All I need to do is pass around byte buffers across the FFI boundaries and flatbuffers handles the rest. The only remaining piece would be a simple FFI library over the read/write functions that reads the data into a flatbuffer compatible buffer. I expect the final implementation might look something like this at a high level:

// Flatbuffer schema
table FileData {
    field1: String;
    field2: uint32;
    field3: uint64;
}

// Rust
use generated::FileData;

impl FileData {
    fn read(&self, p: &Path);
    fn write(&mut self, p: &Path);

    fn from_flatbuffer(&mut self, &[u8]);
    fn to_flatbuffer(&self) -> Vec<u8>;
}

// Rust FFI
pub unsafe extern "C" fn file_data_read(path: *const libc::c_char) -> *mut u8;
pub unsafe extern "C" fn file_data_write(fd: *mut u8, path: *const libc::char);

// C++
class FileData {
public:
    std::string field1;
    std::vector<uint32_t> field2;
    std::vector<uint64_t> field3;

public:
    void read(const &std::string path);
    void write(const &std::string path);
};

# Python
class FileData:
    def __init__(self):
        self.field1 = ""
        self.field2 = []
        self.field3 = []

    def read(self, path):
        # Copy from flatbuffer

    def write(self, path):
        # Copy into flatbuffer

Conclusion

In conclusion, I've found Rust's FFI story to actually be quite good. To wrap a Rust library with a C-FFI is well supported by the language and supporting libraries. From my perspective, exposing Rust functions to multiple programming languages is basically a solved problem thanks to the C-ABI. As discussed above, there's also a variety of ways to setup the API depending on your use case. Things start to get a bit more complicated in the data marshalling space when working with more complex data types. Currently it's not easy to exchange complex data types originating from the Rust side over the FFI boundary without some data wrangling or API changes. There's definitely room here for a crate that can provide some "C-ABI Safe" types that can be used in conjunction with types such as Vec or String.

Finally, I think there's some interesting ideas that can be explored in the metaprogramming space for being able to generate ergonomic cross-language APIs. I'll be experimenting with flatbuffers as my tool of choice but I can envision a Rust-centric tool being especially powerful.

P.S. Thanks for sticking around until the end!