Introduction
I've been spending a good amount of time working in Rust and have come to really enjoy the language. One area that I have not really explored is the macro system. While there are a few resources available, I have to admit that it took me a bit of time to understand how to implement one. I think this is an issue of the incidental complexity of macros being so closely related to the Rust compiler. As someone who is not particularly well-versed in compiler terminology, reading about syntax-trees, ASTs, hygiene and all kinds of compiler-pizzaz was definitely a stumbling block. Anyhow, I was able to scrape by with a basic understanding of how macros work and implemented a simple macro to generate some structs. I wanted to share my results so that others looking to do something similar might find this a useful resource.
Problem: Optional fields
I was implementing a simple config system that could load a configuration from a TOML file using the TOML crate. This crate provides a nice layer ontop of serde to deserialize a TOML string into a struct. Here is an example modified from the docs:
#[macro_use]
extern crate serde_derive;
extern crate toml;
#[derive(Deserialize)]
struct Config {
ip: String,
}
fn main() {
let config: Config = toml::from_str(r#"ip = '127.0.0.1'"#).unwrap();
assert_eq!(config.ip, "127.0.0.1");
}
A key aspect of this crate is that you can make fields optional in the TOML data by specifying it as an Option<T> in your struct. This is a nice feature but I wanted to make the field optional in the TOML but not optional in my config struct and instead have a sane default. Example:
#[macro_use]
extern crate serde_derive;
extern crate toml;
// This is the config I wanted
#[derive(Deserialize)]
struct Config {
foo: String
}
impl Config {
pub fn default() -> Config {
Config {
foo: "default-value".to_string(),
}
}
}
// This won't work
fn main() {
let config: Config = toml::from_str(r#"").unwrap();
}
However, all non-Option<T> fields in the struct must be in the TOML string or else toml::from_str() will fail. This is the error returned for the example above:
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Error { inner: ErrorInner { kind: Custom, line: None, col: 0, message: "missing field `foo`", key: [] } }', /checkout/src/libcore/result.rs:859
Solution: Lame
So an easy way around this is to define a separate struct that has optional fields to work better with the TOML crate and then "merge" the optional struct with the non-optional struct. Example:
#[macro_use]
extern crate serde_derive;
extern crate toml;
// My new struct to hold result from toml::from_str()
#[derive(Deserialize)]
struct ConfigLoader {
foo: Option<String>
}
// My actual config struct I care about
#[derive(Deserialize)]
struct Config {
foo: String
}
impl Config {
pub fn default() -> Config {
Config {
foo: "default-value".to_string(),
}
}
pub fn from_str(&str) -> Config {
let mut c = Config::default();
let cl: ConfigLoader = toml::from_str(r#""#).unwrap();
// If we loaded something, change our config
if let Some(v) = cl.foo {
c.foo = v;
}
c
}
}
fn main() {
// Now everying is just peachy, default values get overridden only if they
// appear in the toml string
let config = Config::from_str(r#"");
}
In this trivial use case this naive-implementation works but it falls flat in more complex scenarios. Primarily, the two biggest issues with this implementation are:
- Adding new fields means duplicating fields in both
ConfigandConfigLoader - Every field needs a repeat of the
if let Some(v)...expression in order to "merge" the two structs
// I mean, this makes me cringe from a mile away
struct ConfigLoader {
a: Option<String>,
b: Option<String>,
...
y: Option<u32>,
z: Option<u8>,
}
struct Config {
a: String,
b: String,
...
y: u32,
z: u8,
}
// And it only gets worse......
if let Some(v) = cl.a {
c.a = v;
}
if let Some(v) = cl.b {
c.b = v;
}
...
if let Some(v) = cl.y {
c.y = v;
}
if let Some(v) = cl.z {
c.z = v;
}
Solution - Macros! Woo!
I KNEW that there was a better way to do this and that I would have to jump into macros in order to get some meta-programming going. From a quick glance at the naive-implementation above you can see there is definitely a repeating pattern to the implementation. What I needed was a way to define my fields once and then to have the structs generated by some mechanism. I discovered that macros are a prime candidate for this sort of thing and here is my solution:
#![feature(struct_field_attributes)]
extern crate toml;
use toml;
/// Macro that generates a `Config` and `ConfigTomlLoader`. Arguments are fields
/// of the struct and associated data.
///
/// # Usage
/// config!(my_field, String, "My defaul value".into(), "My docstring"
macro_rules! config {
($($element: ident, $ty: ty, $def:expr, $doc:expr); *) => {
#[derive(Deserialize)]
struct ConfigTomlLoader { $($element: Option<$ty>),* }
#[derive(Deserialize, Serialize)]
pub struct Config {
$(
#[doc=$doc]
pub $element: $ty
),*
}
impl Config {
pub fn default() -> Config {
Config {
$($element: $def),*
}
}
pub fn from_str(s: &str) -> Config {
let mut c = Config::default();
let cl = ConfigTomlLoader::from_str(s).unwrap();
$(
if let Some(e) = cl.$element {
c.$element = e;
};
)*
c
}
}
}
}
config! {
sqlite_name, String, "blotter.db".into(), "Name of sqlite database"
}
fn main() {
let c = Config::from_str(r#""#);
}
I won't go into too much detail as I am not 100% informed on macros and don't want to pass-along the wrong information but I'll break down the solution into smaller pieces to help explain things.
Invoking the macro
config! {
sqlite_name, String, "blotter.db".into(), "Name of sqlite database"
}
The macro is quite simple. Rather than manually specifying the fields in each struct, I invoke the macro as above and then it will generate two structs for me: Config and ConfigTomlLoader. The arguments are field name, field type, field value and field doc string.
The output of this is something like:
struct ConfigTomlLoader {
sqlite_name: Option<String>,
}
struct Config {
#[doc="Name of sqlite database"]
sqlite_name: String,
}
...
Field docs
#![feature(struct_field_attributes)]
This is an experimental feature from the compiler that is needed because I included the option to document my fields using the macro. So for this line to work #[doc=$doc] then this feature is required.
Pattern match
($($element: ident, $ty: ty, $def:expr, $doc:expr); *) => {
In a nutshell, a macro is a bunch of pattern matching cases where you are matching syntax elements of the language. Thats what ident, ty, expr are. This first book has a good section on these 'fragment specifiers' here. So the basic form is (SOME PATTERN) => { // SOME LOGIC }.
The $(...);* is a special way of saying that I expect a repeating amount of SOMETHING seperated by a semicolon, e.g. something; something; something; something. See here for an explenation of the repetition operator. In this case, I specify exactly what I expect to repeat as: $element: ident, $ty: ty, $def: expr, $doc: expr...which is field name, field type, field default value, field doc string.
Repeat repeat
struct ConfigTomlLoader { $($element: Option<$ty>),* }
So here I use the repitition operator again to create my ConfigTomlLoader but I wrap the type ($ty) in an Option<T> so that I can use it with the TOML crate.
$(
#[doc=$doc]
pub $element: $ty
),*
I do the same here, except this time I DON'T wrap the type with an Option. I also include my doc string here to make a nice output when I run cargo doc.
let mut c = Config::default();
let cl = ConfigTomlLoader::from_str(s).unwrap();
$(
if let Some(e) = cl.$element {
c.$element = e;
};
)*
Finally, I use the repition operator one last time to "merge" any loaded values into my config defaults.
Conclusion
So as you can see, Rust's macro system is quite approachable when solving simple meta-programming problems such as this. In the end I have a straight forward solution that lets me define my information once and then the macro will do some extra lifting for me. I am enjoying Rust a lot and as someone who hasn't used a lot of generic/meta programming patterns before, I am now learning to appreciate their power.