Today, I’ll explain how the shortcomings of Rust’s procedural macros hurting the DevX for some applications. And how some great minds can surpass even those impediments. The credit goes to Leigh and Willem for the innovative unorthodox ways that I’ll explain in this post.
The Problem: Macros Are Strictly Limited By Their Scope
This is a very interesting one. Remember when you define a trait, and provide the default implementation for its methods, and then you implement that trait for your struct, you don’t need to fill in anything for the trait implementation if you want to go with the default implementations, right?
trait Dog {
fn bark(&self) {
println!("Bark!");
}
}
struct MyDog;
impl Dog for MyDog {} // no bark() implementation provided
fn main() {
let d = MyDog;
d.bark(); // uses the default "Bark!" implementation
}
Right?
Right???!!
Well, Stellar says hi :)
No Default Implementations Possible With Macros
In Stellar smart contract development,
- you define a struct for your contract,
- you implement the methods directly on your smart contract struct, and/or
- you implement a trait for your struct,
- and voila! You can call the methods implemented on your smart contract struct (whether direct methods on your struct, or methods implemented on your struct via a trait)
Sounds pretty reasonable, if you ask me.
So, what’s the problem? Let’s see:
This should work, but it doesn’t:
Trait definition file:
#![no_std]
use soroban_sdk::{Env, Symbol};
// Define a trait with a default method
pub trait Hello {
fn say_hello(e: &Env) -> Symbol {
Symbol::new(e, "HelloWorld")
}
}
Contract file:
#![no_std]
use soroban_sdk::{contract, contract_impl};
use my_crate::Hello;
// Define your contract
#[contract]
pub struct HelloWorldContract;
#[contract_impl]
impl Hello for HelloWorldContract {}
// The default methods are not parsed,
// and our smart contract does not have any methods callable...
Instead, we have to do this (only for the contract file):
#![no_std]
use soroban_sdk::{contract, contract_impl, Env, Symbol};
use my_crate::Hello;
// Define your contract
#[contract]
pub struct HelloWorldContract;
#[contract_impl]
impl Hello for HelloWorldContract {
fn say_hello(e: &Env) -> Symbol {
Symbol::new(e, "HelloWorld")
}
}
// although we have the default implementation provided in the trait definition
// we still have to provide it once more in the contract implementation phase...
Why do we have to provide the default implementation? Because of the #[contract_impl] macro. Let me explain:
#[contract_impl] macro does a lot under the hood. For this single method implementation code of 5 lines, it generates 187 lines of additional code, consisting of:
try_*variantsxdrcompatibility- auths and mocks
And the #[contract_impl] procedural macro can only generate these for the code it is annotating. If we provide the default implementation in the struct implementation as well, then the #[contract_impl] macro’s scope is the following code:
#[contract_impl]
impl Hello for HelloWorldContract {
fn say_hello(e: &Env) -> Symbol {
Symbol::new(e, "HelloWorld")
}
}
Thus, it can generate all the smart contract related things for the say_hello() method.
But, if we do not provide the default implementation (which is the expected behavior), then the #[contract_impl] macro’s scope is the following code:
#[contract_impl]
impl Hello for HelloWorldContract {}
Proc-macros get expanded before type-checking, trait resolution, or default method filling happens. That’s why, in the above case, #[contract_impl] does not have access to the say_hello() method, and cannot generate the necessary code.
Although it may seem like I’m talking about Stellar, I believe this problem is not specific to Stellar. If you want your procedural macro to work on the default implementations of a trait, you will face this very problem. And I can imagine many different projects may have come across this problem and decided to change their code accordingly to work around the problem.
Because there is no straightforward way in Rust to get around this problem. In order to solve it, we will do some dark magic.
The Workaround
Our goal is: smart contract developers need not to provide the default implementations once more. The code should be just this, if they want the default behavior:
#[contract_impl]
impl Hello for HelloWorldContract {}
In other words, we want the usual Rust behavior.
One workaround for this, and I do not use the word solution deliberately here, is to have another macro, let’s call it #[default_impl].
#[default_impl] macro will have all the default implementations hardcoded in it, so when it expands, the default implementations will be provided by the macro. It may look something like this:
#[default_impl]
#[contract_impl]
impl Hello for HelloWorldContract {}
Doesn’t look disgusting. I’d say much better than having to copy-paste 10 or 20 default methods. When the #[default_impl] expands (and the #[default_impl] will expand prior to the #[contract_impl] macro due to ordering), you will have the below code:
#[contract_impl]
impl Hello for HelloWorldContract {
fn say_hello(e: &Env) -> Symbol {
Symbol::new(e, "HelloWorld")
}
}
The question is, how will #[default_impl] know about the default implementations for each trait?
Well, you can simply hardcode the necessary default implementations for each trait, and match against the given trait. Something like:
match (trait_name) => {
"Hello" => vec![
syn::parse_quote! {
fn say_hello(e: &Env) -> Symbol {
Symbol::new(e, "HelloWorld")
}
},
],
"OtherTrait" => vec![
syn::parse_quote! {
fn other_method() {
println!("hmm");
}
}
]
}
If you didn’t like having 2 macros on top of the impl block, this is also easily solvable. You can embed this #[default_impl] functionality to #[contract_impl] macro, and it can look like this:
#[contract_impl]
impl Hello for HelloWorldContract {}
For simplicity, let’s assume we are the maintainers/authors of the #[contract_impl] macro, and we can change its behavior if we want to for the rest of this article. If this was not the case in reality, we could have gone with the #[default_impl] additional macro approach.
Downside of This Workaround
Since the macros annotating the impl block cannot access the trait definition in our case (because the trait definition is out of their scope), they cannot find and derive the default implementations on their own. This restriction results in having to hard-code the default implementations in our macro.
If we decide to change the default implementations of our trait, we have to change the source code of the macro as well.
The astute readers may think: “why do we even have the default implementations present in the trait at this point, if they are not going to be used directly?”
And it is correct. If we remove the default implementations from the trait definition, the code will still work. The reasons we decided to keep the default implementation in the trait definition are:
- if the developer wanted to use the traits in some other context, the default implementations will be beneficial,
- if the developer wants to take a look at the default implementations: not having to dive into macro code, but seeing it in the trait definition as expected will be better for the developer experience.
The Solution
The workaround I’ve described in the above section, is not bad per se, and definitely tolerable. So if you don’t fancy complex things, and the above is enough for your needs, there is no blame in selecting the workaround instead of the solution.
What qualifies as a solution in my opinion is to have the above workaround, without the hardcoded part, so the macro can somehow communicate with the trait definition, and embed the default methods from the trait definition into the impl block.
But Proc-Macros Cannot Access The Code Outside of Their Scope?
Yes…
And on top of that, proc-macros cannot even communicate with other proc-macros.
They are strictly bound to the scope they are annotating. Quite annoying, but at the same time makes a lot of sense. I’m not questioning Rust’s design choices here, but merely pointing out the inconvenience we are having.
The question at hand is, “can we somehow manage to establish communication between the trait definition and our macro?”
I thought the answer to this question is a definite “NO!”. But, Leigh and Willem proved me wrong. And I was truly exhilarated to see that is possible!
How to Establish Communication Between Macros
What we want is, our #[contract_impl] macro to:
- have access to the default methods of the trait definition,
- inject these parsed default methods into the
implblock.
Given that #[contract_impl] macro cannot access the original trait definition, we have to be extra creative.
If we can frame the problem better, the solution will be easier to spot.
HIR
As I mentioned earlier, macro expansions happen before HIR (High-Level Intermediate Representation). In plain English, our only option is to somehow copy the raw code from the trait, and paste it into the scope of #[contract_impl]. No other way will work. Remember, all the type checking, trait resolution, etc. will happen AFTER macro expansions.
In other words, we shouldn’t approach this problem from the perspective of traits or generics, or default methods, etc. These don’t mean much before HIR. We are dealing with raw code here.
Copy The Code From Trait Definition?
At this point, you know that it is impossible for our #[contract_impl] to copy code from the trait definition. Because it is not annotating the trait definition, but the impl block.
But we DEFINITELY need to copy the code from the trait definition. That’s why we need another macro that annotates the trait definition. Something like this:
#[contract_trait]
pub trait Hello {
fn say_hello(e: &Env) -> Symbol {
Symbol::new(e, "HelloWorld")
}
}
OK, let’s go step by step. It is definitely possible for this new macro #[contract_trait] to parse all the given default methods of this trait, because it is placed at the correct scope.
Now, the problem is, how to establish communication between #[contract_trait] and #[contract_impl]?
How Can Macros Communicate?
This is where the magic happens.
Normally, macros from different scopes, cannot communicate. They aren’t designed for that.
That’s why what we are doing here can be counted as black magic or an unorthodox approach: we are achieving something that is not intended by design.
And here is the critical question that will guide us to solution. What code can the proc-macros use aside from the code they are annotating?
The answer is actually pretty simple: any code! For example, the macro can use vec!["1", "2", "3"] if it wants to.
The problem is, we don’t know whether vec! is imported. And there isn’t even a way to check if the macro is working in a std or no_std environment. So, although the macro can potentially use any code, the obstacle here is that we don’t know whether the code we want to use is properly imported and available in the first place. Thus, we generally only use the code that is under the scope of the macro.
In this case however, we are the author of both macros #[contract_impl] and #[contract_trait].
If #[contract_trait] exports all the default implementations as raw code, and if #[contract_impl] imports this publicly exported raw code, then it may just work!
The first big problem is, how to export this raw code?
You’d hate me, but another macro (this time declarative) is just perfect for this.
Yet Another Macro
If you are rolling your eyes and thinking “are we going to write another macro?”, we won’t be doing that 👍
We are going to do something much worse: we are going to make #[contract_trait] write a declarative macro for us!
In this post, I don’t want to walk you through the full code. The aim of this post is to abstract away the details and explain the core ideas of how to achieve the solution. Let’s remember how we are planning to use the #[contract_trait] macro:
#[contract_trait]
pub trait Hello {
fn say_hello(e: &Env) -> Symbol {
Symbol::new(e, "HelloWorld")
}
}
So, inside of #[contract_trait] implementation, we will roughly have something like this:
let macro_ident = macro_ident(&input.ident);
let output = quote! {
#[macro_export]
macro_rules! #macro_ident {
(
$impl_ident:ty,
$impl_fns:expr,
) => {
#path::contract_impl_trait_default_fns_not_overridden!(
trait_ident = #trait_ident,
trait_default_fns = [#(#fns),*],
impl_ident = $impl_ident,
impl_fns = $impl_fns,
);
}
}
};
Let me briefly decipher what is going on in the above.
In the below code, we create/export our declarative macro, which has the name #macro_ident, which is the name of our trait.
let macro_ident = macro_ident(&input.ident);
#[macro_export]
macro_rules! #macro_ident {
Now, it is important to remember that this declarative macro we are exporting here will be used later in the impl block. And this declarative macro will take the following input when it will be called in the impl block:
- ident of the impl block
- functions provided in the impl block (to override the default implementations if needed)
macro_rules! #macro_ident {
(
$impl_ident:ty,
$impl_fns:expr,
) => {...}
}
And lastly, it calls an internal function with the following information:
- ident of the trait (
#[contract_trait]macro already embedded this info to the generated declarative macro) - default implementations (
#[contract_trait]macro already embedded this info to the generated declarative macro) - ident of the impl block (will be provided to the declarative macro as an argument)
- functions inside the impl block for override mechanism (will be provided to the declarative macro as an argument)
#path::contract_impl_trait_default_fns_not_overridden!(
trait_ident = #trait_ident,
trait_default_fns = [#(#fns),*],
impl_ident = $impl_ident,
impl_fns = $impl_fns,
);
The above part may be confusing, so I’ll explain what’s going on step-by-step:
#[contract_trait]macro is annotating the trait definition with default functions present#[contract_trait] pub trait Hello { fn say_hello(e: &Env) -> Symbol { Symbol::new(e, "HelloWorld") } }#[contract_trait]macro parses the name(ident) of the trait, and the default functions#[contract_trait]macro generates a declarative macro- this declarative macro already have the name of the trait, and the default functions embedded into itself, thanks to
#[contract_trait]parsing these and passing them to the declarative macro - declarative macro also needs to know about the functions provided in the
implblock (not the trait definition block), so that if the developer wants, they can override the default methods. #[contract_trait]macro has no information about theimplblock at all, because it is annotating the trait definition block- so, the declarative macro will take the necessary information for the
implblock as arguments:$impl_ident:ty(name of the impl block), and$impl_fns:expr(functions provided in theimplblock) - this declarative macro will be called by the
#[contract_impl]macro - since
#[contract_impl]macro has all the necessary information about theimplblock, it can provide the necessary arguments to the declarative macro when calling it - declarative macro will expand and fill the
implblock with all the missing default implementations
But, How to Export and Import This Declarative Macro?
If we can somehow manage to make this declarative macro usable in the scope of #[contract_impl] macro, then we have officially solved the problem!
Let me do a quick rehearsal, and then I’ll show you how to tackle this last problem of importing/exporting the declarative macro as well.
#[contract_trait]macro parses the default implementations#[contract_trait]macro packs these default implementations into a declarative macro, and exports it- this declarative macro right now only have the default implementations in it, but expects the
implblock functions (overriding ones) as its arguments #[contract_impl]macro parses the overriding methods in theimplblock- The
#[contract_impl]macro invokes the declarative macro by providing theimplblock functions as arguments - The declarative macro expands, and fills the
implblock with the necessary default methods. - The
#[contract_impl]macro has the usual business logic placed after the invocation of the declarative macro call, so that it can generate all the smart contract related weird stuff, now that we have all the default and overridden methods available under theimplblock
Let’s tackle our last problem: “how can the #[contract_impl] macro invoke the generated declarative macro?”
This problem is much bigger than it seems. Because the trait can be (and most probably is) defined elsewhere. God knows where… Maybe in another module, maybe in another crate.
I hear some foolish voice mumbling and arguing:
- but, but, isn’t the declarative macro exported by the
#[contract_trait]macro? - and we know that the
#[contract_trait]macro is annotating the trait - and we know that the trait is properly imported in the file where we want to use
#[contract_impl] - so, don’t we actually know where the declarative macro is coming from? It is literally present in the
use {god_knows_which_crate}::TraitName;
Now is a good time to take a look at the revised code of our contract and our library:
Trait definition file:
#![no_std]
use soroban_sdk::{Env, Symbol, contract_trait};
#[contract_trait] // here is our new macro
pub trait Hello {
fn say_hello(e: &Env) -> Symbol {
Symbol::new(e, "HelloWorld")
}
}
Contract file:
#![no_std]
use soroban_sdk::{contract, contract_impl};
use god_knows_which_crate::Hello; // OH, LOOK AT WHAT WE HAVE HERE!
// Define your contract
#[contract]
pub struct HelloWorldContract;
#[contract_impl]
impl Hello for HelloWorldContract {}
// The default methods are not parsed,
// and our smart contract does not have any methods callable...
- so, we just need to put this statement in our
#[contract_impl]macro implementation:use {god_knows_which_crate}::{declarative_macro_name};
Wait, this is not a foolish mumbling voice. This was just me at 1 AM.
Anyway, this won’t work. Because, although we have the statement use god_knows_which_crate present in the file that we are using #[contract_impl] in, there still is no way to hardcode or pass this information to the #[contract_impl] macro.
Here comes the actual black magic that I thought would not be possible to do in Rust. I was truly amazed (not in a bad or good way, just amazed) when I saw it for myself.
You can give the exact same name to the declarative macro and your trait, and with a single import statement, you can import them both, and compiler can resolve them both, and there won’t be any problem.
!!?!?!?!?!??!
Here is a toy example I created. Feel free to test it on your machine if you want to see it for yourself.
stellar--rs-soroban-sdk--pr1507-main/
├── contract/
│ ├── Cargo.toml
│ └── src/
│ └── main.rs
└── library/
├── Cargo.toml
└── src/
└── lib.rs
library/Cargo.toml:
[package]
name = "doglib"
version = "0.1.0"
edition = "2021"
library/lib.rs:
// Export the trait
pub trait Dog {
fn bark(&self) {
println!("Bark!");
}
}
// Declarative macro definition
#[macro_export]
macro_rules! Dog {
() => {
println!("dog");
};
}
See that the library exports both the trait and the declarative macro under the same name Dog.
contract/Cargo.toml:
[package]
name = "contract"
version = "0.0.0"
edition = "2021"
publish = false
[dependencies]
doglib = { path = "../library" }
contract/main.rs:
use doglib::Dog; // import only `Dog`, but which `Dog`?
struct MyDog;
impl Dog for MyDog {}
fn main() {
let d = MyDog;
d.bark(); // works!
Dog!(); // works!!!
}
Although we have only use doglib::Dog, the compiler is not confused.
Both the trait and the declarative macro are successfully imported! And the code works!
Namespaces
Macros and types have different namespaces. This allows the compiler to resolve the mystery.
And here is the detailed article on that for those who want to dive deeper.
A snippet from that article, showcasing how overlapping names in different namespaces can be used unambiguously:
// Foo introduces a type in the type namespace and a constructor in the value
// namespace.
struct Foo(u32);
// The `Foo` macro is declared in the macro namespace.
macro_rules! Foo {
() => {};
}
// `Foo` in the `f` parameter type refers to `Foo` in the type namespace.
// `'Foo` introduces a new lifetime in the lifetime namespace.
fn example<'Foo>(f: Foo) {
// `Foo` refers to the `Foo` constructor in the value namespace.
let ctor = Foo;
// `Foo` refers to the `Foo` macro in the macro namespace.
Foo!{}
// `'Foo` introduces a label in the label namespace.
'Foo: loop {
// `'Foo` refers to the `'Foo` lifetime parameter, and `Foo`
// refers to the type namespace.
let x: &'Foo Foo;
// `'Foo` refers to the label.
break 'Foo;
}
}
Credit goes to Nicolas Garcia for pointing this out!
How It Will Look Like
All of our problems are solved, and we don’t even need to import the declarative macro in the #[contract_impl] macro, because if our declarative macro has the exact same name as our trait, it will be automatically imported!
What kind of sorcery is this?
I’ll give you the final result on how it will look like for the contract developer:
Trait definition:
#![no_std]
use soroban_sdk::{contract, contract_trait, Env, Symbol};
#[contract_trait]
pub trait Hello {
fn say_hello(e: &Env) -> Symbol {
Symbol::new(e, "HelloWorld")
}
}
Contract implementation:
#![no_std]
use soroban_sdk::{contract, contract_impl};
use my_crate::Hello;
#[contract]
pub struct HelloWorldContract;
#[contract_impl]
impl Hello for HelloWorldContract {}
What you see above is the end product that will be used by the smart contract developer. And it will achieve the following:
- Communication between macros
- The default implementation filled by the
#[contract_impl]macro based on the trait name - The default implementations of the relevant trait are derived from the trait definition
- If methods are provided in the
impl Hello for HelloWorldContract, they can still override the default implementations
AWESOME!
For the geeks that are curious about the full implementation, here is the relevant PR
