proc-macro2

What is `proc-macro2`?

`proc-macro2` is a Rust library crate that provides a re-export of the types from `proc_macro` crate (`TokenStream`, `Ident`, `Literal`, `Group`, `Punct`) but makes them available for use *outside* of a `proc-macro` crate. The original `proc_macro` crate types are special and are only available when compiling a `proc-macro` type crate.

Why is it needed?

When writing procedural macros in Rust, you often want to structure your code: parse input tokens, analyze them, generate output tokens, and potentially have helper functions or modules that perform parts of this logic. The problem is that the `proc_macro::TokenStream` and related types are not standard library types; they are tied to the compiler's procedural macro API and are only accessible from `proc-macro` crates.

This restriction makes it difficult to:

1. Share Logic: You can't put common token manipulation logic into a regular library crate and have both your `proc-macro` and other parts of your project (e.g., tests, main application) use it directly, because they won't have access to `proc_macro::TokenStream`.
2. Test Macro Logic: Testing the internal logic of a procedural macro becomes cumbersome without being able to instantiate and manipulate `TokenStream`s directly in your test suite.

`proc-macro2` solves this by providing a set of identical types that are compatible with `proc_macro`'s types. These `proc_macro2::TokenStream`s can be instantiated and manipulated in any Rust crate, including regular libraries, binaries, and test modules. At the boundary of a `proc-macro` function, `proc_macro2::TokenStream` can be easily converted to `proc_macro::TokenStream` (and vice-versa) using `From`/`Into` implementations.

Key Features and Uses:

* Universal `TokenStream`: Provides a `TokenStream` type that can be used anywhere, allowing macro logic to be written and tested outside of the strict `proc-macro` crate context.
* `Ident`, `Literal`, `Group`, `Punct`: Offers equivalent types for identifiers, literals, grouped tokens (like parentheses or braces), and punctuation marks.
* Compatibility: Designed to work seamlessly with other popular macro helper crates like `syn` (for parsing Rust syntax) and `quote` (for generating Rust code from `proc_macro2::TokenStream`). In fact, `syn` parses into `proc_macro2::TokenStream`s, and `quote!` generates them by default when used outside a `proc-macro` crate.
* Span Information: Carries `Span` information, crucial for providing accurate error messages that point to the correct location in the user's source code.

In essence, `proc-macro2` acts as a bridge, enabling robust development, testing, and modularization of procedural macro logic by making token stream manipulation available across the entire Rust ecosystem.

```rust
// In your Cargo.toml, add these dependencies:
// [dependencies]
// proc-macro2 = "1.0"
// quote = "1.0"

use proc_macro2::{Ident, Span, TokenStream};
use quote::quote;

/// This function demonstrates how to work with `proc_macro2::TokenStream`
/// types outside of a `proc-macro` crate. It takes an identifier and generates
/// a module with a public function inside it.
///
/// Such a function could be part of a regular library that a `proc-macro`
/// crate then uses to build its output.
fn generate_custom_module(module_name: &str) -> TokenStream {
    // Create a proc_macro2::Ident for the module name.
    // Span::call_site() is used here for simplicity, in a real macro,
    // you'd typically preserve the original span from the input tokens.
    let mod_ident = Ident::new(module_name, Span::call_site());

    // Create an identifier for the function inside the module.
    let fn_ident = Ident::new(&format!("greet_{}", module_name), Span::call_site());

    // Use the `quote!` macro to construct a `proc_macro2::TokenStream`.
    // The `quote!` macro is designed to work with `proc_macro2::TokenStream`
    // when not compiled within a proc-macro context.
    let generated_code = quote! {
        /// This is a generated module.
        pub mod #mod_ident {
            /// This is a generated function.
            pub fn #fn_ident() {
                println!("Hello from generated module '{}'!");
            }
        }
    };

    generated_code
}

fn main() {
    // Call our helper function to generate a TokenStream.
    let my_module_tokens = generate_custom_module("my_awesome_feature");

    println!("Generated proc_macro2::TokenStream:\n------------------------------------\n{}", my_module_tokens);

    // To demonstrate how this would be used in a real procedural macro:
    // If this code were inside a `proc-macro` crate's `#[proc_macro]` function,
    // you would convert the `proc_macro2::TokenStream` to `proc_macro::TokenStream`
    // and return it.
    //
    // For example:
    // #![proc_macro]
    // extern crate proc_macro;
    // pub fn my_macro_fn(_input: proc_macro::TokenStream) -> proc_macro::TokenStream {
    //     let tokens = generate_custom_module("my_macro_output");
    //     proc_macro::TokenStream::from(tokens) // Conversion happens here
    // }
}
```