![]()
We are excited to announce the open-sourcing of Metalama.Compiler, the Roslyn fork used by Metalama to modify code dynamically during compilation. Metalama.Compiler enables anyone to build arbitrary source transformers.
The Metalama Compiler forms the foundation of Metalama, our high-level meta-programming framework that assists C# developers in reducing boilerplate using aspect-oriented programming and combating architecture erosion with architecture-as-code. The Metalama Compiler, representing several months of work, is a mature project that has seen no significant changes in the last 12 months, except minor bug fixes. By open-sourcing Metalama.Compiler, we aim to make it the preferred platform for Roslyn enthusiasts and SyntaxNode aficionados who want genuine source code generation capabilities. For application developers, building aspects based on Metalama.Framework is significantly simpler – not to mention that they can rely on a library of ready-made aspects on the Metalama Marketplace.
Why Did We Fork Roslyn?
Essentially, we forked Roslyn because we needed the ability to alter the code during compilation for our Metalama product.
When Microsoft introduced Roslyn in 2014, it was a game-changer for C# developers as it turned the traditionally opaque compilation process into a transparent and extendable API. Thanks to Roslyn, developers could see and use the rich information the compiler has about code. The initial extension points of Roslyn allowed any package to report custom warnings and errors using analyzers, suppress redundant warnings, and provide code fixes and refactorings. However, developers could not modify the code model.
In 2020, Microsoft took another step to open the compiler and introduced source generators, an extensibility point that allows any package to generate new source code – typically partial classes – and add them on-the-fly to the current project. The compilation process became open by addition. However, it was still closed to modifications, as there was still no way to modify an existing syntax tree.
At this point, I wrote a long and fruitful technico-philosophical essay on the cognitive nature of programming and how framework and language designers should avoid promoting hacks. As it turned out, my brain continued to think about this topic, and we soon started considering how to take advantage of these new Roslyn features to advance our ultimate mission – to improve the productivity of C# application development teams. We began to consider rewriting PostSharp, based on MSIL rewriting, to Roslyn. The first prototypes were completed between August 2020 and February 2021. This project became known as Metalama.
The first step was to fork Roslyn and open it for code modifications, not just additions. We added a concept of source transformer, just as there already were analyzers and source generators. This fork is Metalama.Compiler. It was maintained in a private repository from August 2020 until today. Metalama.Framework, our high-level meta-programming framework featuring aspect-oriented programming for the masses, acts as a plug-in to Metalama.Compiler at compile time, and as a plug-in of the vanilla Roslyn at design time when it runs under the IDE.
We invested a significant amount of effort into this fork – probably more than six months of real compiler professionals’ time. Initially, we were hesitant to make it publicly available because we instinctively wanted to protect our work. But in the meantime, we realized that our real value was not in opening Roslyn for code modifications, but in making meta-programming simple and accessible for all developers, without requiring in-depth compiler knowledge. Indeed, we now estimate that creating aspects with Metalama.Framework is two orders of magnitude simpler than doing that directly with the Roslyn API. Not surprisingly, the implementation of Metalama.Framework was also much more complex and demanding than the modifications we made to Metalama.Compiler, and our fears that someone is just going to “copy” us faded when we realized that just stabilizing our T# template language took us one year of hard work.
Our initial vision with Metalama.Compiler was to support source transformers in addition to the existing Roslyn extension points, and to address all issues that would arise as a result of this feature: mapping of PDBs, mapping of diagnostics, interaction of this feature with analyzers and source generators, and so on. This vision is now mostly achieved, and we consider Metalama.Compiler as a stable project. We don’t have plans for significant new features at the moment, and if we will have, they will always be features that support our main product, Metalama.
Some areas may be under-tested at the moment and will require more attention, notably the interactions with source generators.
We plan to continue merging changes originated from the Roslyn team at Microsoft. We do this whenever there is a new stable release of Roslyn. We don’t plan to merge any prerelease changes.
Technical Overview
This fork extends the capabilities of analyzer projects and allows them to execute arbitrary transformations of source code during compilation via the ISourceTransformer interface from the Metalama.Compiler.Sdk package. Source transformers can replace entire syntax trees in the compilation. They can also add managed resources.
As an illustration, here is the typical skeleton of a source transformer:
usingMetalama.Compiler;usingMicrosoft.CodeAnalysis;usingMicrosoft.CodeAnalysis.CSharp;usingMicrosoft.CodeAnalysis.CSharp.Syntax;[Transformer]publicclassMySourceTransformer:ISourceTransformer{publicvoidExecute(TransformerContextcontext){varrewriter=newRewriter();foreach(vartreeincompilation.SyntaxTrees){context.ReplaceSyntaxTree(tree,tree.WithRootAndOptions(rewriter.Visit(tree.GetRoot()),tree.Options));}}classRewriter:CSharpSyntaxRewriter{// TODO: Implement transformations.}}
By default, errors and warnings are reported on the source code, and PDB files map the MSIL binary code to the source code as well, so that debugging and breakpoints work as usual. It is possible to toggle that behavior and debug the transformed code by setting some MSBuild properties. The ability to map diagnostics and PDBs back to the source code is by far the most complex feature of Metalama.Compiler.
When there are multiple source transformers in a project, they must be deterministically ordered. One way to order transformers is to use the TransformerOrderAttribute assembly-level custom attribute. A second way is to define the MetalamaCompilerTransformerOrder MSBuild property.
By default, analyzers run after all source transformers. This allows these analyzers to see the final code. However, some analyzers, for instance, the ones checking formatting, need to see the source code. To move the execution of an analyzer before the source transformers, use the MetalamaSourceOnlyAnalyzers MSBuild property.
Metalama.Compiler only covers the compilation use case. It does not cover other Roslyn use cases such as scripting or design-time. The Metalama.Compiler.Sdk package references the vanilla Microsoft.CodeAnalysis.CSharp package, not the forked versions, which makes it possible to have analyzers, diagnostic suppressors, source generators and source transformers in the same project. Source transformers won’t be loaded at design time when the analyzer assembly will be loaded, but the other assets will load successfully.
flowchart BT
YourProject --> Metalama.Compiler.Sdk
Metalama.Compiler.Sdk --> Microsoft.CodeAnalysis
Metalama.Compiler.Sdk --> Metalama.Microsoft.CodeAnalysis
subgraph IDE
Microsoft.CodeAnalysis
end
subgraph Compile-Time
Metalama.Microsoft.CodeAnalysis
end
Microsoft.CodeAnalysis["`_vanilla_
Microsoft.CodeAnalysis`"]
Metalama.Microsoft.CodeAnalysis["`_forked_
Microsoft.CodeAnalysis`"]
Please note that Metalama.Compiler only works for C#. There is no plan to support Visual Basic.
For further details, see the project home on GitHub and the API documentation.
Due to the high complexity and efforts involved in achieving anything significant in source code transformation, we believe that building a source transformer is not for everyone. Metalama Compiler is for researchers and framework developers. However, if you are an application developer, and your job is to deliver business features, then you should probably think more than twice before embarking on the source transformation journey.
The problem is that Roslyn is much too low level. For anything less trivial than making all public methods virtual, you will spend days catering for details that are already handled by a high-level meta-programming framework like Metalama.Framework.
For instance, one significant difference in the design philosophies of Roslyn and Metalama Framework is that Roslyn is compiler sympathetic while Metalama Framework is developer sympathetic. Roslyn exposes various extensibility APIs such as analyzers, diagnostic suppressors, source generators or code fix providers. If you want to implement one aspect directly using these Roslyn APIs, you will need to integrate with all these APIs, which are all their own entry points and have very little integration. However, with Metalama Framework, you implement one aspect class (one entry point) and have access to all the Roslyn features from the same place: reporting or suppressing diagnostics, generating source code or suggesting code fixes.
If you encounter code transformations not supported by Metalama.Framework, your initial response should be to consider the following: firstly, whether there is a supported method to achieve the same result; secondly, if it’s possible to forgo the feature; and thirdly, to construct your Roslyn-level code transformation using Metalama.Framework.Sdk instead of Metalama.Compiler.Sdk. Utilizing Metalama.Compiler.Sdk allows your low-level code to benefit from all the features of Metalama.Framework, letting you focus solely on the code transformation rather than the entire scaffolding.
However, if you wish to construct another aspect-oriented framework, you can certainly build it on top of Metalama.Compiler. It may be an intellectually stimulating task and we wish you the best of luck.
Contributing
You can contribute by reporting bugs, submitting feature requests to the project’s issue tracker, or by proposing pull requests. Both pull requests and feature requests should align closely with the mission of Metalama.Compiler, which is to make the compiler open for code modification in an extensible way. The primary objective of this project is not to serve as a universal fork for all changes rejected by the Roslyn team.
Wrapping Up
We have made Metalama.Compiler, our beloved Roslyn fork, open-source. This project has been in development since August 2020. We hope it will prove useful to syntactic enthusiasts and other semantic (or symbolic, as the case may be) zealots.
My personal journey with meta-programming began in 2004 when I penned the first lines of PostSharp – lines that interpreted bits of IL and abstracted them into an object model. A crucial lesson learned from this journey is that programming is not merely about moving bits. It is a cognitive and social activity. We don’t write code for hardware. We don’t even write it for the compiler. The primary consumers of our code are always humans: you, the future you, your colleagues, and your future colleagues who will still be maintaining, a decade from now, the code you are currently writing. This is why we need advanced programming languages and frameworks: to enable the expression of business or scientific solutions at a level of abstraction that aligns with human understanding – not with compilers or hardware.
Bridging the abstraction gap between bits and minds: this has been our journey, starting from PostSharp and now continuing with Metalama.
When developing a source transformer, always consider the perspective of the developer who will be instructed to work with it. Creating a better programming experience is hard work, but it’s worth it. Metalama Compiler stands as the latest gem in your toolkit.
The new logo has only been active for two weeks, but it already feels as if it has represented us for years.
