XCG Engine

About this document

This document provides an overview of the XCG engine. It contains information needed by those who want to implement extensions, or simply wish to understand XCG a little better.

This document can also be used as a starting point in understanding the XCG source code, as it makes several references to classes and implementation details.

Code Generation in XCG

The XCG engine was designed to meet several criteria, the most important ones being flexibility and extensibility. In order to facilitate this, every part of the process has been componentized, making it clear for extension implementers where they need to hook in.

The basis of the XCG engine lies in generators. Conceptually there is also the generating element, but from the engine's point of view, a generating element is simply a specialized generator. When viewing an XCG source file, it can be viewed as a hierarchical structure of generators. The first step of the process is then to parse the XCG source file, and represent this structure in an in-memory tree. In this tree, represented by the XcgTree class, every tree node is a subclass of the Generator class. The special class GeneratingElement, which of course represents the generating elements in the source code, also derives from Generator. Every generator in XCG or an extension has its own class that is responsible for implementing the functionality of that generator.

To get from this XcgTree to code, XCG uses the .Net provided CodeDom, which represents the syntax tree of a source file in a language-independant format. Each generator that is part of the XcgTree can construct part of the CodeDom. It can also modify and extend the CodeDom objects generated by its child generators.

Once a complete CodeDom tree is constructed, with a CodeCompileUnit as root, a CodeDomProvider, such as the CSharpCodeDomProvider, is used to create the resulting source code.

To summarize, the code generation process consists of the following steps:

Reading in the source file, parsing it to an object model.
Generating the CodeDom tree from that object model.
Using a CodeDom provider to generate either source code or a compiled assembly.

Below we will examine every step in detail.

Step 1: Parsing the source file

In order to provide the greatest amount of flexibility for extensions, the parsing algorithm for XCG source files has been compartmentalized as much as possible. Every extension in XCG uses its own XML namespace URI. The URI 'http://www.liacs.nl/~sgroot/xcg/generators.xsd' is reserved for the XCG intrinsic generators, and generating elements have an empty namespace URI. Every extension should have its own, unique namespace URI. An extension then registers itself with XCG by associating a class that implements IXcgExtension with that namespace URI. XCG itself also uses this mechanism; the class XcgIntrinsic implements IXcgExtension and allows the main parser to handle the XCG intrinsic generators.

The extension class has three tasks: it tells the main parser which namespace URI it handles, it provides it with an XSD schema for that namespace, and it provides it with a method to get the class that implements the functionality of a certain generator.

The constructor of this class will then be invoked, and it is responsible for doing the actual parsing of the current element. The constructor has a choice; it can choose to process the element and its children all the way to the end element, or just the start element and leave the children up to the main parser. This last method is more commonly done.

In outline, the main parser works like this (this is implemented in the XcgTree.ReadXml method):

Read the next element of the XML file.
If the namespace URI is empty, create a GeneratingElement and continue with step 4.
If the namespace URI is not empty:
1. Get the class implementing IXcgExtension for that namespace (this can be XcgIntrinsic).
2. Use the GetGeneratorClass method to get the class that implements the generator for the current element name.
3. Create a new instance of this class.
If the constructor for the class indicates that the element was completed, add it to its parent. If not, we will set this class as the current parent, and add the next elements to this generator until we encounter this element's end element.
Repeat from step 1 until the end of the document has been reaced.

Step 2: Generating the CodeDom tree.

All classes that inherit from the Generator class must implement a method called Process. The Process method is responsible for creating the CodeDom objects for this generator. The Process method returns a CodeTypeDeclaration. A CodeTypeDeclaration represents the declaration of a class, struct or other type in code. A generator may not always want to create a type of its own. In fact, XCG has only one generator that creates a type of its own, and that is gen:Class (extensions can of course define generators that create types). If a generator does not want to create a type, it must still return a CodeTypeDeclaration, but one with a special name. This special name indicates that the members of this temporary CodeTypeDeclaration must be merged with the real CodeTypeDeclaration that is eventually created.

This process of merging CodeTypeDeclarations happens often in XCG, and it is important to understand how it works. The way members of a CodeTypeDeclaration are merged into another CodeTypeDeclaration depends on their type. Most types of members (CodeMemberField, CodeMemberProperty, CodeMemberEvent, CodeSnippetTypeMember and CodeTypeDeclaration, they are simply added to the resulting type. For CodeMemberMethods (which also included CodeConstructor), special action is taken. Whenever two methods are found of the same name (or two constructors are found), they are merged together. Parameters with matching names and types become a single parameter, and parameters that have no equivalent in the other method are simply added. Parameters that match by name but not by type result in an error. This means it is impossible for generators in XCG to create a method or constructor that overloads another method or constructor, since they'll all be merged together. This functionality is necessary; if it hadn't been done, every time you use a parameter expression to initialize a value it would create an overloaded constructor, making it impossible to initialize all of the values at once, which is in 99% of the times what you wanted to do.

A generator can simply merge the CodeTypeDeclarations of its children into its own as they were given to him, but of course it can decide to perform additional actions on them before merging them. Whenever a generator wants to operate directly on a CodeTypeMember of the CodeTypeDeclaration of one of its children, this is where it does it.

However, if a generator wants to use an object such as a field that was generated by one of its children as a target object (that means it wants to use a CodeExpression that refers to it, such as a CodeFieldReferenceExpression for a field), it should not do this by inspecting the CodeTypeDeclaration. Instead it should use the target object mechanism already mentioned in the XCG user documentation. In order to work with a target object, a generator must override the Generator_TargetObjectGenerated method. Here it gets a CodeExpression object that it can use for those purposes. If a generator wants to modify the CodeExpression as it is used by the child generators, it can also do it here.

The reason for the this target object structure is preserving the hierarchy. When a generator gets a CodeTypeDeclaration from a child, it will contain all the members generated by that child and all its descendants. This can be a problem. An example of this is a generating element that represents a collection. Its job is to add all fields generated by its descendants to the collection. It uses target objects for this. Imagine if you have such a generating element, and inside it is a generating element that also is a collection. When looking at the CodeTypeDeclaration, the collection would see all the fields, including those that have already been added to the second collection! For this reason, it uses target objects instead, and stops their propagation. When the inner collection sees a target object to add to the collection, it doesn't propagate it to its parents, so the outer collection doesn't see it. This wouldn't have been possible if they had directly operated on the CodeTypeDeclarations.

The target object event structure is also used for another reason. As indicated in the user documentation, an extension can define additional attributes that can be added to a generating element. These attributes will be passed down together with the target object for the namespace, so that a parent generator from this extension will see them and know what to do with them.

There are a few situations where this structure of calling Process isn't followed. A generating element representing a property of the parent element (another generating element or gen:Class, or perhaps an extension generator that also allows this behaviour) needs another method called, namely ProcessAsPropertyAssignment, to work properly. Certain generators such as gen:Create and gen:Base also have special methods that must be called instead of Process. The pattern here is that these are all generators that need information from their parents to work properly. The Process method only works the other way around: a generator can get information from its children, and return information to its parent, it cannot accept information from its parent, so in cases where this is necessary, different action is taken. The gen:CodeFile generator also has a special method, because it needs to return a CodeCompileUnit instead of a CodeTypeDeclaration.

Lets look at an example,

<gen:CodeFile namespace="ExampleNamespace" xmlns:gen="http://www.liacs.nl/~sgroot/xcg/generators.xsd">
  <gen:Class name="MyClass">
    <gen:Property name="Prop1" field="_prop1" type="System.String" />
    <gen:Property name="Prop2">
      <System.DateTime gen:id="_date" >2005, 02, 14</System.DateTime>
    </gen:Property>
  </gen:Class>
</gen:CodeFile>

In this situation we get the following object tree, all inheriting Generator:

CodeFileGenerator
   ClassGenerator
      PropertyGenerator
      PropertyGenerator
         GeneratingElement

The ProcessRoot method of CodeFileGenerator will be called, which in calls the Process method of ClassGenerator, which in turns calls them on the PropertyGenerators, the second of which will call Process on the GeneratingElement.

The GeneratingElement will create a CodeTypeDeclaration with the special name to indicate it's temporary. It adds a CodeMemberField to it, and a CodeConstructor that does the initialization of the field. In addition, it raises a target object event, which the PropertyGenerator handles. It uses the target object as the field that holds the value of the property. It lets the target object propagate further up, but nothing more is done with it. The GeneratingElement returns its CodeTypeDeclaration. The PropertyGenerator create its own CodeTypeDeclaration, again with the temporary name, and addes a CodeMemberProperty to it. It also merges the GeneratingElement's CodeTypeDeclaration with it, and then returns it. The ClassGenerator then creates a CodeTypeDeclaration, this time with the name "MyClass" instead of the special name, and merges the CodeTypeDeclarations from the two PropertyGenerators into it. The CodeFileGenerator will add this CodeTypeDeclaration to a CodeCompileUnit, and return that to be used in step 3.

You end up with the following CodeDom tree.

CodeCompileUnit
   CodeNamespace (namespace ExampleNamespace)
      CodeTypeDeclaration (class MyClass)
         CodeMemberField (_prop1)
         CodeMemberField (_date)
         CodeConstructor
            CodeAssignmentStatement
               Left=CodeFieldReferenceExpression (_date)
                  TargetObject=CodeThisReferenceExpression
               Right=CodeObjectCreateExpression
                  CreateType=CodeTypeReferenceExpression (System.DateTime)
                  Parameters=CodeExpression[] (CodePrimitiveExpressions for the numbers)
         CodeMemberPropery
      CodeMemberProperty (Prop1)
         SetStatements=CodeStatementCollection
            CodeAssignStatement
               Left=CodeFieldReferenceExpression (_prop1)
                  TargetObject=CodeThisReferenceExpression
               Right=CodePropertySetValueReferenceExpression
         GetStatements=CodeStatementCollection
            CodeMethodReturnStatement
               Expression=CodeFieldReferenceExpression (_prop1)
                  TargetObject=CodeThisReferenceExpression
      CodeMemberProperty (Prop2)
         SetStatements=CodeStatementCollection
            CodeAssignStatement
               Left=CodeFieldReferenceExpression (_date)
                  TargetObject=CodeThisReferenceExpression
               Right=CodePropertySetValueReferenceExpression
         GetStatements=CodeStatementCollection
            CodeMethodReturnStatement
               Expression=CodeFieldReferenceExpression (_date)
                  TargetObject=CodeThisReferenceExpression

Step 3: Generating code

We now use a provider (such as the Microsoft.CSharp.CSharpCodeProvider) to create a source code file from this. This might yield something like this:

public class MyClass
{
    private string _prop1;
    private System.DateTime _date;
    
    public MyClass()
    {
        _date = new System.DateTime(2004, 2, 14);
    }

    public string Prop1
    {
        get
        {
            return this._prop1;
        }
        set
        {
            this._prop1 = value;
        }
    }

    public System.DateTime Prop2
    {
        get
        {
            return this._prop2;
        }
        set
        {
            this._prop2 = value;
        }
    }
}