Compilation Pipeline in the DirectX Compiler

A few months ago I discussed how to invoke the compiler from your code. Today I want to discuss what the invocation does, so you can find your way around if you decide to inspect the code or if you want to learn how your shaders go from text to bytes.

• DxcCompiler: this class is the implementation for the IDxcCompiler interface, and puts together the rest of the pipeline.
• Parser: the parser component setups up a tokenizer/preprocessor and pulls tokens out of a stream; the tokenizer and preprocessor work together to handle #include and #define directives and to break up incoming text into lexical units. As it recognizes structures, it pushes them to the next component.
• Semantic Analyzer: the semantic analyzer (Sema) looks at the parser structures, performs various checks, and generates declarations, statements and expressions, all of which land on the ASTContext that the parser also has access to.
• Code Generator: once the parser is done processing all input, the code generator turns the constructs in the AST context into LLVM IR in a straightforward fashion. We call this form High-Level Intermediate Representation (sometimes HL-IR in code).
• DXIL Lowering and Optimization: a number of passes work by transforming LLVM IR in HL-IR form into LLVM IR in DXIL form. The representation is the same, but DXIL has references and invariants that aren't present in HL-IR. Some optimizations are done before this lowering, while some are done after.
• Container assembly: after all the transformations have been performed, the DXIL program is put into a container that allows other program to have easy access to important information, without having to deserialize and process DXIL.
• Validation: after all the transformations have been performed and the container has been assembled, validation is run to ensure that a number of rules or invariants hold for that program; downstream consumers can then rely on these being present to simplify their work.

If you're familiar with clang and LLVM, you will notice that there is very little that has changed. We mostly do what clang regularly does with sources, then do some post-processing on the IR without targeting a specific backend, and then put things together for DirectX consumption.

Some of these phases have been broken up into other components to enable some additional scenarios, like having the optimizer passes support instrumentation. In particular, the compiler can be made to output HL-IR by using the /fcgl flag. The DxcOptimizer can then perform the work to turn it into DXIL. DxcAssembler will generate the container, and DxcValidator will perform validation on it.

Enjoy!