Recently, I am trying to re-write my graphics code to use D3D12 (instead of D3D11 in Seal Guardian). I need to have a convenient way to compile shader code. While tidying up the MSBuild custom build steps files used in Seal Guardian for my new toy graphics engine, I regret that I did not write a blog post about custom MSBuild at that time, as I remembered, finding such information was hard at that time and I need to look at some of the CUDA custom build files to guess how it works. So this post will just be my personal notes about custom MSBuild and I don't guarantee all information about MSBuild are 100% correct. I have uploaded an example project to compile shader files here. Interested readers may also check out this excellent post about MSBuild written by Nathan Reed.
Custom build steps set up
MSBuild need to have .targets file to describe how the compiler (e.g. dxc/fxc used for shader compilation) are invoked. In the uploaded example project, we have 3 main targets: DXC, JSON, BIN.
- DXC target: described by its name, invoking the dxc.exe to compile HLSL file.
- JSON target: used to invoke the shaderCompiler.exe, which is our internal tool written using Flex & Bison to parse the shader source code to output some meta data, like texture/constant buffer usage for root signature management.
- BIN target: a task that depends on DXC task and JSON task, invoke the dataBuilder.exe, our internal tool for data serialization/deserialization into our binary format, combining the output from DXC and JSON task.
 |
| target dependency |
Although MSBuild can set up the target dependency, but it looks like those independent targets are not executed in parallel. In Seal Guardian, when compiling the surface shaders which generate the shader permutation for lighting, this result in a long compilation time. At the end, I need to create another exe to launch multiple threads to speed up the shader compilation. May be I was setting up MSBuild incorrectly, if anyone knows how to parallelize it, please let me know in the comment below. Thank you!
Incremental Builds
MSBuild use .tlog file to track files modification to avoid unnecessary compilation (also affect which files got deleted when cleaning the project). There are 2 tlog files (read.1.tlog and write.1.tlog), one is for tracking whether the source files are modified, and the other is tracking whether the output file is up to date. We can simply use the WriteLinesToFile task to mark such dependency, e.g.
<WriteLinesToFile File="$(TLogLocation)$(ProjectName).write.1.tlog" Lines="$(TLog_writelines)" />
But doing this only will make the tlog file larger and larger after every compilation. So it is better to read the tlog file content into a PropertyGroup and check whether the file already contains the text we would like to write using a "Conidition" inside the WriteLinesToFile task. For details, please take a look at the example project.
Also, as a side note, do not include $(MSBuildProjectFile) property in the "Inputs" element inside "Target" task. I did it accidentally and it cause the whole project to recompile all the shaders every time a new shader file is added to / removed from the project. This is not necessary as most of the shader files are independent.
Output files
Like every visual studio project, our example project have a Debug and Release configuration. After executing the BIN task described above, we also use a Copy task to copy our compiled shader from Debug/Release config $(OutDir) directory to our content directory. We can also use the Property Function MakeRelative() to maintain the directory hierarchy in the output directory. This is another reason why I use Copy task instead of specifying the $(OutDir) to the content directory, as I cannot get a nested Property Function working inside the .props file (or may be I did something wrong? I don't know...)...
Also, beside output files, another note is about the output log. If we want to write something to the output console of visual studio from your custom exe (e.g. dataBuilder.exe/dataBuilder.exe in the example project), the text must be in a specific format like (but I cannot find the documentation of the exact format, just guess from similar message emitted from visual studio...):
1>C:/shaderFileName.hlsl(123): error : error message
otherwise, those message will not get displayed in output window.
Example project
An example project is uploaded here. It will compile vertex/pixel shaders with dxc.exe and output JSON meta data to the $(IntDir), then combine those data and write to the $(OutDir). Finally those files will be copied to the asset directory with the corresponding relative path to the source directory. Please note that the shaderCompiler.exe used for outputting meta-data is for internal tools, which have some custom restriction on the HLSL grammar for my convenience to create root signature. It is used just as an example to illustrate how to set up a custom MSBuild tool, feel free to replace/modify those files to suit your own need. Thank you.
Reference
[1] https://docs.microsoft.com/en-us/cpp/build/understanding-custom-build-steps-and-build-events?view=vs-2019
[2] http://www.reedbeta.com/blog/custom-toolchain-with-msbuild/
沒有留言:
發佈留言