2017 04 16 Building Visual C++ Without VS Project Files

C:\Program Files (x86)\Microsoft Visual Studio 14.0>cl.exe
Microsoft (R) C/C++ Optimizing Compiler Version 19.00.24215.1 for x86
Copyright (C) Microsoft Corporation.  All rights reserved.

usage: cl [ option... ] filename... [ /link linkoption... ]

Making a start on actual games engineering, I think it’s worth a quick tour of my current build process, both because some people won’t be familiar with doing this manually, and because it gives me a good lead in to some other areas I want to talk about. In case the teaser snippet above didn’t give it away, I compile using the Visual C++ compiler and linker directly (well, almost; using cl.exe from hand-written batch files) and since I’m fairly sure that will sound strange to some people, I’ll start by explaining myself.

As a non-professional “hobbyist” game developer, any time I spend on games is necessarily at the expense of anything else I want, or need, to do. Maintaining motivation is critical to the survival of this kind of project, and it is all too easy for a small obstacle to kill it entirely – as it takes longer to find the uninterrupted time needed to clear a roadblock, dependencies start to pile up, and the risk increases. With this in mind, my priority in how I approach projects (personal or otherwise) is to minimize the “friction” that needs to be overcome to re-start work after an interruption. When it comes to software projects, that begins with actually building and running the code; if I, for example, move to a new computer, I need to be able to check out the code and get to work with a minimum of fuss, else there’s a chance I’ll never pick it up again.

Why fight the tool?

Taking as written that I’m using Visual C++, and that I will use Visual Studio for debugging, the path of least resistance is to use the full Visual Studio IDE as intended, and since I subscribe to the received wisdom that it’s rarely a good idea to “fight the tool”, I ought to justify this choice.

In Visual Studio, a collection of source code that is built into a program or library is managed by a “project file”, which is an XML document that specifies both the list of input files and the various configurations under which to compile them. While technically human-readable, these files are not intended to be read or edited by the developer; source files and configurations are managed instead through the Visual Studio GUI. Ideally, you create a new project from a template using the built in “wizard”, add the source files you need, and never think about it again, but in the real world I have found Visual Studio projects and solutions to be fragile and finicky.

I think there’s a few reasons for this, but chief among them is the fact that when you edit your build configuration through a GUI like Visual Studio, there’s no way to add documentation alongside your configuration changes or to group options together in any way but that chosen by the IDE developer. For example, when you’re changing six settings scattered across four different configuration screens for one reason, how do you know the significance of their current values? Obviously this can be explicitly documented outside of the project itself, but that extra distance and friction tends to lead to documentation that is never read and is less often updated – surely we’ve all seen how easy it is for even an in-line code comment to rot?

Thankfully, Visual Studio does not require a project file to debug an executable, so we’re not tied to their build system – you can simply open an executable from within the IDE as though it were a project file. The debugging symbols (in those .pdb files) store the paths of the source files used, and Visual Studio will happily open and step through them. I’ll also note that while it’s almost certainly possible to create a Visual Studio project that simply delegates to whatever other build system you like, allowing editing, building and debugging in Visual Studio while sidestepping the project file issue, I personally prefer to edit in Sublime Text,1 so I didn’t try it.

Running the compiler

C:\Users\Haddon\projects\example>cl program.cpp
Microsoft (R) C/C++ Optimizing Compiler Version 19.00.24215.1 for x86
Copyright (C) Microsoft Corporation.  All rights reserved.

program.cpp
Microsoft (R) Incremental Linker Version 14.00.24215.1
Copyright (C) Microsoft Corporation.  All rights reserved.

/out:program.exe
program.obj

The Visual C++ compiler (and linker) can be used by running the command line front-end cl.exe, allowing us to treat it much like any other compiler. As you can see above, the command cl program.cpp is sufficient to compile a single file into an executable. It would be nice if this, with the additional arguments and options documented here,2 were all we need to invoke the Visual C++ compiler from the build system of our choice; but unfortunately the Visual C++ compiler expects a variety of paths and variables to be defined in the shell environment in which it is invoked. For this reason, Visual Studio installs a shortcut named “Developer Command Prompt for VS20XX” – this opens a new cmd.exe and initializes the shell environment for use of their command line tools, including cl.exe. It is from this command prompt that I ran the compiler in the snippets above.

For my purposes, wanting to be able to run a build not just from from a command prompt but also from a text editor, the Git for Windows Bash terminal or even Windows Explorer, we’ll either need to ensure the environment is pre-configured for any application we expect to run a build from – which precludes us from choosing the target architecture at build time, and if we’re including Explorer, means changing the system-level environment – or call the underlying script VsDevCmd.bat (or the Visual C++ specific VcVarsAll.bat which allows specification of the target architecture as a command line option). Sadly, this batch file takes over five seconds to run, so there’s no way I’m running it at the start of every single build – what on earth are they doing in there?

Looking after the environment

Since Microsoft didn’t want to help me out with configuring their shell environment in a sensible amount of time, it was down to me, so I started with the bluntest instrument available: set. If you’re familiar with batch files or the Windows command line interpreter, you’ll know that environment variables (more or less the only kind of variables in shell-land) are set using the built-in command set (for example: set name=value). Conveniently for us, set can also be used with no arguments to spit out all environment variables, and it does so in the same name=value format. Knowing this, you can simply run VcVarsAll.bat once, ahead of time, and then record the resulting environment with set >some_file.txt. Then, whenever you need the environment configured (i.e. in your build script), you can just read back the file, calling set with each line in turn, like so:3

for /f "tokens=*" %%a in (some_file.txt) do (
  set %%a
)

As far as I can tell, this is effectively instant, which only makes me wonder quite why VcVarsAll.bat is sleeping on the job. For convenience, I packed up this functionality into a script (see below), which I call from all of my build scripts (call runs the contents of a batch file in the current environment, allowing variables defined inside the file to persist).

Automating the build

Given that ditching project files and using cl.exe frees me to use almost any build system I like, why then would I subject myself to (horror of horrors) batch files? Simply put, they’re good enough for what I want to do: specify some source files, include directories, libraries and compilation options in a text file with comments; and it is ubiquitous on my development platform – I don’t have to install any additional tools to get my build working on a new machine.4 Anyway, regardless of the scripting language used, the steps involved in actually building some code are the same:

None of these steps are particularly interesting or involved, so I’ll skip most of the detail. Generally they just involve building up a few strings (e.g. include directories, linker options) throughout the script, then using them to assemble one large call to cl.exe at the end. There are, however, a few things I do that affect how some of this works: I separate independent parts of my code into packages, each of which gets its own Git repository. This way, the main repository for an actual software item only contains the build script, the code files that are unique to that item (often just the main.cpp containing the entry point), and Git submodule references to the other packages required.5 I also use what is most commonly known as a “unity build”, where only one actual source file is given to the compiler and all other code is #include-d – there are a variety of arguments for and against this,6 but the upshot here is that the list of objects (i.e. .cpp files) to build is defined in code, not in the build script. With these in combination, managing source files to be built is a matter of keeping my #includes in order and adding each required package as an include directory.

I think, at this point I may have rambled long enough about build scripts, so I’ll sign off here. If you’ve read this far, thanks for your patience, and I hope to see you in the not-too-distant future when I return to talk about that enigmatic pre-processor step I so deftly teased above, and perhaps why, when writing game code, I build libraries more often than executables.

Appendix: Visual C++ environment loader

This is the script I use for generating and loading stored shell environment states to avoid running VcVarsAll.bat at build time. It’s hard coded to the default install location for Visual Studio 2015, it doesn’t yet support multiple versions of Visual Studio, and I didn’t bother to make it auto-generate .env files it doesn’t find; but I’ll leave those as an exercise for the reader. Oh, and the syntax highlighter in Jekyll doesn’t seem to support Batch files, so we’ll have to settle for “classic monochrome”. 😊

@echo off

set MSVC_ENV_USAGE=Usage: msvc_env [/generate] [arch]
set MSVC_ENV_TOOL_DIR=%~dp0

set msvc_env_mode=load
set msvc_env_platform=vs14.0
set msvc_env_arch=amd64
set msvc_env_arch_set=false



:label_parse_arguments

if [%1]==[] (
  goto label_no_further_arguments
)

set msvc_env_argument=%1
shift

if [%msvc_env_argument:~0,1%]==[/] (
  set msvc_env_argument=%msvc_env_argument:~1%
) else if [%msvc_env_argument:~0,2%]==[--] (
  set msvc_env_argument=%msvc_env_argument:~2%
) else if [%msvc_env_arch_set%]==[false] (
  set msvc_env_arch_set=true
  set msvc_env_arch=%msvc_env_argument%
  goto label_parse_arguments
) else (
  echo ERROR - unexpected argument: %msvc_env_argument%
  echo %MSVC_ENV_USAGE%
  exit /B 1
)

if [%msvc_env_argument%]==[generate] (
  set msvc_env_mode=generate
) else (
  echo ERROR - unknown option: /%msvc_env_argument%
  echo %MSVC_ENV_USAGE%
  exit /B 1
)

goto label_parse_arguments




:label_no_further_arguments

set msvc_env_file=%MSVC_ENV_TOOL_DIR%\%msvc_env_platform%-%msvc_env_arch%.env

if [%msvc_env_mode%]==[load] (
  goto label_load_env
) else if [%msvc_env_mode%]==[generate] (
  goto label_generate
)

echo ERROR: unexpected mode: %msvc_env_mode%
exit /B 1




:label_load_env

if exist %msvc_env_file% (
  for /f "tokens=*" %%a in (%msvc_env_file%) do (
    set %%a
  )
) else (
  echo ERROR: No file %msvc_env_file%
  echo Unsupported arch %msvc_env_arch%?
  exit /B 1
)

goto end




:label_generate

echo generating %msvc_env_file%

if [%msvc_env_arch%]==[amd64] (
  call "C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\vcvarsall.bat" amd64
) else if [%msvc_env_arch%]==[x86] (
  call "C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\vcvarsall.bat" x86
) else (
  echo ERROR: Unsupported arch "%msvc_env_arch%"
  exit /B 1
)

set >%msvc_env_file%

goto end


:end
  1. Don’t worry, I’m not going to try to sell you it; it’s just what I’m used to using. One advantage of working alone is that you don’t have to choose between asking the whole team to use the same tools and supporting everyone’s peculiar work-flows. 

  2. Or is it here? I guess they’ve rearranged it for VS2017. Nice of them to make that clear on the new page. *sigh* 

  3. I’m not going to pretend to know or understand the cmd for-loop syntax. I have to look the damn thing up every time I use it. 

  4. I did experiment briefly with using Bash scripts for this, as I use the MSYS terminal provided with Git for Windows fairly extensively. This would have given me a significantly less repugnant (though still not amazing, by any measure) language for scripting, but at the time I found that the shell would take a few seconds to initialize before it could run a script. Writing this, I re-tried running scripts with the newer MSYS2-based Git for Windows, and that performs much better; maybe I’ll switch over at some point, but it’s a long way down the priority list so it’s not happening any time soon. It might also be fair to ask “Why not CMake?”; and sure enough, I do begrudgingly install it to build some of my dependencies – I’m looking at you, GLFW – but I simply don’t need the additional layer of abstraction, with the additional complexity that brings. Even if I support multiple platforms in the future, games generally require enough explicit integration with a platform that I wouldn’t expect a common build definition to be as useful as in other domains. (Though since platform abstraction is GLFW’s raison d’être, maybe that’s a bad assumption?) 

  5. Git submodules allow a git repository to track whole other nested repositories as though they were files, storing only a remote URL and commit hash. The command line interface for manipulating these is pretty clunky (who’d have guessed?), but it works just fine and it’s built in to a tool I’m already using, so … 

  6. To be completely honest, I’m using a “unity build” because it was a trendy idea that I’d heard recommended by a few people when I started this project, and I thought I’d give it a try. The theory is that building a lot of independent objects that share common headers duplicates effort, and you can save time by just building the whole program as one big object (this also gives you the benefit of link-time optimization, but mainstream compilers can do that anyway, so it really just comes down to the build performance). The downsides of this approach are that you sacrifice incremental and multi-threaded builds, and hypothetically put a limit on the size of your application when the (still 32-bit only) Visual C++ compiler exhausts its 4GB of address space. I’ve personally found incremental builds to be occasionally flaky, and the fact that parallel builds aren’t enabled by default in Visual Studio is something of a red flag, but I haven’t done any meaningful testing, even to determine whether I’m better off than if I was using a conventional build, and I’m unlikely to find myself building large enough programs for it to hurt me too badly either way, so I’m certainly not in a position to advocate either way.