CruiseControl.NET / Delphi – Dashboard Templates

In my last post I discussed integration CCNet/NAnt with Delphi/BDS 2006. That post however did not cover how to include the build results from Delphi in the project build report.

ERR File
When the bds.exe executes, it doesn’t output anything to STDOUT or STDERR. Instead, it outputs all of the compiler information into the file [ProjectName].err. Here’s an example of what you can expect to find in the file(ignore the various paths):

Delphi Command Line for "Project1.dpr"
$A8 $B- $C+ $D+ $E- $F- $G+ $H+ $I+ $J- $K- $L+ $M- $N+ $O- $P+ $Q- $R- $S- $T- $U- $V+ $W- $X+ $YD $Z1 -B -H+ -W
$M16384,1048576 $K4194304 -CG -Ee:\cc_sandbox\projects\Project1\ OutPutDir=out -N0DCUs
-Ue:\program files\borland\bds\4.0\units;e:\program files\borland\bds\4.0\vcl;
-Oe:\program files\borland\bds\4.0\units;e:\program files\borland\bds\4.0\vcl;
-Re:\program files\borland\bds\4.0\units;e:\program files\borland\bds\4.0\vcl;
PkgDllDir=C:\Documents and Settings\CCBUILD\My Documents\Borland Studio Projects\Bpl
PkgDcpDir=C:\Documents and Settings\CCBUILD\My Documents\Borland Studio Projects\Bpl
-AWinTypes=Windows;WinProcs=Windows;DbiTypes=BDE;DbiProcs=BDE;DbiErrs=BDE; -C0 OBJOutputDir=DCUs
BPIOutputDir=C:\Documents and Settings\CCBUILD\My Documents\Borland Studio Projects\Bpl -W-UNSAFE_TYPE -W-UNSAFE_CODE
-W-UNSAFE_CAST Project1.dpr
[Pascal Warning] BrowserForm.pas(444): W1036 Variable 'list' might not have been initialized


Knowing the basic format of this file, it’s not very hard to scrape it so the information can be included the build report.

NAnt script for .ERR file scraping
Here’s a bit of script to include in your NAnt build scripts. It’s embedded c# code that reads all of the lines of a text file, and dumps them into a single xml document. Each individual line is wrapped inside an xml element named “line”. The document element is named “delphi-console”.

Here’s the script to include in your NAnt builds:

<script language="C#" prefix="script">
<references>
  <include name="System.dll" />
  <include name="System.Xml.dll" />
</references>

<imports>
  <import namespace="System" />
  <import namespace="System.IO" />
  <import namespace="System.Xml" />
</imports>

<code>
  <![CDATA[
  [Function( "read-console-output" )]
  public static string ReadConsoleOutput( string path )
  {
	string[] lines = File.ReadAllLines( path );

	XmlDocument doc = new XmlDocument();
	XmlElement root = doc.CreateElement( "delphi-console" );
	doc.AppendChild( root );

	foreach( string line in lines )
	{
	  XmlElement lineNode = doc.CreateElement( "line" );
	  root.AppendChild( lineNode );

	  lineNode.InnerText = line;
	} //END line loop 

	return doc.OuterXml;
  } //END ReadConsoleOutput method
  ]]>
</code>
</script>

And here is the content of the Project1.err file converted to xml:

<delphi-console>
  <line>Delphi Command Line for "Project1.dpr"</line>
  <line>  $A8 $B- $C+ $D+ $E- $F- $G+ $H+ $I+ $J- $K- $L+ $M- $N+ $O- $P+ $Q- $R- $S- $T- $U- $V+ $W- $X+ $YD $Z1 -B -H+ -W </line>
  <line>  $M16384,1048576 $K4194304 -CG -Ee:\cc_sandbox\projects\Project1\ OutPutDir=out -N0DCUs </line>
  <line>  -Ue:\program files\borland\bds\4.0\units;e:\program files\borland\bds\4.0\vcl;</line>
  <line>  -Oe:\program files\borland\bds\4.0\units;e:\program files\borland\bds\4.0\vcl;</line>
  <line>  -Re:\program files\borland\bds\4.0\units;e:\program files\borland\bds\4.0\vcl;</line>
  <line>  PkgDllDir=C:\Documents and Settings\CCBUILD\My Documents\Borland Studio Projects\Bpl </line>
  <line>  PkgDcpDir=C:\Documents and Settings\CCBUILD\My Documents\Borland Studio Projects\Bpl </line>
  <line>  -AWinTypes=Windows;WinProcs=Windows;DbiTypes=BDE;DbiProcs=BDE;DbiErrs=BDE; -C0 OBJOutputDir=DCUs </line>
  <line>  BPIOutputDir=C:\Documents and Settings\CCBUILD\My Documents\Borland Studio Projects\Bpl -W-UNSAFE_TYPE -W-UNSAFE_CODE </line>
  <line>  -W-UNSAFE_CAST Project1.dpr </line>
  <line>[Pascal Warning] BrowserForm.pas(444): W1036 Variable 'list' might not have been initialized</line>
</delphi-console>

Now that you have the nant script to generate the build output xml file, you need to call it. If the bds.exe process fails, and you have failonerror attribute set to true, the err file scraping process won’t be executed. So for my Delphi build scripts, I wrap my “build” target with a new target that I inventively have named “trybuild”. This target calls the “build” target within a try/catch/finally block(available from the NAntContrib project). This way no matter what happens during the compile, the .err file will be scraped.

The following code listing demonstrates the use of .err file scraping:

<target name="trybuild">
    <trycatch>
      <try>
        <call target="build" />
      </try>
      <catch property="failure">
        <if test="${file::exists(file.result)}">
          <fail>Delphi build failure: ${failure}</fail>
        </if>
        <if test="${not file::exists(file.result)}">
          <fail>General failure:${failure}</fail>
        </if>
      </catch>
      <finally>
        <if test="${file::exists(file.result)}">
          <echo file="${dir.logs}/delphi-console.xml">${script::read-console-output(file.result)}</echo>
        </if>
      </finally>
    </trycatch>
 </target>

Notice the various *if conditions. If the bds.exe fails and the .err file exists, then there was some build related failure. If the .err file does not exist, then there was some other problem.

Also, notice that the way file final xml file is generated is via a call to the echo task, which explicitly calls the read-console-output script we defined earlier.

Once you have that all in place, you must now modify your ccnet.config file to include this output file in the build log via the merge publisher. Example:

<publishers>
	<merge>
		<files>
			<file>logs/delphi-console.xml</file>
		</files>
	</merge>
</publishers>

Now that the .err file has been scraped, and the contents have been merged into ccnet’s build log, we can work on outputting the content to the build report.

This part is relatively simple. I started with a copy of the file compile.xsl, located in the /xsl folder of the dashboard directory. From there, I created an xsl transform that looked for each line in the build log that was prefixed with [Pascal Error] or [Pascal Hint]. The script does not currently look for [Pascal Warning] lines, and treats hints as warnings. I named the xsl script “delphi-console-out.xsl”, and here is its full listing:

<?xml version="1.0"?>
<xsl:stylesheet
    xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">

  <xsl:output method="html"/>

  <xsl:template match="/">

    <xsl:variable name="lines" select="/cruisecontrol/build/delphi-console/line" />
    <xsl:if test="count($lines) > 0">

      <xsl:variable
        name="lines.error"
        select="$lines[(contains(text(), '[Pascal Error]')) or (contains(text(), '[Pascal Fatal Error]'))]" />
      <xsl:variable
        name="lines.error.count"
        select="count($lines.error)" />
      <xsl:variable
        name="lines.hint"
        select="$lines[(contains(text(), '[Pascal Hint]'))]" />
      <xsl:variable
        name="lines.hint.count"
        select="count($lines.hint)" />
      <xsl:variable
        name="lines.total"
        select="count($lines.error) + count($lines.hint)"/>

      <xsl:if test="$lines.error.count > 0">
        <table class="section-table" cellpadding="2" cellspacing="0" border="0" width="98%">
          <tr>
            <td class="sectionheader">
              Delphi Console Output - Errors: (<xsl:value-of select="$lines.error.count"/>)
            </td>
          </tr>
          <tr>
            <td>
              <xsl:apply-templates select="$lines.error"/>
            </td>
          </tr>
        </table>
      </xsl:if>
      <xsl:if test="$lines.hint.count > 0">
        <table class="section-table" cellpadding="2" cellspacing="0" border="0" width="98%">
          <tr>
            <td class="sectionheader">
              Delphi Console Output - Hints: (<xsl:value-of select="$lines.hint.count"/>)
            </td>
          </tr>
          <tr>
            <td>
              <xsl:apply-templates select="$lines.hint"/>
            </td>
          </tr>
        </table>
      </xsl:if>

    </xsl:if>
  </xsl:template>

  <xsl:template match="line">
    <pre class="section-error">
      <xsl:value-of select="text()"/>
    </pre>
  </xsl:template>

</xsl:stylesheet>

After that, you must modify the dashboard.config file to include the xsl. Fortunately, that’s easily done. Locate the dashboard/plugins/buildPlugins/buildReportBuildPlugin/xslFilenames element, and add a new <xslFile> element like so: <xslFile>xsl\delphi-console-out.xsl</xslFile>

Once all of this comes together, you can see at Delphi hints and errors reflected in ccnet’s build report. Here’s a screencap:

Sample output of build report

I hope this helps some one else.

CruiseControl.NET, and… Delphi?

I recently began working on one of my company’s legacy applications, written in Delphi, using Borland Developer Studio 2006. I got sick of every developer doing one-off builds every time some one needed a new executable, so I made it my mission to integrate the project into our Automated Build Environment. It was a long and tortuous process, but completely worth it. During the past 6 months, the build environment has spit out exactly 662 successful builds of this project, averaging 3.6 successful builds per day(including weekends).

Command Line Interface & Licensing
I’m not a Delphi guru, so this entire effort was full of exploration. The starting point was working with Delphi/BDS via the command line. I did some research, and found that there were two command-line candidates: the actual compiler, dcc32.exe, and the executable for the development environment, bds.exe.

I spent maybe a day trying to get dcc32.exe to work. I consulted documentation, checked newsgroups, googled 25 different search terms, and I still couldn’t get it to work. I decided to simplify the problem, and create a new command line project with one source file. I turned on the option in the environment to show the command line. I used the exact same command line that the environment was telling me it was using, and I still couldn’t get it to compile. That was well over 6 months ago, so I can’t remember the exact compiler error messages at this point. Regardless, pursuing dcc32 wasted a day, so I then turned to bds.exe.

The documentation for BDS 2006 states that the bds.exe executable does support a command line interface, so I was in luck. The funny thing is that I couldn’t find any info on the CLI through the integrated help, but when I ran “bds /?” from the console, the BDS environment started up, and opened the integrated help to the correct topic. I don’t know how much it will help, but the url(in the integrated help) for the docs is: ms-help://borland.bds4/bds4ref/html/idecommandline.htm, and the navigation path is Borland Help > Developer Studio 2006 (Common) > Refreence > Command Line Switches > IDE Commnad Line Switches and Options.

With the aid of the documentation, it was pretty simple to run bds from the console. However, there were a few hiccups. First, you must remember that bds.exe is a gui application first that appears to have had a CLI tacked on at the end. This is unfortunate. The executable does not write to STDOUT or STDERR, nor does it support process exit codes. The other problem is that your environment must be setup correctly! More on that in a moment.

After I had decided the practical matter of how I would handle the builds from the console, I needed to check on licensing. BDS 2006 is quite expensive, and the installation/licensing mechanism is quite complex. I needed to check with Borland(or Inprise, or CodeGear, or whatever the company name was that quarter) on the licensing terms. I called, went through phone limbo for a while, until I was finally connected to a sales person. I explained that the company already had x licenses of the product and none were free, and that I needed another so I could perform automated builds in a dedicated build environment. I was transferred multiple times until I finally reached some licensing specialist. I explained the scenario again, and was put on hold again. When the “specialist” returned, I was told that no one had ever inquired about using licenses for this purpose before. I was then told that as long as it was installed in a dedicated environment, and that this new installation would not be used for actual interactive development, that I did not need a separate license. That was good news.

This is a good point to ask a question. I’ve already established that I was the first person in the world to attempt to develop a cruisecontrol.net dashboard plugin, and now it appears that I was the first person to ever contact the company about automated builds. The question: how do I end up doing all these freaky fringe things? Sorry for the tangent.

Anyway, having gotten the OK on the license front, it was time to implement a solution.

Environment Setup is Critical
As I previously mentioned, environment setup is critical. The bds executable is a gui application, and does not behave as one would expect from a CLI. When bds.exe is executed from the console, it actually launches the development environment interactively, loads a specified project, and attempts to compile it. You can actually watch it compile. When its completes, it closes. However, there are cases where the environment does not close. For example, if your project references a component or package that cannot be found, the environment will complain via a modal message box. Also, if the act of compiling changes some file in the project, the environment will ask you if you wish to save the changes before it closes. Specifically, this happens if you have the “Include version information in project” option selected.

This is a Big Deal. If you’re running CruiseControl.NET as a service, and the BDS environment prompts one of these modal dialogs, your process will hang. You can get around this problem by setting a timeout in the ccnet project configuration, but that will cause the build to fail. Here are a few tips for getting around this problem:

Setup your build environment, including installed packages and the like, first.

Do not include version info in your project

Don’t mess with project resources

The point is, getting the environment setup correctly is the most important part of the process. This integration is pretty fragile, and doesn’t take much to break. You’re going to spend quite a bit of time logged in to your build environment, interactively testing your build script, before you finalize it.

Versioning and Project Resources
Our build script handles the versioning of the project. It uses a set of custom NAnt tasks to generate and increment the version number for the build. I’ll cover the minutiae of auto-versioning in a different post. Regardless, after the version number is generated, the build script injects the version number into the .bdsproj file before the bds.exe is launched. This method allows the project output to include the version number, but avoids the problem of hanging the bds environment because a file changed during the compile phase.

The version number is separated into its components: Major, Minor, Build, and Release, via calls to custom embedded c# script functions. That information is then injected into the .bdsproj file via the NAnt xmlpoke task. It is important to note here that Delphi reverses the Build and Release members of a version, at least when compared to a dotnet version structure. So you’ll see in the script that these values are swapped when injected. Here’s the code listing for the version injection:

<!-- extract the version members -->
<property name="version.major" value="${script::get-version-major(version)}" />
<property name="version.minor" value="${script::get-version-minor(version)}" />
<property name="version.build" value="${script::get-version-build(version)}" />
<property name="version.revision" value="${script::get-version-revision(version)}" />

<xmlpoke
	file="${file.proj}"
	xpath="BorlandProject/Delphi.Personality/VersionInfo/VersionInfo[@Name='IncludeVerInfo']"
	value="True" />
<xmlpoke
	file="${file.proj}"
	xpath="BorlandProject/Delphi.Personality/VersionInfo/VersionInfo[@Name='AutoIncBuild']"
	value="False" />
<xmlpoke
	file="${file.proj}"
	xpath="BorlandProject/Delphi.Personality/VersionInfo/VersionInfo[@Name='MajorVer']"
	value="${version.major}" />
<xmlpoke
	file="${file.proj}"
	xpath="BorlandProject/Delphi.Personality/VersionInfo/VersionInfo[@Name='MinorVer']"
	value="${version.minor}" />
<xmlpoke
	file="${file.proj}"
	xpath="BorlandProject/Delphi.Personality/VersionInfo/VersionInfo[@Name='Release']"
	value="${version.build}" />
<xmlpoke
	file="${file.proj}"
	xpath="BorlandProject/Delphi.Personality/VersionInfo/VersionInfo[@Name='Build']"
	value="${version.revision}" />
<xmlpoke
	file="${file.proj}"
	xpath="BorlandProject/Delphi.Personality/VersionInfoKeys/VersionInfoKeys[@Name='FileVersion']"
	value="${version}" />
<xmlpoke
	file="${file.proj}"
	xpath="BorlandProject/Delphi.Personality/VersionInfoKeys/VersionInfoKeys[@Name='ProductVersion']"
	value="${version}" />

As you can see, this code injects the version components individually into the .bdsproj file, and also sets the output’s File version and Product version values. Next, we’re going to deal with resources.

At one point a few months ago, our build process was brought to its knees. It turns out that some one changed the application’s main icon and checked in the modified files. For some reason, this caused some checksum to be out of sync with some other file(I believe it was the project’s .drc file), which caused BDS to believe that some change had been made during the compile phase, which resulted in BDS prompting the user to save changes. Because this was occurring during a non-interactive service, the process hung until it finally timed out. That’s why I mentioned that you do not want to mess with a project’s resources.

To get around the problem of the icon, I figured out how to generate the application’s icon from the build script. I removed the application’s icon, saved the project, and committed the changes. I then added a separate .ico file to the repository, named [Project].ico. Next, I created a resource script named [Project].rc. Finally, before I launch the bds.exe executable, I run brcc32.exe to compile the resource script, which outputs the resource into [Project].res.

The resource script is a simple one line text file. Here’s an example: MAINICON Icon Project1.ico
This little work around has a few benefits. First, bds was no longer prompting us to save changes, and second, to change the app’s icon, one need only commit a new [Project].ico to the repository.

Building the Project
Now that the version information has been set, and the main application resource has been generated, it’s time to finally launch bds.exe and perform the build. This process is actually fairly easy, so I’ll let the following snippet of NAnt script do all the talking:

<target name="build" depends="clean">
	<property name="file.bds.args" value="-ns -nw -b ${file.proj}" />

	<!-- generate the new version number -->
	<call target="gen-version" />

	<!-- delete the current .res file, and generate the new one -->
	<delete file="project.res" />
	<exec program="${env.borland_home}\brcc32.exe" commandline="Project.rc" />

	<exec program="${file.bds}"	commandline="${file.bds.args}" basedir="E:\Program Files\Borland\BDS\4.0\Bin"
	failonerror="true" timeout="600000" />		

	<!-- revert the transient project files -->
	<call target="revert-project-temp-files" />
</target>

File Reversion
The compile phase emits some files and changes a few others. Some of these files are transient, and I don’t want the changed files to be committed back to the repository. Hence the custom target “revert-project-temp-files”.

We’re using cvs as our repository, so the script logs in to cvs, deletes a series of files, and performs a CVS UPDATE on each file that was deleted. This ensures that the transient files are up to date before the next build begins. The list of transient files includes the .bdsproj file(we changed it when we generated the version information), the project.cfg file, the project.drc file, the project.res file, and any .tlb/_TLB.pas files.

Conclusion
That should be enough to get you started, though the process is not yet complete. The next post will discuss handling the .err file, and including the compile-related errors and warnings in the CruiseControl.NET build report.