Introduction to GoCD

In Part 1 of this guide, the concepts of GoCD Server, GoCD Agents, pipelines, materials, stages, jobs and tasks were introduced. You were also able to get a simple pipeline with a single stage, job and task each working.

Let us continue, in this part, by learning about pipelines as materials, about artifacts and about custom tabs.

Using a pipeline as a material

Instead of using a source code repository on GitHub as a material, it is possible to use a stage of any pipeline as a material for another pipeline. This seemingly simple feature provides a lot of power. This allows pipelines to be chained together, allowing very complex workflows to be modeled well and is a basis for the more advanced features in GoCD, such as Value Stream Map (VSM) and fan-in/fan-out.

Concept 5: Pipeline Dependency Material

GoCD Pipeline Dependency Material

A pipeline dependency material links pipelines together. The pipeline which has the material which points to another pipeline is called the "Downstream Pipeline". The actual pipeline which is the dependency material is called the "Upstream Pipeline". Even though it is called a "Pipeline Dependency" the real dependency is to a stage in an upstream pipeline.

As soon as that stage completes successfully, the first stage of each of the configured downstream pipelines triggers. If the stage does not complete successfully, its configured downstream pipelines do not trigger.

In the image above, as soon as "Stage 2" of "UpstreamPipeline" completes successfully, "DownstreamPipeline1" starts. Concurrently, "Stage 3" of "UpstreamPipeline" also starts.

Let's see how we can get a pipeline dependency to work. Clicking on "Admin" and then "Pipelines" takes you to the "Pipeline Configuration" page which lists all the pipelines in the system. It looks like this:

Pipeline Configuration page
Pipeline Configuration page
  1. Create another pipeline, using the "Create a new pipeline within this group" link.
  2. Name this new pipeline "MySecondPipeline" and click "Next" to reach step 2, to choose a material.
  3. Choose "Pipeline" as the Material Type, and you'll be in a screen like the one below.
Start adding a pipeline dependency material
Start adding a pipeline dependency material

As soon as you start typing the name of your first pipeline in the "Pipeline [Stage]" textbox, it will autocomplete and give you the name of the first pipeline and the only stage that is in it. Selecting that will link these two pipelines together.

Autocompletion of the pipeline dependency material
Autocompletion of the pipeline dependency material

Click "Next" and choose some task in the "Initial Job and Task" section and click "Finish".

Create the downstream pipeline
Create the downstream pipeline

That's it! You'll now have a pipeline which will run when un-pause. Un-pause it and let it finish. This is how it will look on the Pipelines list (dashboard) when finished.

Dependent pipelines on the dashboard
Dependent pipelines on the dashboard

That doesn't look very different. But, this allows for some powerful features such as fan-in, fan-out and the ValueStream Map (VSM), which we will learn about in Part 3. Click on the label of the "MyFirstPipeline" (the part highlightedin the image above) for a sneak peek at a small Value Stream Map.

Dependent pipelines in a Value Stream Map
Dependent pipelines in a Value Stream Map

You now know how to chain pipelines together. Let's now learn about artifacts, which are very useful when used with pipeline dependencies.

Publishing, fetching and using artifacts

A very important aspect of GoCD is the way it handles artifacts.

Concept 6: Artifact

An artifact in GoCD is a file or directory which is most often produced during the run of a pipeline. Every job in a pipeline can be configured to publish its own set of artifacts and GoCD will ensure that these artifacts are moved from the Agent where it is created, to the Server and stored there, so that it can be retrieved at any time.

Typically, artifacts are created during a job run, by one of the tasks. Some examples of artifacts are: Test reports, coverage reports, installers, documentation, meta information about the build process itself and anything else that needs to be stored after a pipeline has finished.

These artifacts, published by a job, can be fetched and used by any downstream pipeline or any stage after the one that produced the artifact in the same pipeline, using a special task called a "Fetch Artifact" task. If you recall from "Concept 4" in part 1, jobs are independent of each other. So, a job in the same stage as another job that produced an artifact cannot use that artifact. It needs to be used in a stage after that one.

Let's see how to publish an artifact. In the upstream pipeline, "MyFirstPipeline", let's first declare an artifact. The build script used throughout this guide creates a file called "my-artifact.html" after it finishes. We can use that as the artifact for this example.

  1. Go to "Admin" -> "Pipelines" and the select the stage, "defaultStage" and then the job, "defaultJob".
  2. Click on the "Artifacts" tab in the job.
  3. Enter "my-artifact.html" as the artifact source.
  4. Leave the "Destination" box empty for now.
Adding an artifact
Adding an artifact

Since we have chosen to leave the "Destination" empty, and that means that the artifact will be accessible at the root, by its own name. We have also chosen the type as "Build Artifact", to signify that it is not a "Test Artifact". Marking it as a "Test Artifact" means that GoCD will try and parse the file as a test report and if it can, it will use it in the test aggregation reporting it does. Typically, you'll want to use "Build Artifact" for installers, binaries, documentation, etc.

After saving this change, retrigger "MyFirstPipeline" by going to the pipeline dashboard and clicking on the play button against it. Once it is finished, going to the "Artifacts" tab of the pipeline run shows you the artifact for that run. Every run of that pipeline, from now on, will have that artifact.

The artifacts tab - Every job will have one
The artifacts tab - Every job will have one

Clicking on it will show you its contents:

The contents of the artifact - It is stored on the Go C D Server
The contents of the artifact - It is stored on the GoCD Server

Fetching and using artifacts

We can now use this artifact in any downstream pipeline, or any subsequent stage of the same pipeline. Let's fetch this artifact in the pipeline "MySecondPipeline" and display it as a part of the output. To do this, we go to the task configuration section of the "defaultJob" job inside the "defaultStage" stage of "MySecondPipeline" pipeline, and add a "Fetch Artifact" task.

Adding a Fetch Artifact task
Adding a "Fetch Artifact" task
The details can be autocompleted
The details can be autocompleted

Once you provide all the details and click save, you can move the "Fetch Artifact" task above, so that it is done first. Then, for this demonstration, let us display the fetched file in a "Custom Command" task.

Move the fetch artifact task to the top
Move the fetch artifact task to the top

After it is moved up, you can edit the "Custom Command" task to output the contents of the file (for instance). If you are running this on Windows, use "type" instead of "cat":

Display the contents of the fetched artifact
Display the contents of the fetched artifact
This is how the tasks should look like
This is how the tasks should look like

Running "MySecondPipeline" now will show you the contents of the file created in "MyFirstPipeline" and fetched as an artifact. The importance of fetching an artifact this way is that GoCD will ensure that the correct version of the artifact will be fetched and provided to the job in "MySecondPipeline".

If more instances of "MyFirstPipeline" ran (because it is fast) while "MySecondPipeline" has run fewer times, GoCD will ensure that every time "MySecondPipeline" runs, the correct (and compatible) version of the artifact is fetched and used. When you now check the output of the pipeline, you should see something like this:

The output - The artifact is fetched and displayed
The output - The artifact is fetched and displayed

Concept 7: Ancestor artifacts and building artifacts only once

In GoCD, you can even fetch artifacts from pipelines which are not immediately downstream of the pipeline which produces an artifact. It can be fetched from any pipeline which is downstream, however many levels that is. So, with a Value Stream Map like this:

Value Stream Map

The pipeline "MyFifthPipeline" can be configured to fetch an artifact from "MyFirstPipeline" like this:

Fetching artifacts from ancestor pipelines
Fetching artifacts from ancestor pipelines

GoCD will complete the name of the pipeline and the path through the upstream pipelines for you and you can then use an artifact from an ancestor pipeline in a downstream pipeline.

The ability to do this is important because of how artifacts are expected to be used in a Continuous Delivery pipeline. A core concept of CD is "Only Build Your Binaries Once". This prevents problems such as inconsistencies between a binary which went through various levels of testing and a new "duplicate binary" which is released after being built on a box which has a slightly different environment. So, the ability to move an artifact between pipelines and more importantly, knowing that it is the same binary being used (GoCD ensures that) is important.

We can then take the same binary across different kinds of tests, getting more confident about it as it progresses through the pipelines, eventually reaching a production-deployment pipeline, knowing that the binary that is getting deployed is the same as the one that went through tests.

Custom tabs

The final part of this guide introduces the concept of a custom tab. A question that gets asked all the time is: "Does GoCD integrate with <application-x> and show its output?". The answer is usually: "Yes, as long as that application can generate a report or output that needs to be shown". Here is the concept:

Concept 8: Custom tab

A custom tab in GoCD is a way of showing an artifact as a user-defined tab at the job level. Any file which is published as an artifact by a job can be shown. Usually, these are HTML files, with images. The GoCD Server will serve these files from its internal artifact repository.

Let's use the example artifact we had used earlier as a custom tab. To do that, let's go to the configuration of "MyFirstPipeline" and then to the "Custom Tab" section of the "defaultJob" and create one. It should look like this:

Configuring a custom tab
Configuring a custom tab

That's it! If you now go to the pipeline, and navigate to the job "defaultJob", you'll see a custom tab called "SomeCustomName". Clicking on it will show you the artifact for that run of the pipeline.

A custom tab in action
A custom tab in action

This can be used to integrate with coverage tools, unit test reports and anything else which creates an artifact.

In this part of the guide, you learnt about pipeline dependencies, artifacts and custom tabs. In Part 3 of this guide, we dive deeper into the Value Stream Map and into the powerful fan-in and fan-out concepts.