Skip to end of metadata
Go to start of metadata

We are now using the Jenkins multi-branch pipeline to build the OpenAM project.

Having that pipeline may change your habits but it should be for the best.

Why define the pipeline as code?

As developing new functionality can impact the build process, having the build scripts close to the OpenAM codebase is beneficial.

  • you can make that change atomic and test it in your PR.
  • you don't need to manage different job versions in Jenkins and know what job to trigger for your specific version.

Having a pipeline is also the first step to continuous delivery where every commit could go into production. This means each commit should be treated as a potential release and built the same way.

One advantage of having the same pipeline for our daily work and the release is that we are quite confident that all the stages work. If we run them many time a day, there are fewer chances to spend many weeks at the end of the release cycle to fix gloomy stages. The key to making painful tasks less painful is to do them more often.

Pipeline Execution

The mandatory stages are the stages run by default in the following cases :

  • create a PR
  • update your PR
  • manually via the "Launch pipeline" button in your PR

On updating the PR, any currently running pipelines will be terminated with the exception of manually triggered executions (via the button). This is to account for the 'rebase, push, squash, push' workflow which would otherwise trigger multiple executions.

Stage Tags

Each stage can have a number of tags associated with it. These tags can be used by the pipeline to determine whether or not a stage should be run given the current context.

To see which stages have which tags associated with them, stages on the Engineering Dashboard can be filtered by tag name.

always-run: Stages tagged with this value will always be run, regardless of the context or CI Build Configuration. This tag is used for stages which the pipeline relies on, and will not function correctly without. For example none of the test stages can be run without the build, and stages used by the CI Build Configuration such as temper-pr and commons-pr need to be run on pull requests.

mandatory: All stages tagged with mandatory must pass for this to be considered an acceptable pipeline run. In the context of a pull request, all of the mandatory stages must pass for Jenkins to add an approval to the PR. In the context of the master branch, all of the mandatory stages must be successful for the journey for create-platform-images-pr and it's dependencies to run.

Below is a list of some of the mandatory stage descriptions. These descriptions should be moved into the pipeline rather than be defined here on the wiki.

Below is a list of some of the mandatory stage descriptions:

  • javadoc: Verification that the project JavaDoc compiles
  • image-smoke-tests: backend smoke tests, run against the docker image produced in file based mode
  • pyforge-smoke-tests: A deploy and configuration test which demonstrates that the PyForge framework functions correctly.
  • copyright: Check that the copyright attribution on changed files has been updated for the current year
  • amster-config-upgrader-tests: Runs the amster config upgrader against an old configuration, and attempts to import the upgraded config into the latest AM version.
  • upgrade-rules-check: Checks if any schema change has a rule written for the upgrade (PR only).

Below is a list of some of the other stage descriptions:

  • promote-stash: When all of the mandatory stages for a pipeline succeed, the stashed artifacts are copied to the promoted-stash area of the bucket, and are available for other pipelines to use when they use the mock-build CI Build Configuration option. The linked promoted stash is for master, but each mainline development branch will have it's own area in the bucket, and the version used by mock-build depends on the target branch specified.
  • cross-upgrade-tests: Deploys old version of AM, creates some config, upgrades AM, and then verifies that the upgrade did the right thing (and did not fail)
  • elastic-federation-tests: Runs a set of functional tests that verifies that SAML SSO works in an elastic type deployment
  • file-smoke-tests: Deploys AM with a file based configuration
  • file-sts-tests: Deploys AM with a file based configuration
  • file-functional-tests: Deploys AM with a file based configuration
  • saml2-node-tests-compose: Runs Pyforge functional tests for the SAML2 authentication node
  • upgrade-tests-functional: Deploys old version of AM, upgrades it, then runs the functional smoke tests against the upgraded instance.
  • upgrade-tests-ui-admin: Deploys old version of AM, upgrades it, then runs the UI admin smoke tests against the upgraded instance.
  • upgrade-tests-ui-user: Deploys old version of AM, upgrades it, then runs the UI user smoke tests against the upgraded instance.
  • k8s: Run the Lodestar Kubernetes tests, using the stable ForgeOps branch + the AM commit in the current build
  • pit1: Run the Lodestar PIT1 tests, using the stable ForgeOps branch + the AM commit in the current build


Because we understand that you may have corner cases we have introduced a bit of flexibility in the way Jenkins executes the stages of the pipeline.

How to control the stages executed in your PR

First, Add the following string to the bottom of your PR description “CI build configuration” ( It's case and formatting insensitive )

Here is the list of controls you can specify:

  • only-stages: Run only the stages specified  (mandatory or optional). Can be useful if you already ran the mandatory stages and want to see the state of some optional-stages for the same commit without going through the whole pipeline again. Stages tagged with always-run will still run when using this option.
  • optional-stages: Run the optional stages specified on top of the mandatory ones. By default, the optional stages are not run in your PR.
  • skip-stages: Skip the stages listed here. This should be used with care. Mostly useful when writing a new stage in the pipeline.
  • groups: apply the given test groups to all stages. Can be used in conjunction with the only-stages or optional-stages controls. For example to run only the ciba tests in the smoke tests, use 
        only-stages=smoke-tests
groups=ciba 

Values provided should comma delimited lists without spaces.

optional-stages, skip-stages and only-stages support '*' as a value. If '*' is used as a value it will match every stage.

Note: Master and 6.5.x branch were updated - change how groups are handled. The only option for configuring groups is now `groups=<groups>` in the CI Build Configuration. The intersection of the provided groups with each stage will then be run. This means that running with `groups=oauth2` will run only the oauth2 grouped tests in the mandatory stages. You can also use expressions, for example `groups=or(oauth2, session)` or `groups=and(oauth2, not(oidc))`.


The previous set of commands are useful to tweak the pipeline and get the feeback you need.

The following set of command is useful when you are writing a stage or testing the pipeline :

  • mock-build: This will use a previously built version of openam instead of building one from the source. The version used will be from the promoted-stash of the target branch.
  • fast-mode: This will run a fast version of the stages supporting this options. This often boils down to running a subset of tests. This is not supported by all stages.
  • stop-automatic-build: if set to true, every build triggered by an updated pull request will be aborted. You can still trigger a build manually by clicking the button in your PR. 


Examples of custom build passed in the PR description are :

You already executed the pipeline with the mandatory stages and I want to know the state of the functional tests.











This will execute the build and the functional-tests stage


You want to run the functional-tests and the mandatory stages


This will run the build → smoke tests → security tests → amster tests → functional tests

How to test a change which involves modifying OpenAM and Temper

If you have an OpenAM pull request and a temper pull request you can start the OpenAM pipeline with your temper PR by adding temper-pr command to the “CI build configuration”. You must set the temper pr id as in this example :


Ci build configuration

temper-pr=1466


Where 1466 is the id of your temper PR. This id can be looked up in your PR overview url.

How to test a change which involves modifying OpenAM and Commons

If you have an OpenAM pull request and a Commons pull request you can start the OpenAM pipeline with your commons PR by adding commons-pr command to the “CI build configuration”. You must set the commons pr id as in this example :


Ci build configuration

commons-pr=1466


Where 1466 is the id of your commons PR. This id can be looked up in your PR overview url.

The version of commons used in the AM build will be automatically set to the version that is in the branch the PR is from.

Welcome your new Reviewer 

When a pull request is created a build is triggered automatically and the Jenkins user will join the PR soon after.

The Jenkins user will only approve your PR if all the mandatory stages pass.

A pull-request shouldn’t be merged if Jenkins didn’t approve your PR.

Note that if you skip mandatory stages, Jenkins will not approve your PR and will remove any previous approval unless you didn't push new files between your two builds.

Child pages





  • No labels

6 Comments

  1. The link to the incubating pipeline is broken. Has this pipeline been renamed? Does it still exist?

    1. Good observation, I've removed the section as I think you're right, that pipeline has been removed.

  2. Thanks for the updates to the page.  I have a question:

    Can multiple optional-stages be specified? 

    Something I find useful for OAuth2 related bug fixes and changes (and which used to work recently) is e.g:

    optional-stages=functional-tests,ui-user-tests

    groups=oauth2,oidc

    1. Yes - multiple optional stages can be specified. Should you want to run them all then an * can be used

  3. Can we move this to the new AM Space?  When we do make sure we update the link that Jenkins posts in pull reuests.