Continuous Integration: A Complete Guide to Using Bitbucket Pipelines With Maven Repositories

A complete walkthrough for setting up Bitbucket Pipelines to push and pull from Private Maven Repositories.


format_align_left Read Article

Using Bitbucket Pipelines With Maven Repositories

This tutorial will provide a professionally written, high quality, demonstration on how to use Pipelines and a Maven Repository in order to configure a Continuous Integration (CI) or Continuous Deployment workflow where build artifacts are stored, in a maven repository, for future use.

We accomplish this by providing a walkthrough of two examples:

First, we’ll use an Apache Maven based project in order to build a java library and publish it to an artifact repository. Once this library has been successfully published, we can use the library in other maven projects as a dependency.

If you want to dive right into the library example, you can view all of the source code in the Maven Library Example Bitbucket Repository.

Once the library is available in a repository, we will provide brief instructions on how to modify the first example to pull down the library in subsequent builds.

We believe these examples represent a primary use case for using Maven, building and sharing Java Archive (JAR) files (dependencies).

This article will describe how to publish to CloudRepo, a fully managed, private maven repository server. However, in the interest of providing the greatest benefit, these instructions should apply to any Apache Maven compatible repository.

Assumptions

We assume the usage of the following technologies:

  • Apache Maven
  • Bitbucket Git Repository
  • Bitbucket Pipelines
  • CloudRepo

Apache Maven

Apache Maven is a build tool used by Java software development teams to build software artifacts: libraries, binaries, documentation, etc.

Once an artifact has been successfully built with Maven, it can be deployed to a remote repository where other Maven based builds and processes can access it.

Bitbucket Git Repository

Git is a version control system used by software development teams to manage their source code bases.. Source code is the input to build processes which produce software artifacts.

In the Java/Maven world, source code is typically compiled into Java Virtual Machine (JVM) Object Code and then distributed as JAR files.

Bitbucket Cloud provides cloud based hosting for Git Repositories. A Bitbucket hosted Git Repository is required in order to use Pipelines.

Bitbucket Pipelines

Pipelines provides a Continuous Integration and Continuous Delivery Platform as a Service (PaaS) used by teams of all sizes to build, test, and deliver software in a repeatable deployment pipeline.

Pipelines provides a CI server that can build source code, run automated tests, and automate deployment of your application to your production environment many times each day as part of your development practice and release process.

If you are building maven artifacts, you’ll want to publish them to a shared repository for use in other builds after tests (unit tests, integration tests, etc.) have run successfully.

This article assumes you’ll be using Pipelines (which provides a limited amount of build minutes for free each month) to build libraries and publish them to a central repository hosted by CloudRepo.

CloudRepo

CloudRepo provides hosted private maven repositories so that you can publish your artifacts to persistent, highly available, accessible, and secure storage.

Once an artifact has been published to CloudRepo, it can be retrieved using a maven compatible client or via a HTTPS REST API. This allows you to reuse code by publishing it as a java library and then declaring it as a dependency in any other maven project.

For this example, we will store our artifacts using CloudRepo’s Private Maven Repositories.

Complete Build Process: Git, Apache Maven, Bitbucket Pipelines, and CloudRepo

Tying these technologies together will give you a build pipeline that looks something like this:

Bitbucket Pipelines Pushing to a Maven Repository (CloudRepo)

Examples

There are two major use cases for integrating Pipelines and CloudRepo:

  • Building a Library - Publishing Artifacts
  • Building an Application - Reading Artifacts

This article will first setup a build pipeline which will publish artifacts to CloudRepo using Maven.

Once artifacts have been published we will show how to modify your pom.xml file to read these artifacts in a separate build.

Now that we’ve covered all the background information, let’s jump into the code.

Building and Publishing a Java Library to CloudRepo

The first step to building and publishing your java library is to get the source code into a Bitbucket Git repository.

For this example, we’ve put together a library which performs basic mathematical functions that we’ll later use in an application. The complete source for this example can be found in our Example Git Repository

The functionality of this library is not important to this tutorial. It's just there to provide some code that we can publish as a library.

Bitbucket Pipeline Configuration

This section will cover all the configuration you have to do in the Bitbucket User Interface so that you can configure Maven in subsequent steps.

Enabling Pipelines

In order to use Pipelines, you must first enable them for your repository. Simply navigate to the settings of your repository and find the 'Pipelines' section. Toggle the 'Enable Pipelines' button in the Pipelines 'Settings' menu.

The following screenshot illustrates:

How To Enable Bitbucket Pipelines
Add Secure Environment Variables for Your Pipeline

Private artifact repositories require credentials in order to access them. You don't want to store these credentials in your source code as that is a major no-no and security issue (Pro Tip: never store credentials in a git repository).

Pipelines allows you to set Secure Environment Variables that each pipeline will have access to. This is an ideal place to store your maven credentials and other variables related to our build process.

In your Bitbucket Repository, navigate to the 'Settings' panel, then down to the 'Pipelines' section. Click 'Environment Variables' and add the following:

  • CLOUDREPO_USERNAME
    • This is the username for pushing and pulling artifacts from your maven repo. This will end up in your settings.xml file which we'll create later.
    • Optional: To prevent logging of this username you can set the enviroment variable to secret.
  • CLOUDREPO_PASSPHRASE
    • This is the passphrase for pushing and pulling artifacts from your maven repo. This will end up in your settings.xml file.
    • RECOMMENDED: To prevent logging of this passphrase you can set the enviroment variable to secret.
  • GIT_USER_EMAIL
    • Used only when the 'mvn release:prepare' trigger is run. This is the user email used by Git when Pipelines commits back to this repository.
    • Note: This does not have to correspond to an actual user's email, it will only show up in your Git commit logs as the author's email of any commit performed by Pipelines.
  • GIT_USER_NAME
    • Used only when the 'mvn release:prepare' trigger is run. This is the user name used by Git when Pipelines commits back to this repository.
    • Note: This does not have to correspond to an actual user's name, it will only show up in your Git commit logs as the author's name of any commit performed by Pipelines.

The following screenshot illustrates:

Setting Bitbucket Pipelines Environment Variables
Configuring SSH on Pipelines

When you run the Maven release plugin (ie. mvn release:prepare release:perform), maven will create git commits and attempt to push them back to your source code repository. A Bitbucket Pipeline does not have the permissions to push to a repository so we have to enable this by adding an SSH key associated with a Pipeline.

Create SSH KeyPair (For Maven Releases Only)

We need a key pair to allow Pipelines to git push back to this repo when we are releasing an artifact via a mvn release:perform command.

This can be found in the 'Settings | Pipelines (SSH Keys) | Generate Keys' section of the Bitbucket UI.

Generate a key to associate it with your pipeline as seen in the following screenshot:

Create SSH KeyPair for Bitbucket Pipeline

Add Public Key to Your Accounts list of SSH Keys (For Maven Releases Only)

Once the Key Pair has been generated, copy the Public Key that was created and navigate to your Bitbucket Account Settings (not the Pipelines Settings).

Add a new SSH to your account (Settings | Security | SSH Keys | Add Key).

Paste the public key from the Bitbucket Pipeline here.

Associate Bitbucket Pipeline SSH Key with Bitbucket Account

Now that you've configured the Bitbucket UI, you're ready to proceed with configuring Maven in your pipeline.

Maven POM File Configuration

In order to use Maven to build a project you’ll need to have a valid working Project Object Model (POM) in your source code directory named pom.xml. We provide a pom.xml in our example for reference.

<distributionManagement>

The <distributionManagement> tag is where we tell Maven where to deploy our Maven artifacts when we run commands like ‘mvn deploy’ (snapshots) or ‘mvn release:perform’ (releases).

A best practice for maven repository management is to have a separate repo for snapshots and one for releases.

Our distribution management section looks like this:

<!--
  - Distribution Management tells maven where to deploy your artifacts when the 'mvn deploy' command is used.
  -
  - Repository Parameters:
  -    id: A unique id for each repository, this must correspond to an entry in the settings.xml file used by maven
           usually located in ~/.m2/settings.xml or specified using the mvn command's -s parameter
           (ie: mvn -s settings.xml deploy).

           You can keep the ids as specified below as long as you have only one set (snapshot and release) of
           CloudRepo maven repositories, otherwise you'll need to have a unique id in your settings.xml file for each
           repository.

  -    name: A name to describe your repository, you can keep the example names given here.

  -    url: This is specific to your CloudRepo organization and repository id.  The format of the URL should be:
            https://[cloudrepo-organization-id].mycloudrepo.io/repositories/[repository-id]
  -->
<distributionManagement>
    <!--
    - The <repository> tag defines where release artifacts are pushed to as part of a 'mvn release:perform' command
    -->
    <repository>
        <id>io.cloudrepo.maven.releases</id>
        <name>CloudRepo Release Repositories</name>
        <url>https://cloudrepo-examples.mycloudrepo.io/repositories/maven-releases</url>
    </repository>

    <!--
    - The <snapshotRepository> tag defines where snapshot artifacts are deployed to as part of a 'mvn deploy' command.
    -->
    <snapshotRepository>
        <id>io.cloudrepo.maven.snapshots</id>
        <name>CloudRepo Snapshots Repositories-</name>
        <url>https://cloudrepo-examples.mycloudrepo.io/repositories/maven-snapshots</url>
    </snapshotRepository>
</distributionManagement>

Important: Remember that the <distributionManagement> tag is used only for deploying artifacts TO maven repositories.

For reading artifacts from maven repositories, you'll use the <repositories> tag, which is discussed next:

<repositories>

The <repositories> tag is used when you want to retrieve dependencies from a Maven Repository.

For our example, we will only be deploying artifacts to CloudRepo. However, for completeness we provide the configuration for the <repositories> tag so that you can use it in projects that consume your libraries.

Here's what our <repositories> tag looks like:

<!--
    - The <repositories> tag is used when you wish to define a maven repository that you want to READ artifacts from.
    -
    - While not used in this example, we leave this here in this pom for completeness.  If you want to read artifacts from
    - the repositories that you configured above in <distributionManagement> then these are the tags you'd need.
    -->
<repositories>
    <repository>
        <id>io.cloudrepo.maven.releases</id>
        <name>CloudRepo Release Repositories</name>
        <url>https://cloudrepo-examples.mycloudrepo.io/repositories/maven-releases</url>
        <releases>
            <enabled>true</enabled>
        </releases>
        <snapshots>
            <enabled>false</enabled>
        </snapshots>
    </repository>
    <repository>
        <id>io.cloudrepo.maven.snapshots</id>
        <name>CloudRepo Snapshots Repositories-</name>
        <url>https://cloudrepo-examples.mycloudrepo.io/repositories/maven-snapshots</url>
        <releases>
            <enabled>false</enabled>
        </releases>
        <snapshots>
            <enabled>true</enabled>
        </snapshots>
    </repository>
</repositories>
Snapshots versus Releases?

As you can see, we follow best practices and use separate CloudRepo repositories for snapshots and repositories. We’ll quickly describe the differences between Snapshots and Releases here (skip ahead if this known to you).

Maven Snapshots are used when you’re developing code that isn’t quite ready to be released. Maven snapshots are indicted by the ‘-SNAPSHOT’ suffix appended to a version id. Using Snapshots allows you to point at a changing version of an artifact without having to update your POM file.

Using Maven snapshots should only be used during development. As the contents of a maven snapshot can change, you lose the benefit of repeatable builds by depending on snapshot versions of artifacts.

Maven Releases are considered to be static and should not change. This allows a project to depend on a release version of an artifact and achieve repeatability in a build. Depending on version 1.0.0 of an artifact should always return the same version of an artifact. When declaring dependencies from within your projects, you should attempt to use release versions whenever possible in order to better achieve repeatability.

Maven settings.xml file

Before you can successfully deploy your artifacts to CloudRepo from Pipelines, you’ll need to create a settings.xml file which contains the credentials for the repositories you specified in the <dependencyManagement> section of your POM file.

To simplify this process, we provide a script named `create-settings.sh` which will create a settings.xml file by using the settings.template.xml file as a base and replacing the %CLOUDREPO_USERNAME% and %CLOUDREPO_PASSPHRASE% tokens with values from the CLOUDREPO_USERNAME and CLOUDREPO_PASSPHRASE environment variables.

Testing settings.xml creation

To test the settings.xml creation on a local Linux based machine, execute the `create-settings.sh` script with your CloudRepo username and passphrase:

$ CLOUDREPO_USERNAME=my-user  CLOUDREPO_PASSPHRASE=my-passphrase ./create-settings.sh

This command should run successfully and you should have a new file named settings.xml in your source directory. The contents should be the following:

<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0
      https://maven.apache.org/xsd/settings-1.0.0.xsd">

    <!-- generation text -->

    <!-- Complete Documentation for the settings.xml file can be found at https://maven.apache.org/settings.html -->

    <!--
     - <servers> specify the repositories we deploy or release artifacts to.
     -
     - The <id> tag specified here should match up with an <id> tag for a <repository> or <snapshotRepository>
     - located in the <distributionManagement> section of your project's pom.xml
     -->
    <servers>
        <server>
            <id>io.cloudrepo.maven.releases</id>
            <username>my-user</username>
            <password>my-passphrase</password>
        </server>
        <server>
            <id>io.cloudrepo.maven.snapshots</id>
            <username>my-user</username>
            <password>my-passphrase</password>
        </server>
    </servers>
</settings>

`mvn deploy`- Deploying Snapshots to CloudRepo

Now that you’ve created the settings.xml file, you can test the deployment of snapshots locally by running the following command from your source directory:

mvn deploy -s settings.xml

This command will build and deploy your SNAPSHOT artifacts to the CloudRepo snapshot repository.

bitbucket-pipelines.yml - Pipelines Configuration File

Finally, we’ll need to add a bitbucket-pipelines.yml file to our source code repository. This file describes the steps and scripts that Pipelines should execute when building your library.

In our example, we create two different build steps.

The first step, "Build and Deploy Snapshot Artifact" will build and deploy a SNAPSHOT every time a commit is detected on the master branch.

As you develop, this will be the normal workflow that you use as your team collaborates on a development version of your library. This step is configured to run tests and to ensure that they all pass before a SNAPSHOT is published to maven.

The second step, which must be triggered manually, "Create Release Version" will prepare a Maven RELEASE version, and commit the changes to the master branch. It will then increment the project version and commit the next SNAPSHOT version to the repository.

This results in two commits to your master branch: one for the RELEASE version and one for the new SNAPSHOT version. Both of these commits will be detected by the build trigger in the first step and will be built accordingly.

After these builds complete, you will have the RELEASE version of your library in your repository as well as the new SNAPSHOT version.

Our bitbucket-pipelines.yml file looks like the following:

image: atlassian/default-image:2

  pipelines:
      branches:
          master:  # Trigger this for any pushes to the master branch.
              - step:
                  name: Build and Deploy Snapshot Artifact
                  trigger: automatic
                  caches:
                      - maven # Cache any dependencies we download, speeds up build times.
                  script:
                      - bash create-settings.sh # Create our settings.xml file.  Will fail if environment variables aren't set properly.
                      - mvn -B verify # Ensure all artifacts build successfully before we attempt deploy in order to prevent partial deploys.
                      - mvn -B -s settings.xml deploy # Now that all builds have completed, we can deploy all the artifacts.
              - step:
                  name: Create Release Version # This will create a release version and commit it to master.  It will then be picked up and deployed in the first step.
                  trigger: manual
                  caches:
                      - maven
                  script:
                      - bash create-settings.sh # Create our settings.xml file.  Will fail if environment variables aren't set properly.
                      - bash validate-release-configuration.sh  # Do the best we can to ensure we have the SSH keys and env variables in place before we try to prepare a release.
                      - git config --global user.email "$GIT_USER_EMAIL"
                      - git config --global user.name "$GIT_USER_NAME"
                      - mvn -B -DdryRun=true release:prepare # Ensure that most things will run properly before we do the real work.
                      - mvn -B release:clean release:prepare # This bumps the versions in the poms, creates new commits, which will then get built by the master branch trigger.

Validating Deployment

Now that you have everything setup properly, how do you know it's working? You can validate successful deployment in two places:

  • Bitbucket Pipeline Status
  • CloudRepo Repository Storage

Bitbucket Pipeline Status

You can refer to the 'Pipelines' menu option for your Bitbucket Repository to see the status of your builds.

After successful execution of this example, you should see the following after your first successful build:

Viewing Bitbucket Pipeline Snapshot Builde Status

When you are ready to create a release version of your artifact, click the 'Run' button in the 'Create Release Version' step. This will kick of a 'mvn release:perform' which will create two new commits on your master branch.

When the 'Create Release Versions' step is done, the build results will look like the following:

Viewing Bitbucket Pipeline Releases Build Status

When you go back to the Pipeline Summary you will see that two new builds have been completed, one for the new release version and another for the next version of the snapshot.

Viewing Bitbucket Pipeline Summary Status

If your Bitbucket Pipeline history looks like this then you've completed this example successfully.

CloudRepo Repository Storage

Since these pipelines deployed Snapshot artifacts to CloudRepo, log into your repositories user interface to verify the artifacts have been successfully deployed.

We'll show examples from the CloudRepo Admin Portal below:

Snapshot Release

Navigating to the 'maven-snapshots' repository in CloudRepo, we can validate that the next version of the snapshot has been release:

CloudRepo Snapshot Repository

Release Version

Navigating to the 'maven-releases' repository in CloudRepo, we can validate the 1.0.0 release version has been successfully deployed.

CloudRepo Release Repository

More Details?

The source code for this example contains a README.md file with even more detailed instructions. Please look there if you need further information.

Living Document

This is a living document and so we'd love to hear any feedback on how we can improve this tutorial for others. If you have any suggestions at all, please feel free to email us.

Thank You

While we do our best to ensure a high quality example, we might have missed something. If you hit a snag, please open an issue so that we can fix it for others who are seeking similar knowledge.

This example was brought to you by the engineering team at CloudRepo. If you want to contact us directly, visit our Contact Us Page.

Please feel free to share on Social Media, Medium, or other channels.