Publishing Maven Repo in 5 Minutes

Publishing Maven Repo in 5 Minutes

Chances are you’re already using the Maven Central Repo, which is the default repo when you installed Maven. Today I’ll show you how to publish your artifacts to it.

Backgrounds

The Central Repository (http://search.maven.org/) is an index containing all publicly available Maven artifacts. Maven by default downloads all dependencies from the Central Repo . It does not host the files. Instead, you need to use one of the repo hosting services, and then publish your files to the Central Repo. I’ll demo the following services:

  • Sonatype OSSRH
  • JFrog Bintray

I recommend reading Comparing the Two to choose your preference first.

More Backgrounds on Sonatype

Sonatype’s repo system is a bit complex. I’ll try my best to make it simple (after reading the nexus book).

Sonatype Professional is a repo system. OSSRH is a free publicly available Sonatype Profressional.

The general steps of publishing artifacts are:

# Snapshot (not supported by Bintray)
Build artifacts -> Sign artifacts (optional) -> Snapshot Repo

# Release
Release artifacts -> Sign artifacts -> Staging Repo -> Release Repo

The life cycle of staging repo (notice the open and close states):

Prerequisites

  1. Install gpg
  2. Generate a key pair for signing your artifacts (instructions here)
  3. (For Sonatype) Go to https://issues.sonatype.org/secure/Signup!default.jspa and sign up an account, which will be used to upload artifacts.
  4. (For Sonatype) Register a group ID. To do this, create a Jira ticket like this.
  5. (For Bintray) Register a Bintray account.
  6. Make sure your’re familiar with Maven release plugin. You can read Maven release in 5min.

Warning
It usually takes a couple of days for Sonatype group ID to get approved.
Also, you may need to prove that you own a domain if your group ID is a domain name.
Please read the registration policy.

And start your timer now!

Source Code

You can fork the following project to use for this example (remember to change <groupId> in pom.xml).
https://github.com/ryan-ju/resteasy-demo-01

Using Sonatype OSSRH

Releasing Snapshot

  • Add Sonatype account and gpg keys to ~/.m2/settings.xml
  • Add SCM and repositories to project’s pom.xml
  • Configure gpg and nexus staging plugin to upload snapshots to Sonatype

Configure your .m2/settings.xml to add your Sonatype account and gpg keys.

<settings>
  <servers>
    <server>
      <id>ossrh-server</id>
      <username>your-jira-id</username>
      <password>your-jira-pwd</password>
    </server>
  </servers>
  <profiles>
    <profile>
      <id>ossrh</id>
      <activation>
        <activeByDefault>true</activeByDefault>
      </activation>
      <properties>
        <gpg.executable>gpg(the shell command)</gpg.executable>
        <gpg.passphrase>the_pass_phrase(for the secret key)</gpg.passphrase>
      </properties>
    </profile>
  </profiles>
</settings>

And then add scm and repos to your pom.xml.

<scm>
    <connection>scm:git:git@github.com:ryan-ju/resteasy-demo-01.git</connection>
    <developerConnection>scm:git:git@github.com:ryan-ju/resteasy-demo-01.git</developerConnection>
    <url>git@github.com:ryan-ju/resteasy-demo-01.git</url>
  <tag>HEAD</tag>
</scm>
<distributionManagement>
  <snapshotRepository>
    <id>ossrh-server</id>
    <url>https://oss.sonatype.org/content/repositories/snapshots</url>
  </snapshotRepository>
  <repository>
    <id>ossrh-server</id>
    <url>https://oss.sonatype.org/service/local/staging/deploy/maven2/</url>
  </repository>
</distributionManagement>

Then configure the plugins. See comments for explanation.

<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-source-plugin</artifactId>
            <version>2.4</version>
            <executions>
                <execution>
                    <id>attach-sources</id>
                    <goals>
                        <goal>jar-no-fork</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-javadoc-plugin</artifactId>
            <version>2.10.3</version>
            <executions>
                <execution>
                    <id>attach-javadocs</id>
                    <goals>
                        <goal>jar</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-gpg-plugin</artifactId>
            <version>1.5</version>
            <!-- Automatically gets passphrase from property gpg.passphrase -->
            <executions>
                <execution>
                    <id>sign-artifacts</id>
                    <phase>verify</phase>
                    <goals>
                        <goal>sign</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>
        <plugin>
            <groupId>org.sonatype.plugins</groupId>
            <artifactId>nexus-staging-maven-plugin</artifactId>
            <version>1.6.6</version>
            <!-- This plugin is an extension to the deploy phase, adding a new deployment type -->
            <extensions>true</extensions>
            <configuration>
                <!-- The server ID configured in settings.xml  -->
                <serverId>ossrh-server</serverId>
                <nexusUrl>https://oss.sonatype.org/</nexusUrl>
                <!-- Not important for snapshots -->
                <autoReleaseAfterClose>false</autoReleaseAfterClose>
            </configuration>
        </plugin>
    </plugins>
</build>

nexus-staging-maven-plugin
The available goals include deploy, release and drop. For full list, see here.

Now you’re ready to stage the artifact. Run

mvn clean deploy

and you should see your snapshot artifacts in https://oss.sonatype.org/

Releasing Release

  • Build must satisfy the requirements
  • Prepare your release with release:prepare
  • Release with release:perform
  • (Optional) Execute mvn nexus-staging:release to upload artifacts to

First, you need to run

mvn release:prepare

Then run

mvn release:perform

which will upload the artifacts to a staging repo. This is because the perform goal by default runs a deploy phase, and the nexus plugin’s deploy goal is bound to deploy phase. To configure what phases are run by perform, set the <goals> configuration.

You can now release the artifacts with (reference)

mvn nexus-staging:release

Alternatively, you can do release with Maven release plugin(reference)

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-release-plugin</artifactId>
  <version>2.5.3</version>
  <configuration>
    <autoVersionSubmodules>true</autoVersionSubmodules>
    <useReleaseProfile>false</useReleaseProfile>
    <releaseProfiles>release</releaseProfiles>
    <!-- This will release the artifacts after staging them -->
    <goals>deploy nexus-staging:release</goals>
  </configuration>
</plugin>

and run

mvn release:prepare release:perform

Now you can find your artifacts in the Central Repo http://mvnrepository.com/.

Using JFrog Bintray

Staging Artifacts

  • Add Bintray account to .m2/settings.xml
  • Create a package to hold your artifacts
  • Add SCM and repositories to project’s pom.xml

First, add a server tag to your .m2/settings.xml.

<server>
  <id>bintray</id>
  <username>{bintray-user}</username>
  <password>{bintray-api-key}</password>
</server>

where the api key can be found in you Bintray profile

Then create a package in your Bintray’s Maven repo. This is a Bintray specific tag for organizing artifacts. In my case I created “resteasy-demo-01” package.

Now edit your pom.xml to include

  • The scm tag so the release plugin can update the versions
  • The remote repo
  • The necessary plugins
<scm>
    <connection>scm:git:git@github.com:ryan-ju/resteasy-demo-01.git</connection>
    <developerConnection>scm:git:git@github.com:ryan-ju/resteasy-demo-01.git</developerConnection>
    <url>git@github.com:ryan-ju/resteasy-demo-01.git</url>
    <tag>HEAD</tag>
</scm>
<distributionManagement>
    <repository>
        <id>bintray-repo-maven</id>
        <url>https://api.bintray.com/maven/ryan-ju/maven/resteasy-demo-01/</url>
    </repository>
</distributionManagement>
<!-- Dependencies go here -->
<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-source-plugin</artifactId>
            <executions>
                <execution>
                    <id>attach-sources</id>
                    <goals>
                        <goal>jar-no-fork</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-javadoc-plugin</artifactId>
            <executions>
                <execution>
                    <id>attach-javadocs</id>
                    <goals>
                        <goal>jar</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>
        <plugin>
            <artifactId>maven-release-plugin</artifactId>
            <version>2.5.3</version>
            <configuration>
                <autoVersionSubmodules>true</autoVersionSubmodules>
            </configuration>
        </plugin>
    </plugins>
</build>

Then you can release your artifacts to Bintray

# This creates a tag for the release version in GitHub, and increment to the next SNAPSHOT version (e.g., 1.0.1-SNAPSHOT to 1.0.2-SNAPSHOT).
# This also generates a release.properties file in your work dir
# As long as release.properties exists, this command is idempotent.
mvn release:prepare

# This command releases artifacts to remote repo.  The actual files are from the release version, not the snapshot (obviously).
mvn release:perform

And you should see the artifacts under the package you created.

Up till now, the file aren’t publicly available. To publish them (to your repo, not Maven Central), click the button

Now you can use the repo in other projects by including the following in pom.xml

<repositories>
  <repository>
    <id>bintray</id>
    <url>http://dl.bintray.com/{bintray-user}/{bintray-repo}</url>
    <releases>
      <enabled>true</enabled>
    </releases>
    <snapshots>
      <enabled>false</enabled>
    </snapshots>
  </repository>
</repositories>

Warning
You should also configure the gpg key used to sign your artifacts. This can be done by clicking “Edit” in your Bintray profile and add a public key in the format

-----BEGIN PGP PUBLIC KEY BLOCK-----
Version: GnuPG v1

<key content>

-----END PGP PUBLIC KEY BLOCK-----

Releasing Artifacts

You need to click “Add to JCenter” button in your repo and give a group ID you want to publish under.

and then I guess you just need to wait for it to be approved.

Once that’s done, you should be able to find it in the Central Repo.

Comparing the Two

Both of the hosting services offer staging and publishing ability.

Bintray is easier to set up and can be used straight away (without requesting for a group ID), but can only host release versions of files (no SNAPSHOTs). The web UI is more intuitive and easier to use.

Sonatype is more complicated, but you can stage both snapshot and release versions of your artifacts. The web UI isn’t as pretty as Bintray, but still usable.

References

OSSRH Apache Maven configuration: http://central.sonatype.org/pages/apache-maven.html
Repository management with Nexus: https://books.sonatype.com/nexus-book/reference/staging-deployment.html
A practical guide to uploading to Maven central: http://knowm.org/a-practical-guide-to-uploading-artifacts-to-maven-central/
Nexus staging plugin: https://books.sonatype.com/nexus-book/reference/staging-deployment.html
Bintray tutorial: http://veithen.github.io/2013/05/26/github-bintray-maven-release-plugin.html

Written with StackEdit.

Weave vs Flannel: Network Performance

Weave vs Flannel: Network Performance

Introduction

Weave and Flannel are currently the two main solutions to overlay networks for containers. They both try to solve the problem: how do you assign an IP to each container to connect them, when each container host can only have one IP.

They both employ the IP encapsulation approach: carrying layer2 (link layer) frames inside UDP datagrams.

The difference lies in the implementation details, which you can find in this post. The post also has a network performance test between the two, but doesn’t look at Weave fast datapath.

So in this post, I will compare three different setups: Weave with/without fast datapath, and Flannel.

Setup Weave

I launched 2 EC2 instances (t2.medium on hvm) in the same AZ.

To set up Weave, Weave doc is all we need, and is easy to follow.

Setup Flannel

Setting up Flannel is more involved. The steps are here:
Set up an etcd cluster.

# On both hosts
# Download and untar etcd
$ curl -L  https://github.com/coreos/etcd/releases/download/v2.2.2/etcd-v2.2.2-linux-amd64.tar.gz -o etcd-v2.2.2-linux-amd64.tar.gz
$ tar xzvf etcd-v2.2.2-linux-amd64.tar.gz

# On HOST0, replacing $HOST0 and $HOST1 with the EC2 private IP
$ export ETCD_INITIAL_CLUSTER="infra0=http://$HOST0:2380,infra1=http://$HOST1:2380"
$ export ETCD_INITIAL_CLUSTER_STATE=new
# Start etcd server in the background
$ nohup ./etcd-v2.2.2-linux-amd64/etcd -name infra0 -initial-advertise-peer-urls http://$HOST0:2380 -listen-peer-urls http://$HOST0:2380 -listen-client-urls http://$HOST0:2379,http://127.0.0.1:2379 -advertise-client-urls http://$HOST0:2379 -initial-cluster-token etcd-cluster-1 &

# On HOST1, replacing $HOST0 and $HOST1 with the EC2 private IP
$ export ETCD_INITIAL_CLUSTER="infra0=http://$HOST0:2380,infra1=http://$HOST1:2380"
$ export ETCD_INITIAL_CLUSTER_STATE=new
# Start etcd server in the background
$ nohup ./etcd-v2.2.2-linux-amd64/etcd -name infra1 -initial-advertise-peer-urls http://$HOST1:2380 -listen-peer-urls http://$HOST1:2380 -listen-client-urls http://$HOST1:2379,http://127.0.0.1:2379 -advertise-client-urls http://$HOST1:2379 -initial-cluster-token etcd-cluster-1 &

Install Flannel

# On both hosts
$ curl -L https://github.com/coreos/flannel/releases/download/v0.5.5/flannel-0.5.5-linux-amd64.tar.gz -o flannel.tar.gz
$ tar zxf flannel.tar.gz

# On one host
$ ./etcd-v2.2.2-linux-amd64/etcdctl set /coreos.com/network/config '{ "Network": "10.1.0.0/16", "Backend": { "Type": "vxlan"} }'

Host Connection Test

Install iperf on each host.

# ===TCP===
# On HOST0
$ iperf -f M -i 1 -m -s

# On HOST1
$ iperf -f M -t 60 -i 1 -c $HOST0

# Output
[  3]  0.0-60.0 sec  6887 MBytes   115 MBytes/sec
[  4] MSS size 8949 bytes (MTU 8989 bytes, unknown interface)

# ===UDP===
# On HOST0
$ iperf -f M -i 1 -m -su

# On HOST1
$ iperf -f M -i 1 -t 60 -m -c $SERVER -b 1000M

# Output
[  4]  0.0-10.1 sec   562 MBytes  55.7 MBytes/sec   0.088 ms  450/401197 (0.11%)

Weave without Fast Datapath Test

First, we need to start Weave without fast datapath, using env variable WEAVE_NO_FASTDP.

# HOST0
$ WEAVE_NO_FASTDP=true weave launch
$ eval "$(weave env)"

# HOST1
$ WEAVE_NO_FASTDP=true weave launch $HOST1
$ eval "$(weave env)"

Next, we can launch containers.

# HOST0
$ docker run -it --name test-server ubuntu
## Inside test-server
$ apt-get update; apt-get install iperf

# HOST1
$ docker run -it --name test-client ubuntu
## Inside test-client
$ apt-get update; apt-get install iperf

Then we can run iperf in similar way as above, but you now need to use the IP in the overlay network.

# Output
# ===TCP===
[  3]  0.0-60.0 sec  5168 MBytes  86.1 MBytes/sec

# ===UDP===
[  3]  0.0-60.0 sec  2634 MBytes  43.9 MBytes/sec   0.217 ms 2246629/4125665 (54%)

Note the performance is over 20% less than host connection. Let’s enable fast datapath.

Weave with Fast Datapath Test

Weave requires some work to stop.

# On both hosts.  For HOST1, change test-server to test-client.
$ docker rm -f test-server
# Necessary to put DOCKER_HOST back to its original value
$ eval "$(weave env --restore)"
$ weave stop
$ weave reset

Now restart Weave as above, but without WEAVE_NO_FASTDP=true.

Run tests the same way.

# Output
# ===TCP===
[  4]  0.0-60.1 sec  2546 MBytes  42.4 MBytes/sec

# ===UDP===
[  3]  0.0-60.1 sec  1438 MBytes  23.9 MBytes/sec   0.556 ms 3098522/4124483 (75%)

Er … fast datapath seems to make performance worse ???!!!

Set Proper MTU

It turns out Weave by default has MTU set to 1410, despite AWS VPC can handle 9001. We need to tell Weave to use a higher value, as it can’t detect it automatically (at least I didn’t find out how).

$ WEAVE_MTU=8950 weave launch

And the test result looks much better

# Without fast datapath
# ===TCP===
[  3]  0.0-60.2 sec  5050 MBytes  83.9 MBytes/sec

# ===UDP===
[  3]  0.0-60.0 sec  3895 MBytes  64.9 MBytes/sec   0.291 ms 1349488/4127983 (33%)

# With fast datapath
# ===TCP===
[  4]  0.0-60.0 sec  6897 MBytes   115 MBytes/sec

# ===UDP===
[  3]  0.0-60.2 sec  3245 MBytes  53.9 MBytes/sec   0.518 ms 1808496/4123433 (44%)

Ok the results look more reasonable, expecially that TCP with fast datapath is the same as host speed now (of course there is some error variance, but shouldn’t be that much).

Flannel Test

Flannel seems to be able to automatically detect the network MTU, so we don’t need to care about it.

To start Flannel,

# On both hosts
$ nohup ./flannel-0.5.5/flanneld &
$ source /run/flannel/subnet.env
$ nohup docker daemon --bip=${FLANNEL_SUBNET} --mtu=${FLANNEL_MTU} &

Now Docker daemon is configured to use Flannel’s overlay network.

Launch containers and run tests as before.

# Output
# ===TCP===
[  3]  0.0-60.0 sec  6866 MBytes   114 MBytes/sec
[  3] MSS size 8899 bytes (MTU 8939 bytes, unknown interface)

# ===UDP===
[  3]  0.0-60.2 sec  3086 MBytes  51.2 MBytes/sec  13.546 ms 1920728/4122179 (47%)

As we see, the performance is nearly the same as host speed.

Conclusions

Setup TCP UDP
Host 115MB/s 55.7MB/s
Weave without Fast Datapath, default MTU 86.1MB/s 43.9MB/s
Weave with Fast Datapath, default MTU 42.4MB/s 23.9MB/s
Weave without Fast Datapath, MTU=8950 83.9MB/s 64.9MB/s
Weave with Fast Datapath, MTU=8950 115MB/s 53.9MB/s
Flannel with backend=vxlan 114MB/s 51.2MB/s
  • Weave with Fast Datapath (and correct MTU) seems to have the same performance as Flannel.
  • UDP performance seems to be varying quite a lot during the tests, so the values above may not be representative. Indeed, when I used -l 8950 in iperf, the performance dropped to ~30MB/s. I’m not sure if this is due to some kind of throttling on AWS, but this happens with all of the setups.
  • Weave fast datapath currently doesn’t support encryption, so it’ll be down to the application to do that.

Written with StackEdit.

Thoughts on programming

Thoughts on programming

  • When current build tool lacks some features:
    Java dev: write a Maven plugin
    Scala dev: write an SBT script
    Ruby dev: write a new Rake task
    JS dev: write a new build tool

Written with StackEdit.