Skip to content

Commit a2e5962

Browse files
authored
feat: migrate maven repository and enable snapshots (#2032)
1 parent 7088563 commit a2e5962

14 files changed

+190
-70
lines changed

.github/workflows/publish_assets.yml

Lines changed: 17 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,13 @@ jobs:
1414
runs-on: ubuntu-latest
1515

1616
steps:
17-
- uses: actions/checkout@v3
17+
- uses: actions/checkout@v4
18+
with:
19+
# We need to download all tags so that the axion-release-plugin can resolve the most recent version tag.
20+
# This is useful only for the manual trigger.
21+
# For the release trigger, the tag is available on the current commit.
22+
fetch-depth: 0
23+
1824
# Get the version from the commit. This will depend on the trigger of the workflow
1925
# If the trigger is release, the version will be the tag on the commit.
2026
# If the trigger is a workflow_dispatch, the version will be the branch name, which is
@@ -91,27 +97,22 @@ jobs:
9197
asset_name: javadocs.zip
9298
asset_content_type: application/zip
9399

94-
# The following steps will publish artifacts to a sonatype staging repo with the aim of promoting them to maven central
95-
# Pretty much everything is done through gradle.
96-
# The version used will be according to the axion-release-plugin, meaning it will take a tag if present.
97-
# The tag should follow semantic versioning, e.g. v1.2.3. There could be a suffix, e.g. v1.2.3-TEST
98-
# gradle will build, sign then upload artifacts to a Sonatype staging repo.
99-
# See https://s01.oss.sonatype.org for accessing these repos.
100-
# At this point it should manually be closed, which will trigger acceptance tests for maven central (but not transfer yet)
101-
# Once closed, the repo is available for testing.
102-
# After testing, it can be manually promoted on the sonatype site, which will then publish to maven central.
103-
# Note than once in maven central a release cannot be removed or altered.
104-
105100
- name: Load secrets from 1Password
106101
id: onepw_secrets
107102
uses: ./.github/actions/extract-1password-secret
108103
with:
109104
OP_SERVICE_ACCOUNT_TOKEN: ${{ secrets.OP_SERVICE_ACCOUNT_TOKEN }} # This is required to connect to the vault in our 1Password account.
110-
VARIABLES_TO_EXTRACT: 'MAVEN_GPG_PASSPHRASE, MAVEN_GPG_PRIVATE_KEY, SONATYPE_TOKEN_USERNAME, SONATYPE_TOKEN_PASSWORD'
105+
VARIABLES_TO_EXTRACT: 'MAVEN_GPG_PASSPHRASE, MAVEN_GPG_PRIVATE_KEY, MAVEN_CENTRAL_PORTAL_TOKEN_USERNAME, MAVEN_CENTRAL_PORTAL_TOKEN_PASSWORD'
111106
ONE_PASSWORD_SECRET_REFERENCES: ${{ vars.ONE_PASSWORD_SECRET_REFERENCES }}
112107

113-
- name: Build and Publish to Sonatype
108+
- name: Build and Publish to Maven Central
114109
run: |
115-
# Don't verify since it has already been done when the PR was created.
116-
./gradlew publish --rerun-tasks -x spotlessCheck
110+
# Publishing to Maven Central will fail if the version is a snapshot, i.e. there is no version tag
111+
# directly on the current commit.
112+
# See https://repo1.maven.org/maven2/org/mobilitydata/gtfs-validator/ for the maven central repository.
113+
# Once this step is done, the artefacts will be validated but not yet published.
114+
# See https://central.sonatype.com/publishing/deployments (you need to login as mobilitydata)
115+
# From this page you can examine the list of artefacts, and either drop them or release them.
116+
# Note that once released, the artefacts cannot be removed or altered.
117+
./gradlew publishToMavenCentral
117118
Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
name: Publish Jars Snapshots
2+
3+
on:
4+
push:
5+
branches: [ master ]
6+
paths:
7+
- 'core/src/main/**'
8+
- 'main/src/main/**'
9+
- 'model/src/main/**'
10+
- 'build.gradle'
11+
- '.github/workflows/publish_snapshots.yml'
12+
workflow_dispatch:
13+
14+
env:
15+
java_version: '17'
16+
java_distribution: 'zulu'
17+
18+
jobs:
19+
publish-snapshots:
20+
runs-on: ubuntu-latest
21+
22+
steps:
23+
- uses: actions/checkout@v4
24+
with:
25+
# We need to download all tags so that the axion-release-plugin
26+
# can resolve the most recent version tag.
27+
fetch-depth: 0
28+
29+
- name: Set up JDK ${{ env.java_version }}-${{ env.java_distribution }}
30+
uses: actions/setup-java@v3
31+
with:
32+
java-version: ${{ env.java_version }}
33+
distribution: ${{ env.java_distribution }}
34+
35+
- name: Load secrets from 1Password
36+
id: onepw_secrets
37+
uses: ./.github/actions/extract-1password-secret
38+
with:
39+
OP_SERVICE_ACCOUNT_TOKEN: ${{ secrets.OP_SERVICE_ACCOUNT_TOKEN }} # This is required to connect to the vault in our 1Password account.
40+
VARIABLES_TO_EXTRACT: 'MAVEN_GPG_PASSPHRASE, MAVEN_GPG_PRIVATE_KEY, MAVEN_CENTRAL_PORTAL_TOKEN_USERNAME, MAVEN_CENTRAL_PORTAL_TOKEN_PASSWORD'
41+
ONE_PASSWORD_SECRET_REFERENCES: ${{ vars.ONE_PASSWORD_SECRET_REFERENCES }}
42+
43+
- name: Build and Publish to the snapshot repository
44+
run: |
45+
# Publishing to the snapshot repository will fail if the current commit has a version tag.
46+
# Which in effect makes it a release.
47+
# See https://central.sonatype.com/service/rest/repository/browse/maven-snapshots/org/mobilitydata/gtfs-validator/
48+
# for the snapshot repository.
49+
./gradlew publishToSnapshotRepository
50+

README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -63,7 +63,7 @@ To facilitate easier debugging and logging, we have made our user agent header e
6363
### Run it
6464
Once installed, run the application and you will see the following screen:
6565

66-
![Application-Windows](/docs/Application-Windows.png)
66+
![Application-Windows](/docs/images/Application-Windows.png)
6767

6868
There are two primary options to set:
6969

build.gradle

Lines changed: 66 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -15,14 +15,15 @@
1515
*/
1616
/*
1717
* A note about publishing and signing.
18-
* Maven central requires that artifacts be signed. And upload is done to Sonatype.
18+
* Maven central requires that artifacts be signed. And upload is done via Maven Central Portal.
1919
* To publish you will need these environment variables defined:
20-
* SONATYPE_TOKEN_USERNAME
21-
* SONATYPE_TOKEN_PASSWORD
20+
* MAVEN_CENTRAL_PORTAL_TOKEN_USERNAME
21+
* MAVEN_CENTRAL_PORTAL_TOKEN_PASSWORD
2222
* MAVEN_GPG_PRIVATE_KEY
2323
* MAVEN_GPG_PASSPHRASE
24-
* Suggestion is to put these in a shell script with restricted read permissions, then source it before calling
25-
* ./gradlew publish.
24+
* If run manually, the suggestion is to put these in a shell script with restricted read permissions,
25+
* then source it before calling ./gradlew
26+
* If run from a Github action, make sure these environment variables are defined.
2627
*/
2728
plugins {
2829
id 'java'
@@ -58,6 +59,7 @@ allprojects {
5859

5960
tasks.withType(Javadoc) {
6061
options.encoding = 'UTF-8'
62+
options.addStringOption('Xdoclint:none', '-quiet')
6163
}
6264

6365
// All projects that include the 'java` plugin will have a Test task by default.
@@ -82,16 +84,6 @@ subprojects {
8284
apply plugin: 'java'
8385
apply plugin: libs.plugins.spotless.get().pluginId
8486

85-
// Cannot publish a SNAPSHOT. The provided sonatype url will not accept it.
86-
tasks.withType(PublishToMavenRepository).all { task ->
87-
task.onlyIf {
88-
if (project.version.toString().contains('SNAPSHOT')) {
89-
throw new GradleException("Publishing is not allowed for SNAPSHOT versions. Currently " + project.version)
90-
}
91-
true
92-
}
93-
}
94-
9587
task javadocJar(type: Jar) {
9688
archiveClassifier.set('javadoc')
9789
from javadoc
@@ -111,24 +103,29 @@ subprojects {
111103

112104
// These modules require the same publishing configuration, apart from the name of the module
113105
// Also we want to limit artefact publishing to these modules.
114-
if (project.name == 'main' ||
115-
project.name == 'core' ||
116-
project.name == 'model') {
106+
if (project.name == 'main'
107+
|| project.name == 'core'
108+
|| project.name == 'model'
109+
) {
117110
def fullProjectName = 'gtfs-validator-' + project.name
118111

119112
afterEvaluate {
120113
publishing {
121114
repositories {
122-
// This is the sonatype staging repo for maven.
123-
// Once uploaded, the repo needs to be manually closed, which will trigger acceptance tests for
124-
// maven central (but not transfer yet).
125-
// Once successfully closed, the repo is available for testing.
126-
// After testing, it can be manually promoted on the sonatype site, which will then publish to maven central.
127115
maven {
128-
url = 'https://s01.oss.sonatype.org/service/local/staging/deploy/maven2'
129-
credentials {
130-
username System.getenv("SONATYPE_TOKEN_USERNAME")
131-
password System.getenv("SONATYPE_TOKEN_PASSWORD")
116+
// Snapshot uses a different repository than release.
117+
// The publish in that case will upload to Maven Central snapshot repository.
118+
if (version.endsWith('SNAPSHOT')) {
119+
url = 'https://central.sonatype.com/repository/maven-snapshots/'
120+
credentials {
121+
username = System.getenv("MAVEN_CENTRAL_PORTAL_TOKEN_USERNAME")
122+
password = System.getenv("MAVEN_CENTRAL_PORTAL_TOKEN_PASSWORD")
123+
}
124+
} else {
125+
// For releases, the publish task does not upload to MavenCentral.
126+
// It just the artifacts directory hierarchy on the local disk,
127+
// The actual zipping and uploading is done in the publishToMavenCentral task.
128+
url = "${buildDir}/local-repo"
132129
}
133130
}
134131
}
@@ -172,9 +169,51 @@ subprojects {
172169
useInMemoryPgpKeys(System.getenv('MAVEN_GPG_PRIVATE_KEY'), System.getenv('MAVEN_GPG_PASSPHRASE'))
173170
sign publishing.publications.mavenJava
174171
}
172+
173+
174+
tasks.register('publishToMavenCentral') {
175+
// We want to do call the publish task only if the version is a release (ie not a snapshot).
176+
// Without this conditional, with a snapshot it would do the publish first, then throw the Exception below.
177+
// But since the dependsOn is within the conditional, the dependency is not created for snapshot versions
178+
// so the publish in that case is never executed.
179+
if (!version.toString().endsWith('SNAPSHOT')) {
180+
dependsOn tasks.publish
181+
}
182+
doFirst {
183+
if (version.toString().endsWith('SNAPSHOT')) {
184+
throw new GradleException("Version \"" + version +
185+
"\" is a snapshot. Cannot publish to Maven Central")
186+
}
187+
}
188+
189+
doLast {
190+
exec {
191+
workingDir project.projectDir
192+
commandLine 'bash', "${project.rootDir}/scripts/publish_to_maven_central.sh", project.name, project.version
193+
}
194+
}
195+
}
196+
197+
tasks.register('publishToSnapshotRepository') {
198+
// We want to do call the publish task only if the version is a snapshot.
199+
// Without this conditional, with a release it would do the publish first, then throw the Exception below.
200+
// But since the dependsOn is within the conditional, the dependency is not created for release versions
201+
// so the publish in that case is never executed.
202+
if (version.toString().endsWith('SNAPSHOT')) {
203+
dependsOn tasks.publish
204+
}
205+
doFirst {
206+
if (!version.toString().endsWith('SNAPSHOT')) {
207+
throw new GradleException("Version \"" + version +
208+
"\" is a NOT a snapshot. Cannot publish to the snapshot repository")
209+
}
210+
}
211+
}
175212
}
176213

214+
177215
}
216+
178217
compileJava {
179218
options.compilerArgs << '-parameters'
180219
}

docs/ACCEPTANCE_TESTS.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -102,7 +102,7 @@ The source id can be used to find all datasets versions of a source on the [Mobi
102102
## What do we do with the results?
103103
We follow this process:
104104

105-
<img src="/docs/Acceptance-test-process.jpg" width="750">
105+
<img src="/docs/images/Acceptance-test-process.jpg" width="750">
106106

107107
## Performance metrics within the acceptance reports
108108

docs/BRANCHING.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@
1616
- A maintenance is created if some bug has to be corrected before the next scheduled release.
1717
- As an example see [#1663](https://github.com/MobilityData/gtfs-validator/pull/1653) in the gtfs-validator repository.
1818

19-
![](BranchingDiagram.png)
19+
![](images/BranchingDiagram.png)
2020

2121
## Feature/fix branches
2222

docs/CONTRIBUTING.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -88,7 +88,7 @@ This process is described in more detail in the GitHub documentation [**Contribu
8888

8989
This image shows the tests implemented in the Pull Request process.
9090

91-
<img src="/docs/Pull-Request-process.jpg" width="700">
91+
<img src="/docs/images/Pull-Request-process.jpg" width="700">
9292

9393
## Pull Request comments and reviews
9494
Reviewing Pull Requests is a great way to get familiar with the code & architecture of this tool, and to make sure a functionality meets your needs. Each Pull Request has to be approved by at least one one core developer, but having community members helping with this process is significant for the MobilityData team. Additionally, having the eyes of people from different expertise and backgrounds on a contribution makes it higher quality (nobody can think of everything!).
@@ -131,7 +131,7 @@ A critical step in troubleshooting is being able to reproduce the problem. Instr
131131

132132
The acceptance test is a key part of the Pull Request process. More information about this test is available in the [ACCEPTANCE_TEST.md](/docs/ACCEPTANCE_TESTS.md) file.
133133

134-
<img src="/docs/Acceptance-test-process.jpg" width="650">
134+
<img src="/docs/images/Acceptance-test-process.jpg" width="650">
135135

136136
**How to run tests locally?**
137137

0 commit comments

Comments
 (0)