As some of you will have noticed, my server galileo.mailstation.de has had some performance issues lately. Testing patches with Jenkins could take a long time, building stages was taking up to 20 hours (!) instead of the normal two and Gerrit itself could be slow at times as well.
This was due to aging hardware (that server was assembled in 2010), one horribly performing hard disk and constant over-heating of the machine. I’ve changed a few things over the last few days and performance is much better now but still not to my satisfaction.
Today, I’ve finally had enough of squeezing performance out of rotting hardware and simply ordered a new server. Functionally, it will be identical to good old galileo but on beefier hardware, especially with much more RAM (48 GB instead of 12 GB) which is essential for my uses.
While I’m writing this, I’m copying over the entire galileo to the new server which will hopefully be finished in a few hours from now. Once that’s done, I’m going to
- shut down all non-essential services on galileo (that includes Gerrit, Jenkins, the bots and lots of other stuff),
- re-sync the data to the new server,
- change the DNS records,
- fire up all current services on the new server,
(N. B.: Should you think I’ve forgotten something important here, please comment while you still can. 🙂 )
This update will, thus, cause some downtime (there was some earlier today already when I was preparing the migration). So don’t be alarmed if all services (including my IRC bouncer 🙂 ) are gone for a while – we’ll be back!
Ideally, you’ll only even read this post when all is over. 🙂
Over the last few weeks, I’ve made a few changes to Exherbo’s Gerrit and Jenkins installation:
- Build artifacts are stored in Jenkins:
If a build is unsuccessful, the Paludis build log, config.log (if it exists) and the cave-resolve command used are stored. Since build artifacts are associated with a project (i. e. the repository) and not individual builds, the files have the build number prepended in their file name:
If a build is successful, only the cave-resolve command and the detected dependencies are archived:
This change is effective immediately for all exheres repositories in Jenkins. If you have further suggestions of files to archive, please let me know.
- Due to the recent OpenSSL Heartbleed bug issue, I’ve re-issued the SSL certificates on my machines on April, 11th 2014. If you have a password login on galileo, now is the time to change it. For galileo.mailstation.de, the host key fingerprints are as follows:
1024 a8:8c:7d:87:3d:a6:61:78:8d:59:f5:52:d7:b1:3c:34 (DSA)
256 f1:08:b0:12:66:1c:72:af:2d:d8:c7:15:ca:13:f6:f2 (ECDSA)
256 68:1c:27:e2:8e:b5:42:62:98:8a:f3:04:45:c1:60:f9 (ED25519)
2048 37:38:6e:81:4a:53:24:4d:b0:fb:c7:3e:f0:1d:63:8c (RSA1)
2048 06:57:ff:b7:e6:6f:c8:31:76:c8:9a:7c:37:d7:f5:47 (RSA)
The web server’s certifacte has the following fingerprints:
- SHA-256 Fingerprint:
85 3D 47 8B A5 44 76 A0 09 96 9F 15 4A 9A F8 C5 1C 95 D1 1D 92 4E 78 06 0D F6 E5 7B 1B C5 B2 07
- SHA-1 Fingerprint:
0D E8 77 4F 74 67 AB 61 12 97 A5 57 84 1F B4 51 C1 1D D4 19
- Gerrit was updated to version 2.8.4. There are quite a few bug fixes in the release notes but probably most important for those of you who experienced issues with the side-by-side diff view in the new change screen are the fixes they applied to it.
There’s more to come but that’s going into another post. 🙂
A recent request concerned the possibility to re-test patches patches in Gerrit with Jenkins. I’ve now added a very simple switch for that:
In any given given change (but your own!), you can add a comment with “+1 Retest” set as a “verdict”. On the old change screen (really, change to the new one, it’s much nicer), it looks like this:
On the new change screen, it looks like this:
It’s really, truly easy. You can even do it multiple time – just uncheck the “Retest” checkbox (new screen) or set “0 Retest” (old screen) first and then set it again.
This works for any change as long as you are a registered user – any change but your own. Unfortunately, the Jenkins’ Gerrit-Trigger plugin simply ignores the “CommentAdded” event completely for ones own changes. If you want those re-tested, just amend your commit (no need to actually change anything; the commit timestamp will be updated) and Jenkins will see the patch’s new revision and re-run the test.
On a side note, I’d like to apologise for the instability for about 1,5 days earlier this week. There were some really nasty hardware/software issue that I had to sort out. This is done now, though, and things are running smoothly again.
For quite some time now, I’ve been working on the automatic creation of Exherbo stages for AMD64 (x86_64) and X86 using Jenkins.
Till now, Exherbo’s stages were created manually by one of us Exherbo core developers and usually basically “on-demand” or when they were major changes. This finally changed yesterday.
Using the jobs stage_amd64 and stage_x86 respectively, the stages…
- are built using a modified stage building script – each stage is build from scratch. The latest stage gets unpacked and used to build a new one. I call this: Implicit pro-active quality assurance. No pbins are used in the entire process. Everything is fresh and new, just for you.
- the sources are kept right next to the stages at https://galileo.mailstation.de/stages/sources. You can see right there what sources were used.
- the XZ archive is added to the job (the most recent ten stage archives are kept on their respective job). Like this, you can always check the exact build log for the stage you want and rest assured both are right there at your fingertips.
- next, the successfully built stage is deployed to https://galileo.mailstation.de/stages/. exherbo.org is undergoing maintenance but you really want your Exherbo? Download a stage from this alternative location, sync your repositories from Gerrit and you’re unstoppable!
- and finally automatically transferred to our traditional http://dev.exherbo.org/stages/ where some “extra magic” is performed by Jenkins that creates the “current” symlinks, updates the checksum files, etc.
And this happens every single day. The entire process is completely scalable as well – once the cross stuff is production-ready, I’m going to build ARM stages as well and it will be done by simply copying a Jenkins job and changing a few lines. A milestone for Exherbo.
We just made major changes? Wait a few hours, get them in your stage for your new Exherbo installation.
You somehow got a broken stage? Use an older one, compare them, go nuts with them. There are always at least 10 stages per architecture to choose from.
And if a stage fails to build? Well, the next day it will be fixed because we’re watching over the entire process.
Isn’t this incredibly awesome?
And you can even play a round of Bullshit Bingo using this very post! 😉
As of yesterday, I’ve drastically increased the number of testing machines in Jenkins (from two to 21!). This enables me to poll for changes in git every five minutes and automatically start a testing run for every single commit – regardless of it entering via Gerrit or not. (Note: This is in addition to the Gerrit trigger I’ve been using from day one on, not replacing it.)
Here’s an example from today:
Apart from looking at Jenkins directly, you can join the #exherbo-bots channel where the jenkins-exherbo bot announces all test results. Here’s an example for that:
[03.01.2014 13:42:59] <irker657> Exherbo: pipping arbor:master * 76ec9309669d / packages/sys-apps/grep/grep-2.15.exheres-0 packages/sys-apps/grep/grep-2.16.exheres-0: grep: Bump to 2.16 http://tinyurl.com/pyfhbql
[03.01.2014 13:46:50] <jenkins-exherbo> Project arbor build #67:SUCCESS in 1 min 5 sec: https://galileo.mailstation.de/jenkins/job/arbor/67/
I hope you’ll find this useful and will actually look at the results since it will find out if you missed a dependency. And it potentially shows you automagic dependencies.
Here are a few things I’ve updated on Gerrit, Jenkins and associated tools:
- The Gerrit bot, gerritwk23, has now support for !pl (and pl in a query). It’s just like zebrapig’s !pl.
- I’ve updated Gerrit in mid-Decembre. Since then, you can switch to a newly designed “Change View” which you’ll find under your account’s settings in the “Change View” option. There’s also an associated “Diff View” option to set your preferred diff style if you switch to the new Change View.
- In its build logs, you’ll now find the mounts, the cave resolve command and the (likely) dependencies gathered from the installed package. Dependency information might not be 100% accurate so please take it with a grain of salt but it’s usually a good indicator.
Look out for “**************************************************************” to find that information.
If you have any ideas about what else to improve, please let me know.
If it’s not yet good enough for you that using Gerrit
- is even easier and faster than using zebrapig (you just push your change and you’re done! And if you use git review it’s getting even easier.),
- the review turn-around times are often better when using Gerrit than zebrapig,
- discussing patches can be done both when you’re on IRC and on Gerrit,
then here’s another good reason to use it: I’ve set up Jenkins for all repositories on Gerrit to build every change in a chroot (including sydbox-1, pinktrace-1 and docs).
For now, all reports only go to me (and anyone else who wants to get an email with the result included) and Jenkins will not report back to Gerrit because the setup is quite fragile yet and I want to be sure it works properly before making it do more.
No idea how to use Gerrit? Just read my earlier posts Gerrit Code Review for Exherbo and Gerrit for Core-Devs. Of course, you can ask me, too. 🙂
Exherbo’s Gerrit has left its infancy and, thus, I can now add 3rd party repositories to our Gerrit installation. Please just let me know if you’d like that.
- Your repository must use git.
- You must have an account in Gerrit.
- You must be willing to add a Gerrit user (github: Gerrit-Exherbo; BitBucket: gerrit-exherbo) to your repository as a contributor. Here’s the public key. (You don’t need the public key for github or BitBucket; if you’re unsure, you most likely don’t need it.)
- Your repository should be hosted on one of the big git hosting providers, e. g. github (preferred) or BitBucket. Let me know if your repository is hosted elsewhere.
- Please let me know the push URI for your repository.
Once I’ve set you up, you can immediately work on Gerrit patches for your repository. You are, of course, its sole owner and fully responsible for it.
By default, Exherbo’s core devs (but nobody else unless you explicitly ask me to add someone) will be able to +2 Gerrit changes for your repository, too. This is meant to potentially help you keep your repository up to date (e. g. when mass changes occur). This is entirely optional, though, and you can choose to opt-out at any time if you don’t like it; you only need to let me know.
I’ve just updated the Gerrit installation to the 2.7 release and fixed the pesky “internal server error” that occurred when entering a user name or similar.
I’ve also updated my posts on Gerrit and the FAQ-like post for core devs:
Gerrit Code Review for Exherbo
Gerrit for Core-Devs
After having worked with Samba for almost 20 years, I'm giving up now.
I have submitted patches upstream when Samba (and I :) ) was young,
I've proof-read books on Samba and NTLM but where they're going, I
can't follow them any more.
- Samba 4.1.0 doesn't even build for me any more and I'm too
frustrated to figure it out.
- In Samba 4.1.0, SWAT (the configuration web interface) seems to have
been removed. I might be missing something but if I'm not, that sucks.
- Samba's "new" crappy build system WAF is just horrible and I simply
don't understand it. I asked for help (and it was offered) but nothing
was ever done.
- New versions of Samba 4 try to build against already installed
versions in the live filesystem and I don't understand WAF enough to
I've just removed myself from BUGS_TO. If you want it, take it but if
you do, do me a favour and do a better job than me and make sure
you're weird enough to understand (and ideally like) WAF.