My favorite continuous integration providers (May 2018)
Seeing the following tweet yesterday, which links to a great article comparing multiple CI/CD providers
made me realize that I did indeed go through most of them at some point in time. So very shortly, here are my two cents:
I agree with the article, it's great, I use it to do CI/CD for this very website. Nothing much to say. The build UI is not great on mobile though. Cannot do iOS/Android builds though?
Still use it, gets the job done. Nothing more to add. It's running the danger-todoist builds.
I found the configuration weird in the beginning, but you can get used to it. Major plus: Can do both my iOS and Android builds for PitBuddy App for private repositories.
I think Nevercode aquired Buddybuild at some point. I was using Buddybuild for iOS builds and when I started to get an Android build going it just was impossible to setup and massively unreliably. Can't say much about Nevercode, but I am happy I left Buddybuild for Bitrise.
Obviously I just recently started using it, so far so good. Can't do iOS/Android. Build speeds are great up until now.
Is this the solution to doing both web, iOS and Android in one tool? Hosting the code there as well? Seems tempting! I like Gitlab the SCM tool, and Gitlab CI has come a long way.
That's it, just a quick reminder on what I already tried and would like to try.
FTP lives: Deploying a static site with Bitbucket pipelines
The code for the site itself is versioned in a private repo on bitbucket.org, and bitbucket does come with their version of CI/CD tooling called "pipelines". I wanted to use this since setting up yet another external CI integration like travis or circleci felt a bit overkill for this. Granted, the site did not have any tests/verification at all so far.
BackstopJS once more
Having no tests or verification mechanism at all is not a great prerequisite for a continuous delivery toolchain, even for a static site. This would mean broken versions of the site would be deployed without warning. So once more backstopjs to the rescue. Setting up backstopjs was again without hassle, it can resolve
file:// urls no problem. So a set of reference images were quickly created and I could look at reproducing those images in the pipeline.
Bitbucket pipelines are configured through a
bitbucket-pipelines.yml file. No surprises here, thats in essence how most of the CI providers do it. The two cornerstones of the pipeline are two docker images. One to run the visual regression tests, the other to do the deployment. That's a "test->deploy" pipeline in a nutshell. This is the configuration in it's entirety:
The test step, run directly on the official backstopjs docker container, is simply executing the
test task of backstopjs. This compares three reference images (three because I set up three viewport sizes once more) against what is currently rendered when opening the
index.html. When they are identical, the deploy pipeline step is triggered. Deploy is, as mentioned, as simple as copying the relevant files to the correct directory on the webspace. Already a while ago I found a git plugin to accomplish this. It is called git-ftp and solves my issue very smoothly. As for backstop, a docker image already exists just to run
git ftp. So all I needed to do (and I have struggled to get this working in the past) was to create a separate ftp user (for security reasons), add the
FTP_URL environment variables to the bitbucket pipelines settings and voilá, continuous deployment for a static site with the safety net of visual regression tests it is 🎉.
git ftp can use a separate ignore file to tune what content to deploy or omit.
- There is a docker container for almost anything
- Does this approach scale? Probably not. Do I care? No.
- I need to run backstopjs using the docker container locally as well, since there are text rendering (aliasing) differences on macOs vs. dockerized ubuntu.
Shipping code from the iPhone
After traveling abroad for more than two weeks and then spending two days laying by the pool I naturally got a bit of a tech craving. To spend the afternoon I set myself the challenge of fixing a small bug on this blog, having nothing more than some Hotel WiFi™ and my iPhone at hand. But that must be enough, right? Let‘s see how it went.
In this blog I display a permalink hash next to the blog posts title, which is a link to view the post separately as well. When using e.g. Safari‘s „Reader“ mode, this hash symbol is of no use and should not be there.
So how can we get started on this? First we need access to the repo of course. I had come across an app called Working Copy, and looking that up in the App Store was promising: I can check out the repo to my iPhone. What a time to be alive!
Not only can I check out the source code, but it comes with text editing capabilities as well. Making the change was pretty easy it seemed. I first browsed for the test that verifies the permalink, and removed the expected hash symbol from there.
Which lead to the first realization: I cannot run the tests on my iPhone. While that definitely kills the quick feedback loop I so much long for, it is fine for very simple changes (because CI will save me) and the occasional holiday-deploy. Alas I made the change, committed and pushed, and opened the PR. Opening the PR can be done on github.com, however strangely only via the desktop version. This triggers the continuous integration build on circleci.com, and duly notified me that I forgot to make the production code change to go with the test change! Oh well, another commit and push and I was looking good, build passing. At least concerning the crystal code. But what about the CSS? For now I had only removed the hash symbol, but we still want it to be there through CSS. So that‘s limitation two: I have no way to play around with the CSS on the blog. Firstly because of the lack of an actual dev environment on the phone, but also because I don’t yet run visual regression testing on CI. My way out of this was to fire up codepen.io and use that as a playground. Again, because the change was pretty trivial and I just needed to confirm one assumption I had. That proved true and with it I was ready to merge the PR, using GitHub's website. As continuous deployment was already set up as well this was actually all there was to it. The build job passed as expected, sending off the deploy job straight after it and after 5 minutes or so I could visit the live page and confirm that indeed everything still looks the same, but the hash symbol is not showing up in Reader mode. Cool ✅
Switching between GitHub's mobile and desktop version was kind of clunky and still didn't allow me to upload any screenshots. I then realized that my twitter feed had mentioned Ryan Nystroms GitHawk in the past. One more trip to the App Store and yes yes yes, that is much more pleasant than the website!
Editing issues / PRs is way more comfortable and image upload works as intended. That will be a much better combo I think, I am looking forward to using that for the next exercise.
- Making trivial code changes directly from within @WorkingCopyApp
- Pushing that change to GitHub
- Opening a pull request via the desktop version of github.com
- Updating issues/PRs in @githawk
What didn‘t work
- Building/pushing a new docker image. I could have tried to install via the Dockerfile, but without building and pushing the new image this still gets me nowhere
- Uploading screenshots to issues on the github.com website
- Quickly seeing the effects of both crystal code as well as CSS changes. Waiting on CI works for the code, CSS is still push and pray (working on it)
- Publishing a blog post about this journey. I can only create new posts locally and sync the dev database to production from my development environment. I guess that needs to change!? (Might be the excuse to build a json api powered endpoint and an iPhone app to go with it 😎)
Of course this exercise was purely recreational. I was in no way forced to write or deploy code from halfway across the world on my iPhone, but I kind of wanted to.
One shortcoming of this permanently-work-in-progress blog of mine was the rendering on mobile devices. The experience of browsing the blog on a phone or tablet was less than ideal: text touching the borders of the screen, images overflowing the main section, social media links being out of place, and many more. It was a long standing issue. So I set off on fixing this. While digging straight into the first CSS changes and fiddling in the developer console of Chrome, I remembered what I wrote in the Snapshot TDD post:
Things of visual nature are not unit-tested easily, which is why they are often simply untested. We usually don't test stylesheets, colors, images etc. However we can't say those things are unimportant.
Unhappy about the workflow I just started and taking into account the above thought, I typed some words into google and emerged with this awesome tool: BackstopJS. It's headlined with "Visual regression testing for web apps" and was exactly what I was looking for. It provides a safety net for changing the visual nature of something rendered in a browser by taking and comparing screenshots. Basically exactly what I manually and crudely setup for the e-ink dashboard, just so much more awesome.
Setup and usage
Both setting up and using BackstopJS is dead simple and can be explained in this short snippet:
You will get a really nice page telling you that the tests were failing. Why? Because you have not approved any reference images yet. Once you look at those images and assert that this is currently the way things are looking, go ahead and
backstop approve those. Now you are golden and able to make changes to your pages CSS without having to fear to a) break thinks if you are refactoring, or b) having a super quick way to get an overview of the changes you made across multiple pages of your blog in different viewports.
With my page being a blog, the content obviously will continuously change, making the comparison of the current look against reference images unfeasable. Luckily this is an easy fix: I created a new database containing exactly one sample post, which uses all the usual tags like h1-h6, lists, blockquotes to showcase most of the styles coming into play.
This is how the current reference image looks in tablet mode:
As always, you can follow along to the PR on github. One example of a change to the reference image can be seen here. I must say I am mega impressed with BackstopJS and will try to use that again in the future. One open task is to get that running on CircleCI as well.
One of my recent weekend side projects, an e-ink / raspberrypi driven build status dashboard, was a great playground for doing TDD powered by visual snapshots. But let's rewind a bit.
What I actually wanted to achieve was the following: Build a semi-decent python class to draw a dashboard type interface, which I can feed to my e-ink display. I had already prototyped such a script, but it was a "make it work in the quickest possible way in 1 hour" mess. Nothing I wanted to maintain or even look at for even five more minutes. I also didn't want to start completely from scratch regarding the output, because I was happy enough with the result this script produced, which is shown here:
So how could I develop the code from scratch, while making sure I got the exact same output in the end? Right, creating myself a feedback loop that will quickly compare the reference image to the current output. To quote from jest:
Snapshot tests are a very useful tool whenever you want to make sure your UI does not change unexpectedly.
A typical snapshot test case for a mobile app renders a UI component, takes a screenshot, then compares it to a reference image stored alongside the test.
This is powerful, because how else would I test this? Things of visual nature are not unit-tested easily, which is why they are often simply untested. We usually don't test stylesheets, colors, images etc. However we can't say those things are unimportant. So I set out to do TDD with snapshots and iterate myself toward the reference result.
Lot of work left to do, sure, but that set myself up for about a two second feedback cycle. The process, which I packaged into a simple
npm test bound to
<leader>t in VIM so I can invoke it in one keystroke, is this:
- Run unit tests in python (This is just one dumb test for the constructor, I should remove it)
- Render current image to
actual.png in an "integration" test
- Create image diff with pixelmatch
- Open this diff in Preview so it jumps into my face
See the process encoded here, and yes, the irony of having a node based test invocation for a python script is not lost on me. Computers 🤷🏼♂️
Let's walk through one of my commits together. I really enjoyed working like this. A few minutes in, I had the rendering of the header, header title, project text to the left all fleshed out with minimal differences to the reference. I assume something regarding the font-rendering on the raspberrypi/debian vs. my mac is to be blamed for the tiny deviations around the text. No clue though. So here I was:
Lets add some code to render the badge text on the right:
<leader>t, and see this:
So obviously I got the alignment wrong. Lets fix it:
Re-run the tests, see this:
Less red! That's basically what I did over and over again. Feel free to have a look at the commits for more examples.
Next page »
- Fast feedback is gold
- Even visual feedback is good
- I wouldn't have wanted to unit test this, so quick visual feedback is way better than no feedback. Let's remember that coding this up, syncing the code to the rasbperrypi, actually running the code and see the output on the e-ink display is a multi-minute process!
- Keep snapshot tests in your toolbelt, there is a place and time for them, and it's not only react!
- Updating the reference image is displayed really nicely in github