How to

If you’re a Bitbucket user then you probably already know about Bitbucket Pipelines. If you’ve not heard of Bitbucket Pipelines then they are Bitbucket’s continuous delivery mechanism. They allow you to run commands, on a new commit for example, to run tests and then deploy your code to production (if you want).

I’ve been using Jenkins for a while but wanted to try out Pipelines on one of my Django projects. Specifically I wanted to automatically run my test suite when code was committed – I’ll think about using Bitbucket Pipelines for deployment later. Bitbucket Pipelines uses Docker containers to run the tests but has the constraint that you can only use one Docker image and doesn’t currently support things like Docker compose. This makes things hard for testing applications that have a number of dependencies and, in most cases, your applications are going to have dependencies (such as needing a database).

Currently there are two solutions to this:

  1. Use a single Docker image that contains all your dependencies. You can search through the Docker hub for something suitable or create your own.
  2. Set up your pipeline/application to use one or more Bitbucket Pipelines services.

In this blog post I’m going to focus on using the MySQL service with Django in Bitbucket Pipelines. I’m writing this post because I didn’t find the existing documentation to be sufficient to get things going and, in some cases, I think there may be errors or ambiguities that cost me quite a lot of time so hopefully this will save other people time. That said, this is just how I got things working, it may not be the only way and probably isn’t the best way. Some things I don’t intend to cover are configuring Bitbucket Pipelines for your project in Bitbucket, Django installation, running tests (bar including the command to do this).

Basic requirements

For this project I have a number of requirements:-

  • Python 2.7
  • Django (for this example I’m using 1.8 but it doesn’t matter)
  • MySQL
  • pip (to install a number of other dependencies from requirements.txt)

There is some documentation showing how to set up your bitbucket-pipelines.yml and how to add a MySQL service to the build. So, I started off with something like:

Error 1 – bitbucket-pipelines.yml file must be a map

However, pipelines wouldn’t run and I kept getting the following error:

This was caused by the MYSQL_RANDOM_ROOT_PASSWORD line. It turns out there was an error (well two) in the Bitbucket documentation and the relevant section needs to have values quoted and MYSQL_USERNAME should be MYSQL_USER. Bitbucket have now fixed the docs so hopefully no-one else will have these issues. For completeness the section should be:

Error 2 – django.db.utils.OperationalError: (2002, “Can’t connect to local MySQL server through socket ‘/var/run/mysqld/mysqld.sock’ (2)”)

So, this took a while to sort out and appears to be an issue related to ‘localhost’. The Bitbucket docs say the MySQL database will be available on ‘localhost’ and the Django settings for databases say leave the database setting for HOST as ” for localhost. However, I found that this wouldn’t work and neither would: HOST: ‘localhost’ in my Django settings file. I had to set it as follows:

It is also important to note that the ‘NAME’ for the database and test database for Django must be the same. When running the pipeline you only get one database and Django will, by default, prefix the default database name with ‘test_’ and that database won’t exist… so you’ll get errors. To get around this, set the test database name explicitly. I set this up in a separate set of settings that I only use for my Bitbucket Pipeline.

Error 3 – Django can’t create or drop the test database

At the start of the test run Django will try and drop and recreate the database but it can’t and the pipeline stops waiting for the answer to:-

To solve this issue we need to get the tests to use the existing database. To do this we need to use the –keepdb flag when running tests. So, we change our bitbucket-pipelines.yml to use:

Error 4 – Django test leakage

I’ve found that Django isn’t always great at keeping tests isolated (this is unrelated to Bitbucket Pipelines). This means that tests that run earlier can make database changes that affect the outcomes of later tests. This seems to be particularly evident when running my integration tests immediately after my unit tests. To avoid this I found I need to run the sets of tests separately and flush the database between the runs. So I added the following to my bitbucket-pipelines.yml:

It’s important to note that ‘flush’ doesn’t drop and create the database it only flushes the data tables (it doesn’t flush the Django migrations table). It’s also important to use the ‘–noinput’ flag to prevent the pipeline hanging whilst waiting for you to confirm the flush (which you can’t).

Error 5 – OperationalError: (1366, Incorrect string value…

OK, so we’re getting close now. At this point most of my tests are running but the suite itself is ultimately failing because of an “OperationalError” and reporting an “Incorrect string value”.

The test that is failing is a test to make sure my application works with UTF-8 characters. The tests pass locally so I know there’s nothing wrong with my code and it must be the collation used on the MySQL database. The MySQL database that pipelines creates uses the default collation (latin1_swedish_ci) and so the tests fail when inserting and retrieving UTF-8.

At this point I tried a number of different things to try and configure the collation (through MySQL settings such as “–character-set-server=utf8” and “–collation-server=utf8_unicode_ci”) but none of them seemed to work. As a result I resorted to using the code from this StackOverflow answer to change the collation for the existing database tables before I run my tests. It is worth noting that you need to change the “host” setting to be “127.0.0.1” rather than “localhost”. Once that’s done I stored this code in a separate directory (with a few other pipeline specific things such as test settings etc.) and can then run the code to fix the database prior to running my tests using:

Conclusion

After all this I now have a pipeline that’s able to run my Django test suite against a MySQL database using the correct collation. My final ‘bitbucket-pipelines.yml’ file looks something like:

As I said, I hope this helps get you started with Bitbucket pipelines but I’m sure it’s not the only way to resolve the issues I came across and, over time, I’m sure Bitbucket/Atlassian will address some of these issues and I can remove the workarounds.

Back in 2014 I blogged about how to solve the “Unable to convert classes into dex format” Unity error.

I thought I’d post a follow up as I had the same error recently but with a slightly different message and cause. The error was:

I looked for ExceptionReporter and found it in:

The original error report also said:

When the asset is renamed it causes the problem. However, in the end I found that the problem was caused by these duplicate files:

So, you can fix the issue by removing one of the files (or choosing a different architecture if that’s appropriate for your project).

I recently came across the following error when trying to build my game in Unity “Unable to convert classes into dex format. See the Console for details.”

Unity error dialog stating: Unable to convert classes into dex format. See the Console for details.

The console error is pretty long but starts with:

After a bit of digging I found this: Multiple plugin conflict

The answer does a really good job of explaining the issue, essentially you have two classes named the same. In most cases this is because you’ve more than one copy of the same JAR file in your build. I did a quick search of ‘*.jar’ and looked for duplicates. In my case I had:

Having deleted one copy I could then re-run the build successfully.

For many years I maintained an instance of Gallery 2 (http://galleryproject.org/) to share some of our photos with family and friends. However, life and technology moves on and I simply don’t have time to maintain the installation of the Gallery software* and couldn’t justify the time to upgrade it to version 3. Similarly, we don’t tend to use it for our photos any more and instead share things via Flickr, Facebook or just plain email.

That said, I didn’t want to remove the site entirely… just “make it safe” and make it trivial to host/maintain. To that end, and because I had no requirement to keep on updating the site, I looked around for solutions on how to make a static copy of the site (i.e. one that could just be served as static files without needing PHP or a MySQL database etc). The first version of Gallery had a static mode that you could put the site into and then crawl to create a static copy. Version 2 doesn’t have that functionality.

In the end I settled on crawling the site with ‘wget’ and then using a series of Perl one liners to tidy up the resulting HTML. This results in a reasonable copy of the site in static form. The only downside is that the image sizes are lost so the pages just show the images at the default size – no great loss though as we have the originals anyway.

In case anyone is faced with a similar task, here are the commands and the bash script used to clean up the pages.

-m = mirror the site
-R “reject,list” = list of file patterns to reject

The above command can take many hours to run, on my site the spidering took seven hours: Downloaded: 30453 files, 526M in 27m 7s (331 KB/s)

Once the spidering is done you have copies of the pages and the images etc but the pages still have references to PHP files and so on… so you need to do a bit of cleaning up.

To do this I created the following shell script:

Use the above at your own risk… 😉

* It turns out that the core Gallery team are also taking a break from the software too and it’s “in hibernation”