Testing Django applications with Bitbucket Pipelines and MySQL

If you’re a Bitbucket user then you probably already know about Bitbucket Pipelines. If you’ve not heard of Bitbucket Pipelines then they are Bitbucket’s continuous delivery mechanism. They allow you to run commands, on a new commit for example, to run tests and then deploy your code to production (if you want).

I’ve been using Jenkins for a while but wanted to try out Pipelines on one of my Django projects. Specifically I wanted to automatically run my test suite when code was committed – I’ll think about using Bitbucket Pipelines for deployment later. Bitbucket Pipelines uses Docker containers to run the tests but has the constraint that you can only use one Docker image and doesn’t currently support things like Docker compose. This makes things hard for testing applications that have a number of dependencies and, in most cases, your applications are going to have dependencies (such as needing a database).

Currently there are two solutions to this:

  1. Use a single Docker image that contains all your dependencies. You can search through the Docker hub for something suitable or create your own.
  2. Set up your pipeline/application to use one or more Bitbucket Pipelines services.

In this blog post I’m going to focus on using the MySQL service with Django in Bitbucket Pipelines. I’m writing this post because I didn’t find the existing documentation to be sufficient to get things going and, in some cases, I think there may be errors or ambiguities that cost me quite a lot of time so hopefully this will save other people time. That said, this is just how I got things working, it may not be the only way and probably isn’t the best way. Some things I don’t intend to cover are configuring Bitbucket Pipelines for your project in Bitbucket, Django installation, running tests (bar including the command to do this).

Basic requirements

For this project I have a number of requirements:-

  • Python 2.7
  • Django (for this example I’m using 1.8 but it doesn’t matter)
  • MySQL
  • pip (to install a number of other dependencies from requirements.txt)

There is some documentation showing how to set up your bitbucket-pipelines.yml and how to add a MySQL service to the build. So, I started off with something like:

image: python:2.7.13
    - step: 
        - mysql
        - pip install -r requirements.txt
        - python manage.py test myapp.tests.unit_tests
        - python manage.py test myapp.tests.integration_tests
      image: mysql 
        MYSQL_DATABASE: test_pipelines 
        MYSQL_USERNAME: test_user 
        MYSQL_PASSWORD: test_user_password

Error 1 – bitbucket-pipelines.yml file must be a map

However, pipelines wouldn’t run and I kept getting the following error:

The 'environment' section in your bitbucket-pipelines.yml file must be a map.

This was caused by the MYSQL_RANDOM_ROOT_PASSWORD line. It turns out there was an error (well two) in the Bitbucket documentation and the relevant section needs to have values quoted and MYSQL_USERNAME should be MYSQL_USER. Bitbucket have now fixed the docs so hopefully no-one else will have these issues. For completeness the section should be:

      image: mysql 
        MYSQL_DATABASE: 'test_pipelines' 
        MYSQL_USER: 'test_user' 
        MYSQL_PASSWORD: 'test_user_password'

Error 2 – django.db.utils.OperationalError: (2002, “Can’t connect to local MySQL server through socket ‘/var/run/mysqld/mysqld.sock’ (2)”)

So, this took a while to sort out and appears to be an issue related to ‘localhost’. The Bitbucket docs say the MySQL database will be available on ‘localhost’ and the Django settings for databases say leave the database setting for HOST as ” for localhost. However, I found that this wouldn’t work and neither would: HOST: ‘localhost’ in my Django settings file. I had to set it as follows:

    'default': {
        'ENGINE': 'django.db.backends.mysql',
        'NAME': 'test_pipelines',
        'USER': 'test_user',
        'PASSWORD': 'test_user_password',
        'HOST': '',
        'PORT': '3306',
        'OPTIONS': {
               "init_command": "SET default_storage_engine=MyISAM",
        'TEST': {
            'NAME': 'test_pipelines',
            'CHARSET': 'UTF8',

It is also important to note that the ‘NAME’ for the database and test database for Django must be the same. When running the pipeline you only get one database and Django will, by default, prefix the default database name with ‘test_’ and that database won’t exist… so you’ll get errors. To get around this, set the test database name explicitly. I set this up in a separate set of settings that I only use for my Bitbucket Pipeline.

Error 3 – Django can’t create or drop the test database

At the start of the test run Django will try and drop and recreate the database but it can’t and the pipeline stops waiting for the answer to:-

Type 'yes' if you would like to try deleting the test database 'test_pipelines', or 'no' to cancel: 

To solve this issue we need to get the tests to use the existing database. To do this we need to use the –keepdb flag when running tests. So, we change our bitbucket-pipelines.yml to use:

- python manage.py test myapp.tests.unit_tests --failfast --keepdb

Error 4 – Django test leakage

I’ve found that Django isn’t always great at keeping tests isolated (this is unrelated to Bitbucket Pipelines). This means that tests that run earlier can make database changes that affect the outcomes of later tests. This seems to be particularly evident when running my integration tests immediately after my unit tests. To avoid this I found I need to run the sets of tests separately and flush the database between the runs. So I added the following to my bitbucket-pipelines.yml:

- python manage.py test myapp.tests.unit_tests --failfast --keepdb
- python manage.py flush --noinput
- python manage.py test myapp.tests.integration_tests --failfast --keepdb

It’s important to note that ‘flush’ doesn’t drop and create the database it only flushes the data tables (it doesn’t flush the Django migrations table). It’s also important to use the ‘–noinput’ flag to prevent the pipeline hanging whilst waiting for you to confirm the flush (which you can’t).

Error 5 – OperationalError: (1366, Incorrect string value…

OK, so we’re getting close now. At this point most of my tests are running but the suite itself is ultimately failing because of an “OperationalError” and reporting an “Incorrect string value”.

The test that is failing is a test to make sure my application works with UTF-8 characters. The tests pass locally so I know there’s nothing wrong with my code and it must be the collation used on the MySQL database. The MySQL database that pipelines creates uses the default collation (latin1_swedish_ci) and so the tests fail when inserting and retrieving UTF-8.

At this point I tried a number of different things to try and configure the collation (through MySQL settings such as “–character-set-server=utf8” and “–collation-server=utf8_unicode_ci”) but none of them seemed to work. As a result I resorted to using the code from this StackOverflow answer to change the collation for the existing database tables before I run my tests. It is worth noting that you need to change the “host” setting to be “” rather than “localhost”. Once that’s done I stored this code in a separate directory (with a few other pipeline specific things such as test settings etc.) and can then run the code to fix the database prior to running my tests using:

- python myapp/pipelines/fix_db.py


After all this I now have a pipeline that’s able to run my Django test suite against a MySQL database using the correct collation. My final ‘bitbucket-pipelines.yml’ file looks something like:

    - step:
        image: python:2.7.13
          - mysql
          - mv pipelines/pipelines_extra_settings.txt private_settings.py
          - pip install -r requirements.txt
          - python myapp/pipelines/fix_db.py
          - python manage.py test myapp.tests.unit_tests --failfast --keepdb
          - python manage.py flush --noinput
          - python manage.py test myapp.tests.integration_tests --failfast --keepdb
      image: mysql
      command: mysqld --character-set-server=utf8 --collation-server=utf8_unicode_ci --default-storage-engine=MyISAM
        MYSQL_DATABASE: 'test_pipelines' 
        MYSQL_USER: 'test_user' 
        MYSQL_PASSWORD: 'test_user_password'

As I said, I hope this helps get you started with Bitbucket pipelines but I’m sure it’s not the only way to resolve the issues I came across and, over time, I’m sure Bitbucket/Atlassian will address some of these issues and I can remove the workarounds.

How to: Solve the “Unable to convert classes into dex format” Unity error – the return….

Back in 2014 I blogged about how to solve the “Unable to convert classes into dex format” Unity error.

I thought I’d post a follow up as I had the same error recently but with a slightly different message and cause. The error was:


java.lang.IllegalArgumentException: already added:

I looked for ExceptionReporter and found it in:


The original error report also said:

Created unique file for imported asset as file exists with different
GUID (Assets/Plugins/Android/MainLibProj/libs/play-games-plugin-support.jar
=> Assets/Plugins/Android/MainLibProj/libs/play-games-plugin-support

When the asset is renamed it causes the problem. However, in the end I found that the problem was caused by these duplicate files:

Assets/Plugins/Android/android-support-v4.jar (ARMv7)
Assets/Plugins/Android/libs/android-support-v4.jar (ARMv7)

So, you can fix the issue by removing one of the files (or choosing a different architecture if that’s appropriate for your project).

How to: Solve the “Unable to convert classes into dex format” Unity error

I recently came across the following error when trying to build my game in Unity “Unable to convert classes into dex format. See the Console for details.”

Unity error dialog stating: Unable to convert classes into dex format. See the Console for details.

The console error is pretty long but starts with:

Error building Player: CommandInvokationFailure: Unable to convert classes into dex format. See the Console for details.
C:\Program Files\Java\jdk1.6.0_38\bin\java.exe -Xmx1024M -Dcom.android.sdkmanager.toolsdir="E:/adt-bundle-windows-x86-20131030/sdk\tools" -Dfile.encoding=UTF8 -jar "E:/unity/Editor/Data/BuildTargetTools/AndroidPlayer\sdktools.jar" -


java.lang.IllegalArgumentException: already added: Lcom/google/android/gms/common/internal/safeparcel/a;
	at com.android.dx.dex.file.ClassDefsSection.add(ClassDefsSection.java:122)
	at com.android.dx.dex.file.DexFile.add(DexFile.java:161)
	at com.android.dx.command.dexer.Main.processClass(Main.java:685)
	at com.android.dx.command.dexer.Main.processFileBytes(Main.java:634)
	at com.android.dx.command.dexer.Main.access$600(Main.java:78)
	at com.android.dx.command.dexer.Main$1.processFileBytes(Main.java:572)
	at com.android.dx.cf.direct.ClassPathOpener.processArchive(ClassPathOpener.java:284)
	at com.android.dx.cf.direct.ClassPathOpener.processOne(ClassPathOpener.java:166)
	at com.android.dx.cf.direct.ClassPathOpener.processDirectory(ClassPathOpener.java:229)
	at com.android.dx.cf.direct.ClassPathOpener.processOne(ClassPathOpener.java:158)
	at com.android.dx.cf.direct.ClassPathOpener.process(ClassPathOpener.java:144)
	at com.android.dx.command.dexer.Main.processOne(Main.java:596)
	at com.android.dx.command.dexer.Main.processAllFiles(Main.java:498)
	at com.android.dx.command.dexer.Main.runMonoDex(Main.java:264)
	at com.android.dx.command.dexer.Main.run(Main.java:230)
	at com.android.dx.command.dexer.Main.main(Main.java:199)
	at com.android.dx.command.Main.main(Main.java:103)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at SDKMain.main(SDKMain.java:129)
1 error; aborting

After a bit of digging I found this: Multiple plugin conflict

The answer does a really good job of explaining the issue, essentially you have two classes named the same. In most cases this is because you’ve more than one copy of the same JAR file in your build. I did a quick search of ‘*.jar’ and looked for duplicates. In my case I had:


Having deleted one copy I could then re-run the build successfully.

How to: Create a static copy of a Gallery 2 site

For many years I maintained an instance of Gallery 2 (http://galleryproject.org/) to share some of our photos with family and friends. However, life and technology moves on and I simply don’t have time to maintain the installation of the Gallery software* and couldn’t justify the time to upgrade it to version 3. Similarly, we don’t tend to use it for our photos any more and instead share things via Flickr, Facebook or just plain email.

That said, I didn’t want to remove the site entirely… just “make it safe” and make it trivial to host/maintain. To that end, and because I had no requirement to keep on updating the site, I looked around for solutions on how to make a static copy of the site (i.e. one that could just be served as static files without needing PHP or a MySQL database etc). The first version of Gallery had a static mode that you could put the site into and then crawl to create a static copy. Version 2 doesn’t have that functionality.

In the end I settled on crawling the site with ‘wget’ and then using a series of Perl one liners to tidy up the resulting HTML. This results in a reasonable copy of the site in static form. The only downside is that the image sizes are lost so the pages just show the images at the default size – no great loss though as we have the originals anyway.

In case anyone is faced with a similar task, here are the commands and the bash script used to clean up the pages.

wget -m -R "*g2_v*,*g2_h*,*g2_c*,*g2_i*" http://photos.example.com/

-m = mirror the site
-R “reject,list” = list of file patterns to reject

The above command can take many hours to run, on my site the spidering took seven hours: Downloaded: 30453 files, 526M in 27m 7s (331 KB/s)

Once the spidering is done you have copies of the pages and the images etc but the pages still have references to PHP files and so on… so you need to do a bit of cleaning up.

To do this I created a shell script which I have included below.

Edit 13/02/2020 – the original shell script was created and run on Ubuntu Linux back in 2014. It uses “rename” but Daniel M. Drucker kindly got in contact with me to point out that the behaviour of this command is different on different Linux distros and macOS and may not work. I’ve left the original command in the script below but if it doesn’t work then you may be able to replace that line (4) with this one. However, I no longer have a gallery2 site to test this on so use at your own risk:

find . -name '*g2_*' | while read file; do newfile=`echo $file | awk -F [=.] '{print "." $2 "_page_" $4 ".html" }'`; mv $file $newfile; done;

Original shell script:

# Tidy up offline gallery

# Rename files
find . -name '*g2_*' -exec rename s'/index\.html\?g2_page=(\d+)/index_page_$1.html/' {} \;

# Fix links to the renamed files above
perl -0777 -spi -e 's!\?g2_page=(\d+)!index_page_$1.html!gs' `find . -name \*.html`

# Remove various links to scripts and functionality we don't have now
perl -0777 -spi -e 's!<script.*?</script>!!gs' `find . -name \*.html`
perl -0777 -spi -e 's!<a href=\"/main\.php.*?</a>!!s' `find . -name \*.html`
perl -0777 -spi -e 's!<form\s+id=\"search_SearchBlock\".*?</form>!!gs' `find .  -name \*.html`
perl -0777 -spi -e 's!<div id=\"gsFooter\">!<div id=\"gsFooter\">Was once powered by:<br />!gs' `find . -name \*.html`
perl -0777 -spi -e 's!<div class=\"block-core-PhotoSizes.*?</div>!!s' `find .  -name \*.html`

# Fix first breadcrumb link
# CHANGEME = Update to the name of your site
perl -0777 -spi -e 's!<a href=\"/main\.php.*? class=\"BreadCrumb-1\".*?</a>!<a href="/" class="BreadCrumb-1">Someone'\''s photos</a>!gs' `find . -name \*.html`

# Fix image links
perl -0777 -spi -e 's!<a href=\".*?g2_imageViewsIndex=[\d+]\">[\s*]<img(.*?)/>[\s*]</a>!<img$1/>!g' `find . -name \*.html`

# Remove any remaining links to main.php
perl -0777 -spi -e 's!<a href="\&quot;/main\.php.*?</a" data-mce-href="\&quot;/main\.php.*?</a">!!gs' `find . -name \*.html`</a>

Use the above at your own risk… 😉

* It turns out that the core Gallery team are also taking a break from the software too and it’s “in hibernation”