BitBucket Pipelines and Docker for PHP Developers

I’ve been meaning to look into Docker for a long while now.  But, as always, time is the issue.  In the last couple of days though I’ve been integrating BitBucket Pipelines into our workflow.  BitBucket Pipelines is a continuous integration solution, which runs your project tests in a Docker container.  So, naturally, I had to get a better idea of how the whole thing works.

Docker for PHP Developers” article was super useful.  Even though it wasn’t immediately applicable to BitBucket Pipelines, as they don’t currently support multiple containers – everything has to run within a single container.

The default BitBucket Pipelines configuration suggests the phpunit/phpunit image.  If you want to run PHPUnit tests only, that works fine.  But if you want to have a full blown Nginx and MySQL setup for extra bits (UI tests, integration tests, etc), then you might find smartapps/bitbucket-pipelines-php-mysql image much more useful.  Here’s the full bitbucket-pipelines.yml file that I’ve ended up with.

MySQL, PHP and “Integrity constraint violation: 1062 Duplicate entry”

Anna Filina blogs about an interesting problem she encountered with when working on a PHP and MySQL project:

MySQL was complaining about “Integrity constraint violation: 1062 Duplicate entry”. I had all the necessary safeguards in my code to prevent duplicates in tha column.

I gave up on logic and simply dumped the contents of the problematic column for every record. I found that there was a record with and without an accent on one of the characters. PHP saw each as a unique value, but MySQL did not make a distinction, which is why it complained about a duplicate value. It’s a good thing too, because based on my goal, these should have been treated as duplicates.

She also mentions two possible solutions to the problem:

My solution was to substitute accented characters before filtering duplicates in the code. This way, similar records were rejected before they were sent to the database.

and

As pointed out in the comments, a more robust and versatile solution would be to check the collation on the column.

I’m sure this will come in handy one day.

WWW SQL Designer

www sql designer

I came across the WWW SQL Designer today, and I have only one thing to say…

Holy Molly!  I’ve been looking for a tool like this for a long long time!  It is a web-based database designer, which can export designs into MySQL.  It’s super easy to use and it does exactly what it is supposed to.  No non-sense.  Simply amazing!

Cipherli.st – strong ciphers for Apache, Nginx and Lighttpd

Cipherli.st – provides ready to use cipher configurations for a variety of applications, such as Apache, Nginx, Lighttpd, HAProxy, Exim, Postfix, Dovecot, OpenSSH, and others.  This is a huge time-saver for those of us not well versed in cryptography and security.

Don’t forget to use Qyalis SSL Labs SSL Server Test tool for the complete analysis of where you went wrong.

WordPress Benchmark of MySQL server on Amazon EC2

I have a friend who is a newcomer to the world of WordPress.  Until recently, he was mostly working with custom-built systems and a PostgreSQL database engine, so there are many topics to cover.

One of the topics that came up today was the performance of the database engine.  A quick Google search brought up the Benchmark plugin, which we used to compare results from several servers.  (NOTE: you’ll need php-bcmath installed on your server for this plugin to work.)

My friend’s test server showed a rather poor 48 requests / second result.  And that’s on an Intel Core2 Duo E4500 machine with 4 GB of RAM and 160 GB 7200 RPM SATA HDD, running Ubuntu 12.04 x86-64.

So, I tried it on my setup.  My setup is all on Amazon EC2, using the smallest possible t2.micro servers (that’s Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz, with 1 GB of RAM and god knows what kind of hard disk, running Amazon AMI).

First, I ran the benchmark on the test server, which hosts about 20 sites with low traffic (I didn’t want to bring up a separate instance for just a single benchmark run).  MySQL runs on the same instance as the web server.  And here are the results:

Your System Industry Average
CPU Speed: 38,825 BogoWips 24,896 BogoWips
Network Transfer Speed: 97.81 Mbps 11.11 Mbps
Database Queries per Second: 425 Queries/Sec 1,279 Queries/Sec

Secondly, I ran the benchmark on one of the live servers, which also hosts about 20 sites with low traffic. Here though, Nginx web server runs on one instance and the MySQL database on another. Here are the results:

Your System Industry Average
CPU Speed: 37,712 BogoWips 24,901 BogoWips
Network Transfer Speed: 133.91 Mbps 11.15 Mbps
Database Queries per Second: 1,338 Queries/Sec 1,279 Queries/Sec

In both cases, MySQL is v5.5.42, running on the /usr/share/doc/mysql55-server-5.5.42/my-huge.cnf configuration file. (I find it ironically pleasing that the tiniest of Amazon EC2 servers fits perfectly for the huge configuration shipped with documentation.)

The benchmark plugin explains how the numbers are calculated. Here’s what it says about the database queries:

To benchmark your database I use your wp_options table which uses the longtext column type which is the same type used by wp_posts. I do 1000 inserts of 50 paragraphs of text, then 1000 selects, 1000 updates and 1000 deletes. I use the time taken to calculate queries per second based on 4000 queries. This is a good indication of how fast your overall DB performance is in a worst case scenario when nothing is cached.

So, it’s a good number to throw around, but it’s far from the realistic site performance, as your WordPress site will mostly get SELECTs, not INSERTs or UPDATEs or DELETEs. And then, you’ll obviously need to see how many SQL queries do you need per page. And then you’ll need to examine all the caching in play – from browser, web server, WordPress, MySQL, and the operating system. And then, and then, and then.

But for a quick measure, I think, this is a good benchmark. It’s obvious that my friend can get a lot more out of his server without digging too deep. It’s obvious that separating web and database server into two Amazon instances gives you quite a boost. And it’s obvious that I don’t know much about performance measuring.

SchemaSpy – Graphical Database Schema Metadata Browser

SchemaSpy – Graphical Database Schema Metadata Browser.  This is a tool written in Java that helps one to generate database schema documentation.  Have a look at some sample pages.  Those familiar with Graphviz will immediately realize that the tools is using dot for graphing tables and their relationships.  Those familiar with SugarCRM documentation will immediately notice that SchemaSpy is used for the API documentation.

On Amazon EC2 instances

I am staring at the t2.micro (the smallest available instance type) server running MySQL 5.5.40 (using the my-huge.cnf example configuration shipped with MySQL, which ironically matches t2.micro specs).  Here’s why (as reported by Nagios for the last few hours):

Queries per second avg: 12888.839

The number is fluctuating between about 12,500 and 13,500.  Server load is moving between 0.05 and 0.08.

I think this answers the question of whether or not I am happy with the Amazon EC2 instance performance with a “hell yeah!” bang.

MySQL view processing algorithms

I had a last work session last night, troubleshooting one of the project’s database performance issues.  Without giving more details (at least for now), I want to save the link to MySQL view processing algorithms for future me.

For UNDEFINED, MySQL chooses which algorithm to use. It prefers MERGE over TEMPTABLE if possible, because MERGE is usually more efficient and because a view cannot be updatable if a temporary table is used.

A reason to choose TEMPTABLE explicitly is that locks can be released on underlying tables after the temporary table has been created and before it is used to finish processing the statement. This might result in quicker lock release than the MERGE algorithm so that other clients that use the view are not blocked as long.

A particular heavy query, using views, kept going into “Copying to tmp table” state, locking up the server and slowing everything to a crawl.  Upon closer examination, the view was created without specifying the algorithm (UNDEFINED).  Changing the view to use TEMPTABLE made everything so much faster.

I knew there were reasons for me being against using views in MySQL, but I could never remember them.  This is one.  Views not supporting indexes is another.