testing

PHP Static Analysis Tool – discover bugs in your code without running it!

Ondřej Mirtes shares the idea behind the creation of PHPStan – a static analysis tool for PHP:

Compiled languages need to know about the type of every variable, return type of every method etc. before the program runs. This is why the compiler needs to make sure that the program is “correct” and will happily point out to you these kinds of mistakes in the source code, like calling an undefined method or passing a wrong number of arguments to a function. The compiler acts as a first line of defense before you are able to deploy the application into production.

On the other hand, PHP is nothing like that. If you make a mistake, the program will crash when the line of code with the mistake is executed. When testing a PHP application, whether manually or automatically, developers spend a lot of their time discovering mistakes that wouldn’t even compile in other languages, leaving less time for testing actual business logic.

I’d like to change that.

This made sense to me, so I rushed to the repository. I have quite a few projects to try this on. I hurried so much that I didn’t pay attention to the important notes (aka prerequisities). These are:

PHPStan requires PHP 7.0. You have to run it in environment with PHP 7 but the actual code does not have to use PHP 7 features. (Code written for PHP 5.6 and earlier can run on 7 mostly unmodified.)

PHPStan works best with modern object-oriented code. The more strongly-typed your code is, the more information you give PHPStan to work with.

Properly annotated and typehinted code (class properties, function and method arguments, return types) helps not only static analysis tools but also other people that work with the code to understand it.

Erm … if I had properly annotated and typehinted code, which is nicely organized into objects, I think, I wouldn’t need PHPStan as much as I need it now. Anybody can analyze beautiful code. Try figuring out what’s going on in a WordPress theme with 150 PHP files, split into classes, functions and chunks of unmaintainable code. That’s where I wanted PHPStan to help me.

But OK. Let’s see what it can do. Gladly, my laptop already runs PHP 7 – here is a good first use for it.

Intstalling PHPStan with composer was easy. All I had to do was resolve the nikic/php-parser dependency conflict between PHPStan and Sami, which is our source code documentation tool of choice (the newer version uses a much more recent version of the PHP Parser, so it wasn’t rocket science).

Once installed, a simple “vendor/bin/phpstan analyse ./src” command produced a report with a few issues. Most of those were false positives, which can be fixed with a bit of PHPStan configuration. But a few real problems that were found, were indeed bits that sneaked through our automated and manual testing. For example:

------ ---------------------------------
 Line   src/Shell/EmailShell.php
------ ---------------------------------
 37      Return typehint of method App\Shell\EmailShell::getOptionParser() has invalid type App\Shell\ConsoleOptionParser.
------ ---------------------------------

I don’t think we’ll use PHPStan across all our code base just yet. It’ll be too noisy for some projects. And the PHP 7 requirement is not that easy to satisfy just yet. But maybe sometime next year, once we finalize our move to PHP 7, I will integrate it into our automatic testing process.

All in all, it’s quite a useful tool and much needed for larger code bases.

How to Read and Improve the C.R.A.P Index of your code

crapclasscompletetest

Levi Hackwith has an excellent post explaining “How to Read and Improve the C.R.A.P Index of your code“:

The C.R.A.P. (Change Risk Analysis and Predictions) index is designed to analyze and predict the amount of effort, pain, and time required to maintain an existing body of code.

It iterates over the old bits of wisdom – write simpler code and cover it with unit tests – but it does so in a very simple and measurable way.

He also reminds us that:

…software metrics, in general, are just tools. No single metric can tell the whole story; it’s just one more data point. Metrics are meant to be used by developers, not the other way around – the metric should work for you, you should not have to work for the metric. Metrics should never be an end unto themselves. Metrics are meant to help you think, not to do the thinking for you. ~Alberto Savoia

BitBucket Pipelines and Docker for PHP Developers

I’ve been meaning to look into Docker for a long while now. But, as always, time is the issue. In the last couple of days though I’ve been integrating BitBucket Pipelines into our workflow. BitBucket Pipelines is a continuous integration solution, which runs your project tests in a Docker container. So, naturally, I had to get a better idea of how the whole thing works.

“Docker for PHP Developers” article was super useful. Even though it wasn’t immediately applicable to BitBucket Pipelines, as they don’t currently support multiple containers – everything has to run within a single container.

The default BitBucket Pipelines configuration suggests the phpunit/phpunit image. If you want to run PHPUnit tests only, that works fine. But if you want to have a full blown Nginx and MySQL setup for extra bits (UI tests, integration tests, etc), then you might find smartapps/bitbucket-pipelines-php-mysql image much more useful. Here’s the full bitbucket-pipelines.yml file that I’ve ended up with.

3 serious (but common) misconceptions about software testing

QA Symphony looks at 3 serious (but common) misconceptions about software testing:

Testing is a Cost Center
Legacy Tools are Good Enough
Testing Is Easy

These are indeed very common.

Let me just ~~briefly~~ focus on the last one. Consider how quickly the complexity escalates. You are a building a simple website – nothing fancy, just some modern design and a few pages of content to represent your company. Let’s say you have just four pages: front page, about us, services, and contact us. You can quickly check how these four pages look in your browser. Easy right?

But wait. People use different browsers to access the web. So just checking it in your favorite one is not that enough. Which browsers should you test for? Let’s say we take the major ones – Google Chrome, Firefox, Microsoft Internet Explorer, and Safari. These should about cover us. Until now, your entire test was 4 page loads. Now you are multiplying it by 4 browsers. That’s 16 page loads. So far so good?

Not really. Each browser has multiple versions, which render pages differently. Not everyone is using the latest version. And new versions are released continuously. Let’s say you decided to support the latest two major versions of each browser. So now, that’s each browser times two. So 4 pages by 8 browser comes up to 32 page loads.

But we were just talking about the desktop browsers. You do want your site looking good on tablets and mobiles, right? Of course you do. What does that mean in terms of testing? Let’s say we support only iOS and Android – two major platforms, both for mobile and tablets. And we only support the default browser on each of those. That adds 4 more browsers.

By the way, when people browse on mobiles and tablets, sometimes they view your page holding a device vertically, and sometimes horizontally. You should probably test for those things as well. You know, just to make sure, nothing breaks through to the right, or requires too much scrolling down.

Remember that we are still talking about the simplest of all the websites here. Nothing fancy.

Let’s throw in an additional language. Here in Cyprus, for example, most websites are in English and Greek. Some add Russian. Some have a few languages. I’ve worked in the companies supporting over a dozen languages on the website. Each language means that you need to do more tests. In each of those browsers and on each of the supported devices.

You get my drift.

Something else. So far our tests were simple page loads, just to have a quick look whether or not the page looks fine. What if we have multiple tests per page. You want the page to look fine. But you also don’t want to have any syntax or grammar errors in the content. Especially in those foreign languages, that you don’t speak yourself. And you want the page to be optimized for search engines. And you want it to be fast (performance testing).

By now, you’ll probably give and agree that testing is not easy. Not even for the simplest of sites. Consider just how much more complicated it can get if your site is slightly more complex than that.

Remember that contact us page. That thing was just showing the company mailing address and the phone number. Now you want an integrated Google Map and a contact us form. Nothing wrong with that, right?

Well. Google services are banned in many countries. Will your contact us page still work if the user cannot access the Google Maps service? You’ll need to test for that. What about your contact form? Does it actually work? You can’t be sure from it just appearing on the page. You need to test it now too.

We are still in the domain of simple websites. And this is getting too long. Let me just through a few more things at you:

more content (imagine a website with dozens or hundreds of pages … this very blog has almost 10,000 blog posts, which are organized into categories, tags, date archives, etc).
more functionality (fancy things, dynamic page loads, form validations, etc)
even more functionality (online shop, user-driven content, etc)
more integrations with other services (Google, social networks, company CRM/ERP/etc)
spreading the site across multiple servers for performance (multiple web servers, caching servers, application servers, database servers, etc)

With each and every bit that you throw into your website, the testing gets exponentially more complex. You have to consider functionality testing, user interface testing, performance testing, security testing, and so on and so forth.

When you get to all of that, you probably have multiple people working in several teams. They all need to communicate and coordinate. Which is another layer of complexity.

I’ll stop here.

Next time you think testing is easy, think again.

Update (September 10, 2016): here are a few more misconceptions, which are as common as the ones above.

Analyzing 2+ Million Travis Builds

TravisCI – a continuous integration service – shares some of the insights from over 2,000,000 builds they’ve run, in an blog post called “What We Learned about Continuous Integration from Analyzing 2+ Million Travis Builds“. For me, the most valuable bit is about the reasons for failing builds, which clearly indicates the need for and the importance of unit, integration, and UI tests:

Around 20% of all builds fail. There is a variation based on the language – for some programming languages, testing is part of the process and culture – for others it’s an acquired tool. Once you do implement testing, most of your builds will run. You’ll cancel very few. But about 20% will fail due to failed unit tests, configurations, or environment setups. Catching these 20% before it hits production is super important.