Algorithm Economy and Containers

Containers (Docker, et al) have been getting all the hype recently.  I’ve played around with these a bit, but I’m not yet convinced this is the next greatest thing for projects that I am involved with currently.  However, it helps to look at these from different perspectives.  Here’s a blog post that ties containers to a new term that I haven’t heard before – algorithm economy.

The “algorithm economy” is a term established by Gartner to describe the next wave of innovation, where developers can produce, distribute, and commercialize their code. The algorithm economy is not about buying and selling complete apps, but rather functional, easy to integrate algorithms that enable developers to build smarter apps, quicker and cheaper than before.

Best Practices for Designing a Pragmatic RESTful API

An API is a user interface for developers. Put the effort in to ensure it’s not just functional but pleasant to use.

Vinay Sahni has a rather lengthy, detailed, and well-rounded post on how to design a good RESTful API.  It covers pretty much everything from URL structures and parameters, request methods, to error handling, documentation, and coding style.

Common files in PHP packages

Jordi Boggiano looks at some common files in PHP packages, using Packagist as a data source.  There are some interesting metrics in there.  For example:

  • 58% of packages include a src/ directory and 5% a lib/ one. That’s surprisingly low to me, that means a lot have the code simply in the root folder.
  • 4% have a bin/ directory, including some sort of CLI executables.
  • 55% have a LICENSE file, that’s.. pretty disastrous but hopefully a lot of those that don’t at least indicate in the README and composer.json
  • 49% have some file or directory indicating the presence of tests (phpunit.xml & co). I am not sure if this is good or bad news to be honest, that depends on your expectations.

How fast has computing become?

Jeff Atwood has an excellent blog post, about the increase in computing powers of the modern CPUs and GPUs and the affects of those on things around us.  In particular – games such as chess and Go, and password cracking.

Every time you see a new video card release, don’t think “slightly nicer looking games” think “wow, hash cracking and AI just got 2× faster … again!”

Fascinating read!

tus.io – open protocol for resumable file uploads

tus.io, in their own words:

People share more and more photos and videos every day. Mobile networks remain fragile however. Platform APIs are a mess and every project builds its own file uploader. There are a thousand one week projects that barely work, when all we need is one real project. One project done right.

We are going to do this right. We aim to solve the problem of unreliable file uploads once and for all. tus is a new open protocol for resumable uploads built on HTTP. It offers simple, cheap and reusable stacks for clients and servers. It supports any language, any platform and any network.

Adventure in composer private repositories

First of all, I would like to take this opportunity and wish composer a happy birthday and many more years to come.  It’s been five years, and the world of PHP has changed so drastically that not many people remember how it used to be before.

I would have completely missed the birthday if it wasn’t for all the Google searching I did while working on a weird problem.  But before I get to the problem, let me set the scene.

The big part of composer success is Packagist.org – the repository of almost 100,000 packages (93,572, according to statistics, as of this writing, to be precise), which help one to solve pretty much any problem that PHP can be used for.

As good as the Packagist is, there is often a need for a repository or a package elsewhere.  Whether it’s a commercial library, or sensitive corporate code, having an ability to store it outside of public eye and handle with the same ease and the same tool as the rest of the dependencies is a very welcome feature.

Now we are getting closer to what actually happened.  A while back, I’ve setup deployment and development process for WordPress-based projects.  WordPress doesn’t support composer natively, but there are ways to make it work.  Huge thanks here goes to Roots.

So.  The composer.json file was filled with additional repositories and requirements which told composer where from to fetch things, and where to install them.  Here’s an example:

"repositories": [
    {
        "type": "package",
        "package": {
            "name": "wordpress",
            "type": "webroot",
            "version": "4.4.1",
            "dist": {
                "type": "zip",
                "url": "https://github.com/WordPress/WordPress/archive/4.4.1.zip"
            },
            "require": {
                "fancyguy/webroot-installer": "1.1.0"
            }
        }
    },
    // ...

This got especially useful, when we need to work with commercial plugins, which we couldn’t push to our public repositories, and which we didn’t want to re-distribute with every project.

So far so good. Fast-forward to a few month ago. We are setting up similar development and deployment process, but now for CakePHP-based projects. Things are much easier, since CakePHP 3 natively supports composer for the application itself and for its plugins.

But we still have the need for private repositories here and there, so we follow the same setup as we did for WordPress. This week, we had an unexpected roadblock, which wasn’t making much sense to me. When trying to pull a CakePHP plugin from a private repository, we’d get a nasty exception like this:

$ composer update
Loading composer repositories with package information
Updating dependencies (including require-dev)
  - Installing qobo/cakephp-test-foo (1.0.1)
    Loading from cache


                                                                                                                                              
  [RuntimeException]                                                                                                                          
  Unable to get primary namespace for package qobo/cakephp-test-foo.                                                                          
  Ensure you have added proper 'autoload' section to your plugin's config as stated in README on https://github.com/cakephp/plugin-installer  
                                                                                                                                              

This was confusing. First of all, I’ve never seen this error before. Secondly, I’ve read the README and had the autoloader section in the composer.json as instructed. Thirdly, very similar plugins were working fine.

Long story short, the issue wasn’t related to whether or not the GitHub/BitBucket repository was private or public. It was related to how the repository was configured in the composer.json. Reading and re-reading composer documentation about Repositories finally helped.

You can have a look at our test setup:

CakePHP plugin installer requires the autoload section in composer.json.  But, when the repository is of type “package“, for some reason, the autoload section is ignored altogether.  The RuntimeException “Unable to get primary namespace for package” is thrown by cakephp/plugin-installer (src/Installer/PluginInstaller.php line 274).  The problem is not actually in the cakephp/plugin-installer, but somewhere in the composer.  Maybe it’s intentional or maybe it’s not.  I didn’t have the time to investigate and understand it further.

Switching the repository to type “vcs” fixes the problem.  It also simplifies the composer.json file and removes the need for multiple version definitions, as now composer is using BitBucket/GitHub API to fetch available versions.

How To Keep Your Best Programmers

I really liked this article – How To Keep Your Best Programmers.  It’s not your average three paragraphs and a link, I admit. It’s somewhat of a long read.  But it does a good job of explaining why people in general, and good developers in particular choose to leave or stay in the company.

It’s difficult to quote as it flows continuously, but if I had to choose, I’d use this as a teaser:

For some background, check out this video from RSA Animate. The video is great watching, but if you haven’t the time, the gist of it is that humans are not motivated economically toward self-actualization (as widely believed) but are instead driven by these three motivating factors: the desire to control one’s own work, the desire to get better at things, and the desire to work toward some goal beyond showing up for 40 hours per week and collecting a paycheck.

Frustration with organizational stupidity is usually the result of a lack of autonomy and the perception of no discernible purpose.

Not that I am a good programmer, but it helped me understand some of my own career jumps…

Single Sign-On Between SugarCRM and Request Tracker

As mentioned before, over the last few month I’ve been involved in quite a few integration projects, using mostly SugarCRM and Request Tracker.  One of the interesting challenges was the Single Sign-On (SSO) between the two.

Continue reading “Single Sign-On Between SugarCRM and Request Tracker”

Working with encrypted values in SugarCRM 6.5

SugarCRM comes with a variety of modules that store values in the database.  Some of those values are encrypted.  For example, mailbox passwords for inbound and outbound email configurations.

When you create this configurations through the web interface or the API, you don’t need to worry about encryption, as SugarCRM handles that all by itself.  But sometimes, you need to access those values from third-party code.  The easiest way would be of course to use the same API functionality, but this is not always possible (different machines, different technology stack, etc).

It is still possible decrypt the values in the database, if you know where to look.

First of all, here is a little side note for InboundEmail and OutboundEmail modules.  InboundEmail is a full-featured module, which you can find in modules/InboundEmail folder.  OutboundEmail is however not – it lives in include/OutboundEmail .  This might seem surprising, but the reason for this (probably, as I don’t know for sure) is that outbound email configuration is much simpler.  Inbound emails are linked with folders, which are then used to subscribe users, etc.  Outbound emails are just SMTP configurations to use, directly linked to users.

Anyways.  Let’s get back on track.

Most of the encryption and decryption magic happens in include/utils/encryption_utils.php.  If you look through the code, you’ll notice that it deals with mostly two things:

  1. Generating or reading an existing encryption key.
  2. Encrypting or decrypting text with Blowfish, using the encryptionkey.

Encryption keys are stored in custom/blowfish/ folder.  The files that you’ll find there have weird names and a .php extension.  The name of the file comes from the module, for which the key will be used.  ROT13 algorithm is used to convert the name of the module into the file name.  (Note, that for outbound email, the name of the module is OutBoundEmail, not OutboundEmail).

If the encryption key file does not exist, a new one will be generated.  The file will contain a PHP snippet like this:

<?php // created: 2016-04-18 10:00:00 
  $key = array ( 0 => 'a0a0a0a0-b1b1-c3c3-d4d4-e5e5e5e5e5e5',
);

If you accidentally remove the file, then you won’t be able to decrypt any of the values, encrypted with this key, so make sure you backup this up.  Especially considering that this folder might be in your .gitignore, as a sub-folder of custom/ which stored lots of auto-generated stuff.

Note that the file actually defines a $key variable, which, if you will include it in your code, can overwrite your $key variable. So, be warned.

Now, the encryption and decryption is handled with the Crypt_Blowfish library from Pear.  You can find it in include/Pear/Crypt_Blowfish folder.

A little note for the above as well.  The Blowfish.php file which contains the Crypt_Blowfish class, requires the Blowfish/DefaultKey.php file (from the setKey() method).  That requirement uses relative path, but not based on the current file.  Yeah, I know.  So, if you just copy over the library somewhere else, you might need to adjust either path variables, or the setKey() method.

Armed with this knowledge, you can now work with encrypted values stored by the SugarCRM in the database.  Good luck!

Open Source software is so reassuring …

There’s nothing like working on a problem for a few days and getting to the reassuring code snippet like this:

sub PSGIApp {
    my $self = shift;

    # XXX: this is fucked
    require HTML::Mason::CGIHandler;
    require HTML::Mason::PSGIHandler::Streamy;
    my $h = RT::Interface::Web::Handler::NewHandler('HTML::Mason::PSGIHandler::Streamy');

    $self->InitSessionDir;

    my $mason = sub {
        my $env = shift;

        # mod_fastcgi starts with an empty %ENV, but provides it on each
        # request.  Pick it up and cache it during the first request.
        $ENV{PATH} //= $env->{PATH};

        # HTML::Mason::Utils::cgi_request_args uses $ENV{QUERY_STRING} to
        # determine if to call url_param or not
        # (see comments in HTML::Mason::Utils::cgi_request_args)
        $ENV{QUERY_STRING} = $env->{QUERY_STRING};

The first comment is misleading. It throws you off. Almost make you close the file and go somewhere else. But that’s just a little frustration from the last few days. The solution to my problem is here too… And that’s when the warm, cozy feeling I have for the Open Source Software kicks in.

P.S.: both the problem and the solution will be posted separately.