Blog of Leonid Mamchenkov

After SugarCRM was deployed, we were experiencing some lock ups.  Not frequent or dangerous, but annoying.  About once a week or every ten days or so, SugarCRM would lock up and won’t answer any queries at all.  Not even the login was possible.  A brief investigation showed that somehow it was locking up the MySQL database - about 15 processes (using “show full processlist”) in Locked state, with no data being sent back or forth.  All locked queries were rather complex, with several JOINs.  The load on the system was somewhat high, since we have about a few dozen operators working on it at the same time.

A similarly brief Google search suggested (see here and here) and explained converting MySQL tables from MyISAM to InnoDB.  A test has been performed and everything went OK.  Our SugarCRM database is about 600 MBytes and it was converted from MyISAM to InnoDB in under 20 minutes.  The best part is that it takes even less to convert back to MyISAM, in case you change your mind.

It’s been a few days now since we did the conversion and it looks OK.  Also, the CRM itself feels a bit faster.

I noticed this ticket in WordPress Trac - Change enum to varchar and went in to see if there is any heated discussion.  The issue is around field types used in SQL scheme for WordPress tables.  Certain fields, such post status employed ENUM type with a set of allowed values.  The proposed change in the ticket is to convert them to VARCHAR type.

Why the change?  Well, VARCHAR is just a text field.  Anyone can put pretty much any string into it.  It has more flexibility for plugin developers and future changes - no need to tweak the SQL scheme.  ENUM on the other hand works a little bit faster.

Side note: I also thought that ENUM provides some extra data validation, assuming the ENUM field is set to NOT NULL, but it turns out this is not the case.  If you insert a record with a value which is not in the list, the NULL is used. 

The change has been approved, the patch was attached, and the world will see it in the next WordPress release.  Once again, it has been proven that human time is much more valuable than machine time.  Making it easier for plugin developers to extend and change the system has more value than that of a few extra CPU cycles to lookup in strings instead of numbers.

Steve Yegge has posted yet another of his excellent (and long) rants.  This time he talks about the size of code and why one should jump out of its skin to keep it minimal.

 Most programmers have successfully compartmentalized their beliefs about code base size. Java programmers are unusually severe offenders but are by no means the only ones. In one compartment, they know big code bases are bad. It only takes grade-school arithmetic to appreciate just how bad they can be. If you have a million lines of code, at 50 lines per “page”, that’s 20,000 pages of code. How long would it take you to read a 20,000-page instruction manual? The effort to simply browse the code base and try to discern its overall structure could take weeks or even months, depending on its density. Significant architectural changes could take months or even years.

As I said, it’s a long piece. But it’s worth every paragraph. Even though some Java programmers might be slightly offended by the article, I’m sure it’s not intentional.

I was just reminded about this small thing, which is so easy to forget - regular expressions that have markers of line start (^) and/or line end($) are so much faster than those regexps that don’t have these markers. The thing is that with line start/end marker regexp engine needs to make only one match/substution, whereas when there is no such markers, it has to repeat the match/substitution operation at every character of the string.

In practice, it’s unbelievable how much difference this can make. Especially when using complex regular expressions over large data sets.

P.S.: I understand that it is not always possible to use these markers, but I think that they can be used much more often than they are. Everywhere.

After I’ve complained on #perl channel that dprofpp (perl profiler) dies on me, I received a number of suggestions on how to profile the code without using dprofpp. Ideas ranged from elemetary insertion of ‘print “Started”, time,”\n”‘ to all sorts of dynamic subroutine overloading. Here is one elegant solution that was suggested:

sub foo { print "foo (@_)\n" }
sub bar { print "bar (@_)\n" }

for (qw(foo bar)) {
  my $name = "main::" . $_;
  my $old = *$name{CODE};
  *$name = sub {
    print scalar(localtime), " Started $name\n";
    &$old;
    print scalar(localtime), " Finished $name\n";
  };
}

foo(qw(one two));
bar(qw(three four));

Oh, and someone also mentioned that dprofpp is known to be buggy. Here is a quote from the “Profiling Perl” article at perl.com:

The other problem you might face is that Devel::DProf, being somewhat venerable, occasionally has problems keeping up on certain recent Perls, (particularly the 5.6.x series) and may end up segfaulting all over the place.