Blog of Leonid Mamchenkov

We upgraded our development server to Fedora 10 over the weekend.  Among other things, it runs RT3 - excellent support, issue management, and bug tracking tool.  Once the upgrade was over, we ended up with a semi-working setup of RT3.  The emails were going through just fine, but the web interface was giving out a blank screen with no content or errors or warnings.

Googled a bit, but that didn’t help a lot.   Went through server logs and found an out of memory shout from Storable.pm:

2325:Callback called exit at
../../lib/Storable.pm (autosplit into ../../lib/auto/Storable/thaw.al)
line 415.

Googled for that, but it turned out that quite a few people have the problem with this module running out of memory.  And not only in RT3.

So I left it where it was and had some good night sleep.  And it helped.  In the morning, englightment came in, and I tried reloading the page with cookies and cache cleaned.  It worked.  And then it didn’t work again.  Cleaning cookies was helping for a couple of page views.  So I dived back into the RT_Config.pm file to see my options.  There it was.

=item C<$WebSessionClass>
C<$WebSessionClass> is the class you wish to use for managing Sessions.
It defaults to use your SQL database, but if you are using MySQL 3.x and
plans to use non-ascii Queue names, uncomment and add this line to
F<RT_SiteConfig.pm> will prevent session corruption.
=cut
# Set($WebSessionClass , 'Apache::Session::File');

Once I enabled Apache::Session::File, the problem went away.  We are now back to work, enjoying the new web 2.0 round corners interface, pastel colors, and more.

Open source software activity usually bumps up quite a lot before and during Christmas.  This time around I am waiting for:

• Fedora 10, which should be released today.
• WordPress 2.7, which should be released “some time in November“.
• Firefox 3.1, which is approaching its second beta.

What are your waiting for this year?

The software is checked very carefully in a bottom-up fashion. First, each new line of code is checked, then sections of code or modules with special functions are verified. The scope is increased step by step until the new changes are incorporated into a complete system and checked. This complete output is considered the final product, newly released. But completely independently there is an independent verification group, that takes an adversary attitude to the software development group, and tests and verifies the software as if it were a customer of the delivered product. There is additional verification in using the new programs in simulators, etc. A discovery of an error during verification testing is considered very serious, and its origin studied very carefully to avoid such mistakes in the future. Such unexpected errors have been found only about six times in all the programming and program changing (for new or altered payloads) that has been done. The principle that is followed is that all the verification is not an aspect of program safety, it is merely a test of that safety, in a non-catastrophic verification. Flight safety is to be judged solely on how well the programs do in the verification tests. A failure here generates considerable concern.

The above was written by R. P. Feynman, in Feynman’s Appendix to the Rogers Commission Report on the Space Shuttle Challenger Accident, 1986. More than 20 years ago. Much recommended reading.

Found via Richard Feynman, the Challenger Disaster, and Software Engineering.

I have a rather large MP3 collection.  The directories and files are named correctly more or less, but ID3 tags used to be a mess until very recently.  Two applications helped me to bring some order in that mess.

EasyTag, a GUI application, that helped me to fix lots of broken and add lots of missing comments to my MP3 files.  The smart thing about this program is that it can figure out a lot of data from the names of the files and directories, and that it can grab and replicate partial data from within the albums.

The second program that I wanted to mention, I just found out about today (thanks to Michael Stepanov’s delicious bookmarks).  It’s called tag2utf.  It’s a little Python script that converts the encoding of ID3 tags from koi8 or cp1251 (two most widely used Russian encodings) to utf8.  It’s very easy to install (the only requirement my system needed was python-eye3d library, which exists in Fedora repository) and use.  Just run it from the command line with no parameters and it will recursively look in the current directory for any files that have ID3 tags in non-utf8 encoding.  It will then give you a choice of two encodings to select from (koi8 or cp1251), a “skip” option, and a “manual” option.  All you will have to do is take a quick look at the files, and chose to either convert them from one of the two options, skip them or convert manually one by one.  You will have to make this choice for every directory with non-utf8 files.  Optionally, you can specify on the command line which directories to scan.  In case you need to convert from some other non-Russian encoding to utf8, the script is trivial to modify.

Both tools are excellent pieces of software.  It took me practically no time at all to fix my mp3 collection.  Now I can search it better, and all files display nicely in any mp3 player.  Brilliant stuff!

Since we started working on SugarCRM in the office, one of the hardest tasks that we had was solving the deployment issue.  On one hand, SugarCRM comes with some really nice GUI tools, such Studio and Module Builder.  On the other hand, the system is large and complex and should be developed and tested in a separate, non-production environment.

We’ve spent a lot of effort over the last couple of month trying to solve the puzzle.  The problem is that there is a tricky combination of files updates and database changes, some of which can be just copied over while others have to be executed from the destination machine’s administration.

So, what we did first was complete separation of environments.  Each developer had his own machine on which he could install and configure as many instances of SugarCRM as he saw fit.  Also, each developer had a separate branch in the Subversion, so that he could work on his own stuff without being afraid to run into conflict with anyone.

After that, we created a development server with a checkout of common trunk.  For extra insurance, we did a checkout from a system user, who does not have any write permissions in the repository.  In this case, even if someone will accidentally try to commit from the development server, we would be sure that it fails.

Now, each developer had to merge his changes into trunk, and then test them on the development server.  This procedure is very similar to the production deployment and consisted of two parts.  Firts part was updating all the relevant files (a bit more on this in a moment) with svn update.  Second part was logging into SugarCRM and doing Admin -> Repair -> Quick Repair and Rebuild.

The graphical tools that come with SugarCRM are powerful, but a bit confusing.  The biggest confusion for me was (and maybe still is) between Module Builder and Studio.  Studio can be used to customize core modules that are shipped with SugarCRM.  The results of these customizations are stored in custom/modules directory, and when loaded into the database, can be observed in _cstm tables (for example, accounts_cstm).  This is where new custom fields and things like that are going.  Module Builder is a tool which can help you customize existing modules or build the new ones. The confusion here is because both of these tools can be used to do the same things.  But with Module Builder you’d be working closer to the core system and modifying “original” functionality.  You can build your own modules too, by the way.  The results of the Module Builder work with go into modules/ directory, and changes in the database will take place in the original tables.  One thing to remember though, is that you’ll need to push Save & Deploy button every time you are finished with changes in Module Builder.  This is like compiling and building a module.  If you forget this step, then your module will hang in its source somewhere around custom/modulebuilder directory.

Another thing to keep in mind is the sillyness of the machine trying to figure out another machine.  Meaning that Subversion will often have issues trying to figure out the changes from the last commit, and these issues would be often caused by a lot of automatically generated code by SugarCRM.  In most of these problematic cases, Subversion will just merge the changes, and this would often result in a broken system.  I’ve found at least two reasons behind these: small context size that Subversion uses (3 lines or so) confuses it sometimes, bringing it to a wrong place in the file to do the merging; and rather messy automatically generated stuff by SugarCRM - unnecessary reordering and mixed (DOS and UNIX) ends of lines in a single file.  These problems are mostly related to vardef files (vardef.php and anything *def.php) and language dictionaries (anything with *en_us*php, or whatever your locale is).  The solution we are using at the moment is simple, although a bit heavy on the manual work - instead of merging the changes and checking them every time we simply remove old versions of files and add the new ones in two separate commits.  These way Subversion treats the files as completely different ones and real removes and re-creates them instead of trying to merge.

We follow exactly the same procedure now to deploy to the production server.  We just merge code from trunk/ to branches/stable , commit, then update the files on the production server, and then do the Quick Repair and Rebuild.

The thing about Quick Repair and Rebuild is that it takes the update definitions of your forms and layouts and rebuilds compiled templates.  It also compares the structure of the database with the update definitions in the files and, if needed, updates the database scheme too.  Sometimes you’d get an error of missing table (usually custom tables with _cstm suffix) - just create an empty table manually.  Put a couple of standard fields like id_c, date_modified, and date_entered.  After that, field modifications should be OK.  In case you run into a problem with updates to several fields at once, make sure that SugarCRM put a semicolon (;) at the end of each SQL statement that it shows you in a popup window.  For some weird reason, sometimes it just works, and sometimes it tries to execute several queries without separating them one from another.

So far the setup seems to be working for us just fine, but I’m sure that we’ll have a few changes here and there.  I’ll let you know once we find any better way of doing things.  In the meantime, here are some links that might help your development efforts:

• SugarCRM Model-View-Controller (MVC)
• SugarCRM SugarBean
• SugarCRM Vardefs
• SugarCRM Performance Tweaks for Large Systems