Firefox extensions

Yesterday was a somewhat slow day, so I spent some time on the housekeeping of my Firefox browser.  Somehow I managed to accumulate a lot of extensions, themes, plugins, bookmarks, bookmarklets, and what not.  It felt like a good time to clean the mess up a bit.

I spent about two hours going through the list of all installed pieces, upgrading outdated versions, changing old solutions to the modern alternatives, getting new tools, and so on.   I have to say that after that effort my Firefox works faster, and it suits me better now.  One of the biggest changes from my previous setup became the use of Tab Kit extension.  Among its many features, it has the one that I’ve been passively looking for for a long time now – tab bar on the right side, instead of top, but not as a part of the sidebar, and with a tree view.  Combined with Aging Tabs extension, the result is exactly the way I wanted it.

(it is better in real life than it is on the screenshot)

Now when I open links from the site in the new tabs, these new tabs are organized in a tree like structure.  Tabs that I haven’t yet visited are highlighted in green.  Current tab is highlighted in blue, as usual.  And the rest of the tabs are coloured in different shades of grey, depending on how long ago I last viewed them.  Also, because the tab bar is separate from the sidebar, I can get an additional panel on demand, with an application that I need the most at the moment, without sacrificing my precious tabs.

And just in case you are wondering which other extensions I am using, continue reading for the list of all extensions, which was generated by one of the extensions on that list.

Continue reading Firefox extensions

SugarCRM deployment efforts

Since we started working on SugarCRM in the office, one of the hardest tasks that we had was solving the deployment issue.  On one hand, SugarCRM comes with some really nice GUI tools, such Studio and Module Builder.  On the other hand, the system is large and complex and should be developed and tested in a separate, non-production environment.

We’ve spent a lot of effort over the last couple of month trying to solve the puzzle.  The problem is that there is a tricky combination of files updates and database changes, some of which can be just copied over while others have to be executed from the destination machine’s administration.

So, what we did first was complete separation of environments.  Each developer had his own machine on which he could install and configure as many instances of SugarCRM as he saw fit.  Also, each developer had a separate branch in the Subversion, so that he could work on his own stuff without being afraid to run into conflict with anyone.

After that, we created a development server with a checkout of common trunk.  For extra insurance, we did a checkout from a system user, who does not have any write permissions in the repository.  In this case, even if someone will accidentally try to commit from the development server, we would be sure that it fails.

Now, each developer had to merge his changes into trunk, and then test them on the development server.  This procedure is very similar to the production deployment and consisted of two parts.  Firts part was updating all the relevant files (a bit more on this in a moment) with svn update.  Second part was logging into SugarCRM and doing Admin -> Repair -> Quick Repair and Rebuild.

The graphical tools that come with SugarCRM are powerful, but a bit confusing.  The biggest confusion for me was (and maybe still is) between Module Builder and Studio.  Studio can be used to customize core modules that are shipped with SugarCRM.  The results of these customizations are stored in custom/modules directory, and when loaded into the database, can be observed in _cstm tables (for example, accounts_cstm).  This is where new custom fields and things like that are going.  Module Builder is a tool which can help you customize existing modules or build the new ones. The confusion here is because both of these tools can be used to do the same things.  But with Module Builder you’d be working closer to the core system and modifying “original” functionality.  You can build your own modules too, by the way.  The results of the Module Builder work with go into modules/ directory, and changes in the database will take place in the original tables.  One thing to remember though, is that you’ll need to push Save & Deploy button every time you are finished with changes in Module Builder.  This is like compiling and building a module.  If you forget this step, then your module will hang in its source somewhere around custom/modulebuilder directory.

Another thing to keep in mind is the sillyness of the machine trying to figure out another machine.  Meaning that Subversion will often have issues trying to figure out the changes from the last commit, and these issues would be often caused by a lot of automatically generated code by SugarCRM.  In most of these problematic cases, Subversion will just merge the changes, and this would often result in a broken system.  I’ve found at least two reasons behind these: small context size that Subversion uses (3 lines or so) confuses it sometimes, bringing it to a wrong place in the file to do the merging; and rather messy automatically generated stuff by SugarCRM – unnecessary reordering and mixed (DOS and UNIX) ends of lines in a single file.  These problems are mostly related to vardef files (vardef.php and anything *def.php) and language dictionaries (anything with *en_us*php, or whatever your locale is).  The solution we are using at the moment is simple, although a bit heavy on the manual work – instead of merging the changes and checking them every time we simply remove old versions of files and add the new ones in two separate commits.  These way Subversion treats the files as completely different ones and real removes and re-creates them instead of trying to merge.

We follow exactly the same procedure now to deploy to the production server.  We just merge code from trunk/ to branches/stable , commit, then update the files on the production server, and then do the Quick Repair and Rebuild.

The thing about Quick Repair and Rebuild is that it takes the update definitions of your forms and layouts and rebuilds compiled templates.  It also compares the structure of the database with the update definitions in the files and, if needed, updates the database scheme too.  Sometimes you’d get an error of missing table (usually custom tables with _cstm suffix) – just create an empty table manually.  Put a couple of standard fields like id_c, date_modified, and date_entered.  After that, field modifications should be OK.  In case you run into a problem with updates to several fields at once, make sure that SugarCRM put a semicolon (;) at the end of each SQL statement that it shows you in a popup window.  For some weird reason, sometimes it just works, and sometimes it tries to execute several queries without separating them one from another.

So far the setup seems to be working for us just fine, but I’m sure that we’ll have a few changes here and there.  I’ll let you know once we find any better way of doing things.  In the meantime, here are some links that might help your development efforts:

Vacancy for Python or C++ programmers and Linux sysadmins

Point Nine is looking for some talanted people.  They are working in the financial sector, and seeking Python and C++ programmers, as well as Linux system administrators.  They seem to be an interesting company, judging both by their technology stack and hiring techniques.   For example, I’ve spotted their job ad on The Daily WTF.  Cool stuff.  If I wasn’t working where I am working, I would have applied for the job already.

Migrating MS Dynamics CRM to SugarCRM

I’ve been a bit quiet for the last couple of month.  That’s because I was leading an ambicious project at my new job – migration of a Microsoft Dynamics CRM version 3 to SugarCRM Community Edition version 5.0.0.  There were only three people involved, non of us could afford to work full time on the project, and we only had three weeks to do it.

Read on for a story on why it took us longer, how we did, and if it was a success at all.

Continue reading Migrating MS Dynamics CRM to SugarCRM

BLOB is bad for your (mental) health

If you ever mention that your web application uses database to store files, you risk being flamed into oblivion.  Indeed, in most cases, it is a bad idea, since file system is more effecient when it comes to files.  However, there are cases when it makes sense to have files saved in the database.

Maybe I am doing something wrong, but in the last six month, I had to develop at least three systems that used MySQL for file storage (uploaded files that have to be synchronized across several hosts, etc).  Yesterday, for the third time I stumbled across the same problem, that almost drove me insane.

MySQL has four data types for storing binary data – TINYBLOB, BLOB, MEDIUMBLOB, and LONGBLOB.  Somehow I always forget about these and use BLOB.  BLOB works just fine, but it has a limit on size, which is rather low – 64 KBytes.  The mean thing here is that it will work just fine with most of the test data – text files, short PDFs, and small pictures.  Once the application is tested and put into production, the corrupted files will start coming in.  Re-writing all parts that deal with uploading, moving, cleaning, escaping, and encrypting binary data takes time.  Going through file reading and writing routines is boring too, and it won’t help either.

By the time, the issue is discovered and all fields are changed to LONGBLOB, it is often very late, and you’ve lost your weekend, as well as a lot of large files. This post is an attempt to save my (and your) sanity.

Reminder: use LONGBLOB instead of BLOB for file storage, unless you are absolutely sure about the maximum size of incoming data.