Google services via command line

I don’t know how I missed the announcement and why there is no noise around this release, but here it goes.  GoogleCL is a tool that you use from the command line to access Google web services.  I just installed it on my Fedora 13 laptop and its awesome!

Here is what you need to do to intall

$ sudo yum install python-gdata
$ wget http://googlecl.googlecode.com/files/googlecl-0.9.7.tar.gz
$ tar xzvf googlecl-0.9.7.tar.gz
$ cd googlecl-0.9.7
$ sudo python setup.py install

Here are some of the things that you can do.  And I guess more is coming shortly.

The first time you’ll connect to any of the Google web services using this tool, you’ll need to authenticate and grant access to the application.  After that – it’s all pure honey.

Copy SSH key to remote machine

Those of us who use secure shell (SSH) for logging in to remote machines, already know about key authentication, which is so much easier and sometimes more secure than password authentication.  We also know that in order to make it work you need to:

  • generate a pair of keys with ssh-keygen command
  • copy public key from the local machine to authorized_keys file on the remote machine
  • fix the permissions of the .ssh/ folder and authorized_keys file on the remote machine

And that’s just what we have been doing.  Or at least me.  Today, after approximately 10 years of using secure shell, I’ve learned that there is a ssh-copy-id command, which will automatically add your current public key to a remote machine’s authorized_keys file and arrange for correct permissions.   Wow!

Thanks to @commandlinefu and top 10 one-liners blog post.

Monitoring PHP errors, warnings, and notices

There are a number of ways to monitor PHP errors, warnings, and notices.   You can have your application code trigger some error handling, you can use PHP built-in methods, you can have some scripts running in the background analyzing logs, etc.  While you already probably do some of it, here is something that you’ll find handy.

First of all, don’t log all PHP noise into a single file.   You can easy make separate logs for each project.  Somewhere at the top of your project, when it only starts loading, add the following configuration settings:

ini_set('error_reporting', E_ALL);
ini_set('log_errors', '1');
ini_set('error_log', '/path/to/project/logs/php_errors.log');
ini_set('display_errors', '0');

This will enable logging of all errors, warnings, and notices into a file that you specified. And, at the same time, it will disable the display of all the logs to your visitors (something that you should definitely do for a production server).

One you’ve done that, you’ll notice another problem. If your application is of any considerable size and/or if it uses a lot of third-party code, you’ll get buried in all those warnings and notices. The file will quickly become very large and boring and leave your attention span. Not good. While you can fight the size of the file with a tool like logrotate, the boredom is a more serious problem. The same notices and warnings appear over and over and over. You’ll fix some of them and the others will stay there forever. What you need as a way to have a quick overview of what is broken and what is noisy.

Today I wrote a quick cronjob to do just that. Here it is in all its entirety.

#!/bin/bash

# This script parses the project PHP errors logs every hour, creates the summary of all
# errors/warnings/notices/etc and emails that summary to the email specified below.

EMAIL="me@here.com"
SUBJECT="here.com PHP errors summary for the last hour"
PHP_ERRORS_FILE="/path/to/project/logs/php_errors.log"

# The log starts with timestamp like [01-Mar-2010 12:48:56]. Timestamp + 1 stamp occupy about 24 bytes
ONE_HOUR_AGO=`date +'[%d-%b-%Y %H:' -d '1 hour ago'`

# We only need that double backslash because date pattern uses square bracket
grep "^\\$ONE_HOUR_AGO" $PHP_ERRORS_FILE | cut -b 24- | sort | uniq -c | sort -n -r | mail -s "$SUBJECT" $EMAIL

You can drop this file into /etc/cron.hourly/report_php_errors.sh, change permissions to executable, and wait for the next run of hourly scripts. If you’ve updated the variables inside the script to reflect the correct email address and path to log file, you’ll get an email every hour which will look something like this:

From: cron@your.host
To: me@here.com
Subject: here.com PHP errors summary for the last hour

  14 PHP Notice:  Use of undefined constant PEAR_LOG_DEBUG - assumed 'PEAR_LOG_DEBUG' in /some/path/to/some/file.php on line 17
    12 PHP Notice:  Undefined index:  is_printed in /path/to/something.php on line 2035
     9 PHP Notice:  Undefined index:  blah in /some/foo/bar.php on line 42
     7 PHP Notice:  Undefined offset:  1 in /some/verifier/script.php on line 120

The email will not be limited to 3 or 4 lines. It will actually contain each and every individual notice, error, and warning that occurred during the last hour in your project. The list will be sorted by how often each warning occurred, with the most frequent entries at the top.

With this list you can start fixing your most frequently seen problems, and you can also notice weird activity much faster than just checking the log file and hoping to catch it with your own eyes.

Enjoy!

Fedora Linux history tour

Last weekend I went through a somewhat lengthy process of upgrading one of my servers from Fedora 6 to Fedora 12.  The server is vital for a company that uses it, there is more than 2 TBytes of data on that machine, and I only had a weekend to go through the upgrade.

Fedora is a very dynamic distribution, with new releases coming out roughly every 6 month.  An upgrade backward compatibility is maintained only for the last 2 releases.  So, I had to first upgrade from Fedora 6 to Fedora 8, then from Fedora 8 to Fedora 10, and then finally from Fedora 10 to Fedora 12.

Of course such a long path would pretty much guarantee that things would break.  But gladly I didn’t have to fix them for every upgrade, only once, after Fedora 12 upgrade was complete.  So the actual upgrade routine was rather simple: insert DVD with the new version, reboot, upgrade, remove DVD, reboot.  I was attempting to boot the system at least once into each new version to see how much stuff would break and if I notice anything going horribly wrong.  Everything was going smooth, except for once machine refused to boot into the new version (Fedora 8, if I remember correctly).  That didn’t stop me though.  Just upgrade to the next one, and then to the next one, etc.

After the upgrade was finished, I installed the updates for Fedora 12 and started fixing things.  The thing that I was worried for the most was Request Tracker (aka RT3) installation, which is a Perl application.  As any proper Perl application, RT3 utilizes a whole lot of Perl modules from CPAN and every time Perl version is changed signifficantly, these modules should be downloaded and installed.  Before, it was a rather slow, boring, and time consuming task.  Now however things are much simpler.  Before any perl upgrade just create an autobundle using the command “perl -MCPAN -e autobundle“.  This will create a bundle with all your current Perl modules.  After the upgrade is done, run “perl -MCPAN -e ‘install Bundle::Snapshot_2010_01_27_00‘” (where 2010_01_27_00 is the bundle version, as given to you by autobundle).  Now Perl will download all modules and their requirements from CPAN and install them automatically.  Pure magic.

Apart from RT3 only one thing broke.  One that I would expect to break because I don’t follow the development of it that close.  It was Samba.  After the upgrade to Fedora 12 none of the users could connect to any of the file shares.  “Access denied” was given no matter which username and password was used and what was the access level to the share.  A quick Google search revealed the fix.  Apparently, somewhere in between Fedora 6 and Fedora 12, Samba changed default back-end for storing credentials.  A fix was as simple as adding a single line (“passdb backend = smbpasswd) to the configuration file, that switched Samba back to the old backend.

And that’s it!  That’s all that broke and had to be fixed after an upgrade between 6 versions of a rather dynamic Linux distribution.  Once again, I am really amazed by how well things are managed in Fedora.  Kudos and congrats!