Those of us who use secure shell (SSH) for logging in to remote machines, already know about key authentication, which is so much easier and sometimes more secure than password authentication.  We also know that in order to make it work you need to:

• generate a pair of keys with ssh-keygen command
• copy public key from the local machine to authorized_keys file on the remote machine
• fix the permissions of the .ssh/ folder and authorized_keys file on the remote machine

And that’s just what we have been doing.  Or at least me.  Today, after approximately 10 years of using secure shell, I’ve learned that there is a ssh-copy-id command, which will automatically add your current public key to a remote machine’s authorized_keys file and arrange for correct permissions.   Wow!

Thanks to @commandlinefu and top 10 one-liners blog post.

There are a number of ways to monitor PHP errors, warnings, and notices.   You can have your application code trigger some error handling, you can use PHP built-in methods, you can have some scripts running in the background analyzing logs, etc.  While you already probably do some of it, here is something that you’ll find handy.

First of all, don’t log all PHP noise into a single file.   You can easy make separate logs for each project.  Somewhere at the top of your project, when it only starts loading, add the following configuration settings:

ini_set('error_reporting', E_ALL);
ini_set('log_errors', '1');
ini_set('error_log', '/path/to/project/logs/php_errors.log');
ini_set('display_errors', '0');

This will enable logging of all errors, warnings, and notices into a file that you specified. And, at the same time, it will disable the display of all the logs to your visitors (something that you should definitely do for a production server).

One you’ve done that, you’ll notice another problem. If your application is of any considerable size and/or if it uses a lot of third-party code, you’ll get buried in all those warnings and notices. The file will quickly become very large and boring and leave your attention span. Not good. While you can fight the size of the file with a tool like logrotate, the boredom is a more serious problem. The same notices and warnings appear over and over and over. You’ll fix some of them and the others will stay there forever. What you need as a way to have a quick overview of what is broken and what is noisy.

Today I wrote a quick cronjob to do just that. Here it is in all its entirety.

#!/bin/bash

# This script parses the project PHP errors logs every hour, creates the summary of all
# errors/warnings/notices/etc and emails that summary to the email specified below.

EMAIL="me@here.com"
SUBJECT="here.com PHP errors summary for the last hour"
PHP_ERRORS_FILE="/path/to/project/logs/php_errors.log"

# The log starts with timestamp like [01-Mar-2010 12:48:56]. Timestamp + 1 stamp occupy about 24 bytes
ONE_HOUR_AGO=`date +'[%d-%b-%Y %H:' -d '1 hour ago'`

# We only need that double backslash because date pattern uses square bracket
grep "^\\$ONE_HOUR_AGO" $PHP_ERRORS_FILE | cut -b 24- | sort | uniq -c | sort -n -r | mail -s "$SUBJECT" $EMAIL

You can drop this file into /etc/cron.hourly/report_php_errors.sh, change permissions to executable, and wait for the next run of hourly scripts. If you’ve updated the variables inside the script to reflect the correct email address and path to log file, you’ll get an email every hour which will look something like this:

From: cron@your.host
To: me@here.com
Subject: here.com PHP errors summary for the last hour

  14 PHP Notice:  Use of undefined constant PEAR_LOG_DEBUG - assumed 'PEAR_LOG_DEBUG' in /some/path/to/some/file.php on line 17
    12 PHP Notice:  Undefined index:  is_printed in /path/to/something.php on line 2035
     9 PHP Notice:  Undefined index:  blah in /some/foo/bar.php on line 42
     7 PHP Notice:  Undefined offset:  1 in /some/verifier/script.php on line 120

The email will not be limited to 3 or 4 lines. It will actually contain each and every individual notice, error, and warning that occurred during the last hour in your project. The list will be sorted by how often each warning occurred, with the most frequent entries at the top.

With this list you can start fixing your most frequently seen problems, and you can also notice weird activity much faster than just checking the log file and hoping to catch it with your own eyes.

Enjoy!

Last weekend I went through a somewhat lengthy process of upgrading one of my servers from Fedora 6 to Fedora 12.  The server is vital for a company that uses it, there is more than 2 TBytes of data on that machine, and I only had a weekend to go through the upgrade.

Fedora is a very dynamic distribution, with new releases coming out roughly every 6 month.  An upgrade backward compatibility is maintained only for the last 2 releases.  So, I had to first upgrade from Fedora 6 to Fedora 8, then from Fedora 8 to Fedora 10, and then finally from Fedora 10 to Fedora 12.

Of course such a long path would pretty much guarantee that things would break.  But gladly I didn’t have to fix them for every upgrade, only once, after Fedora 12 upgrade was complete.  So the actual upgrade routine was rather simple: insert DVD with the new version, reboot, upgrade, remove DVD, reboot.  I was attempting to boot the system at least once into each new version to see how much stuff would break and if I notice anything going horribly wrong.  Everything was going smooth, except for once machine refused to boot into the new version (Fedora 8, if I remember correctly).  That didn’t stop me though.  Just upgrade to the next one, and then to the next one, etc.

After the upgrade was finished, I installed the updates for Fedora 12 and started fixing things.  The thing that I was worried for the most was Request Tracker (aka RT3) installation, which is a Perl application.  As any proper Perl application, RT3 utilizes a whole lot of Perl modules from CPAN and every time Perl version is changed signifficantly, these modules should be downloaded and installed.  Before, it was a rather slow, boring, and time consuming task.  Now however things are much simpler.  Before any perl upgrade just create an autobundle using the command “perl -MCPAN -e autobundle“.  This will create a bundle with all your current Perl modules.  After the upgrade is done, run “perl -MCPAN -e ‘install Bundle::Snapshot_2010_01_27_00‘” (where 2010_01_27_00 is the bundle version, as given to you by autobundle).  Now Perl will download all modules and their requirements from CPAN and install them automatically.  Pure magic.

Apart from RT3 only one thing broke.  One that I would expect to break because I don’t follow the development of it that close.  It was Samba.  After the upgrade to Fedora 12 none of the users could connect to any of the file shares.  ”Access denied” was given no matter which username and password was used and what was the access level to the share.  A quick Google search revealed the fix.  Apparently, somewhere in between Fedora 6 and Fedora 12, Samba changed default back-end for storing credentials.  A fix was as simple as adding a single line (“passdb backend = smbpasswd) to the configuration file, that switched Samba back to the old backend.

And that’s it!  That’s all that broke and had to be fixed after an upgrade between 6 versions of a rather dynamic Linux distribution.  Once again, I am really amazed by how well things are managed in Fedora.  Kudos and congrats!

While reading yet another sad story of the data loss, I came across a really awesome picture.  Whether you like it or not, it perfectly describes the disaster recovery plan and implementation in way too many companies.

Google announced a Public DNS service, which is extremely easy to configure and which will improve your web browsing speed and security.  This service is not revolutionary however.  There were a few ones before, and the one that seems most popular these days is OpenDNS.  In case you wonder what’s the difference between OpenDNS and Google Public DNS, take a look at this Google Groups discussion.

From the end-user point of view:

Right now the difference is that Google Public DNS does not use any sort of redirection or display any ads. If a host (domain name, web address, etc…) doesn’t resolve, it will just fail. With OpenDNS, they hijack these failures and redirect you to a search page that displays ads and makes them money.

From the administrator or customer point view there are things like stats, control panels, and more – all in OpenDNS.  Google Public DNS seems to be focused differently. At least for now.

Update: Jason Kottke explains why Google did it.

I’ve upgraded my laptop to Fedora 12.  I know, that wasn’t the smartest move, since the conference which I am attending is not over yet.  But I just couldn’t wait.

The upgrade process was as simple as:

• Open up the terminal, switch to root user, and run “preupgrade“.
• Select Fedora 12 from the list of available options and click Next.
• Watch the downloads in progress.  Or, as I did, go and have a few beers and socialize.
• When you are back from the beers, before you crash into bed, click the Next button for the installation to start.
• Go to sleep.
• Wake up to a shiny Fedora 12 distribution on your computer.

The only thing that got me worried for a second was that after the laptop rebooted into Fedora 12, for some reason the resolution of the screen went down to 800×600 or something like that.  But all I had to do was login into Gnome, navigate to System / Administration / Display and switch resolution back to what it was before the upgrade.

I didn’t have much time to explore things yet, just read through release notes and features list.  However things do look to be faster and more polished.  I’ll have to use it for a few days to know for sure.

Last time I wrote about KDE 4, I said that it wasn’t very usable for me.  Recenlty, the much praised upgrade to KDE 4.2 became available via Fedora updates.  So I got it and switched to KDE for a couple of days.

What can I say?  Quite an improvement indeed.  Mostly stable, with only a few issues, which are nicely balanced out by nice desktop effects, overall graphics, and plasmoids.  A few things moved around and I had to look for them (such as keyboard shortcuts for switching between desktops), but overall it was a pleasant experience.

Why am I back to Gnome for now?  Because of the following:

• couldn’t manage to make KDE 4.2 work properly with two monitors (major issue)
• got a bit annoyed by KDE 4.2 not waking up properly from suspend  (not always, but often enough)
• also got annoyed by it not always coming back properly from screensaver mode
• couldn’t find graphical configuration for power managment (my laptop kept suspending when idle for 5 minutes with no power plugged in)
• a few other minor things here and there

I’m sure I’ll be back to KDE very soon.  The progress between KDE 4 and KDE 4.2 is huge and if it goes like this, then I won’t have to wait long.

It’s been bugging me for a while now that advanced search is extremely slow in our RT3.  I thought it was something related to the famous Perl bug, but apparently it wasn’t.  Then I was I waiting for Fedora 10 to come out, so that we’d upgrade our RT3 installation to version 3.8.  And that didn’t solve the problem either.  Finally, we got bored and annoyed enough by this problem to actually do soemthing about it.  The solution was, as often, just a Google search away.  Here is the quote from this discussion:

Faulty rights on a specific queue caused the owner list to be quite long, which RT didn’t like. (By mistake someone had given the own ticket right on the queue to all unprivileged users)

I went through all the queues to check the rights, and there it was – a test queue had “Own Ticket” assigned to “Everyone”.  Immediately, after remove this access levels things got back to normal.

If it so happens that your Fedora install suddenly fails to boot, giving some error messages or a simple “GRUB ” string, then I advise you to boot into rescue mode, install all updates, regenerate initrd image and reboot.  All should be nice and sweet now.

Those of you who need more info, scroll through Common Fedora 10 Bugs wiki page.

We upgraded our development server to Fedora 10 over the weekend.  Among other things, it runs RT3 – excellent support, issue management, and bug tracking tool.  Once the upgrade was over, we ended up with a semi-working setup of RT3.  The emails were going through just fine, but the web interface was giving out a blank screen with no content or errors or warnings.

Googled a bit, but that didn’t help a lot.   Went through server logs and found an out of memory shout from Storable.pm:

2325:Callback called exit at
../../lib/Storable.pm (autosplit into ../../lib/auto/Storable/thaw.al)
line 415.

Googled for that, but it turned out that quite a few people have the problem with this module running out of memory.  And not only in RT3.

So I left it where it was and had some good night sleep.  And it helped.  In the morning, englightment came in, and I tried reloading the page with cookies and cache cleaned.  It worked.  And then it didn’t work again.  Cleaning cookies was helping for a couple of page views.  So I dived back into the RT_Config.pm file to see my options.  There it was.

=item C<$WebSessionClass>
C<$WebSessionClass> is the class you wish to use for managing Sessions.
It defaults to use your SQL database, but if you are using MySQL 3.x and
plans to use non-ascii Queue names, uncomment and add this line to
F<RT_SiteConfig.pm> will prevent session corruption.
=cut
# Set($WebSessionClass , 'Apache::Session::File');

Once I enabled Apache::Session::File, the problem went away.  We are now back to work, enjoying the new web 2.0 round corners interface, pastel colors, and more.