Integrating RT3 with Subversion

As I have mentioned a few times before, I am a big fan of using BestPractical RT3 for all sorts of things, including, but not limited to, bug tracking during project development.  I see a great benefit in having a single system for both technical support and development departments.  Bugs can be reported by customers, investigated by technical support department, passed on to developers, fixed and tested, and then passed back to technical support department to verify with the customer and resolve.

Needless to say, integrating RT3 with Subversion can be of great benefit.  In this case, not only you will have full history of bug reports, but you’ll also see which code changes were made for each bug report.  Learning from previous bug fixes and having a quick way to see why something was changed is priceless.

Read more to see how RT3 can be integrated with Subversion.  You can also easily adopt the same approach to other version control systems.

A quick Google search for integration of RT3 with Subversion will suggest using RT::Integration::SVN package and will also point to https://rt.cpan.org/ for an example of such RT3 bug tracking setup.  While that approach is nice and handy, it’s not the one that I prefer.  I like simple and universal things.  So here is how I do it.

First of all, you should have a working instance of RT3.  Secondly, you should have some Subversion repository.  Got those?  We can proceed.

Step 1. Create a separate queue for commit messages.

I prefer having a separate commits queue for each project.  This way I can better manage who gets notified about commits.  Here are the settings for the queue:

• Queue name: project.commits
• Subject tag: Project
• Reply address: project.commits@rt.domain
• Comment address: project.commits-comment@rt.domain

Step 2. Create a separate user group.

This is, of course, not a requirement.  You an utilize any of your existing user groups.  But I prefer to have a user group for each queue.  This way I have better control on who has access where.  When things get complicated with a lot of groups, I can always group the groups into more abstract groups.  Like group Developers consisting of groups project.bugs and project.commits.

For this example, I create a user group called project.commits and assign a developers who work on this project to this group.

Step 3. Assign user group to the queue.

Now, obviously, I need to configure the queue to use the new user group.  Usually I create the user group first, and then the queue, but it would have been more difficult to explain.  So for now you’ll have to go to queue editing for the second time in two minutes.  Here is what I change in project.commits queue now:

• Watchers: add project.commits user group as AdminCC
• Group rights: add CommentOnTicket, CreateTicket, ForwardMessage, ModifyCustomField, ModifyTicket, OwnTicket, ReplyToTicket, SeeCustomField, SeeQueue, ShowOutgoingEmail, ShowTicket, ShowTicketComments, TakeTicket, Watch, WatchAsAdminCc rights to project.commits group

Step 4. Modify Subversion post-commit hook.

You probably already know that Subversion has a wonderful system of hooks, which are just scripts triggered in different stages of different operations. One of such hooks is a post-commit hook, which is triggered just after the commit goes into the repository.  Here is the source of my post-commit hook, which you should drop into the hooks folder of your Subversion repository installation and make executable.  See comments in the source for better understanding of what is what.

#!/bin/sh

# Path to the repository
REPOS="$1"

# The number of the revision which was just commited
REV="$2"

# Path to svnlook executable which was installed as part of the Subversion
SVNLOOK=/usr/bin/svnlook

# Default email for RT3 commit. You should have the user in project.commits with such email
EMAIL="rt@rt.domain"

# Find out who was the author of this commit
AUTHOR=`$SVNLOOK author "$REPOS" -r "$REV"`

# In our case, Subversion usernames are different from usernames in RT3, so
# we map Subversion users to emails of developers as they are registered in RT3
case "$AUTHOR" in
	"developer1")
		EMAIL="developer1@company.domain"
		;;
	"subcontractor1")
		EMAIL="someone@some.domain"
		;;
	"consultant1")
		EMAIL="weirdguy@surname.domain"
		;;
esac

# Default queue for commits. If we can't figure out for which project this commit is, we send the notification there
RT_EMAIL="misc.commits@rt.domain"

# Choose the queue to send the commit message to based on the project.
# In our scenario, we have the following structure of folders in  Subversion:
# /projects/
# /projects/one/
# /projects/one/branches/
# /projects/one/tags/
# /projects/one/trunk/
# ...
# /projects/more/
# /projects/more/branches/
# /projects/more/tags/
# /projects/more/trunk/
#
# If you structure is different, you'll have to play with this line a bit.
PROJECTS=`$SVNLOOK changed $REPOS --revision $REV | awk '{print $2}' | grep '^projects' | sed -e 's#projects/##' -e 's#/.*##' | sort | uniq`

# Since we examined the path of every committed file,
# we can have a situation where one commit modifies more than one project.
# This is bad, but theoretically might happen.  In such a case, we send commit
# notification to the queue of the last file.  Feel free to change the behavior as
# you see fit.
for PROJECT in `echo $PROJECTS`
do
	case "$PROJECT" in
		"project1")
			RT_EMAIL="project.commits@rt.domain"
			;;
		"project2")
			RT_EMAIL="other.commits@rt.domain"
			;;
	esac
done

# Send commit notification email.  There are many scripts online that do the same, but I prefer
# this one.  You can install it via 'cpan install SVN::Notify'.
# If you the structure of your Subversion repository is different from ours, you'll need to tweak some
# of these parameters.
/usr/bin/svnnotify \
			--repos "$REPOS" \
			--revision "$REV" \
			--to "${RT_EMAIL}" \
			--from "$EMAIL" \
			--set-sender \
			--subject-prefix "[r%d] " \
			--subject-cx \
			--strip-cx-regex '^projects/' \
			--strip-cx-regex '^people/' \
			--strip-cx-regex '/trunk.*$' \
			--strip-cx-regex '/tags.*$' \
			--strip-cx-regex '/branches.*$' \
			--ticket-map "rt=https://rt.domain/rt3/Ticket/Display.html?id=%s" \
			--with-diff

Try doing a commit to the project now. If you’ve done everything OK, you should get a nice email with commit notification to the project.commits queue, from where RT will distribute it to all the members of the project.commits user group.

That’s nice, but that’s not all.  We are still missing the important bit – linking of commits to tickets.  For that, we need to do one more step.

Step 5. Add AutoReference scrip to RT3.

I found this among many contributions in RT3 wiki.  You can set the script only for specific queue, but this one is so useful that I usually add it as a global scrip for all queues.  In short, AutoReference scrip looks for a string pattern like “ticket #123” in ticket updates (both comments and replies) and if finds, automatically links current ticket to ticket #123 (or whatever the number is).   The pattern matching is case insensitive, so “ticket #123”, “Ticket #123”, and “tIcKeT #123” will all work.  Also, pattern matching is done in the whole text of the update, so you won’t have to waste a whole lunch just for ticket number.  This allows you to do things like “Fixed a bug from ticket #123 and cleaned up the SQL a bit”.  Also, the scrip will handle multiple pattern matches with no problem.  So you can do things lke “Fixed a bug from ticket #123 and cleaned up the SQL as per ticket #124”.  Both tickets will be linked to the current one.

Here are the parameters for the scrip:

• Description: AutoReference
• Condition: OnTransaction
• Action: UserDefined
• Template: Global Template: Blank
• Stage: TransactionCreate

And here are the code chunks that you will need:

Custom condition:

my $txn = $self->TransactionObj;
return undef if $txn->Type =~ /^AddLink$/i;
return undef if $txn->Type =~ /^DeleteLink$/i;
return 1;

Custom action preparation code:

return 1;

Custom action cleanup code:

my $attachObj = $self->TransactionObj->Attachments->First;
my $content = ($attachObj && $attachObj->Content) ? $attachObj->Content : '';
my @matches = $content =~ m/ticket\s+#(\d+)/gi;

foreach my $ticket_id (@matches) {
  chomp($ticket_id);
  my $ticket = $self->TicketObj->new($RT::SystemUser);
  $ticket->LoadById($self->TransactionObj->ObjectId);
  my ($j,$k) = $ticket->AddLink('Target'=>$ticket_id,'Type'=>'RefersTo');
}

Now that the AutoReference is in and your commit notifications are going into RT3, the magic kicks in. First of all, you can reference tickets in your commit log messages. Commit log messages will be a part of commit notification email, which is sent to the RT3, which in turn will get parsed for ticket references by AutoReference scrip. But that gets even better. If you check back at the parameters to svnnotify in the post-commit hook above, you’ll notice the “–with-diff” parameter. Guess what? This will cause a diff of the commit’s changes to be inside the email body. Not a big deal, eh? That’s until you realize that that will get parsed by AutoReference too. Which means that if you have ticket references inside your code, the commit message will get linked to those tickets also. This is extremely handy! You can use ticket references in comments to your code, and every time you modify a line with such reference (either add, edit, or remove), the commit will get linked to the mentioned ticket number.

The best part of it is that in RT3 links are always two ways. Not only you will be able to see the link to original bug from the commit ticket, but also from the original bug you will links to all commits that were done about this ticket. This provides with valuable information on when each change was done, who was working on the ticket, how much time was spent, and so on, and so forth. Also, this helps with related commits, which are not that easy to find using other ways. For example, when you commit code changes separately from the unit test. If both commits reference the original ticket number with the bug, than from one commit you’ll go to the bug, and from the bug you’ll go the other related commit ticket.

On top of that, tickets in your commit queue are created using the accounts of the developers who did the commits. With that, you can have a quick and easy access to some charts, using the RT3 charting system. For example, you can create a chart by Requestor of all tickets in project.commits queue, which will show you something like this:

Elegant, isn’t it?

Also, this being an RT3, you can now comment and discuss specific commits, create tickets based on commits (refactoring tasks, for example), and so on and so forth.

The only drawback I might think of is that now you have a queue with project.commits tickets which are created fast, but nobody bothers to close them.  For that there is an easy solution – cron job to automatically resolve old tickets.  Again, there are a few scripts online to handle the task.  Here is one that I am using, and I don’t remember where from I got it.  Let me know, if you know, so I link to the author.

#!/usr/bin/perl -w

# This script resolves old tickets in commit queue. Based on rt-escalate script
#
### Configuration

# Queues to operate on (default is all)
my @queues = qw ( project.commits );

# How many days should ticket live?
my $old = 2;

# Location of RT3's libs
use lib ("/usr/lib/perl5/vendor_perl/5.8.8/RT", "/var/lib/rt3");

### Code

use strict;
use Carp;

# Pull in the RT stuff
package RT;
use RT::Interface::CLI qw(CleanEnv GetCurrentUser GetMessageContent loc);

# Clean our the environment
CleanEnv();

# Load the RT configuration
RT::LoadConfig();

# Initialise RT
RT::Init();

# Drop any setgid permissions we might have
#RT::DropSetGIDPermissions();

use RT::Date;
use RT::Tickets;
use Date::Calc qw(Delta_Days Today);

die "No queues defined" unless @queues;

foreach my $queuename (@queues) {
    # Load in the queue
    my $queue = new RT::Queue($RT::SystemUser);
    $queue->Load($queuename);

    # Get hold of new and open tickets only
    my $tickets = new RT::Tickets($RT::SystemUser);
    $tickets->LimitStatus(VALUE => 'open');
    $tickets->LimitStatus(VALUE => 'new');
    $tickets->LimitQueue(VALUE => $queue->Id);

    # Process each ticket
    while (my $ticket = $tickets->Next) {
	my ($date_str,$time_str) = split(/\s+/, $ticket->LastUpdated);
	my ($y,$m,$d) = split(/-/, $date_str);
	my ($H,$M,$S) = split(/:/, $time_str);

	my ($ny, $nm, $nd) = Today();

	my $diff = Delta_Days($y, $m, $d, $ny, $nm, $nd);
	if ($diff >= $old) {
	    $ticket->SetStatus('resolved');
	}
    }
}

# Disconnect before we finish off
$RT::Handle->Disconnect();
exit 0;

Create the file like /etc/cron.daily/rt-resolve.pl, make it executable, and paste the above code into it. From now on, all tickets in queues specified in that script will be checked, and those tickets that haven’t been updated for more than 2 days will be automatically resolved. This way you can still have discussions and comments on your commits, but once the discussion dies out the ticket will be automatically closed. Also, keeping tickets from the last couple of days gives an easy way of reviewing the most recent changes to the project, in case, for example, something breaks.

I’ve played with different ways of integrating RT3 with Subversion in at least 3 different companies, and found that the above works the best.  It’s simple, straight-forward, and easy to explain to anyone who works either with RT3 or Subversion.  Also, it can be easily extended with additional scrips to handle commands from the emails to do things like resolve tickets via Subversion log messages.  But since I never had the really used it (only proof of concept configurations were done), I’m not going to explain it here.

Enjoy!