{"id":9587,"date":"2005-10-19T05:06:26","date_gmt":"2005-10-19T02:06:26","guid":{"rendered":"https:\/\/mamchenkov.net\/wordpress\/?p=9587"},"modified":"2005-10-20T07:02:05","modified_gmt":"2005-10-20T04:02:05","slug":"generating-ultimate-movie-wishlist-with-perl-and-imdb","status":"publish","type":"post","link":"https:\/\/mamchenkov.net\/wordpress\/2005\/10\/19\/generating-ultimate-movie-wishlist-with-perl-and-imdb\/","title":{"rendered":"Generating ultimate movie wishlist with Perl and IMDB"},"content":{"rendered":"\n

One of the things that will go into history with the year 2005 is the number of bad movies produced by Hollywood. IMDB says<\/a>:<\/p>\n

Hollywood is mired in its biggest box-office slump in over 20 years.<\/p><\/blockquote>\n

With all those bad movies around, finding something worth the time and effort becomes increasing difficult. Luckily, there are these two things that can make our lives easier – IMDB<\/a> and Perl<\/a>.<\/p>\n

Combining the two all sorts of interesting things can be achieved. Particularly, an ultimate movie wishlist can be generated.<\/p>\n

If you are here just for the script, than here is the movie_wishlist.pl<\/a>. If you want just the result, than here is wishlist.html<\/a>. Otherwise read on for the explanations on how it works and how you can make it better.<\/p>\n

<\/p>\n

I’ll start from the very beginning.<\/p>\n

Writing the script<\/h4>\n

Trying to figure out where to get good movies, I asked Martin – DVD club owner – to bring some. Our tastes in movies differ a lot, so it was smart of him to ask for a specific list of films that I wanted to see. I already have a movie wishlist<\/a>, but most of the stuff there was put becaus of some vague references that I don’t even remember. I needed something better.<\/p>\n

IMDB has a number of lists with Top Movies<\/a>. All I had to do was select those lists that I liked and import them into a single list that I would give to Martin. Which lists did I select? Not all of them, that’s for sure. I wasn’t interested in box office. I wasn’t interested in video. I wanted the ones with best ratings. There is Top 250 Movies of All-Time<\/a>, which is a good start. Additionally, I chose a few lists by genre – Action<\/a>, Comedy<\/a>, and few others. Than I added a few lists by decades – 2000<\/a> and 1990<\/a>.<\/p>\n

Now, what do I have to do to combine them all nicely in one list? I decided to download the lists, so that I could inspect them more closely. Nothing fancy here – simple Ctrl+S<\/code> to ‘Save As’ did the job.<\/p>\n

After closer examination two things became obvious:<\/p>\n

    \n
  1. Dups.<\/strong> Some movies appeared in more than one list. That means I would have to somehow remove dups leaving a single copy in the list.<\/li>\n
  2. Parsing.<\/strong> The structure of HTML was complex. Separating the list from the rest of the page could be a problem.<\/li>\n<\/ol>\n

    Solving the first problem – removing dups – would be much easier if I had the simple lists to deal with, and not complex HTML pages. So I decided to start looking for a solution for the second problem – parsing.<\/p>\n

    As with any other task in Perl – TIMTOWTDI – There Is More Than One Way To Do It. Two possible ways to solve the problem were:<\/p>\n

      \n
    1. Write some regular expressions to clean-up HTML and get the list of films.<\/li>\n
    2. Check out CPAN<\/a> for some helper module.<\/li>\n<\/ol>\n

      Since I am a lazy guy and not a big fan of writing HTML-parsing regular expressions, I chose the second way.<\/p>\n

      Let’s see what it has to offer. List of results<\/a> for query “html” at http:\/\/search.cpan.org<\/a> was all that I needed. The third match was HTML::LinkExtractor<\/a> – exactly what I needed. You see, items in all those IMDB top movie lists look like this:<\/p>\n