Blog of Leonid Mamchenkov

If you ever mention that your web application uses database to store files, you risk being flamed into oblivion.  Indeed, in most cases, it is a bad idea, since file system is more effecient when it comes to files.  However, there are cases when it makes sense to have files saved in the database.

Maybe I am doing something wrong, but in the last six month, I had to develop at least three systems that used MySQL for file storage (uploaded files that have to be synchronized across several hosts, etc).  Yesterday, for the third time I stumbled across the same problem, that almost drove me insane.

MySQL has four data types for storing binary data - TINYBLOB, BLOB, MEDIUMBLOB, and LONGBLOB.  Somehow I always forget about these and use BLOB.  BLOB works just fine, but it has a limit on size, which is rather low - 64 KBytes.  The mean thing here is that it will work just fine with most of the test data - text files, short PDFs, and small pictures.  Once the application is tested and put into production, the corrupted files will start coming in.  Re-writing all parts that deal with uploading, moving, cleaning, escaping, and encrypting binary data takes time.  Going through file reading and writing routines is boring too, and it won’t help either.

By the time, the issue is discovered and all fields are changed to LONGBLOB, it is often very late, and you’ve lost your weekend, as well as a lot of large files. This post is an attempt to save my (and your) sanity.

Reminder: use LONGBLOB instead of BLOB for file storage, unless you are absolutely sure about the maximum size of incoming data.

This is a quick follow-up to yesterday’s post - “Where did all the PHP programmers go?“.

First of all, let me take the moment and say “Wow!”.  Somebody submitted the post to Reddit and it made it to the front page and got an unbelievable amount of comments.  Almost 500, and still coming.  Thank you all.

Secondly, the comments on this blog are fixed finally.  Murphy’s Law in action - they got broken just before the wave came in and they got fixed shortly after.

Thirdly, I should clear up a few things.  My apologies for getting you guys confused.  I never asked any candidate to compare sorting algorithms, much less to implement them.  I asked to sort an array.  I was expecting one of those PHP function calls in return.  But I only got it a few times.  Many candidates didn’t know how to sort an array (apparently they use MySQL to sort an array).  A few suggested “bubble sort”.  Probably thinking that the tasks for testing sorting algorithms.  One even went as far as implementing a bubble sort in PHP.  With pen and paper.  This one was the toughest to decide about, by the way.

Fourthly, the correction.  The language is indeed called Ruby, not Ruby on Rails. I am aware of that.  I was just trying to catch a thought.  Thanks for pointing it out though.

Fifthly, explanation for the pen and paper.  Yes, I know that programmers are used to typing code.  I know that they are used to their tools and online references.  But.  This is an interview.  My time is limited and I have to make a decision.  If I give all the tools and references to my mother, she will be able to solve the problem I am giving in reasonable time.  She is not a PHP developer.  She has no experience with PHP.  But she has enough of common sense to do it.  If I take everything away - she won’t be able to do that.  But any semi-decent programmer will do.  Further on, I am not feeding the resulting paper into the machine.  The only parser that sees that code is the one embedded in my brain.  And I assure you it is very tolerant to minor syntax errors and missing parameters.  I want to see the process.  The approach. Some data structures and algorithms.  A bit of style in variable names, indentation, and empty lines, if I am lucky.  That’s all.

Sixthly, on the exercise itself.  I like to think that I am pretty flexible with answers.  For this particular exercise, a Perl programmer inside me thinks associative array is the best data structre.  (And yes, before you start bashing further, I know that associative arrays in PHP aren’t the same as hashes in Perl.)  I can accept an OOP solution just fine.  What I find hard to accept is a single dimensional array with hopping over a pre-defined number of fields per record.

Seventhly, this post, once it got to reddit and then furthermore to other news streams, generated more candidates and hints to where to find them, then all of my prevoius efforts.  Thanks to all of you who sent me resumes, links, and pointers.  My inbox is a bit overwhelmed right now, but I’ll reply to everyone over the next few days.

Thanks a lot to all of you.

I’ve recently posted some of my thoughts on PHP.  If you enjoy the topic, here is an excellent post over at Coding Horror, which reminds why PHP sucks so badly and also why it will stick around for some time to come.

Here is a quote from a wise comment in the discussion of the “Linux system programming” book review:

Build your code with -Wall -Werror (or your compiler’s equivalent). Once you clean up all the crud, that pops up, crank it up with -W -Wno-unused-parameter -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith. Once there — add -Wreturn-type -Wcast-qual -Wswitch -Wshadow -Wcast-align and tighten up by removing the no in -Wno-unused-parameter. The -Wwrite-strings is essential, if you wish your code to be compiled with a C++ compiler some day (hint: the correct type for static strings is ” const char *”).
For truly clean code, add -Wchar-subscripts -Winline -Wnested-externs -Wredundant-decls.
The people, who wrote and maintain the compiler, are, most likely, several levels above you in understanding programming in general and C-programming in particular. Ignoring the advice their code generates is foolish on your part…
As a minimum, solved warnings will make your code more readable by reducing/eliminating the “Why is he doing this?” questions. More often than not, they point out bugs you would otherwise spend hours chasing with a debugger later.
And they make your code more portable. But if you don’t understand, why a warning is generated — ask around. Don’t just “shut it up”. For example, initializing a variable at declaration is usually a no-no. If the compiler thinks, the variable may be used before being initialized, scrutinize your program’s flow. If you can’t figure out, it may some times be better to disable this one warning temporarily with -Wno-uninitialized to move on, instead of shutting it up for ever by a bogus “= 0″ or some such…

Today I came across something that rather puzzled me at first, seemed irresponsible and such, but was cleared later, upon reading the manual.  When uploading files in PHP, variable $_FILES stores a bunch of information about each file.  One of those stored bits is the MIME type of the file.  I was puzzled with how easy it was to trick PHP into setting a wrong MIME type.  However, documentation clearly says that:

The mime type of the file, if the browser provided this information. An example would be “image/gif”. This mime type is however not checked on the PHP side and therefore don’t take its value for granted.