GitLab horror story : backup / restore failure

As I am reading this story – GitLab.com melts down after wrong directory deleted, backups fail and these details – every single hair I have, moves … I don’t (and didn’t) have any data on GitLab, so I haven’t lost anything.  But as somebody who worked as a system administrator (and backup administrator) for years, I can imagine the physical and psychological state of the team all too well.

Sure, things could have been done better.  But it’s easier said than done.  Modern technology is very complex.  And it changes fast.  And businesses want to move fast too.  And the proper resources (time, money, people) are not always allocated for mission critical tasks.  One thing is for sure, the responsibility lies on a whole bunch of people for a whole bunch of decisions.  But the hardest job is right now upon the tech people to bring back whatever they can.  There’s no sleep.  Probably no food.  No fun.  And a tremendous pressure all around.

I wish the guys and gals at GitLab a super good luck.  Hopefully they will find a snapshot to restore from and this whole thing will calm down and sort itself out.  Stay strong!

And I guess I’ll be doing test restores all night today, making sure that all my things are covered…

Update: you can now read the full post-mortem as well.

composer-patches – Simple patches plugin for Composer

composer-patches is a plugin for Composer which helps with applying patches to the installed dependencies.  It supports patches from URLs, local files, and from other dependencies.

I think this is absolutely brilliant!

It’s quite often that one finds bugs and issues in external dependencies.  Once the bug (or even the pull request with the fix) is submitted to the vendor, it can take anywhere from a few hours to a few weeks to be resolved and a new version to be released.

If you have a fix for the problem and need it in your project right away, and can’t wait until the vendor releases the new version, your best choice is to fork the dependency, fix the problem, and use your repository instead of the vendor’s package.  This works, but it’s messy.

With the patches plugin to composer, you can still use the vendor’s package and just apply a patch with composer, until the new version is available.  Clean and simple.

This also helps with testing things and working with different changes by different people, if you want to try things out – no need to choose between multiple repositories.  Just select the patches that you want and apply them at the environment you need.

Given that most development work is happening on GitHub these days, this composer plugin is even more useful than what I might think at first.  You see, GitHub provides patch and diff URL for each commit – all you need to do is add the extension to the URL.  For example, take this recent commit to my dotfiles repository.

Commit screen

If you add a “.patch” extension to this URL, you’ll get a patch output, which is useful for git am, and other commands (more on using git with email):

Patch screen

 

If you add a “.diff” extension to this URL, you’ll get a unified diff output, which you can either apply with diff and patch utils, or use with the composer-patches plugin.

Unified diff screen

So, this gives you a way of applying any commit on GitHub (and other repositories) via composer to any of your dependencies.  This is mind blowing!

 

Sharing constants between PHP classes

When writing larger applications, it is often useful to have some constants defined, which can then be shared between different parts of the application.  There are several ways to do this, and there is no real rocket science here.

However, the question is: what’s the best way to do so?

Until now I haven’t seen an elegant solution to this problem.  The old school way was defining the constants in the separate include file or in a class, and then accessing them through the global scope or via a specific class.  With a moderately modern PHP version, class constants can also be redefined by an inheriting class, which is, I think, more useful than painful.

Today, I came across an implementation that I really liked.  It’s simple and elegant: define constants in the interface and let classes implement it.  I saw it in the HTTP Message Util library, which defines HTTP methods, status codes and the like for the libraries and applications implementing PSR-7 standard (HTTP message interface).  Look here, for example.

I played around for a few minutes with this and it worked really well.  Until I stumbled upon something.  PHP classes can override the constants from the classes they inherit.  But not from the interface.  Here’s what PHP manual on Object Interfaces has to say about that:

It’s possible for interfaces to have constants. Interface constants works exactly like class constants except they cannot be overridden by a class/interface that inherits them.

This seems reasonable.  After all, the interface is a contract, and whatever is part of it should be precisely so in the implementing classes.  If you try to redefine the constants in the class:

<?php
interface FooInterface
{
    const FOO = 1;
}

class Foo implements FooInterface
{
    const FOO = 1;
}

echo "Foo = " . Foo::FOO . "\n";

you get an error:

PHP Fatal error: Cannot inherit previously-inherited or override constant FOO from interface FooInterface in foo.php on line 7

That sounds about right.  But!  It looks like you can overwrite the interface constants by extending the class.  The following snippet happily prints out two and throws no errors or warnings (my Fedora 25 runs PHP 7.0.14, if you were wondering):

<?php
interface FooInterface
{
    const FOO = 1;
}

class Foo implements FooInterface
{
}
class Bar extends Foo
{
    const FOO = 2;
}

echo "Foo = " . Bar::FOO . "\n";

Weird, right?  But there is more.  If the extending class implements the interface, rather than inherit its implementation from the parent, then the error is back:

<?php
interface FooInterface
{
    const FOO = 1;
}

class Foo implements FooInterface
{
}
class Bar extends Foo implements FooInterface
{
    const FOO = 2;
}

echo "Foo = " . Bar::FOO . "\n";

results in the familiar error, with the adjusted line number:

PHP Fatal error: Cannot inherit previously-inherited or override constant FOO from interface FooInterface in foo.php on line 10

Additionally, PHP provides the final keyword, which can be used to prevent overwriting of the inherited method by the child classes.  However, final keyword is not allowed on the constants, which makes results a bit more difficult to predict.  Especially so with the late static binding.  Here is an example:

<?php
interface FooInterface
{
    const FOO = 1;

    public function hello();
}

class Foo implements FooInterface
{
    public function hello()
    {   
        echo __FUNCTION__ . ": Self " . self::FOO . "\n";
        echo __FUNCTION__ . ": Static " . static::FOO . "\n";
    }   

    final public function bye()
    {   
        echo __FUNCTION__ . ": Self " . self::FOO . "\n";
        echo __FUNCTION__ . ": Static " . static::FOO . "\n";
    }   
}
class Bar extends Foo 
{
    const FOO = 2;
}

echo "\nFrom Foo\n";
$foo = new Foo();
$foo->hello();
$foo->bye();

echo "\nFrom Bar\n";
$bar = new Bar();
$bar->hello();
$bar->bye();

Which results in:


From Foo
hello: Self 1
hello: Static 1
bye: Self 1
bye: Static 1

From Bar
hello: Self 1
hello: Static 2
bye: Self 1
bye: Static 2

I can’t say for sure whether overwriting interface constants in inherited classes is a bug or not, but the current behavior does feel weird.

RethinkDB: why we failed

Startups are born and gone every single day.  Much more often so in technology sector.  Most of these just disappear into the ether.  RethinkDB at least leaves the useful trace of analysis of what happened and why they failed.

When we announced that RethinkDB is shutting down, I promised to write a post-mortem. I took some time to process the experience, and I can now write about it clearly.

In the HN discussion thread people proposed many reasons for why RethinkDB failed, from inexplicable perversity of human nature and clever machinations of MongoDB’s marketing people, to failure to build an experienced go-to-market team, to lack of numeric type support beyond 64-bit float. I aggregated the comments into a list of proposed failure reasons here.

Some of these reasons have a ring of truth to them, but they’re symptoms rather than causes. For example, saying that we failed to monetize is tautological. It doesn’t illuminate the reasons for why we failed.

In hindsight, two things went wrong – we picked a terrible market and optimized the product for the wrong metrics of goodness. Each mistake likely cut RethinkDB’s valuation by one to two orders of magnitude. So if we got either of these right, RethinkDB would have been the size of MongoDB, and if we got both of them right, we eventually could have been the size of Red Hat[1].

Thank you, guys.  There are valuable lessons in there.  And three points, of course:

If you remember anything about this post, remember these:

  • Pick a large market but build for specific users.
  • Learn to recognize the talents you’re missing, then work like hell to get them on your team.
  • Read The Economist religiously. It will make you better faster.