Two years at Qobo

Today marks the completion of my second year at Qobo Ltd.  The first year was quite a ride.  But the second one was even wilder.  As always, it’s difficult (and lengthy) to mention everything that happened.  A lot of that stuff is under the non-disclosure agreement (NDA) terms too.  But here are a few generic highlights:

  • Vision and strategy – most of my first year has been spent in putting out fires, fixing things big and small, left, right, and center.  The technology boost was necessary across the board, so it didn’t leave much time for the vision and strategy.  I feel that we’ve made a huge progress in this area in the last 12 month.  We have a clear vision.  We have all the stakeholders agreeing on all key elements.  We have worked out a strategy on how to move forward.  And we’ve started implementing this strategy (hey, Qobrix!).  In terms of achievements, I think this was the most important area and I am pretty happy with how things are shaping up.
  • Team changes – much like in the first year, we had quite a few changes in the team.  Some of them were unfortunate, others not so much.  The team is still smaller than what we want and need, but I think we are making progress here.  If our World Domination plans will work out to even some degree, we’ll be in a much better place very soon.
  • Technology focus – we’ve continued with our goal of doing fewer things but doing them better.  Our expertise in WordPress, CakePHP and SugarCRM grew a lot.  We’ve signed and deployed a variety of projects, which resulted in more in-depth knowledge, more networking with people around each technology, more tools and practices that we can reuse in our future work.
  • Open Source Softwareour GitHub profile is growing, with more repositories, pull requests, releases, features, and bug fixes.  We’ve also contributed to a variety of Open Source projects.  Our involvement with Open Source Software will continue to grow – that’s one of those things that I am absolutely sure about.
  • Hosting, continuous integration and delivery (CI/CD), and quality assurance – again, the trend continued this year.  We are using (and understanding) more of the cloud infrastructure in general and Amazon AWS in particular.  We have a much better Zabbix setup.  And our love and appreciation of Ansible grows steeply. Let’s Encrypt is in use, but we’ll grow it to cover all our projects soon.  We are also experimenting with a variety of quality assurance tools.  We are using TravisCI for most of our Open Source work.  And we are on the brink of using recently announced BitBucket Pipelines for our private repositories (sorry Jenkins, we’ve tried you, but … not yet).  We’ve also jumped into ChatOps world with HipChat and its integrations, to the point that it’s difficult to imagine how could we have worked without it just a few month ago. has also proved to be useful.
  • Projects, projects, projects – much like the previous year, we’ve completed a whole lot of projects (see some of our clients).  Some were simple and straightforward.  Others were complicated and challenging. And we have more of these in the pipelines.  Overall, we’ve learned how to do more with less.  Our productivity, technical expertise, and confidence grows day-to-day.  I hope we keep it up for years to come.
  • Website – one thing that we wanted to do for ages is to update our website.  Which we did, despite all the crazy things going on.  It’s not a complete redesign, but it’s a nice refreshment.  And we’ve also got our blog section, which I promised you last year.  All we need to do now is to use it more. ;)

There are a couple of major updates coming soon, but I am not at liberty to share them right now.  But they are very, very exciting – that’s all I can say today.  Keep an eye our blog – we’ll be definitely sharing.

As I said, it was quite an intense year, with lots of things going on everywhere.  There were tough times, and there were easy times.  There were challenges and there were accomplishments.  There were successes, and there were mistakes and failures.  But I wouldn’t have it any other way!

After two years, I am still excited about this company and about my job here.  (Which, looking at my career so far, is not something that happens often.)  I hope the next year will continue the adventure and by the end of it I’ll be able to proudly show you a few more things.


101 Most Common Interview Questions with Pass or Fail Answers

Following the recent post “10 Favorite Job Interview Questions for Linux System Administrators“, here is a more generic, but a much more comprehensive resource – “101 Most Common Interview Questions with Pass or Fail Answers“.  It’s not as technical, but it provides a good summary of common interview questions, from the generic ones like “Why do you want to leave your current company”, through brainteasers like “How many gas stations are there in the United States?”, to stress and communication ones like “What did you do when you had a boss you didn’t get along with?”.  The good thing is that you’ll find not only the questions, but also the suggestions on how to answer them.

Altogether, it’s a great resource to go through before your next interview.  Most of these questions are very common, no matter which position you are applying to.

What is a Senior Developer?

I’ve been hiring, firing, and working with developers of all sorts for the last couple of decades.  In those years, I realized that each developer is very unique – their strong and weak sides, knowledge gaps, working rhythm, social interaction, communication abilities, etc.  But regardless of how unique each developer is, it is often useful to group them into expertise levels, like junior and senior.  Companies do that for a variety of reasons – billing rates, expectations, training required, responsibility, etc.

And this is where things get tricky.  One needs a good definition of what a senior developer is (other definitions can be derived from this one one too).  There is no standard definition that everybody agrees upon, so each one has their own.

I mostly consider a senior developer to be self-sufficient and self-motivated.  It’s somebody who has the expertise to solve, or find ways of solving any kind of technical problems.  It’s also someone who can see the company’s business needs and issues, and can find work to do, even if nothing has been recently assigned to him.  A senior developer would also provide guidance and mentorship to the junior teammates. I’ve also came to believe that people with the real expertise have no problem discussing complex technical issues in simple terms, but that’s just a side note.

Anyway, recently, I came across this very short blog post, which sent me a spree of pages, charts, and discussions:

Because of this “What is a senior developer?” conversation on Reddit, I am reminded of the Construx Professional Development Ladders, as mentioned to me long ago by Alejandro Garcia Fernandez. Here is a sample ladder for developers.

The original article for the Reddit discussion – “The Conjoined Triangles of Senior-Level Development” is absolutely brilliant.  In the beginning it provides a chart of the conjoined triangles of senior-level development, which reflects my definition and understanding:


But it doesn’t stop there. It dives deeper into the problem, and, eventually features this Venn diagram:


.. and more.  By now, I’ve read the article three times, but I keep coming back to it – it just makes me think and rethink over and over again.  Once it settles in my head a bit, I’ll look deeper into the Professional Development Ladder and it’s example application to the senior developer.

Overall, this is a very thought provoking bunch of links.

Why Some People Get Promoted (And Others Don’t)

I enjoyed reading the article “Why Some People Get Promoted (And Others Don’t)“.  Unlike many other in this domain, it is simple, direct, and to the point.  TLDR version:

  1. Do great things.
  2. Tell people.

There are quite a few links to external resources, with research and insightful quotes.  Here are a couple of my favorite bits:

‘[S]ent does not mean received’ is a profound thing. Half of your job in this studio is doing your work, the other half of your job is communicating that it’s been done. Because if you do it, and I don’t hear about it, how do I know what’s going on? I’m not trying to control everything, but in an intimate work environment, where we’re really trying to develop something complex, a nod, saying, ‘I got it,’ helps move things along.

And this part, which resonates with my inner blogger:

Asking for help is part of getting better at your job.

3. Work where people can see you.

Gaining visibility might require going outside your office. Maybe you have a side project, or maybe your work culture isn’t a healthy environment to pursue visibility.

Promoting yourself doesn’t have to be on someone else’s terms. Write a book, start a blog, make a side-project, collaborate with new people outside of work, or speak at panels and conferences. Tell people about what you’ve done, what you’re doing, why it’s important, and how you did it. Give talks, teach others, raise your hand for new projects.

Bad project

CommitStrip nails one of the ways of getting into a bad project …

bad project

I remember reading an interview with Matt Mullenweg (though can’t seem to find a reference now), where he said that this sort of thing happened with Automattic.  People were asking them for commercial support, but they didn’t want to do it, so they started with an insane amount of like $5,000 per month and all of a sudden found themselves with a queue of people outside.

And they were not alone, of course.

Upgrading Amazon EC2 instance type

By now everybody knows that one of the major benefits to using cloud services rather than hosting on your own hardware is the ease to scale quickly.  Many web applications and large companies benefit from this, but what about smaller customers?  How about a single server?

Well, today one of our web servers was experiencing some pick loads.  It hosts a whole array of small websites built with WordPress, CakePHP, and other popular tools.  There was no time to update all these projects to work with multiple web servers.  And even redeploying them to multiple individual servers would have taken a few hours.  Instead, we’ve decided to upgrade the server hardware.

Pause for a second and imagine the situation with your own server.  Or a dedicated hosting account for that matter.  So much to configure.  So much to backup and restore.  So much to test.

Here’s how to do it, if your projects are on the Amazon EC2 instance (our was also inside a virtual private cloud (VPC), but even if it wasn’t, the difference would be insignificant):

  1. Login to the Amazon AWS console.
  2. Navigate to the Amazon EC2 section.
  3. Click on Instances in the left sidebar.
  4. Click on the instance that you want to upgrade in the list of your instances.
  5. Click Actions -> Instance State -> Stop.
  6. Wait a few seconds for the instance to stop.  You can use the Refresh button to update the list.
  7. (While your instance is still selected in the list of instances:) Click Actions -> Instance Settings -> Change Instance Type.
  8. In the popup window that appeared, select an Instance Type that you want.
  9. Click Apply.
  10. Click Actions -> Instance State -> Start.
  11. Wait a few seconds for the instance to start.
  12. Enjoy!

The whole process literally takes under two minutes.  You get exactly the same configuration – hostname, IP addresses (both internal and external), mounted EBS volumes, all your OS configuration, etc.  It’s practically a reboot of your machine. But into a different hardware configuration (CPU/RAM).

Coincidentally, earlier this morning I had to pack up a rack-mountable server – screws, cables, dusty boxes, the whole shebang.  It’s been a while since I’ve done that last time.

Another day in the office #work #cables #sysadmin

A photo posted by Leonid Mamchenkov (@mamchenkov) on

But I can tell you that I much prefer clicking a few buttons and moving on with my day.  Maybe I’m just not the hardware type.

Hard working #selfie #me #work #office

A photo posted by Leonid Mamchenkov (@mamchenkov) on


WTF with Amazon and TCP

Here goes the story of me learning a few new swear words and pulling out nearly all my hair.  Grab a cup of coffee, this will take make a while to tell…

First of all, here is a diagram to make things a little bit more visual.


As you can see, we have an office network with NAT on the gateway.  We have an Amazon VPC with NAT on the bastion host.  And then there’s the rest of the Internet.

The setup is pretty straight forward.  There are no outgoing firewalls anywhere, no VLANs, no network equipment – all of the involved machines are a variety of Linux boxes.  The whole thing has been working fine for a while now.

A couple of weeks ago we had an issue with our ISP in the office.  The Internet connection was alive, but we were getting extremely high packet loss – around 80%.  The technician passed by, changed the cables, rebooted the ADSL modem, and we’ve also rebooted the gateway.  The problem was fixed, except for one annoying bit.  We could access all of the Internet just fine, except our Amazon VPC bastion host.  Here’s where it gets interesting.

Continue reading “WTF with Amazon and TCP”