Two years at Qobo

Today marks the completion of my second year at Qobo Ltd.  The first year was quite a ride.  But the second one was even wilder.  As always, it’s difficult (and lengthy) to mention everything that happened.  A lot of that stuff is under the non-disclosure agreement (NDA) terms too.  But here are a few generic highlights:

  • Vision and strategy – most of my first year has been spent in putting out fires, fixing things big and small, left, right, and center.  The technology boost was necessary across the board, so it didn’t leave much time for the vision and strategy.  I feel that we’ve made a huge progress in this area in the last 12 month.  We have a clear vision.  We have all the stakeholders agreeing on all key elements.  We have worked out a strategy on how to move forward.  And we’ve started implementing this strategy (hey, Qobrix!).  In terms of achievements, I think this was the most important area and I am pretty happy with how things are shaping up.
  • Team changes – much like in the first year, we had quite a few changes in the team.  Some of them were unfortunate, others not so much.  The team is still smaller than what we want and need, but I think we are making progress here.  If our World Domination plans will work out to even some degree, we’ll be in a much better place very soon.
  • Technology focus – we’ve continued with our goal of doing fewer things but doing them better.  Our expertise in WordPress, CakePHP and SugarCRM grew a lot.  We’ve signed and deployed a variety of projects, which resulted in more in-depth knowledge, more networking with people around each technology, more tools and practices that we can reuse in our future work.
  • Open Source Softwareour GitHub profile is growing, with more repositories, pull requests, releases, features, and bug fixes.  We’ve also contributed to a variety of Open Source projects.  Our involvement with Open Source Software will continue to grow – that’s one of those things that I am absolutely sure about.
  • Hosting, continuous integration and delivery (CI/CD), and quality assurance – again, the trend continued this year.  We are using (and understanding) more of the cloud infrastructure in general and Amazon AWS in particular.  We have a much better Zabbix setup.  And our love and appreciation of Ansible grows steeply. Let’s Encrypt is in use, but we’ll grow it to cover all our projects soon.  We are also experimenting with a variety of quality assurance tools.  We are using TravisCI for most of our Open Source work.  And we are on the brink of using recently announced BitBucket Pipelines for our private repositories (sorry Jenkins, we’ve tried you, but … not yet).  We’ve also jumped into ChatOps world with HipChat and its integrations, to the point that it’s difficult to imagine how could we have worked without it just a few month ago. has also proved to be useful.
  • Projects, projects, projects – much like the previous year, we’ve completed a whole lot of projects (see some of our clients).  Some were simple and straightforward.  Others were complicated and challenging. And we have more of these in the pipelines.  Overall, we’ve learned how to do more with less.  Our productivity, technical expertise, and confidence grows day-to-day.  I hope we keep it up for years to come.
  • Website – one thing that we wanted to do for ages is to update our website.  Which we did, despite all the crazy things going on.  It’s not a complete redesign, but it’s a nice refreshment.  And we’ve also got our blog section, which I promised you last year.  All we need to do now is to use it more. ;)

There are a couple of major updates coming soon, but I am not at liberty to share them right now.  But they are very, very exciting – that’s all I can say today.  Keep an eye our blog – we’ll be definitely sharing.

As I said, it was quite an intense year, with lots of things going on everywhere.  There were tough times, and there were easy times.  There were challenges and there were accomplishments.  There were successes, and there were mistakes and failures.  But I wouldn’t have it any other way!

After two years, I am still excited about this company and about my job here.  (Which, looking at my career so far, is not something that happens often.)  I hope the next year will continue the adventure and by the end of it I’ll be able to proudly show you a few more things.


Upgrading Amazon EC2 instance type

By now everybody knows that one of the major benefits to using cloud services rather than hosting on your own hardware is the ease to scale quickly.  Many web applications and large companies benefit from this, but what about smaller customers?  How about a single server?

Well, today one of our web servers was experiencing some pick loads.  It hosts a whole array of small websites built with WordPress, CakePHP, and other popular tools.  There was no time to update all these projects to work with multiple web servers.  And even redeploying them to multiple individual servers would have taken a few hours.  Instead, we’ve decided to upgrade the server hardware.

Pause for a second and imagine the situation with your own server.  Or a dedicated hosting account for that matter.  So much to configure.  So much to backup and restore.  So much to test.

Here’s how to do it, if your projects are on the Amazon EC2 instance (our was also inside a virtual private cloud (VPC), but even if it wasn’t, the difference would be insignificant):

  1. Login to the Amazon AWS console.
  2. Navigate to the Amazon EC2 section.
  3. Click on Instances in the left sidebar.
  4. Click on the instance that you want to upgrade in the list of your instances.
  5. Click Actions -> Instance State -> Stop.
  6. Wait a few seconds for the instance to stop.  You can use the Refresh button to update the list.
  7. (While your instance is still selected in the list of instances:) Click Actions -> Instance Settings -> Change Instance Type.
  8. In the popup window that appeared, select an Instance Type that you want.
  9. Click Apply.
  10. Click Actions -> Instance State -> Start.
  11. Wait a few seconds for the instance to start.
  12. Enjoy!

The whole process literally takes under two minutes.  You get exactly the same configuration – hostname, IP addresses (both internal and external), mounted EBS volumes, all your OS configuration, etc.  It’s practically a reboot of your machine. But into a different hardware configuration (CPU/RAM).

Coincidentally, earlier this morning I had to pack up a rack-mountable server – screws, cables, dusty boxes, the whole shebang.  It’s been a while since I’ve done that last time.

Another day in the office #work #cables #sysadmin

A photo posted by Leonid Mamchenkov (@mamchenkov) on

But I can tell you that I much prefer clicking a few buttons and moving on with my day.  Maybe I’m just not the hardware type.

Hard working #selfie #me #work #office

A photo posted by Leonid Mamchenkov (@mamchenkov) on


WTF with Amazon and TCP

Here goes the story of me learning a few new swear words and pulling out nearly all my hair.  Grab a cup of coffee, this will take make a while to tell…

First of all, here is a diagram to make things a little bit more visual.


As you can see, we have an office network with NAT on the gateway.  We have an Amazon VPC with NAT on the bastion host.  And then there’s the rest of the Internet.

The setup is pretty straight forward.  There are no outgoing firewalls anywhere, no VLANs, no network equipment – all of the involved machines are a variety of Linux boxes.  The whole thing has been working fine for a while now.

A couple of weeks ago we had an issue with our ISP in the office.  The Internet connection was alive, but we were getting extremely high packet loss – around 80%.  The technician passed by, changed the cables, rebooted the ADSL modem, and we’ve also rebooted the gateway.  The problem was fixed, except for one annoying bit.  We could access all of the Internet just fine, except our Amazon VPC bastion host.  Here’s where it gets interesting.

Continue reading “WTF with Amazon and TCP”

One year at Qobo

Today marks a year since I joined Qobo as a CTO.  And what a year it has been!  On one hand, it flew by like a week.  On the other hand, looking at how much has been done, it feels like a decade.  Here are just a few highlights, in no particular order:

  • Team changes – a few people left, a few people joined.  A gadzillion people were interviewed, met, and even worked with on a project basis.  Those who are in the office now have also grown and got better both individually and as a team.  I’m always excited to see change in people.
  • Technology focus – a variety of technologies from earlier company history were consolidated into fewer, but better solutions.  We are a PHP shop now, with solid expertise in WordPress, CakePHP, and SugarCRM.  Focusing on fewer technologies allowed us to gain deeper knowledge and to perfect our tools.  Which leads me to the next point …
  • Open Source Software.  I’ve been a fan and an advocate for Open Source Software for years.  But it was always difficult to push it in the corporate world.  I got better with it at each previous work place, and Open Source grew up with years too.  But it wasn’t until Qobo that I got to the level that I wanted.  Have a look at our GitHub profile – there are forks of projects that we contribute to (mostly third-party WordPress plugins), our project templates, experiments, and tools.  And we are far from done yet.
  • Hosting consolidation and the cloud.  When I joined a year ago, the company was using a variety of hosting companies around the world.  While that provided an immediate base for some of the platforms in use, it was somewhat difficult to manage.  Since then, we’ve moved everything to Amazon Virtual Private Cloud.  We use quite a bit of the AWS stack, mainly – VPC, EC2, Route53, and S3.  And it works great for us!
  • Deployment automation, quality assurance, monitoring, etc.  We’ve built (yet again) and (finally) Open Sourced our deployment automation tools.   I’ve built this kind of tools before, but I was never quite happy with them, and I had to reiterate again and again.  Finally, we have something that works quite well.  There’s always room for improvement, of course, but it served us well for a few month now.
  • Projects, projects, projects.  We’ve worked on client projects big and small all year round.  And there are quite a few that we are proud of.  The Portfolio page for our website is coming soon too, so we can properly showcase those.
  • Office move.  We’ve moved offices in the shortest possible time,  optimizing and completely restructuring our internal infrastructure.  We are now in better premises with a better setup.  We need less hardware and things are much easier to handle.
  • Podcasts and audio books.  In the last year, I’ve driven over 40,000 kilometers, mostly going back and forth between Limassol and Nicosia.  Most of this time was spent listening to podcasts and audio books, from which I’ve learned a lot and got a billion ideas.

There’s plenty more, of course.  But the most important out of this all, I think, is that I’ve learned a whole lot about a whole lot, I’ve tried plenty of things I wanted to try for a while, and I’ve had a tonne of fun, even though sometimes that meant long nights and no weekends.

It’s been a great year and I hope I’ll have many more like this one.  To all, who were around – a big thank you and please don’t go away.  To those who parted ways – thanks anyway and good luck.  I hope more people get to experience what I’ve experienced this year.

P.S.: And the blog section of our website is coming soon too.  Once it’s up, we’ll keep you all updated on our endeavors through there.

Mr.Meat – meat online shop for Nicosia, Cyprus

Mr.Meat is the first project going live since I joined Qobo.  I’ve only had a minor role in this one, but it’s still delicious.  There aren’t many online shops in Cyprus, and even fewer that handle the delivery, so this new addition is very welcome, even if they only cover Nicosia delivery for now.  The thing that I love the most about this site is the photography – the images are just salivating!

Goodbye Easy Forex, hello Qobo

I’ve spent the last two and a half years working for Easy Forex.  I went from a consultant through senior web developer, team leader to the director of web development.  I’ve worked on a variety of projects and managed several teams.  I’ve had great fun and I’ve learned a lot.  (Thank you all!  You guys are awesome!)

But the time has come to make a change.  Today is my last day at Easy Forex.  Tomorrow is the first day of my new adventure – Qobo, where I will assume the position of the Chief Technical Officer.  Qobo is a Nicosia-based company that develops mostly web-based software for the enterprise needs.

I worked in this industry before, and I think now is a good time for me to return.