Scalable DevOps – It is missing

I follow a few people (online, don’t worry) and read quite the blog-posts about all new fancy things related to IT, development and DevOps. I read about CI, cloud, AWS and how the “big guys” are doing it. Awesome, and it looks great. To be honest there are some discussions on what path you should take, for instance should you use CI , AWS or go for Docker etc. All fair but in the end the solution is solid nonetheless. Why? Because the tools support your application 100% in every way.

Here is my problem; I do not run 1 single application. I run 300. All with their own configurations, demands, requirements and the need to keep up with today’s technology.

DevOps tools suck when you need to scale

There, I said it.

And for the record; I do not mean to scale the application, but to scale the DevOps tools.

Real life example; and I do not say this is the core of devops or automatization but take a look at DeployHQ:  Do not get me wrong, I kinda like these guys but if you look at the package you’ll get my point.

Ultimate+  200 projects

So, you take the Awesome “Ultimate PLUS” package and you are limited to 200 projects.

and Bamboo is the same..

It is only recent they have added a new feature: which might make things a bit more easy.

Configuration as code is now available in Bamboo! You can start storing your build plan configuration as code for easier automation, change tracking, validation, and much more. Read all about other benefits of using configuration as code in Bamboo Specs. We’ve also prepared a tutorial that will help you create a simple plan in no time. For behind-the-scenes information about the Bamboo Specs library, see our detailed reference documentation.

I mean great, I can extend my current “plan” or project with as much build plans I want. I can use branches and even those can be integrated easy into your current project. The issue is that I want all the power of Bamboo for 300 projects. I want to make a base-plan, a configuration file and integrate all my projects into a great build & deploy plan with each project following my “build plan”.  Yet each project should have different options. For instance some might need different resources, some extra post-deploy scripts, some builds needs to be pushed over 3 servers, others only on 1 or 2. It is NOT possible to automate this.

For some reason it is utterly weird to have 300+ projects, and you end up in some grindfest to actually setup your projects, rather than that the tools are helping you with everything.

Plus a webserver is not easy

Yes, there are many Ansible scripts out there that can easy setup a semi-decent webserver with all the basic stuff in it. The point is, “semi-decent” is not good enough when you are working in a professional team and that many websites. The chance that something happens to your website is maybe a few percent. I mean; bad performance, mysql or php crashes/failures,  bad configuration, not handling well under somewhat load from a crawler, etc.  It is “ok” if you run that hobby website, not ok when 300+ servers are not performing and the setup takes ages. You are simply not done with a simple script that installs  apache2, PHP & MySQL and places a vhost with a single domain.

So what we do?

Well, we (the company where I work) invested quite some time and energy to migrate from a fixed managed hosting to our own in-house cloud-based solution based on Ansible for the servers. A mix on deployments depending on the project (so either deploying via ssh, or building via bamboo).

The core is an Ansible setup that consists over 15.000 lines of code.

We can chose different roles on our needs. We still have legacy projects, and we also have newer projects. Therefor the sole configuration is mental:

  • Ubuntu 14, 16
  • PHP 5.x, 7.x
  • Apache, Nginx
  • MySQL, PostGre
  • SSL or not
  • Multiple domains

Configurations for

  • fail2ban based on the above settings
  • logrotate
  • Nagios setup including sending the server info to the nagios server
  • PHP/Nginx/MySQL based on the server
  • SSH-keys (for devs, deployments)
  • Test-acceptance and production configurations

It does not end because some projects require

  • standalone database server
  • special frontend tools (gulp/bower etc.)
  • nginx loadbalancer

and I actually can keep on going.

So what is the problem?

The problem is that we actually had to make this ourself. Really, if there is a good solution I will never re-invent the wheel.

In all the technology of today, it is to hard to get a decent setup for your projects. When you run a single application you only have to make a decent structure and you can keep building, deploying and whatnot. It is a one-time thing and for the rest you do not have to worry about it anymore. It does not work that way for us.

If you look at all the possible configurations, it is just to much to keep up with it for a relatively small company. I only covered a small part of the building/deploying but essentially it is the same as the server part.

Why is there not a company that focus his energy in creating stable, manageable, configurable and automated servers. Why aren’t tools also including scaling on a different level: masses? I would like to focus on the quality, the latest techniques. Yet all the time I’m wasting my time on making considerations on what part of a tool I can actually use and combining those to actually get some half decent A to Z product.

Possible solution?

So we have big companies giving resources;  AWS, openstack etc. etc. They just provide the infrastructure and that is it. On the other hand we have quite expensive managed hosting solutions. So either you pay little to nothing for server power and endless options. The other option is to pay a shitload for a fixed server with hardly options at all.

I need the middle.

I’m going to be greedy, but I also need all current tools to actually give me the option / possibility to use it in mass. I need those to work with hundreds of projects in a hour. Not 3 weeks of migrations and fixing configurations to just be compatible with the utter misery of said tool.

I guess this blogpost ended in a somewhat rant. I actually hope people could share their experiences with me and their problems & solutions.

Leave a Reply

Your email address will not be published. Required fields are marked *