Backend Architecture Decisions

Feature flags are a tool which allow code to be committed to a main branch and deployed at a later date. This is incredibly useful for software developers and engineers since they can see how code would act in a production environment, and they can have the feature flag lay dormant and simply toggle it on when they want users to see it.

Feature flags also help minimize risk. Teams can determine who has access to the flag and gradually roll it out to more users as it performs well. If there are any bugs, the feature toggle can be used as a kill switch to instantly roll it back.

With all the benefits of feature flags, they are a tool which makes software developers and engineers' lives much easier. DevCycle is a tool which helps manage teams feature flags to ensure they’re deployed with minimal risk, saving your engineers time. To make sure we were building the best possible feature flagging tool, our team knew we had to have a strong foundation, and that meant using the best backend architecture tools on the market.

There’s a lot that goes into making backend architecture decisions. Picking the right tools is critical to your success. When selecting which tools we were going to use for DevCycle, we made decisions based on the requirements demanded of publicly traded, fortune 100 companies that require solutions that focus on speed and scalability. 

To ensure the best backend architecture possible we started from scratch. Our entire dev team took their favorite features from what was currently available in the market as the starting point to build a feature flagging tool that we love and want to use for ourselves.

‍

Table of contents:

• Kubernetes vs. AWS Elastic Container Service (ECS)
• Atlantis vs. Terraform Cloud
• Mongo DB Atlas vs. Self Hosting
  

Kubernetes vs ECS

We chose Kubernetes (EKS) over ECS for several reasons, but the main one was due to its ability to scale up and scale down faster, making it a very effective method. Based on the amount of documentation and actual development being done on the underlying systems; Kubernetes is much more recent and updated than ECS. In addition, it allows developers to have a far less complex infrastructure setup, along with much more complete tooling.  

Kubernetes makes it easier to see and understand everything that's going on with all your deployments. In ECS we had several different services running in different places, but there was no way to see an overview of everything that was running and what was being deployed. 

In addition, Kubernetes has several third-party extensions you can add to further improve your experience. For example, our dev team is using Keel which watches for new versions of our services to be pushed and then deploys them automatically.  

Another reason to use Kubernetes is because it’s a system made up of several standards that can “run anywhere”. This means if we ever had to do on-prem again it would be easy to port our infrastructure to an on-prem Kubernetes cluster because it speaks the same language. It’s also possible to run a Kubernetes cluster on your own laptop if you want to mirror what was deployed onto a local machine.

Finally, Kubernetes is a more industry-standard system than ECS that exists on every cloud provider, not just AWS. Plus, it's easier these days to find developers who understand Kubernetes compared to ECS.

Atlantis vs. Terraform Cloud

Right before building our new feature flagging product, we began using Atlantis. Atlantis is a tool we used to abstract out the permissions required to apply our Terraform configurations to AWS and other places we use like Snowflake. Atlantis as a concept is what Terraform Cloud is by HashiCorp. To give some background, Atlantis came before Terraform Cloud because the people maintaining Atlantis were acquired and they now work for HashiCorp which makes Terraform. 

We were looking into different options of basically abstracting our Terraform, either using Terraform Cloud or Atlantis and we found Atlantis to be a better fit. It does everything we wanted Terraform Cloud to do with nothing we didn't want it to do.

MongoDB Atlas vs. Self Hosting

We decided to use MongoDB Atlas for our Mongo database instead of having to self-host it which allows anyone in the engineering department to apply a Terraform change that they need. 

Let’s say TSO wanted more perms to be able to run Athena queries. Normally that's something that would require a dev on the architecture team to do because it requires such high-level permission to apply that Terraform change. 

Now with Atlantis, all we have to do is have someone create a PR and from that PR they can specify the changes that are going to be made. From there, someone from the architecture team can approve it, and then they can apply it with the PR.

That doesn't require their user account specifically to have the same level of permissions than the architecture team. Previously, making a Terraform change would be a lengthy process. A user would have to check out the code from GitHub, make the change locally, make a PR, have someone approve it, merge it, plan the changes to Terraform, and send architecture the plan.

From there, someone from the architecture team would have to approve the plan. Then if it's something that requires an architecture dev to do it, the architecture dev would have to apply it themselves. Alternatively, if it's something that someone else can do, like a version bump to prod, then they would have to apply the plan and do this all locally.

Atlantis removes that humongous chain from our ecosystem and eliminates a lot of the back and forth between teams. This has saved us a lot of time and has taken a task that usually takes anywhere from 5 to 15 minutes and made it nearly automatic for us. In addition, it’s given the engineering team confidence around making infrastructure updates, without worrying about breaking things.

Our team spent a lot of time ensuring we’re using the right tools. Picking the right backend tools from the getgo can really help you build a great product while saving time, money and dev resources down the road.

‍