How to Use Datadog for Monitoring

Adam Wootton

7/16/2023

A key component to any engineering team is having a strong monitoring system. Monitoring is a way to track and monitor for bugs or potential errors. When building DevCycle we needed to have an extensive monitoring system that would allow us to easily track bugs. We also needed that system to keep track of feature flags being turned on and off, so we could correlate that with the things we’re monitoring.

The main tool we used is Datadog which allowed the engineering team to see a compiled list of all the errors that happen in one place rather than having to switch between multiple tools to track them down.

How do we use DataDog with DevCycle?

Prior to building DevCycle our engineering team had only used a few of Datadog’s functionalities. However, now we use a large portion of their featureset for better and easier monitoring. Datadog is easy to use and provides useful features, like collecting a list of all your errors so you can easily track them back to when a feature flag was turned on.

When we were picking what tool to use we wanted to make sure we used something that allowed us to have everything in one place. Some examples of things we are tracking with Datadog are infrastructure metrics, frontend user sessions, errors, incoming requests, feature flags being turned on and off, and new versions of code being deployed. We also set up monitors that combine all of this information to make it easier to track.

One of the best parts of Datadog is their Watchdog feature that lets us know if any abnormalities were observed in our systems. Say things are taking longer to load than usual; Watchdog lets us know about it, which allows us to look into what’s causing things to be slow. This way you have the opportunity to catch something that could be an error before it becomes an issue.

Tips for Using Datadog

1️⃣ Use unified tagging

When you’re monitoring, make sure to use unified tagging so your metrics are correlated across different sources. When something goes wrong, you can easily view all metrics that may be related to the issue.

2️⃣ Connect Watchdog to your alerting platform

Ensure your new Watchdog alerts are connected to an alerting platform like Pagerduty or Slack so you can easily get updates on your alerts as they happen. This allows you to react to them in a timely manner and allows you to keep Watchdog automated alerts organized alongside your other system alerts.

3️⃣ Set up a dashboard

When using Datadog, consider setting up a dashboard with key metrics you want to monitor every time a new feature is released. This is an easy way to see if all the key metrics you’re looking for are still normal after turning something on. Examples of things to monitor include your error rates, CPU usage, and HTTP request latency.

How feature flags help with monitoring

Feature flags make monitoring easier since turning one on or off is a noticeable event that happens frequently. This makes it easy to track an error back to a specific flag and the code paths controlled by it. If our team wasn't using feature flags, it would be much harder to determine which change caused an issue.

Having good monitoring in place means that users will have a better experience. It allows us to prevent them from seeing errors and have a faster resolution time. Using Watchdog provides us the opportunity to fix a potential error before it even happens. If there’s an outlier in performance, we can look into it and catch a tiny error before it becomes a big problem.

How feature flags save time

Feature flags save developers time by allowing for continuous deployment. With feature flags, unfinished features can safely be deployed to production which minimizes the amount of errors and bugs that will arise in the future. As a result, releasing new code is no longer a big event. Engineers can slowly roll out a new feature to small groups when ready, and watch Datadog to spot any potential issues. If Datadog detects a problem, it’s simple to rollback the feature without redeploying.

Getting started with feature flags

Using feature flags allows teams to reach achievable milestones for their project. With feature flags, you start with an MVP, then implement a version two with a bunch of improvements. Each stage can have its own flag which you can turn on before moving to the next phase. This allows you to break down a larger project into digestible parts, and lets you celebrate with your team for reaching each milestone. It also helps with monitoring since you can monitor each small release and quickly catch errors as they’re added to your main branch.

Breaking projects down into smaller digestible parts, or using feature flags for just one isolated project at first, is a great way to get your team comfortable with using feature flags and allows you to see just how valuable they can be for your workflow.