OpenTelemetry

An integration guide for Opentelemetry.
​OpenTelemetry is a vendor-agnostic, open-source, observability framework that allows the ingestion, transformation, and transport of data to an observability back-end. You can integrate it with various well-known systems, such as Prometheus, ElasticSearch, etc. [1]. 
The integration uses a so-called collector that has to be installed in your infrastructure to send data to the Rely platform.
Currently we only support a Prometheus-based integration for OpenTelemetry. This integration does not require you to install Prometheus, as it acts as a drop-in replacement for Prometheus. Further details can be found in our step-by-step guide.

Should you be interested in other integrations via OpenTelemetry, please reach out to us.
Why use OpenTelemetry?
OpenTelemetry is rapidly being adopted as a standard within the quickly evolving SRE space. By using OpenTelemetry you benefit from its large and active community, as well as being able to use community-vetted open source code for compliance and security reasons. Additionally, you can integrate it with a large range of systems, allowing you to simplify & standardize your infrastructure. Lastly, by using OpenTelemetry you can integrate with the Rely platform without giving us access to any of your systems, as would be the case for most other integrations.
This integration guide assumes you are already instrumenting your services with insightful metrics.
If you need support with this, OpenTelemetry provides clients in several programming languages that you could use [6]. In order to save time, you may also opt for any of their auto-instrumentation alternatives available for a wide variety of libraries [7].
Getting Started
The following guide describes how to integrate OpenTelemetry with the Rely platform by following these steps:
1.Configure the OpenTelemetry collector to connect to the Rely platform
2.Configure the OpenTelemetry collector to collect data from your systems
3.Deploy the OpenTelemetry collector in your infrastructure
If you are new to OpenTelemetry, we recommend fully configuring the configuration file before deploying the collector by following the step-by-step guide below.

If you already have OpenTelemetry installed and configured in your systems, simply adjust the configuration to integrate with the Rely platform and restart your OpenTelemetry deployment. You can skip all remaining steps.
The Rely platform requires you to use the OpenTelemetry contrib distribution since our systems are secured by the OAuth2 protocol which is not available in the regular distribution.
1. OpenTelemetry Collector basics
Every OpenTelemetry collector installation, regardless of deployment option is based on a configuration file. This configuration file has four configuration sections:
1.Receivers: This section specifies how you collect data from which of your systems.
2.Processors: This section specifies optional processing steps for the data. This is not required for the Rely platform integration.
3.Exporters: This section specifies where and how the data is exported to other systems, such as the Rely platform. Rely uses the HTTP exporter.
4.Extensions: This section allows the configuration of additional features unrelated to data processing. This allows the collector to securely connect to the Rely platform via the OAuth2 protocol.
Lastly, the service section specifies which of the above configurations are active.
Here is an example configuration of how this configuration file could look when integrating with Rely:
receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: "otel-collector"
          scrape_interval: 5s
          static_configs:
            - targets: ["localhost:8888"]
​
extensions:
  oauth2client:
    client_id: *****
    client_secret: *****
    endpoint_params:
      audience: *****
    token_url: *****
​
exporters:
  otlphttp:
    metrics_endpoint: *****
    auth:
      authenticator: oauth2client
​
service:
  extensions: [oauth2client]
  pipelines:
    metrics:
      exporters: [otlphttp]
2. Connect to the Rely platform
Open the Rely platform and start by navigating to the data-sources page. Click the "Add Data Source" button and select "OpenTelemetry".
This will prompt a modal to appear asking you for the information necessary to successfully integrate OpenTelemetry with your Rely.io account. 
Give an expressive name to your data source instance by filling in the “Collector Name” field. This is an unique name that will allow you to distinguish between multiple data source instances.
Once you assign the collector's name, go ahead and hit Submit. You should now be able to see your OpenTelemetry data colector instance under the Active section of the Data Sources page.
In order for Rely.io to gain access to your metric and label data, you must configure your OpenTelemetry collector to send your metrics to our data ingestion API. You may download an example snippet that provides you the necessary information to do so. Navigate to your collector's settings and hit "Download Configuration".
 This will download a YAML file that looks somewhat like the following.
extensions:
  oauth2client:
    client_id: *****
    client_secret: *****
    endpoint_params:
      audience: *****
    token_url: *****
​
exporters:
  otlphttp:
    metrics_endpoint: *****
    auth:
      authenticator: oauth2client
​
service:
  extensions: [oauth2client]
  pipelines:
    metrics:
      exporters: [otlphttp]
The configuration is split into three parts, as described in the OpenTelemetry Basics section:
Authentication: This defines the credentials in order for your OpenTelemetry collector to authenticate itself with our data-ingestion API. This is done using the oauth2client extension supported by the OpenTelemetry-contrib distribution [2, 3].
Exporter: This defines the HTTP exporter that will be responsible for sending your data to us.
Service: This adds the previously defined exporter to your metrics's pipeline.
3. Connect to your systems
In this step we will add the receivers section to the configuration file. This step is different for the system you would like to integrate with. Currently, we only support a Prometheus-based integration. 
The OTel Prometheus Receiver is a drop-in replacement for Prometheus, with a few limitations. This means you can reuse your Prometheus scrape_config.
To integrate with your systems, you need to add the receivers section to the config file from the previous step. For further details, see the excellent OTel Prometheus receiver Readme.
A simple example is depicted below. Prometheus provides many examples on their Github space that you can use to get started.
receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: "otel-collector"
          scrape_interval: 5s
          static_configs:
            - targets: ["localhost:8888"]
4. Deploy the Collector
Now that you have fully configured the OpenTelemetry config file, it is time to deploy the system. As stated in the documents [4], the OpenTelemetry collector provides a single binary and two deployment methods:
Agent: A Collector instance running with the application or on the same host as the application (e.g. binary, sidecar, or daemonset).
Gateway: One or more Collector instances running as a standalone service (e.g. container or deployment) typically per cluster, data center or region.
We recommend using the Gateway approach to limit the number of integrations with the Rely platform.
There are several options to install OpenTelemetry in your infrastructure. Below you will find instructions for a Kubernetes and a standalone Docker image deployment. Further options can be found in the official OpenTelemetry documentation.
Kubernetes Deployment
If you are using Kubernetes, we recommend using the OpenTelemetry Collector Helm Chart. 
This Helm chart uses the contrib image that is required by Rely by default.
A simple configuration that will get you started is shown below. For advanced users, there are many more configuration options available as described in the Helm chart repository. They also provide many configuration examples.
mode: deployment
​
# Optional resource adjustments
resources:
  limits:
    cpu: 2
    memory: 4Gi
​
# replace everything below config with the contents of your config.yaml
# we configured in the previous steps
config:
  receivers:
    prometheus:
      config:
        scrape_configs:
          - job_name: "otel-collector"
            scrape_interval: 5s
            static_configs:
              - targets: ["localhost:8888"]
​
  extensions:
    oauth2client:
      client_id: *****
      client_secret: *****
      endpoint_params:
        audience: *****
      token_url: *****
​
  exporters:
    otlphttp:
      metrics_endpoint: *****
      auth:
        authenticator: oauth2client
​
  service:
    extensions: [oauth2client]
    pipelines:
      metrics:
        exporters: [otlphttp]
Standalone Docker Deployment
The exact steps depend on the specific type of Docker deployment you are using. Most cloud providers give you an easy way to deploy Docker images with custom files. 
In general you want to run the OTel Collector contrib image and point it at the configuration file we configured in the previous steps.
For example, with plain Docker this could look as follows:
1.Run docker pull otel/opentelemetry-collector-contrib
2.Place/mount the config.yaml file under /etc/otel-collector-config.yaml 
3.Run the docker image with
docker run otel/opentelemetry-collector-contrib --config=/etc/otel-collector-config.yaml 
Prometheus
Next - Service Level Objectives
Concepts and Definitions
Last modified 2mo ago