deploy v0.0.2, see CHANGELOG

Author: thomasdesplaces

CHANGELOG.md

@@ -0,0 +1,16 @@
+0.0.2 - 2022-12-15
+===================
+
+### Features
+- Add [Application stack](./generate_data_application/).  
+- Add CHANGELOG.md.  
+- Add MAINTAINERS.md.  
+
+### Updating
+- Update README with the architecture of application stack and a new source.  
+- Update [agent.yaml](./agent/agent.yaml) for the application stack.  
+- Change log level for Grafana in [config.ini](./grafana/config.ini).  
+- Update [Loki config](./loki/loki-config.yaml) for the application stack.  
+- Update log level for Mimir in [mimir.yaml](./mimir/mimir.yaml).  
+- Update log level for NGinx in [nginx.conf](./nginx/nginx.conf).  
+- Update log level for Tempo in [tempo.yaml](./tempo/tempo.yaml).  

MAINTAINERS.md

@@ -0,0 +1 @@
+- [@tonyglandyl28](https://github.com/tonyglandyl28)

README.md

@@ -2,8 +2,10 @@
     <img src="./assets/observability.png" alt="Observability" width="500"/>
 </p>
 
+![alt version](https://img.shields.io/badge/Version-0.0.2-violet)
 ![alt production](https://img.shields.io/badge/Production--Ready-No-green)
 ![alt docker](https://img.shields.io/badge/Deployment_tool-Docker--Compose-orange)
+![alt ARM](https://img.shields.io/badge/Platform-ARM--M1-red)
 
 # Why
 
@@ -15,12 +17,20 @@ This for testing purpose.
 
 # Architecture
 
-Mimir, Loki and Tempo are deploy in Monolithic mode with 3 instances of each.  
+The observability stack is composed by Mimir, Loki and Tempo which are deployed in Monolithic mode with 3 instances of each.  
+The application stack is composed by a FastAPI application with some routes and PostgreSQL database with 1 table.  
+Logs are written in a log file, scrape by [Grafana Promtail](https://grafana.com/docs/loki/latest/clients/promtail/) and send to Grafana Agent.  
+Metrics are exposed via [Prometheus Client](https://github.com/prometheus/client_python) and scraped by Grafana Agent.  
+Traces are sent via [Opentelemetry SDK](https://github.com/open-telemetry/opentelemetry-python) to Grafana Agent.  
+Unit tests are performed by [k6.io](https://k6.io).
 
 ![alt schema](./assets/Observability-Grafana_Stack.png)
 
 # Prerequisite
 
+This repository is run on Mac M1 (ARM architecture).  
+If you want to use another architecture, be sure to modify the Dockerfiles which import arm binary (ex. promtail on the app Dockerfile).  
+
 1. Install [Docker](https://docs.docker.com/engine/install/).  
 2. Create Docker network :  
 `docker network create observability-network`
@@ -29,10 +39,12 @@ Mimir, Loki and Tempo are deploy in Monolithic mode with 3 instances of each.
 
 1. Clone this repository :  
 `git clone https://github.com/tonyglandyl28/observability_docker.git`  
-2. Build & deploy :  
+2. Build & deploy observability stack :  
 `docker-compose --profile grafana up --build`  
 3. Access to Grafana for visualization :  
 *http://localhost:3000*
+4. Build & deploy application stack :  
+`docker-compose --profile application up --build`  
 
 # How to send logs/metrics/traces
 
@@ -73,9 +85,9 @@ Grafana --> Nginx : 4702 --> Mimir : 3703
 # TODO
 
 - Tests Prometheus Exemplars.
-- Add comments in YAML & Docker files.
+- ~~Add comments in YAML & Docker files.~~ (v0.0.2)
 - Add Alert Manager.
-- Add a data generator (logs, metrics, traces) - a React frontend with FastApi & PostgreSQL backend.
+- Add a data generator (logs, metrics, traces) - a React frontend ~~with FastApi & PostgreSQL backend~~. v(0.0.2)
 - Create dashboards for these application stack.
 - Configure all components for distributed installation.
 
@@ -89,4 +101,5 @@ Grafana --> Nginx : 4702 --> Mimir : 3703
 | [<img src="./assets/loki.png" alt="Grafana Loki" width="200"/>](https://grafana.com/docs/loki/latest/) | v2.7.0 | Used to store Logs (like Elasticsearch). Based on https://github.com/grafana/loki/tree/main/examples/getting-started |
 | [<img src="./assets/grafana.png" alt="Grafana" width="200"/>](https://grafana.com/docs/grafana/latest/) | v9.3.1 | Used to visualize data (like Kibana). Based on https://github.com/grafana/grafana |
 | [<img src="./assets/minio.png" alt="Minio" width="200"/>](https://min.io) | Latest | Used to load balance traffic between each instance (on Cloud, use S3, Google Cloud Storage or similar). Based on https://github.com/minio/minio |
-| [<img src="./assets/nginx.png" alt="NGinx" width="200"/>](https://www.nginx.com/) | Latest | Used to load balance traffic between each instance. Based on https://github.com/nginx/nginx |
+| [<img src="./assets/nginx.png" alt="NGinx" width="200"/>](https://www.nginx.com/) | Latest | Used to load balance traffic between each instance. Based on https://github.com/nginx/nginx |
+| [<img src="./assets/blueswen.png" alt="Blueswen" width="200"/>](https://github.com/Blueswen/fastapi-observability) | Latest | Used to configure FastAPI application. Thanks to [@Blueswen](https://github.com/blueswen) |

agent/agent.yaml

@@ -1,38 +1,43 @@
 server:
-  log_level: "debug"
-  log_format: "logfmt"
+  log_level: debug
+  log_format: logfmt
 
 metrics:
   global:
     scrape_interval: 5s
     external_labels:
-      cluster: demo
-      namespace: demo
+      cluster: test
+      namespace: monitoring
     remote_write:
       - url: http://nginx:3702/api/v1/push
         send_exemplars: true
         # Add X-Scope-OrgID header so that Mimir knows what tenant the remote write data should be stored in.
-        # In this case, our tenant is "myapp"
+        # In this case, our tenant is "backend"
         headers:
-          X-Scope-OrgID: myapp
+          X-Scope-OrgID: backend
   wal_directory: "/tmp/agent/prom"
   configs:
-    - name: "myapp-metrics"
+    - name: "backend-metrics"
       scrape_configs:
-        - job_name: "agentprometheus"
+        - job_name: "backend_fastapi"
           static_configs:
-          - targets: ['api:8000']
+          - targets: ['backend:5050']
+    # - name: "myapp-metrics"
+    #   scrape_configs:
+    #     - job_name: "backend_flask"
+    #       static_configs:
+    #       - targets: ['api:8000']
 
 logs:
   configs:
-    - name: "myapp-logs"
+    - name: "backend-logs"
       clients:
         - url: http://nginx:3502/loki/api/v1/push
       positions: 
         filename: "/tmp/agent/loki.yaml"
       scrape_configs:
       # This value should be a problem --> https://github.com/grafana/loki/issues/7275
-        - job_name: "agentloki"
+        - job_name: "backend_fastapi"
           loki_push_api:
             server:
               http_listen_port: 3501
@@ -41,12 +46,12 @@ logs:
 
 traces:
   configs:
-  - name: "myapp-traces"
+  - name: "backend-traces"
     attributes:
       actions:
         - action: insert
           key: tempo.attribute.labels
-          value: "grafana_agent"
+          value: "backend_fastapi"
     receivers:
       otlp:
         protocols:

assets/blueswen.png

@@ -74,6 +74,7 @@ services:
       - "./grafana/dashboard.yaml:/etc/grafana/provisioning/dashboards/default.yaml"
       - "./grafana/catalogue.json:/var/lib/grafana/dashboards/catalogue.json"
       - "./grafana/container_process.json:/var/lib/grafana/dashboards/container_process.json"
+      - "./grafana/config.ini:/etc/grafana/grafana.ini"
     labels:
       namespace: monitoring
     environment:
@@ -284,3 +285,63 @@ services:
   ######################################################################################
   ######################################################################################
   # TODO Add Alert Manager
+
+  ######################################################################################
+  ######################################################################################
+  ############################## BACKEND APPLICATION ###################################
+  ######################################################################################
+  ######################################################################################
+  backend:
+    profiles:
+      - application
+    container_name: backend
+    image: backend
+    build:
+      context: generate_data_application/backend
+      dockerfile: Dockerfile
+    ports:
+      - 5050:5050
+      - 8000:8000
+    depends_on:
+      - db
+    labels:
+      namespace: monitoring
+  
+  ######################################################################################
+  ######################################################################################
+  ############################## DATABASE APPLICATION ##################################
+  ######################################################################################
+  ######################################################################################
+  db:
+    profiles:
+      - application
+    container_name: db
+    image: db
+    build:
+      context: generate_data_application/db
+      dockerfile: Dockerfile
+    volumes:
+      - ./generate_data_application/db/postgres.conf:/etc/postgresql/postgresql.conf
+    command: postgres -c config_file=/etc/postgresql/postgresql.conf
+    ports:
+      - 5432:5432
+    labels:
+      namespace: monitoring
+  
+  ######################################################################################
+  ######################################################################################
+  ################################### Unit tests #######################################
+  ######################################################################################
+  ######################################################################################
+  k6:
+    profiles:
+      - application
+    container_name: k6
+    image: k6
+    build:
+      context: generate_data_application/load_tests
+      dockerfile: Dockerfile
+    depends_on:
+      - backend
+    labels:
+      namespace: monitoring

docker-compose.yaml

@@ -0,0 +1,24 @@
+FROM python:bullseye
+RUN apt-get update && apt-get install --no-install-recommends -y systemctl=1.4.4181-1.1 \
+    && apt-get clean \
+    && rm -rf /var/lib/apt/lists/*
+WORKDIR /usr/local/bin/
+SHELL ["/bin/bash", "-eo", "pipefail", "-c"]
+
+# Configure Promtail to send logs to Grafana Agent
+RUN curl -s -q https://api.github.com/repos/grafana/loki/releases/latest | grep browser_download_url |  cut -d '"' -f 4 | grep promtail-linux-arm64.zip | wget -i - && \
+    unzip promtail-linux-arm64.zip && \
+    mv ./promtail-linux-arm64 ./promtail
+COPY config/promtail-config.yaml /etc/promtail/
+RUN useradd --system promtail
+COPY config/promtail.service /etc/systemd/system/
+
+# Configure FastAPI application
+WORKDIR /usr/src/app
+RUN chmod a+rwx /usr/src/app
+COPY config/requirements.txt ./
+RUN pip install --no-cache-dir -r requirements.txt
+COPY app .
+COPY config/start.sh ./config/
+EXPOSE 5050
+CMD ["/bin/sh", "./config/start.sh"]

generate_data_application/backend/Dockerfile

@@ -0,0 +1,125 @@
+"""
+Function to define FastAPI routes
+"""
+
+from fastapi import APIRouter, Depends, HTTPException
+from sqlalchemy.orm import Session
+from sqlalchemy.exc import OperationalError
+from schemas import ClientsBase, Clients
+from responsesSettings import responsesMessages
+import connection, crud_clients
+from observability import LOGGER, tracer
+
+traces = tracer.get_tracer(__name__)
+
+LOGGER = LOGGER.getChild(__name__)
+
+router = APIRouter(
+    tags=["clients"],
+    responses={
+        400: responsesMessages['400'],
+        403: responsesMessages['403'],
+        404: responsesMessages['404'],
+        405: responsesMessages['405'],
+        422: responsesMessages['422'],
+        500: responsesMessages['500']
+    }
+)
+
+def get_db():
+    """
+    Database connexion
+    """
+    db = connection.SessionLocal()
+    try:
+        LOGGER.debug("Open database connexion")
+        yield db
+    finally:
+        LOGGER.debug("Close database connexion")
+        db.close()
+
+@router.post(path="/clients",
+    response_model=Clients, 
+    status_code=201,
+    summary="Client creation"
+)
+async def post_client(client: ClientsBase, db: Session = Depends(get_db)):
+    """
+    Create client with all parameters :
+
+    - **firstname**: Firstname of the client
+    - **lastname**: Lastname of the client
+    """
+    with traces.start_as_current_span("Router GET client by ID"):
+        LOGGER.info("Start POST Client")
+        new_client = crud_clients.create(db=db, client=client)
+        return new_client
+
+@router.get(path="/clients",
+    response_model=list[Clients],
+    status_code=200,
+    summary="List clients"
+)
+async def get_clients(db: Session = Depends(get_db)):
+    """
+    List all clients
+    """
+    with traces.start_as_current_span("Router GET clients List"):
+        LOGGER.info("Start GET Clients List")
+        try:
+            list_clients = crud_clients.get_list(db=db)
+            return list_clients
+        except OperationalError as error:
+            LOGGER.error(error)
+            return ""
+
+@router.get(path="/clients/{client_id}",
+    response_model=Clients,
+    status_code=200,
+    summary="Details for specific client")
+async def get_client(client_id: str, db: Session = Depends(get_db)):
+    """
+    Details for specific client
+    """
+    with traces.start_as_current_span("Router GET client by ID"):
+        LOGGER.info("Start GET Client by ID")
+        client = crud_clients.get_client(db=db, client_id=client_id)
+        LOGGER.debug("Return client : %s" % str(client))
+        return client
+
+@router.put(path="/clients/{client_id}",
+    response_model=Clients,
+    status_code=200,
+    summary="Modify details for a specific client")
+async def put_client(client_id: str, client: ClientsBase, db: Session = Depends(get_db)):
+    """
+    Modify details for a specific client :
+
+    - **firstname**: Firstname of the client
+    - **lastname**: Lastname of the client
+    """
+    with traces.start_as_current_span("Router Modify clients by ID"):
+        LOGGER.info("Start PUT Client by ID")
+        try:
+            modify_client = crud_clients.put(db=db, client_id=client_id, client=client)
+            return modify_client
+        except OperationalError as error:
+            LOGGER.error(error)
+            return ""
+
+@router.delete(path="/clients/{client_id}",
+    response_model=None, 
+    status_code=204,
+    summary="Delete a specific client"
+)
+async def delete_client(client_id: str, db: Session = Depends(get_db)):
+    """
+    Delete a specific client
+    """
+    with traces.start_as_current_span("Router DELETE clients by ID"):
+        LOGGER.info("Start DELETE Client by ID")
+        delete_client = crud_clients.get_client(db=db, client_id=client_id)
+        if delete_client is None:
+            raise HTTPException(status_code=405)
+        crud_clients.delete(db=db, client_id=client_id)
+        return None