日志收集 | Orange's

日志处理的组件

收集
存储
检索

日志收集的组件

LogStash
FileBeat
Alloy
ProTail

日志存储组件

Loki
ElaticSearch
Promentheus (时序数据库)

日志的检索组件

kibana
Grafana

日志处理的两种解决方案

重量级日志: ELK
- ElaticSerach (存储)
- LogStash (收集)
- Kibana (检索, 可视化)
轻量级: GLA
- Grafana (检索)
- Loki (存储)
  - breakend
  - Loki write
  - Loki read
  - Nginx (gateWay)
- Alloy (收集)
  - 配置服务标签, 根据标签信息给日志进行分类

区别:

ELK:
- Es使用java开发, 启动时占用内存大
- 存储: Es在存储时, 将日志的所有消息存储起来, 在查找时是一条条查找,检索速度慢
GLA:
- Loki使用go语言开发, 占用内存小
- 引入携程优化, 速度更快
- Loki在存储时, 通过给日志打上不同的标签, 通过标签来查找日志, 也就是将标签设置未索引, 在查找日志信息时, 可以根据标签查找日志, 可以过滤掉一些没用的日志, 查找速度会更快

GLA底层工作原理

Loki中的组件

Nginx: 网关, Loki中的组件通过nginx进行通信
backend:
- 作用是堆收集的日志做处理
- 它主要由两个组件组成
- loki write:
  1. 日志收集组件, 将日志信息发送给loki write, loki write根据配置中的标签规则, 为该日志打上标签
  2. 将打好的日志发送给nginx, 由nginx发送日志信息,存储在三分对象或文件存储中,比如minio
- loki read: 日志读取组件
  1. 在Grafana发送检索语句LogQL个Loki read时,loki read 根据检索语句,做出处理
  2. 如何将处理好的检索请求发生给nginx, 由nginx 请求三方存储组件. 获取到对应的日志信息

GLA使用流程

[loki官网:](Grafana Loki | Grafana Loki documentation) Grafana Loki | Grafana Loki documentation

Docker 部署

创建文件夹
1
2
mkdir evaluate-loki
cd evaluate-loki

下载Loki和Alloy的默认配置文件, 下载docker compose 配置文件

1
2
3

wget https://raw.githubusercontent.com/grafana/loki/main/examples/getting-started/loki-config.yaml -O loki-config.yaml
wget https://raw.githubusercontent.com/grafana/loki/main/examples/getting-started/alloy-local-config.yaml -O alloy-local-config.yaml
wget https://raw.githubusercontent.com/grafana/loki/main/examples/getting-started/docker-compose.yaml -O docker-compose.yaml

Loki的配置文件: 主要配置Loki中的组件的信息: nginx \ write \ read \ grafana \ minio的信息

Alloy的配置文件: 主要编写拉取Log的服务和 Log的标签规则. 在存储和检索时, 根据标签存储和检索

loki-config.yaml

---
server:
  http_listen_address: 0.0.0.0
  http_listen_port: 3100

memberlist:
  join_members: ["read", "write", "backend"]
  dead_node_reclaim_time: 30s
  gossip_to_dead_nodes_time: 15s
  left_ingesters_timeout: 30s
  bind_addr: ['0.0.0.0']
  bind_port: 7946
  gossip_interval: 2s

schema_config:
  configs:
    - from: 2023-01-01
      store: tsdb
      object_store: s3
      schema: v13
      index:
        prefix: index_
        period: 24h
common:
  path_prefix: /loki
  replication_factor: 1
  compactor_address: http://backend:3100
  storage:
    s3:
      endpoint: minio:9000
      insecure: true
      bucketnames: loki-data
      access_key_id: loki
      secret_access_key: supersecret
      s3forcepathstyle: true
  ring:
    kvstore:
      store: memberlist
ruler:
  storage:
    s3:
      bucketnames: loki-ruler

compactor:
  working_directory: /tmp/compactor

alloy-local-config.yaml

discovery.docker "flog_scrape" {
	host             = "unix:///var/run/docker.sock"
	refresh_interval = "5s"
}

discovery.relabel "flog_scrape" {
	targets = []

	rule {
		source_labels = ["__meta_docker_container_name"]
		regex         = "/(.*)"
		target_label  = "container"
	}
}

loki.source.docker "flog_scrape" {
	host             = "unix:///var/run/docker.sock"
	targets          = discovery.docker.flog_scrape.targets
	forward_to       = [loki.write.default.receiver]
	relabel_rules    = discovery.relabel.flog_scrape.rules
	refresh_interval = "5s"
}

loki.write "default" {
	endpoint {
		url       = "http://gateway:3100/loki/api/v1/push"
		tenant_id = "tenant1"
	}
	external_labels = {}
}

docker-compose.yaml

---
networks:
  loki:

services:
  read:
    image: grafana/loki:latest
    command: "-config.file=/etc/loki/config.yaml -target=read"
    ports:
      - 3101:3100
      - 7946
      - 9095
    volumes:
      - ./loki-config.yaml:/etc/loki/config.yaml
    depends_on:
      - minio
    healthcheck:
      test: [ "CMD-SHELL", "wget --no-verbose --tries=1 --spider http://localhost:3100/ready || exit 1" ]
      interval: 10s
      timeout: 5s
      retries: 5
    networks: &loki-dns
      loki:
        aliases:
          - loki

  write:
    image: grafana/loki:latest
    command: "-config.file=/etc/loki/config.yaml -target=write"
    ports:
      - 3102:3100
      - 7946
      - 9095
    volumes:
      - ./loki-config.yaml:/etc/loki/config.yaml
    healthcheck:
      test: [ "CMD-SHELL", "wget --no-verbose --tries=1 --spider http://localhost:3100/ready || exit 1" ]
      interval: 10s
      timeout: 5s
      retries: 5
    depends_on:
      - minio
    networks:
      <<: *loki-dns

  alloy:
    image: grafana/alloy:latest
    volumes:
      - ./alloy-local-config.yaml:/etc/alloy/config.alloy:ro
      - /var/run/docker.sock:/var/run/docker.sock
    command:  run --server.http.listen-addr=0.0.0.0:12345 --storage.path=/var/lib/alloy/data /etc/alloy/config.alloy
    ports:
      - 12345:12345
    depends_on:
      - gateway
    networks:
      - loki

  minio:
    image: minio/minio
    entrypoint:
      - sh
      - -euc
      - |
        mkdir -p /data/loki-data && \
        mkdir -p /data/loki-ruler && \
        minio server /data
    environment:
      - MINIO_ROOT_USER=loki
      - MINIO_ROOT_PASSWORD=supersecret
      - MINIO_PROMETHEUS_AUTH_TYPE=public
      - MINIO_UPDATE=off
    ports:
      - 9000
    volumes:
      - ./.data/minio:/data
    healthcheck:
      test: [ "CMD", "curl", "-f", "http://localhost:9000/minio/health/live" ]
      interval: 15s
      timeout: 20s
      retries: 5
    networks:
      - loki

  grafana:
    image: grafana/grafana:latest
    environment:
      - GF_PATHS_PROVISIONING=/etc/grafana/provisioning
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
    depends_on:
      - gateway
    entrypoint:
      - sh
      - -euc
      - |
        mkdir -p /etc/grafana/provisioning/datasources
        cat <<EOF > /etc/grafana/provisioning/datasources/ds.yaml
        apiVersion: 1
        datasources:
          - name: Loki
            type: loki
            access: proxy
            url: http://gateway:3100
            jsonData:
              httpHeaderName1: "X-Scope-OrgID"
            secureJsonData:
              httpHeaderValue1: "tenant1"
        EOF
        /run.sh
    ports:
      - "3000:3000"
    healthcheck:
      test: [ "CMD-SHELL", "wget --no-verbose --tries=1 --spider http://localhost:3000/api/health || exit 1" ]
      interval: 10s
      timeout: 5s
      retries: 5
    networks:
      - loki

  backend:
    image: grafana/loki:latest
    volumes:
      - ./loki-config.yaml:/etc/loki/config.yaml
    ports:
      - "3100"
      - "7946"
    command: "-config.file=/etc/loki/config.yaml -target=backend -legacy-read-mode=false"
    depends_on:
      - gateway
    networks:
      - loki
    

  gateway:
    image: nginx:latest
    depends_on:
      - read
      - write
    entrypoint:
      - sh
      - -euc
      - |
        cat <<EOF > /etc/nginx/nginx.conf
        user  nginx;
        worker_processes  5;  ## Default: 1

        events {
          worker_connections   1000;
        }

        http {
          resolver 127.0.0.11;

          server {
            listen             3100;

            location = / {
              return 200 'OK';
              auth_basic off;
            }

            location = /api/prom/push {
              proxy_pass       http://write:3100\$$request_uri;
            }

            location = /api/prom/tail {
              proxy_pass       http://read:3100\$$request_uri;
              proxy_set_header Upgrade \$$http_upgrade;
              proxy_set_header Connection "upgrade";
            }

            location ~ /api/prom/.* {
              proxy_pass       http://read:3100\$$request_uri;
            }

            location = /loki/api/v1/push {
              proxy_pass       http://write:3100\$$request_uri;
            }

            location = /loki/api/v1/tail {
              proxy_pass       http://read:3100\$$request_uri;
              proxy_set_header Upgrade \$$http_upgrade;
              proxy_set_header Connection "upgrade";
            }

            location ~ /loki/api/.* {
              proxy_pass       http://read:3100\$$request_uri;
            }
          }
        }
        EOF
        /docker-entrypoint.sh nginx -g "daemon off;"
    ports:
      - "3100:3100"
    healthcheck:
      test: ["CMD", "service", "nginx", "status"]
      interval: 10s
      timeout: 5s
      retries: 5
    networks:
      - loki
  

  flog:
    image: mingrammer/flog
    command: -f json -d 200ms -l
    networks:
      - loki

启动docker compose
1
docker compose up -d
进入Grafana的主页面: http://localhost:3000

日志的LogQL语句分类

json数据
结构化数据
非结构化数据

具体处理: Simple LogQL simulator | Grafana Loki documentation

在部署容器时,先指定日志格式为json格式,便于LogQL的编写