Cloud Native Monitoring Notes

/ dev
# notes

Cloud Native Monitoring

Observability and Analysis

Observability 是指一种 从系统外部输出能够理解到系统的程度 的 系统特性。
比如我们可以从系统的cpu占用，内存使用量等来观察计算机。

Analysis 指的是分析这些可观测数据并进行理解。

为了确保系统不中断，需要对系统的各个方面进行观测和分析。
Observability和Analysis工具涵盖了logging, monitoring, tracing, 和 chaos engineering。

Monitoring

监控是指对系统进行检测以收集、汇总和分析日志和指标，以提高我们对其行为的理解。
而一个好的监控能够让操作人员快速响应异常，甚至能够自动的进行处理，同时也能够监控系统的健康程度，甚至系统的任何变动。

同时，监控是一个高效系统重要的组成部分。

BuzzwordsCNCF Projects
Monitoring
Time series
Alerting
Metrics		Prometheus (graduated)
Cortex (incubating)
Thanos (incubating)
Fonio (sandbox)
Kuberhealthy (sandbox)
OpenMetrics (sandbox)
Pixie (sandbox)
Skooner (sandbox)
Trickster (sandbox)

Prometheus

Overview

Architecture：

组件：

Metric DataModel

Metric Definition：

1
2
3
<metric name>{<label name>=<label value>, ...} value [timestamp]
# e.g.
api_http_requests_total{method="POST", handler="/messages"} 5 1395066363000

Text Data Format

1
2
3
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 1871.15625

文本协议基于 行，忽略空行，使用 \n 分割，最后一行必须是换行符。

# 是注释行，但是如果后面是 HELP 和 TYPE：

1
2
# 时间戳是UTC毫秒时间戳
metric_name [ "{" label_name "=" `"` label_value `"` { "," label_name "=" `"` label_value `"` } [ "," ] "}" ] value [ timestamp ]

Metric Types

Counter

是一个逐渐累加的metric数据，比如已完成请求数量等等，不能使用counter来表示一个可以减少的数据，比如当前线程数量。

Gauge

是一个可任意增减的metric数据，且仅能是一个数字，比如内存使用量、温度等等。

Histogram

Histogram包含了一个时间段内以同一个前缀命名的多个时序数据：

Summary

Summary包含一个时间段内以同一个前缀命名的多个时序数据：

1
2
3
4
5
6
7
8
9
# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 2.83e-05
go_gc_duration_seconds{quantile="0.25"} 6.83e-05
go_gc_duration_seconds{quantile="0.5"} 8.95e-05
go_gc_duration_seconds{quantile="0.75"} 0.0001084
go_gc_duration_seconds{quantile="1"} 0.000557999
go_gc_duration_seconds_sum 0.203637424
go_gc_duration_seconds_count 1894

Histogram 和 Summary 差异：

JOBS AND INSTANCES

instance 是指一个可以抓取数据的 endpoint，通常也是对应一个进程。

具有相同目的的 instance 称为一个 job。

ServiceMonitor & PodMonitor

prometheus通过 ServiceMonitor 监控 service，通过 PodMonitor 来监控 pod。

默认配置的是可以获取集群中所有的monitor资源，前提是配置好权限，如果需要限制prometheus监控的monitor范围，
可以在 prometheus-prometheus.yaml
中进行配置：

1
2
3
4
5
6
7
8
9
10
11
12
13
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  labels:
    prometheus: k8s
  name: k8s
  namespace: monitoring
spec:
# 默认没有进行配置，可以对所有 ns 和 monitor进行监控
  serviceMonitorNamespaceSelector: {}
  serviceMonitorSelector: {}
  podMonitorNamespaceSelector: {}
  podMonitorSelector: {}

Deploy

kubernetes 部署方式支持 operator 和 kube-prometheus，
这里使用 kube-prometheus 来进行部署。

1
2
3
4
# 安装 operator
kubectl create -f manifests/setup
# 安装 prometheus 服务端和各个组件
kubectl create -f manifests/

需要注意的是上面的配置文件创建在 monitoring 命名空间下，同时为了支持跨 ns 的监控需要修改服务端的权限配置，
prometheus-clusterRole.yaml ：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus-k8s
rules:
  - apiGroups:
      - ""
    resources:
      - nodes/metrics
    verbs:
      - get
  - nonResourceURLs:
      - /metrics
    verbs:
      - get
  - apiGroups:
      - ""
# 资源和操作权限
    resources:
      - services
      - pods
      - endpoints
    verbs:
      - get
      - list
      - watch

部署成功后默认配置了下面的 service ：

1
2
3
4
5
6
7
8
9
10
NAME                    TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
alertmanager-main       ClusterIP   10.233.43.107   <none>        9093/TCP                     9d
alertmanager-operated   ClusterIP   None            <none>        9093/TCP,9094/TCP,9094/UDP   9d
grafana                 ClusterIP   10.233.24.21    <none>        3000/TCP                     9d
kube-state-metrics      ClusterIP   None            <none>        8443/TCP,9443/TCP            9d
node-exporter           ClusterIP   None            <none>        9100/TCP                     9d
prometheus-adapter      ClusterIP   10.233.7.187    <none>        443/TCP                      9d
prometheus-k8s          ClusterIP   10.233.31.173   <none>        9090/TCP                     9d
prometheus-operated     ClusterIP   None            <none>        9090/TCP                     9d
prometheus-operator     ClusterIP   None            <none>        8443/TCP                     9d

需要关注的是下面几个 service ：

先将所有的ui服务通过ingress暴露出来：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
# ingress
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: monitor-ingress
  namespace: monitoring
  annotations:
spec:
  rules:
    - http:
        paths:
          - path: /
            backend:
              serviceName: prometheus-k8s
              servicePort: web
      host: prome.minei.test
    - http:
        paths:
          - path: /
            backend:
              serviceName: grafana
              servicePort: http
      host: grafana.minei.test
    - http:
        paths:
          - path: /
            backend:
              serviceName: alertmanager-main
              servicePort: web
      host: alert.minei.test

Querying

PromQL数据类型：

字面量：

选择器：

Operators

运算符优先级：

  1. ^
  2. *, /, %, atan2
  3. +, -
  4. ==, !=, <=, <, >=, >
  5. and, unless
  6. or

Functions

Counter指标增长率

1
2
3
rate(go_memstats_frees_total[5m])
increase(go_memstats_frees_total[5m]) / 300
irate(go_memstats_frees_total[5m])

increase() rate() 更偏向于平均增长率，而 irate() 是瞬时增长率，各自适合的场景不一样。

Gauge指标

1
predict_linear(go_memstats_alloc_bytes[10m], 3600)

Histogram指标分位数

1
histogram_quantile(0.9, rate(apiserver_response_sizes_bucket[10m]))

聚合函数

<aggregation>_over_time()：

Exporters

官方支持多种硬件、数据库、消息系统、存储等等的支持，可查看官方 支持列表 。

exporter 用于导出 metric 数据，prometheus 服务来拉取。

Node Exporter

上面提到的 kube-prometheus 部署方式已经默认在集群内部部署了集群节点个数的 node-exporter。
配置文件：

1
2
3
4
5
6
7
node-exporter-clusterRole.yaml
node-exporter-clusterRoleBinding.yaml
node-exporter-daemonset.yaml
node-exporter-prometheusRule.yaml
node-exporter-service.yaml
node-exporter-serviceAccount.yaml
node-exporter-serviceMonitor.yaml

监控集群外虚拟机

node exporter 也支持安装包部署，部署成功默认在 9100 端口暴露 metric 数据。

1
curl localhost:9100

配置 prometheus ：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
apiVersion: v1
kind: Service
metadata:
  name: external-server
  namespace: monitoring
  labels:
    k8s-app: external-server
spec:
  type: ClusterIP
  clusterIP: None
  ports:
  - name: metrics
    port: 9100
    protocol: TCP
    targetPort: 9100
---
apiVersion: v1
kind: Endpoints
metadata:
  name: external-server
  labels:
    k8s-app: external-server
  namespace: monitoring
subsets:
- addresses:
  - ip: 192.168.1.125
  - ip: 192.168.1.129
  - ip: 192.168.1.179
  - ip: 192.168.1.119
  ports:
  - name: metrics
    port: 9100
    protocol: TCP
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: external-server
  labels:
    k8s-app: external-server
  namespace: monitoring
spec:
  endpoints:
  - port: metrics
    interval: 30s
    scheme: http
  selector:
    matchLabels:
      k8s-app: external-server
  namespaceSelector:
    matchNames:
    - monitoring

可以在 service-discovery 进行查看。

JMX Exporter

JMX Exporter 是一个基于 JVM 应用的一个 exporter，
通过javaagent的方式来进行监控，同时需要配置目标应用的jmx：

1
2
3
4
5
6
7
# 使用启动参数配置 jmx
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=9010
-Dcom.sun.management.jmxremote.rmi.port=9010
-Dcom.sun.management.jmxremote.local.only=false
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false

配置agent：

1
2
# 8080指定的是暴露metric数据的端口
-javaagent:/path/to/your/agent/jmx_prometheus_javaagent-0.16.1.jar=8080:/path/to/your/exporter/config/jmx-exporter-config.yaml"

exporter配置：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
---
startDelaySeconds: 0
# 这里配置的是的目标jmx
hostPort: 127.0.0.1:9010
username: 
password: 
# jmxUrl: service:jmx:rmi:///jndi/rmi://127.0.0.1:1234/jmxrmi
ssl: false
lowercaseOutputName: false
lowercaseOutputLabelNames: false
whitelistObjectNames: ["org.apache.cassandra.metrics:*"]
blacklistObjectNames: ["org.apache.cassandra.metrics:type=ColumnFamily,*"]
rules:
  - pattern: 'org.apache.cassandra.metrics<type=(\w+), name=(\w+)><>Value: (\d+)'
    name: cassandra_$1_$2
    value: $3
    valueFactor: 0.001
    labels: {}
    help: "Cassandra metric $1 $2"
    cache: false
    type: GAUGE
    attrNameSnakeCase: false

配置 ServiceMonitor ：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
apiVersion: v1
kind: Service
metadata:
  name: less-svc
  namespace: monitoring
  labels:
    java-app: less-svc
spec:
  type: ClusterIP
  clusterIP: None
  ports:
  - name: jmx
    port: 8080
    protocol: TCP
    targetPort: 8080
  selector:
    external-app: less-svc
---
apiVersion: v1
kind: Endpoints
metadata:
  name: less-svc
  labels:
    java-app: less-svc
    external-app: less-svc
  namespace: monitoring
subsets:
- addresses:
  - ip: 192.168.1.39
  ports:
  - name: jmx
    port: 8080
    protocol: TCP
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: java-app
  labels:
    java-app: less-svc
  namespace: monitoring
spec:
  endpoints:
  - port: jmx
    interval: 30s
    scheme: http
  selector:
    matchLabels:
      java-app: less-svc
  namespaceSelector:
    matchNames:
    - monitoring

How To Debug ServiceMonitor ?

  1. 检查servicemonitor标签是否成功被Prometheus成功筛选到；
  2. 检查service的标签是否成功被servicemonitor标签筛选到；
  3. 检查Prometheus对目标service是否有权限；
  4. 目标service是否能够正常访问到pod，endpoint是否工作正常；

ServiceMonitor 是否被 Prometheus 筛选到？

查看 配置 中是否有你配置的 job。

Push Metrics

Pushgateway 允许临时和批量任务向prometheus推送 metric 数据。

When to use pushgateway

使用 Pushgateway 的缺陷：

备选方案

如果是防火墙或者是NAT导致prometheus无法拉取数据，可以使用 PushProx 让 Prometheus穿透NAT或者防火墙，正常的拉取数据。

Deploy Pushgateway & Push metrics

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: prome-push-gateway
  name: prome-push-gateway
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prome-push-gateway
  template:
    metadata:
      labels:
        app: prome-push-gateway
    spec:
      containers:
        - image: prom/pushgateway:v1.4.2
          name: prome-push-gateway
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 9091
          env:
            - name: less-svc_hazelcast_kubernetes_namespace
              value: "product"

---
# pod service
apiVersion: v1
kind: Service
metadata:
  labels:
    app: prome-push-gateway
  name: prome-push-gateway
  namespace: monitoring
spec:
  selector:
    app: prome-push-gateway
  ports:
    - name: push
      port: 9091
      targetPort: 9091
      protocol: TCP

---
# ingress
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: prome-push-gateway-ingress
  namespace: monitoring
  annotations:
spec:
  rules:
    - http:
        paths:
          - path: /
            backend:
              serviceName: prome-push-gateway
              servicePort: push
      host: pushgateway.minei.test

---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: push-gateway
  namespace: monitoring
spec:
  endpoints:
  - port: push
    interval: 30s
    scheme: http
    # pushgateway 的 monitor需要将该配置设置为 true，否则 job 将会被 exported_job 替代
    # 同样 instance 也会被 exported_instance 替代
    honorLabels: true
  selector:
    matchLabels:
      app: prome-push-gateway
  namespaceSelector:
    matchNames:
    - monitoring

大部分exporters都不支持pushgateway，所以需要自己push metric数据。

Pushgateway 支持类似 restful 的metric api，同时也支持admin api。

push一个metric数据（注意所有的换行都是 \n ，结尾也需要一个 \n 可以参考官方的客户端数据格式说明：
Prometheus Client Data Exposition Format ）：

1
2
3
4
5
6
# TYPE test_metric counter
test_metric{label="val1"} 42
# TYPE another_metric gauge
# HELP another_metric Just an example.
another_metric 2398.283

查询metric：

1
curl http://pushgateway.minei.test/api/v1/metrics

Pushgateway在 /metrics 暴露了所有的metrics数据，所以metric定义一定不能出现冲突，比如相同名称的metric必须有相同的类型和标签，如果冲突了push数据的时候会返回400。

Java Client

DataStorage

Prometheus支持本地存储和远程存储系统。

容器中默认的数据存储结构：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
./prometheus
├── 01BKGV7JBM69T2G1BGBGM6KB12
│   └── meta.json
├── 01BKGTZQ1SYQJTR4PB43C8PD98
│   ├── chunks
│   │   └── 000001
│   ├── tombstones
│   ├── index
│   └── meta.json
├── 01BKGTZQ1HHWHV8FBJXW1Y3W0K
│   └── meta.json
├── 01BKGV7JC0RY8A6MACW02A2PJD
│   ├── chunks
│   │   └── 000001
│   ├── tombstones
│   ├── index
│   └── meta.json
├── chunks_head
│   └── 000001
└── wal
    ├── 000000002
    └── checkpoint.00000001
        └── 00000000

存储配置

如果保留时间和大小同时配置了，哪个先触发就先使用哪个，过期的Block清理发生在后台。同时，Block只有在完全过期了才会被移除。

Prometheus每个数据平均占用空间为 1-2 bytes，可以用下面的公式来粗略计算存储占用：

1
needed_disk_space = retention_time_seconds * ingested_samples_per_second * bytes_per_sample

数据抓取频率可以在servicemonitor中进行配置：ServiceMonitor.spec.endpoints.interval。

If your local storage becomes corrupted for whatever reason, the best strategy to address the problem is to shut down Prometheus then remove the entire storage directory. You can also try removing individual block directories, or the WAL directory to resolve the problem. Note that this means losing approximately two hours data per block directory. Again, Prometheus’s local storage is not intended to be durable long-term storage; external solutions offer extended retention and data durability.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  labels:
    prometheus: k8s
  name: k8s
  namespace: monitoring
spec:
  alerting:
    alertmanagers:
    - name: alertmanager-main
      namespace: monitoring
      port: web
  image: quay.io/prometheus/prometheus:v2.22.1
  nodeSelector:
    kubernetes.io/os: linux
  podMonitorNamespaceSelector: {}
  podMonitorSelector: {}
  probeNamespaceSelector: {}
  probeSelector: {}
  replicas: 2
  resources:
    requests:
      memory: 400Mi
  ruleSelector:
    matchLabels:
      prometheus: k8s
      role: alert-rules
  securityContext:
    fsGroup: 2000
    runAsNonRoot: true
    runAsUser: 1000
  serviceAccountName: prometheus-k8s
  serviceMonitorNamespaceSelector: {}
  serviceMonitorSelector: {}
  version: v2.22.1
  # 数据保留时间和最大存储配置
  retention: 30d
  retentionSize: 1GB
  storage:
  # 存储配置
    volumeClaimTemplate:
      spec:
        storageClassName: nfs-storage
        resources:
          requests:
            storage: 1Gi
        accessModes:
        - ReadWriteOnce

Pushgateway Storage

Pushgateway默认不持久化数据，可以使用 --persistence.file 来配置持久化的文件。

Remote Storage

Remote Storage List

AlertManager

配置告警和通知流程：

  1. 配置和部署 Alertmanager；
  2. Configure Prometheus to talk to the Alertmanager 在Prometheus中配置 Alertmanager；
  3. Prometheus中配置规则；
1
2
3
4
5
              load alertmanager
 Prometheus  ------------------->  Alertmanager
     ↓            alerting              ↓
     ↓                                  ↓
PrometheusRule                   AlertmanagerConfig

Alertmanager配置包含3部分：

receivers支持多种通知方式，email、webhook、wechat等等。
Alertmanager通过 Alertmanager.spec.alertmanagerConfigNamespaceSelector Alertmanager.spec.alertmanagerConfigSelector 来筛选配置。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
  name: alertmanager-config
  namespace: monitoring
  labels:
    alertmanagerConfig: test
spec:
  route:
    groupBy: ['instance']
    groupWait: 30s
    groupInterval: 5m
    repeatInterval: 12h
    receiver: 'pushover'
    matchers:
    - name: app
      value: less-svc
  receivers:
  - name: 'pushover'
    pushoverConfigs:
      userKey:
        name: pushover-userkey
        key: userkey
      title: Alert
      token:
        name: pushover-userkey
        key: token
---
apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: pushover-userkey
  namespace: monitoring
data:
  userkey: xxx
  token: xxx

配置Alertmanager：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
apiVersion: monitoring.coreos.com/v1
kind: Alertmanager
metadata:
  labels:
    alertmanager: main
  name: main
  namespace: monitoring
spec:
# 一定要配置，否则无法正确加载到配置
  alertmanagerConfigSelector:
    matchLabels:
      alertmanagerConfig: test
  image: quay.io/prometheus/alertmanager:v0.21.0
  nodeSelector:
    kubernetes.io/os: linux
  replicas: 3
  securityContext:
    fsGroup: 2000
    runAsNonRoot: true
    runAsUser: 1000
  serviceAccountName: alertmanager-main
  version: v0.21.0

Prometheus 通过 Prometheus.spec.alerting 来配置alertmanager，通过 Prometheus.spec.ruleNamespaceSelector 和 Prometheus.spec.ruleSelector 来配置rule。

Prometheus Rules

Prometheus 包含两种规则，一种是 RecordingRules, 一种是 AlertingRules。

RecordingRules

记录规则提供了 提前计算经常需要用的数据 或者 计算量大 的表达式，并且把计算结果保存到新的时序数据中，常见的就是一些复杂的监控面板数据。

AlertingRules

通过Prome表达式来定义报警条件，同时发送通知。

PrometheusRule.spec.groups.rules：

FieldDescriptionSchemeRequiredFor
recordrecord rules metric namestringfalseRecording
alertalerting rules namestringfalseAlerting
expr表达式intstr.IntOrStringtrueBoth
for触发表达式后告警前等待时间stringfalseAlerting
labels标签，覆盖方式添加到metric或者alertmap[string]stringfalseBoth
annotations添加到alertmap[string]stringfalseAlerting

Templates

告警中使用的模板基于 Go templating。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: prometheus-alert-rules
  namespace: monitoring
  labels:
    prometheus: k8s
    role: alert-rules
spec:
  groups:
  - name: test.rules
    rules:
    - alert: TestAppOffline
      expr: count without() (up{endpoint="jmx", job="less-svc"}) == 0
      for: 1m
      labels:
        app: less-svc
      annotations:
      # 注意如果模板中需要使用标签，那么表达式最终结果也是需要带上这些标签的
        description: app {{ $labels.job }} offline, instance {{ $labels.instance }}

Security

Exporters & Pushgateway

官方exporters和pushgateway支持tls和basic authentication，可以通过启动参数指定配置 --web.config.file="web-config.yml"：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
tls_server_config:
  # Certificate and key files for server to use to authenticate to client.
  cert_file: <filename>
  key_file: <filename>

  # Server policy for client authentication. Maps to ClientAuth Policies.
  # For more detail on clientAuth options: [ClientAuthType](https://golang.org/pkg/crypto/tls/#ClientAuthType)
  [ client_auth_type: <string> | default = "NoClientCert" ]

  # CA certificate for client certificate authentication to the server.
  [ client_ca_file: <filename> ]

  # Minimum TLS version that is acceptable.
  [ min_version: <string> | default = "TLS12" ]

  # Maximum TLS version that is acceptable.
  [ max_version: <string> | default = "TLS13" ]

  # List of supported cipher suites for TLS versions up to TLS 1.2. If empty,
  # Go default cipher suites are used. Available cipher suites are documented
  # in the go documentation:
  # https://golang.org/pkg/crypto/tls/#pkg-constants
  [ cipher_suites:
    [ - <string> ] ]

  # prefer_server_cipher_suites controls whether the server selects the
  # client's most preferred ciphersuite, or the server's most preferred
  # ciphersuite. If true then the server's preference, as expressed in
  # the order of elements in cipher_suites, is used.
  [ prefer_server_cipher_suites: <bool> | default = true ]

  # Elliptic curves that will be used in an ECDHE handshake, in preference
  # order. Available curves are documented in the go documentation:
  # https://golang.org/pkg/crypto/tls/#CurveID
  [ curve_preferences:
    [ - <string> ] ]

http_server_config:
  # Enable HTTP/2 support. Note that HTTP/2 is only supported with TLS.
  # This can not be changed on the fly.
  [ http2: <bool> | default = true ]

# Usernames and hashed passwords that have full access to the web
# server via basic authentication. If empty, no basic authentication is
# required. Passwords are hashed with bcrypt.
basic_auth_users:
  [ <string>: <secret> ... ]

同时在servicemonitor中 servicemonitor.spec.endpoints.tlsConfig 配置抓取的tls信息，servicemonitor.spec.endpoints.basicAuth 配置basic authentication。

API Security

管理相关的API旨在使用简单的cURL工具访问，所以并没有做CSRF保护。在向外部不可信用户暴露的时候可以使用反向代理来避免CSRF，
对一些不信任的输入进行进行转义。

Pushgateway

由于一般都开启了 honor_labels ，所以能够访问到Pushgateway的用户都能创建任意的时间序列。
如果开启了 --web.enable-admin-api 则可以通过admin api操作任意的数据。

Best Practices

Naming

指标命名：

使用标签来区分被观测事物的特征：

基础单位：

FamilyBase unitRemark
Timeseconds
Temperaturecelsiuscelsius is preferred over kelvin for practical reasons. kelvin is acceptable as a base unit in special cases like color temperature or where temperature has to be absolute.
Lengthmeters
Bytesbytes
BitsbytesTo avoid confusion combining different metrics, always use bytes, even where bits appear more common.
PercentratioValues are 0–1 (rather than 0–100). ratio is only used as a suffix for names like disk_usage_ratio. The usual metric name follows the pattern A_per_B.
Voltagevolts
Electric currentamperes
Energyjoules
PowerPrefer exporting a counter of joules, then rate(joules[5m]) gives you power in Watts.
Massgramsgrams is preferred over kilograms to avoid issues with the kilo prefix.

Recording rules

复合 level:metric:operations 的命名规则：

HISTOGRAMS AND SUMMARIES

/HistogramSummary
配置选择适合观察值的预期范围的bucket选择所需的 φ 分位数和滑动窗口。 其他未选择的 φ 分位数和滑动窗口无法稍后计算。
客户端性能由于只需要counters，所以不怎么消耗性能计算流分位数非常消耗性能
服务器性能需要服务器来计算q分位数，很消耗性能，但是可以使用recording rules来预计算性能消耗小
时间序列数量 (除了 _sum 和 _count 序列)每一个bucket都有一个序列每一个配置的分位数都有一个序列
分位数误差 (see below for details)选择合适的bucketsError is limited in the dimension of φ by a configurable value.
φ分位数和滑动窗口定义通过PromeQL定义通过客户端定义
聚合可通过PromeQL聚合一般不可聚合
查询方式histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))http_request_duration_seconds_summary{quantile="0.95"}

histogram_quantile 采用线性插值：

计算方式：分位值=起始bucket大小+(本bucket宽度)*(目标分位数在本bucket排行/本bucket记录数)

如果bucket选择合适，那么得到的分位数误差就较小。

1
2
3
4
5
6
7
8
9
10
prometheus_http_request_duration_seconds_bucket{le="0.05"} 199881
prometheus_http_request_duration_seconds_bucket{le="0.1"} 212210
prometheus_http_request_duration_seconds_bucket{le="0.2"} 215395
prometheus_http_request_duration_seconds_bucket{le="0.4"} 319435
prometheus_http_request_duration_seconds_bucket{le="0.8"} 419576
prometheus_http_request_duration_seconds_bucket{le="1.6"} 469593
prometheus_http_request_duration_seconds_bucket{le="+Inf"} 519593
# 计算0.75分位数
0.75 * 519593 = 389694.75 位于 0.4~0.8之间
0.4 + (0.8-0.4) * ((389694.75 - 319435) / (419576 - 319435))

如何选择：

Grafana

Datasource

支持多种数据源，通过kube-prometheus安装的时候已经集成了Prometheus数据源。

1
2
3
4
5
6
7
-rw-r--r--. 1 root root     550 Oct 27 10:37 grafana-dashboardDatasources.yaml  # 配置数据源
-rw-r--r--. 1 root root 1403539 Oct 27 10:37 grafana-dashboardDefinitions.yaml  # 配置面板
-rw-r--r--. 1 root root     454 Oct 27 10:37 grafana-dashboardSources.yaml
-rw-r--r--. 1 root root    7629 Oct 27 10:37 grafana-deployment.yaml
-rw-r--r--. 1 root root      86 Oct 27 10:37 grafana-serviceAccount.yaml
-rw-r--r--. 1 root root     208 Oct 27 10:37 grafana-serviceMonitor.yaml
-rw-r--r--. 1 root root     201 Oct 27 10:37 grafana-service.yaml

Dashboards

支持json导入，手动配置，配置文件配置，官方面板应用商城 id导入。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
Default
├── DashBoard
│   ├── panel1
│   │   └── query
|   |   └── alert
│   ├── panel2
│   │   └── query
|   |   └── alert
General
│   ├── panel3
│   │   └── query
|   |   └── alert
│   ├── panel4
│   │   └── query
|   |   └── alert

Alert

告警配置跟随面板一起配置的：

本站所有文章除特别声明外，均采用 BY-NC-SA 4.0 许可协议。转载请注明出处！