Kubernetes

DOIK-Study

가시다님과 함께하는 스터디는 항상 즐겁다. 이번 스터디엔 라이브로 못해서..일단 바닐라쿠버 배포하고 시작했다.

Headless 서비스는 ClusterIP가 None으로 설정하며 kubeproxy를 통하지않고 pod의 endpoint가 svc에 연결된다.

나는 먼저 NFS 서버를 Headless 로 배포하기로 했다.

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: nfs-server-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
      
---

kind: Service
apiVersion: v1
metadata:
  name: nfs-service
spec:
  type: ClusterIP
  clusterIP: None
  selector:
    role: nfs
  ports:
    # Open the ports required by the NFS server
    # Port 2049 for TCP
    - name: tcp-2049
      port: 2049
      protocol: TCP

    # Port 111 for UDP
    - name: udp-111
      port: 111
      protocol: UDP
      
    # Port 20048 for TCP
    - name: tcp-20048
      port: 20048
      protocol: TCP
---

apiVersion: v1
kind: ReplicationController
metadata:
  name: nfs-server
spec:
  replicas: 1
  selector:
    role: nfs-server
  template:
    metadata:
      labels:
        role: nfs-server
    spec:
      containers:
      - name: nfs-server
        image: gcr.io/google_containers/volume-nfs:0.8
        ports:
          - name: nfs
            containerPort: 2049
          - name: mountd
            containerPort: 20048
          - name: rpcbind
            containerPort: 111
        securityContext:
          privileged: true
        volumeMounts:
          - mountPath: /exports
            name: nfs-export
      volumes:
        - name: nfs-export-fast
          persistentVolumeClaim:
            claimName: nfs-server-pvc-fast
            

🐤

yaml을 deploy 하면 다음과같다.

이제 프로비저너 셋팅이 좀 필요하다. 가시다님께서는 친절하게 프로비저너 셋팅도 다해주셨지만 나는 내가만든 NFS 서버를 사용할거기 때문에 프로비저너를 다시 배포할거다.

#지우고
helm delete -n kube-system nfs-provisioner
#다시 설치하고
helm install nfs-provisioner -n kube-system nfs-subdir-external-provisioner/nfs-subdir-external-provisioner --set nfs.server=nfs-service.default.svc.cluster.local --set nfs.path=/exports
NAME: nfs-provisioner
LAST DEPLOYED: Thu May 26 16:10:31 2022
NAMESPACE: kube-system
STATUS: deployed
REVISION: 1
TEST SUITE: None

🐤

응 잘됬다.

(🐤 |DOIK-Lab:default) root@k8s-m:~# k get svc
NAME          TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)            AGE
kubernetes    ClusterIP   10.200.1.1   <none>        443/TCP            4h30m
nfs-service   ClusterIP   None         <none>        2049/TCP,111/UDP   6m29s

(🐤 |DOIK-Lab:default) root@k8s-m:~# k get ep
NAME          ENDPOINTS                            AGE
kubernetes    192.168.10.10:6443                   4h30m
nfs-service   172.16.158.2:2049,172.16.158.2:111   6m48s

(🐤 |DOIK-Lab:default) root@k8s-m:~# k get pod -o wide
NAME             READY   STATUS    RESTARTS   AGE    IP             NODE     NOMINATED NODE   READINESS GATES
nfs-server-pod   1/1     Running   0          7m1s   172.16.158.2   k8s-w1   <none>           <none>

🐤

정상적으로 NFS서버가 잘 배포된것을 확인할수 있다.

apiVersion: v1
kind: PersistentVolume
metadata:
  name: mysql-nfs-pv
  labels:
    type: mysql-nfs-pv
spec:
  storageClassName: nfs-client
  capacity:
    storage: 4Gi
  accessModes:
  - ReadWriteOnce          # ReadWriteOnce RWO (1:1 마운트, 읽기 쓰기)
  nfs:
    server: 172.16.184.9 # NFS-Server 의 IP
    path: /1       # NFS 저장소
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: wp-nfs-pv
  labels:
    type: wp-nfs-pv
spec:
  storageClassName: nfs-client
  capacity:
    storage: 4Gi
  accessModes:
  - ReadWriteOnce
  nfs:
    server: 172.16.184.9 # NFS-Server 의 IP
    path: /2      # NFS 저장소

🐤

nfs-service.svc.cluster.local domain을 이용하려 하였으나, PV 에서 domain으로 설정시 nfs-provisioner 정상적으로 마운트 되지 않았다.

headless NFS를 하려고 한것이나, 실패하였다.
지원하지 않는다.(결론)

다음과 같은 증상이었다. IP로 프로비저너 설치후엔 잘되었다.

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  labels:
    app: wordpress
  name: mysql-pv-claim
spec:
  storageClassName: nfs-client
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 4Gi
  selector:
    matchLabels:
      type: "mysql-nfs-pv"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  labels:
    app: wordpress
  name: wp-pv-claim
spec:
  storageClassName: nfs-client
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 4Gi
  selector:
    matchLabels:
      type: "wp-nfs-pv"

🐤

selector를 이용하여 PV를 사용하도록 설정해 주었다

Every 2.0s: kubectl get svc,pods,pv,pvc -o wide                                                                                                                                                                    k8s-m: Thu May 26 18:10:41 2022

NAME                      TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)                      AGE     SELECTOR
service/kubernetes        ClusterIP   10.200.1.1    <none>        443/TCP                      7h56m   <none>
service/nfs-service       ClusterIP   None          <none>        2049/TCP,111/UDP,20048/TCP   71m     role=nfs
service/wordpress         NodePort    10.200.1.33   <none>        80:30387/TCP                 3m25s   app=wordpress,tier=frontend
service/wordpress-mysql   ClusterIP   None          <none>        3306/TCP                     3m25s   app=wordpress,tier=mysql

NAME                                   READY   STATUS    RESTARTS   AGE     IP              NODE     NOMINATED NODE   READINESS GATES
pod/nfs-server-rxvf7                   1/1     Running   0          71m     172.16.184.9    k8s-w2   <none>           <none>
pod/wordpress-859f989bbb-msppd         1/1     Running   0          3m25s   172.16.158.21   k8s-w1   <none>           <none>
pod/wordpress-mysql-66fb7cfb68-z9vj5   1/1     Running   0          3m25s   172.16.158.20   k8s-w1   <none>           <none>

NAME                                                        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                         STORAGECLASS   REASON   AGE     VOLUMEMODE
persistentvolume/mysql-nfs-pv                               4Gi        RWO            Retain           Bound    default/mysql-pv-claim        nfs-client              3m32s   Filesystem
persistentvolume/pvc-adc24c97-ca67-4700-b3c5-2fc51c4cce01   10Gi       RWO            Delete           Bound    default/nfs-server-pvc-fast   local-path              71m     Filesystem
persistentvolume/wp-nfs-pv                                  4Gi        RWO            Retain           Bound    default/wp-pv-claim           nfs-client              3m32s   Filesystem

NAME                                        STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE     VOLUMEMODE
persistentvolumeclaim/mysql-pv-claim        Bound    mysql-nfs-pv                               4Gi        RWO            nfs-client     3m25s   Filesystem
persistentvolumeclaim/nfs-server-pvc-fast   Bound    pvc-adc24c97-ca67-4700-b3c5-2fc51c4cce01   10Gi       RWO            local-path     71m     Filesystem
persistentvolumeclaim/wp-pv-claim           Bound    wp-nfs-pv                                  4Gi        RWO            nfs-client     3m25s   Filesystem

🐤

서비스가 잘 작동하는것을 확인하였다.

(🐤 |DOIK-Lab:default) root@k8s-m:~/yaml/yaml# k exec nfs-server-rxvf7 -it -- /bin/bash
[root@nfs-server-rxvf7 /]# cd /exports/
[root@nfs-server-rxvf7 exports]# ll
total 32
drwxr-xr-x 5 systemd-bus-proxy root 4096 May 26 09:07 1
drwxr-xr-x 5                33   33 4096 May 26 09:07 2
-rw-r--r-- 1 root              root   16 May 26 07:59 index.html
[root@nfs-server-rxvf7 exports]# cd 1
[root@nfs-server-rxvf7 1]# ll
total 110608
-rw-rw---- 1 systemd-bus-proxy input       56 May 26 09:07 auto.cnf
-rw-rw---- 1 systemd-bus-proxy input 50331648 May 26 09:07 ib_logfile0
-rw-rw---- 1 systemd-bus-proxy input 50331648 May 26 09:07 ib_logfile1
-rw-rw---- 1 systemd-bus-proxy input 12582912 May 26 09:07 ibdata1
drwx------ 2 systemd-bus-proxy input     4096 May 26 09:07 mysql
drwx------ 2 systemd-bus-proxy input     4096 May 26 09:07 performance_schema
drwx------ 2 systemd-bus-proxy input     4096 May 26 09:07 wordpress
[root@nfs-server-rxvf7 1]# cd ..
[root@nfs-server-rxvf7 exports]# cd 2
[root@nfs-server-rxvf7 2]# ll
total 192
-rw-r--r--  1 33 33   418 Sep 25  2013 index.php
-rw-r--r--  1 33 33 19935 Jan  2  2017 license.txt
-rw-r--r--  1 33 33  7413 Dec 12  2016 readme.html
-rw-r--r--  1 33 33  5447 Sep 27  2016 wp-activate.php
drwxr-xr-x  9 33 33  4096 Oct 31  2017 wp-admin
-rw-r--r--  1 33 33   364 Dec 19  2015 wp-blog-header.php
-rw-r--r--  1 33 33  1627 Aug 29  2016 wp-comments-post.php
-rw-r--r--  1 33 33  2764 May 26 09:07 wp-config-sample.php
-rw-r--r--  1 33 33  3154 May 26 09:07 wp-config.php
drwxr-xr-x  4 33 33  4096 Oct 31  2017 wp-content
-rw-r--r--  1 33 33  3286 May 24  2015 wp-cron.php
drwxr-xr-x 18 33 33 12288 Oct 31  2017 wp-includes
-rw-r--r--  1 33 33  2422 Nov 21  2016 wp-links-opml.php
-rw-r--r--  1 33 33  3301 Oct 25  2016 wp-load.php
-rw-r--r--  1 33 33 34327 May 12  2017 wp-login.php
-rw-r--r--  1 33 33  8048 Jan 11  2017 wp-mail.php
-rw-r--r--  1 33 33 16200 Apr  6  2017 wp-settings.php
-rw-r--r--  1 33 33 29924 Jan 24  2017 wp-signup.php
-rw-r--r--  1 33 33  4513 Oct 14  2016 wp-trackback.php
-rw-r--r--  1 33 33  3065 Aug 31  2016 xmlrpc.php

🐤

목적이었던 NFS도 정상적으로 작동한다.

(🐤 |DOIK-Lab:default) root@k8s-m:~/yaml# k scale deployment wordpress --replicas=3

(🐤 |DOIK-Lab:default) root@k8s-m:~/yaml# k get pod
NAME                               READY   STATUS    RESTARTS   AGE
nfs-server-rxvf7                   1/1     Running   0          77m
wordpress-859f989bbb-8r5zh         1/1     Running   0          47s
wordpress-859f989bbb-msppd         1/1     Running   0          8m38s
wordpress-859f989bbb-xhbs9         1/1     Running   0          47s
wordpress-mysql-66fb7cfb68-z9vj5   1/1     Running   0          8m38s

(🐤 |DOIK-Lab:default) root@k8s-m:~/yaml# k get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                         STORAGECLASS   REASON   AGE
mysql-nfs-pv                               4Gi        RWO            Retain           Bound    default/mysql-pv-claim        nfs-client              9m6s
pvc-adc24c97-ca67-4700-b3c5-2fc51c4cce01   10Gi       RWO            Delete           Bound    default/nfs-server-pvc-fast   local-path              77m
wp-nfs-pv                                  4Gi        RWO            Retain           Bound    default/wp-pv-claim           nfs-client              9m6s
(🐤 |DOIK-Lab:default) root@k8s-m:~/yaml# k get pvc
NAME                  STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
mysql-pv-claim        Bound    mysql-nfs-pv                               4Gi        RWO            nfs-client     9m2s
nfs-server-pvc-fast   Bound    pvc-adc24c97-ca67-4700-b3c5-2fc51c4cce01   10Gi       RWO            local-path     77m
wp-pv-claim           Bound    wp-nfs-pv                                  4Gi        RWO            nfs-client     9m2s

🐤

볼륨도 잘공유하여 프로비저닝 된것을 확인할수 있다. ㅜㅜ

EKS-prometheus-grafana

https://docs.aws.amazon.com/ko_kr/eks/latest/userguide/prometheus.html

먼저 프로메테우스를 설치한다.

cat << EOF | k apply -f -
 ---
 apiVersion: v1
 kind: PersistentVolumeClaim
 metadata:
   name: grafana-pvc
 spec:
   accessModes:
     - ReadWriteOnce
   resources:
     requests:
       storage: 1Gi
 ---
 apiVersion: apps/v1
 kind: Deployment
 metadata:
   labels:
     app: grafana
   name: grafana
 spec:
   selector:
     matchLabels:
       app: grafana
   template:
     metadata:
       labels:
         app: grafana
     spec:
       securityContext:
         fsGroup: 472
         supplementalGroups:
           - 0
       containers:
         - name: grafana
           image: grafana/grafana:7.5.2
           imagePullPolicy: IfNotPresent
           ports:
             - containerPort: 3000
               name: http-grafana
               protocol: TCP
           readinessProbe:
             failureThreshold: 3
             httpGet:
               path: /robots.txt
               port: 3000
               scheme: HTTP
             initialDelaySeconds: 10
             periodSeconds: 30
             successThreshold: 1
             timeoutSeconds: 2
           livenessProbe:
             failureThreshold: 3
             initialDelaySeconds: 30
             periodSeconds: 10
             successThreshold: 1
             tcpSocket:
               port: 3000
             timeoutSeconds: 1
           resources:
             requests:
               cpu: 250m
               memory: 750Mi
           volumeMounts:
             - mountPath: /var/lib/grafana
               name: grafana-pv
       volumes:
         - name: grafana-pv
           persistentVolumeClaim:
             claimName: grafana-pvc
 ---
 apiVersion: v1
 kind: Service
 metadata:
   name: grafana
 spec:
   ports:
     - port: 3000
       protocol: TCP
       targetPort: http-grafana
   selector:
     app: grafana
   sessionAffinity: None
   type: LoadBalancer
 EOF

https://grafana.com/docs/grafana/latest/installation/kubernetes/

설치는 위링크를 참조하고 grafana svc type 만 LoadBalancer 로 변경한다.

k get svc
NAME         TYPE           CLUSTER-IP       EXTERNAL-IP                                                                   PORT(S)          AGE
grafana      LoadBalancer   172.20.237.228   af7fa7486f6eb4ad4a6bde897210f4a9-206885623.ap-northeast-2.elb.amazonaws.com   3000:32317/TCP   32m

그라파나의 서비스가 다만들어지면 URL로 접근이 가능하다.

패스워드는 admin / admin 이다.

로그인후 할일은 data source 를 지정하는것이다. 우리는 prometheus 를 이용할것이다.

서비스이름/네임스페이스/svc:port 로 지정한다.

save & test 눌러서 잘되는지 확인하자.

그리고 dashboard를 import 하자.

https://grafana.com/grafana/dashboards/11074

많은 사람이 애용하는 dashboard를 사용할것이다. import 는 ID로 넣으면된다 이경우엔 11074 를 입력하자

VictoriaMetrics 를 프로메테우스로 지정하자. 그리고 Import 하면 대시보드가 뜬다.

대략 이런 대시보드가 자동으로 수집된다.

https://grafana.com/grafana/dashboards/13770

그라파나는 사람들이 만들어놓은 대시보드를 이용하기 쉽다.

그리고 node-exporter 로 만들어내는 매트릭리스트를 파악하여 원하는 지표를 사용할수 있다.

https://prometheus.io/docs/guides/node-exporter/

위URL을 참고해서 매트릭을 확인하여 보자.

예를 들어서 Dropped packet를 확인하려 한다면 다음 매트릭을 확인할수 있다.

읽어주셔서 감사하다!

올해의 가시다님 과의 스터디가 마무리되었다. 같이 EKS 스터디에 참여해주신분들께 감사를 드리며, 평안한 하루되시라!

Web-Performance-test

 k run test -it --rm --image=dos65/httperf --restart=Never --  httperf --server default-productpage-84356-8135672-aff6ae07c7cc.kr.lb.naverncp.com --port 9080 --uri /productpage?u=normal --rate 100 --num-conn 1000 --num-call 10 --timeout 10
If you don't see a command prompt, try pressing enter.
Maximum connect burst length: 1

Total: connections 1000 requests 3319 replies 2766 test-duration 37.384 s

Connection rate: 26.7 conn/s (37.4 ms/conn, <=1000 concurrent connections)
Connection time [ms]: min 10017.5 avg 27192.2 max 34318.0 median 28058.5 stddev 4668.2
Connection time [ms]: connect 497.2
Connection length [replies/conn]: 8.563

Request rate: 88.8 req/s (11.3 ms/req)
Request size [B]: 138.0

Reply rate [replies/s]: min 7.2 avg 61.3 max 150.2 stddev 44.8 (7 samples)
Reply time [ms]: response 2895.1 transfer 0.3
Reply size [B]: header 250.0 content 4474.0 footer 0.0 (total 4724.0)
Reply status: 1xx=0 2xx=2766 3xx=0 4xx=0 5xx=0

CPU time [s]: user 17.72 system 19.55 (user 47.4% system 52.3% total 99.7%)
Net I/O: 353.3 KB/s (2.9*10^6 bps)

Errors: total 737 client-timo 737 socket-timo 0 connrefused 0 connreset 0
Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0
pod "test" deleted

httperf 를 이용한 부하테스트

AKOS-Study-Manual-EKS-istio

클러스터를 먼저 프로비저닝 했다. 30분이상이 걸리는 작업이므로 시작해놓고 기다린다.

eksctl create cluster --vpc-public-subnets $WKSubnets --name $CLUSTER_NAME --region $AWS_REGION --version 1.21 \
> --nodegroup-name $CLUSTER_NAME-nodegroup --node-type t3.medium --nodes 3 --nodes-min 3 --nodes-max 6 \
> --with-oidc --node-volume-size=20 --ssh-access --ssh-public-key $MySSHKeypair
2021-09-04 11:29:11 [ℹ]  eksctl version 0.63.0
2021-09-04 11:29:11 [ℹ]  using region ap-northeast-2
2021-09-04 11:29:12 [✔]  using existing VPC (vpc-094808933b68add7c) and subnets (private:map[] public:map[ap-northeast-2a:{subnet-0a603a222db0cce10 ap-northeast-2a 10.0.11.0/24} ap-northeast-2b:{subnet-007964ce4a003361a ap-northeast-2b 10.0.12.0/24} ap-northeast-2c:{subnet-007813cf58631ef3b ap-northeast-2c 10.0.13.0/24}])
2021-09-04 11:29:12 [!]  custom VPC/subnets will be used; if resulting cluster doesn't function as expected, make sure to review the configuration of VPC/subnets
2021-09-04 11:29:12 [ℹ]  nodegroup "first-eks-nodegroup" will use "" [AmazonLinux2/1.21]
2021-09-04 11:29:12 [ℹ]  using EC2 key pair %!q(*string=<nil>)
2021-09-04 11:29:12 [ℹ]  using Kubernetes version 1.21
2021-09-04 11:29:12 [ℹ]  creating EKS cluster "first-eks" in "ap-northeast-2" region with managed nodes
2021-09-04 11:29:12 [ℹ]  will create 2 separate CloudFormation stacks for cluster itself and the initial managed nodegroup
2021-09-04 11:29:12 [ℹ]  if you encounter any issues, check CloudFormation console or try 'eksctl utils describe-stacks --region=ap-northeast-2 --cluster=first-eks'
2021-09-04 11:29:12 [ℹ]  CloudWatch logging will not be enabled for cluster "first-eks" in "ap-northeast-2"
2021-09-04 11:29:12 [ℹ]  you can enable it with 'eksctl utils update-cluster-logging --enable-types={SPECIFY-YOUR-LOG-TYPES-HERE (e.g. all)} --region=ap-northeast-2 --cluster=first-eks'
2021-09-04 11:29:12 [ℹ]  Kubernetes API endpoint access will use default of {publicAccess=true, privateAccess=false} for cluster "first-eks" in "ap-northeast-2"
2021-09-04 11:29:12 [ℹ]  2 sequential tasks: { create cluster control plane "first-eks", 3 sequential sub-tasks: { 4 sequential sub-tasks: { wait for control plane to become ready, associate IAM OIDC provider, 2 sequential sub-tasks: { create IAM role for serviceaccount "kube-system/aws-node", create serviceaccount "kube-system/aws-node" }, restart daemonset "kube-system/aws-node" }, 1 task: { create addons }, create managed nodegroup "first-eks-nodegroup" } }
2021-09-04 11:29:12 [ℹ]  building cluster stack "eksctl-first-eks-cluster"
2021-09-04 11:29:12 [ℹ]  deploying stack "eksctl-first-eks-cluster"
2021-09-04 11:29:42 [ℹ]  waiting for CloudFormation stack "eksctl-first-eks-cluster"
2021-09-04 11:30:12 [ℹ]  waiting for CloudFormation stack "eksctl-first-eks-cluster"
2021-09-04 11:31:12 [ℹ]  waiting for CloudFormation stack "eksctl-first-eks-cluster"
2021-09-04 11:32:12 [ℹ]  waiting for CloudFormation stack "eksctl-first-eks-cluster"
2021-09-04 11:33:12 [ℹ]  waiting for CloudFormation stack "eksctl-first-eks-cluster"
2021-09-04 11:34:12 [ℹ]  waiting for CloudFormation stack "eksctl-first-eks-cluster"
2021-09-04 11:35:12 [ℹ]  waiting for CloudFormation stack "eksctl-first-eks-cluster"
2021-09-04 11:36:12 [ℹ]  waiting for CloudFormation stack "eksctl-first-eks-cluster"
2021-09-04 11:37:12 [ℹ]  waiting for CloudFormation stack "eksctl-first-eks-cluster"
2021-09-04 11:38:12 [ℹ]  waiting for CloudFormation stack "eksctl-first-eks-cluster"
2021-09-04 11:39:12 [ℹ]  waiting for CloudFormation stack "eksctl-first-eks-cluster"
2021-09-04 11:40:13 [ℹ]  waiting for CloudFormation stack "eksctl-first-eks-cluster"
2021-09-04 11:41:13 [ℹ]  waiting for CloudFormation stack "eksctl-first-eks-cluster"
2021-09-04 11:45:14 [ℹ]  building iamserviceaccount stack "eksctl-first-eks-addon-iamserviceaccount-kube-system-aws-node"
2021-09-04 11:45:14 [ℹ]  deploying stack "eksctl-first-eks-addon-iamserviceaccount-kube-system-aws-node"

EKS를 Setup 하는 과정에 대해선 이전포스팅을 참고하기 바란다.

간단한 실습이 있지만..음 istio는 못참지.

가즈아!

먼저 istioctl을 설치하자.

curl -L https://istio.io/downloadIstio | ISTIO_VERSION=1.10.4 TARGET_ARCH=x86_64 sh -
tree istio-1.10.4/ -L 2
mv istio-1.10.4/bin/istioctl /usr/local/bin/istioctl
istioctl version

버전과 상황에 따라 설치 방법이 다를 수 있다.

istioctl install --set profile=demo -y

demo로 프로파일을 설정하게되면 istio에서 사용하는 모든 오브젝트를 설치해준다. 그러므로 만약 프로덕션에서 사용한다면 원하는 오브젝트만 따로 설치하자.

nginx pod에 istio inject 명령어로 yaml 에 istio를 주입하면 이렇게 된다.

#전
apiVersion: v1
kind: Pod
metadata:
  name: pod1
spec:
  containers:
  - name: nginx
    image: nginx
    ports:
    - containerPort: 80
#후    
istioctl kube-inject -f pod1.yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubectl.kubernetes.io/default-container: nginx
    kubectl.kubernetes.io/default-logs-container: nginx
    prometheus.io/path: /stats/prometheus
    prometheus.io/port: "15020"
    prometheus.io/scrape: "true"
    sidecar.istio.io/status: '{"initContainers":["istio-init"],"containers":["istio-proxy"],"volumes":["istio-envoy","istio-data","istio-podinfo","istio-token","istiod-ca-cert"],"imagePullSecrets":null}'
  creationTimestamp: null
  labels:
    istio.io/rev: default
    security.istio.io/tlsMode: istio
    service.istio.io/canonical-name: pod1
    service.istio.io/canonical-revision: latest
  name: pod1
spec:
  containers:
  - image: nginx
    name: nginx
    ports:
    - containerPort: 80
    resources: {}
  - args:
    - proxy
    - sidecar
    - --domain
    - $(POD_NAMESPACE).svc.cluster.local
    - --serviceCluster
    - pod1.default
    - --proxyLogLevel=warning
    - --proxyComponentLogLevel=misc:error
    - --log_output_level=default:info
    - --concurrency
    - "2"
    env:
    - name: JWT_POLICY
      value: third-party-jwt
    - name: PILOT_CERT_PROVIDER
      value: istiod
    - name: CA_ADDR
      value: istiod.istio-system.svc:15012
    - name: POD_NAME
      valueFrom:
        fieldRef:
          fieldPath: metadata.name
    - name: POD_NAMESPACE
      valueFrom:
        fieldRef:
          fieldPath: metadata.namespace
    - name: INSTANCE_IP
      valueFrom:
        fieldRef:
          fieldPath: status.podIP
    - name: SERVICE_ACCOUNT
      valueFrom:
        fieldRef:
          fieldPath: spec.serviceAccountName
    - name: HOST_IP
      valueFrom:
        fieldRef:
          fieldPath: status.hostIP
    - name: CANONICAL_SERVICE
      valueFrom:
        fieldRef:
          fieldPath: metadata.labels['service.istio.io/canonical-name']
    - name: CANONICAL_REVISION
      valueFrom:
        fieldRef:
          fieldPath: metadata.labels['service.istio.io/canonical-revision']
    - name: PROXY_CONFIG
      value: |
        {}
    - name: ISTIO_META_POD_PORTS
      value: |-
        [
            {"containerPort":80}
        ]
    - name: ISTIO_META_APP_CONTAINERS
      value: nginx
    - name: ISTIO_META_CLUSTER_ID
      value: Kubernetes
    - name: ISTIO_META_INTERCEPTION_MODE
      value: REDIRECT
    - name: ISTIO_META_WORKLOAD_NAME
      value: pod1
    - name: ISTIO_META_OWNER
      value: kubernetes://apis/v1/namespaces/default/pods/pod1
    - name: ISTIO_META_MESH_ID
      value: cluster.local
    - name: TRUST_DOMAIN
      value: cluster.local
    image: docker.io/istio/proxyv2:1.10.4
    name: istio-proxy
    ports:
    - containerPort: 15090
      name: http-envoy-prom
      protocol: TCP
    readinessProbe:
      failureThreshold: 30
      httpGet:
        path: /healthz/ready
        port: 15021
      initialDelaySeconds: 1
      periodSeconds: 2
      timeoutSeconds: 3
    resources:
      limits:
        cpu: "2"
        memory: 1Gi
      requests:
        cpu: 10m
        memory: 40Mi
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop:
        - ALL
      privileged: false
      readOnlyRootFilesystem: true
      runAsGroup: 1337
      runAsNonRoot: true
      runAsUser: 1337
    volumeMounts:
    - mountPath: /var/run/secrets/istio
      name: istiod-ca-cert
    - mountPath: /var/lib/istio/data
      name: istio-data
    - mountPath: /etc/istio/proxy
      name: istio-envoy
    - mountPath: /var/run/secrets/tokens
      name: istio-token
    - mountPath: /etc/istio/pod
      name: istio-podinfo
  initContainers:
  - args:
    - istio-iptables
    - -p
    - "15001"
    - -z
    - "15006"
    - -u
    - "1337"
    - -m
    - REDIRECT
    - -i
    - '*'
    - -x
    - ""
    - -b
    - '*'
    - -d
    - 15090,15021,15020
    image: docker.io/istio/proxyv2:1.10.4
    name: istio-init
    resources:
      limits:
        cpu: "2"
        memory: 1Gi
      requests:
        cpu: 10m
        memory: 40Mi
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        add:
        - NET_ADMIN
        - NET_RAW
        drop:
        - ALL
      privileged: false
      readOnlyRootFilesystem: false
      runAsGroup: 0
      runAsNonRoot: false
      runAsUser: 0
  volumes:
  - emptyDir:
      medium: Memory
    name: istio-envoy
  - emptyDir: {}
    name: istio-data
  - downwardAPI:
      items:
      - fieldRef:
          fieldPath: metadata.labels
        path: labels
      - fieldRef:
          fieldPath: metadata.annotations
        path: annotations
      - path: cpu-limit
        resourceFieldRef:
          containerName: istio-proxy
          divisor: 1m
          resource: limits.cpu
      - path: cpu-request
        resourceFieldRef:
          containerName: istio-proxy
          divisor: 1m
          resource: requests.cpu
    name: istio-podinfo
  - name: istio-token
    projected:
      sources:
      - serviceAccountToken:
          audience: istio-ca
          expirationSeconds: 43200
          path: istio-token
  - configMap:
      name: istio-ca-root-cert
    name: istiod-ca-cert
status: {}
---

istio의 sidecar가 nginx pod에 삽입되게 된다.

      limits:
        cpu: "2"
        memory: 1Gi
      requests:
        cpu: 10m
        memory: 40Mi

사용하는 자원의 제한은 위와같다. istio-init(initcontainer) proxy(envoy) 가 추가된다.

kubectl label namespace default istio-injection=enabled
namespace/default labeled
kubectl get ns -L istio-injection
NAME              STATUS   AGE   ISTIO-INJECTION
default           Active   46m   enabled

namespace 에 라벨을 붙이면 자동으로 그뒤론 NS 에 sidecar가 붙게된다.

k run nginx-istio --image=nginx --restart=Never
pod/nginx-istio created
k get pod
NAME          READY   STATUS            RESTARTS   AGE
nginx-istio   0/2     PodInitializing   0          4s
pod1          2/2     Running           0          5m11s

이제 sidecar를 본격적으로 확인해보자.

kubectl apply -f istio-1.10.4/samples/addons

아까 다운로드한 istio 에서 샘플로제공된 애드온을 설치한다. 위와같은 명령어를 치면 모든 애드온이 설치된다. 애드온내부에 있는 특정 애드온만도 설치가능하니 필요하면 특정 애드온만 설치해도 된다.

kiali.yaml 를 설치할때 kind 에 MonitoringDashboard 가 있어야 설치가 되는데 처음에 한꺼번에 다 배포를 하면 실패한다 그럼 그냥 쿨하게 명령어 한번더 입력해주자.

이제 애드온으로 접근하기위해선 애드온의 서비스를 퍼블릭하게 변경해줘야하는데, 나는 이전에는 yaml를 손수 수정했는데 이부분이 싱크빅하다.

k get svc -n istio-system grafana -o yaml | sed -e "s/type: ClusterIP/type: LoadBalancer/" | kubectl apply -f -
service/grafana configured
k get svc -n istio-system kiali -o yaml | sed -e "s/type: ClusterIP/type: LoadBalancer/" | kubectl apply -f -
service/kiali configured
k get svc -n istio-system tracing -o yaml | sed -e "s/type: ClusterIP/type: LoadBalancer/" | kubectl apply -f -
service/tracing configured

sed 로 수정해서 바로 적용한다. 와우..당연히 내가 못하는건 아닌데 관념의 차이로 인하여 이런 사용을 생각못했다. 다음엔 써먹어야지

ubectl get svc -n istio-system
NAME                   TYPE           CLUSTER-IP      EXTERNAL-IP                                                                    PORT(S)                                                AGE
grafana                LoadBalancer   172.20.162.75   a6d32baedc66b4633bb7fbb0875c6132-465014933.ap-northeast-2.elb.amazonaws.com    3000:30572/TCP                                                3m51s
istio-egressgateway    ClusterIP      172.20.129.21   <none>                                                                         80/TCP,443/TCP                                                21m
istio-ingressgateway   LoadBalancer   172.20.95.93    a0e6177dd9cb64884bd2893028c04328-781274984.ap-northeast-2.elb.amazonaws.com    15021:31227/TCP,80:30590/TCP,443:32395/TCP,31400:32264/TCP,15443:32750/TCP   21m
istiod                 ClusterIP      172.20.90.49    <none>                                                                         15010/TCP,15012/TCP,443/TCP,15014/TCP                                        21m
jaeger-collector       ClusterIP      172.20.99.248   <none>                                                                         14268/TCP,14250/TCP                                                3m51s
kiali                  LoadBalancer   172.20.96.205   a313dbdb158064d578d88c0a022bc845-1007771282.ap-northeast-2.elb.amazonaws.com   20001:30296/TCP,9090:30713/TCP                                               3m51s
prometheus             ClusterIP      172.20.50.6     <none>                                                                         9090/TCP                                                3m50s
tracing                LoadBalancer   172.20.58.118   a9da5b64099ed4fd3b5abdf3b1cd9ebe-68617878.ap-northeast-2.elb.amazonaws.com     80:30295/TCP                                                3m51s
zipkin                 ClusterIP      172.20.76.230   <none>                                                                         9411/TCP                                                3m51s

샘플 manifest 중에 bookinfo 가 있다.

샘플에서 보여주는것은 트래픽이 어떻게 흐르는지 시각화로 보여주는것이다.

문제가 생길경우 다음과같이 UI 와 로깅으로 확인이 가능하다.

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: reviews
spec:
  hosts:
    - reviews
  http:
  - route:
    - destination:
        host: reviews
        subset: v2
      weight: 50
    - destination:
        host: reviews
        subset: v3
      weight: 50

위의 에러는 기본적인 destination rule 을 설정하지 않은 상태로 review 에 대한 룰을 설정해서 그렇다.

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: productpage
spec:
  host: productpage
  subsets:
  - name: v1
    labels:
      version: v1
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: reviews
spec:
  host: reviews
  subsets:
  - name: v1
    labels:
      version: v1
  - name: v2
    labels:
      version: v2
  - name: v3
    labels:
      version: v3
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: ratings
spec:
  host: ratings
  subsets:
  - name: v1
    labels:
      version: v1
  - name: v2
    labels:
      version: v2
  - name: v2-mysql
    labels:
      version: v2-mysql
  - name: v2-mysql-vm
    labels:
      version: v2-mysql-vm
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: details
spec:
  host: details
  subsets:
  - name: v1
    labels:
      version: v1
  - name: v2
    labels:
      version: v2
---

destination rule 을 설정하고 보면 reviews rule 이 정상적으로 작동하는것을 알수있다.

적용전
적용후

가중치에 의하여 v2/v3로만 라우팅 되는것을 확인할수 있다.

istio는 조만간 블로그에 적용후에 더 자세히 다뤄보도록 하겠다.

좋은 주말되시라!

NKS-Linuxer-Blog-trouble-shooting-lifecycle-not-working

블로그를 이전한지 얼마안됬기 때문에 집중모니터링 기간이다.
먼저 자원부터 본다.

k top node
NAME                  CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
nks-pool-1119-w-gzi   223m         5%     1265Mi          16%       
nks-pool-1119-w-kvi   172m         4%     1540Mi          20%       

k top pod
NAME                                             CPU(cores)   MEMORY(bytes)   
php-fpm-nginx-deployment-6bc7b6df77-fbdx9        9m           138Mi           
storage-nfs-client-provisioner-5b88c7c55-dvtlj   2m           8Mi             

자원은 얼마안쓰지만..혹시나 사용량이 늘어날까봐 scale을 늘렸다.

k scale deployment php-fpm-nginx-deployment --replicas=3
deployment.apps/php-fpm-nginx-deployment scaled

그리고 pod 를 확인했는데...

k get pod
NAME                                             READY   STATUS              RESTARTS   AGE
php-fpm-nginx-deployment-6bc7b6df77-bpf2g        2/2     Running             0          19s
php-fpm-nginx-deployment-6bc7b6df77-fbdx9        2/2     Running             3          32h
php-fpm-nginx-deployment-6bc7b6df77-rfpb2        0/2     ContainerCreating   0          19s
storage-nfs-client-provisioner-5b88c7c55-dvtlj   1/1     Running             0          10h

생성단계에서 멈춘 pod 가 있었다. 상태를 확인해보니

Events:
  Type     Reason               Age        From                          Message
  ----     ------               ----       ----                          -------
  Normal   Scheduled            <unknown>  default-scheduler             Successfully assigned default/php-fpm-nginx-deployment-6bc7b6df77-rfpb2 to nks-pool-1119-w-gzi
  Normal   Pulled               28s        kubelet, nks-pool-1119-w-gzi  Container image "linuxer-regi.kr.ncr.ntruss.com/php-fpm:12" already present on machine
  Normal   Created              28s        kubelet, nks-pool-1119-w-gzi  Created container php-fpm
  Normal   Started              28s        kubelet, nks-pool-1119-w-gzi  Started container php-fpm
  Normal   Pulled               28s        kubelet, nks-pool-1119-w-gzi  Container image "nginx:1.21" already present on machine
  Normal   Created              28s        kubelet, nks-pool-1119-w-gzi  Created container nginx
  Normal   Started              28s        kubelet, nks-pool-1119-w-gzi  Started container nginx
  Warning  FailedPostStartHook  28s        kubelet, nks-pool-1119-w-gzi  Exec lifecycle hook ([/bin/sh -c chmod 777 /run/php-fpm.sock]) for Container "nginx" in Pod "php-fpm-nginx-deployment-6bc7b6df77-rfpb2_default(6978da29-8045-49a0-9745-6be3cc48c364)" failed - error: command '/bin/sh -c chmod 777 /run/php-fpm.sock' exited with 1: chmod: cannot access '/run/php-fpm.sock': No such file or directory

Exec lifecycle hook ([/bin/sh -c chmod 777 /run/php-fpm.sock]) for Container "nginx" in Pod "php-fpm-nginx-deployment-6bc7b6df77-rfpb2_default(6978da29-8045-49a0-9745-6be3cc48c364)" failed - error: command '/bin/sh -c chmod 777 /run/php-fpm.sock' exited with 1: chmod: cannot access '/run/php-fpm.sock': No such file or directory

lifecycle hook 이 정상동작하지 않았다. 음...이벤트상으론 컨테이너가 생성전에 hook이 동작한건데 이건좀 확인해봐야겠다.

조금있다가 컨테이너가 자동으로 재시작되며 Runing 상태로 변경됬다.

k logs php-fpm-nginx-deployment-6bc7b6df77-rfpb2 -c nginx 
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: can not modify /etc/nginx/conf.d/default.conf (read-only file system?)
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
2021/09/02 00:30:55 [notice] 1#1: using the "epoll" event method
2021/09/02 00:30:55 [notice] 1#1: nginx/1.21.1
2021/09/02 00:30:55 [notice] 1#1: built by gcc 8.3.0 (Debian 8.3.0-6) 
2021/09/02 00:30:55 [notice] 1#1: OS: Linux 5.4.8-050408-generic
2021/09/02 00:30:55 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576
2021/09/02 00:30:55 [notice] 1#1: start worker processes
2021/09/02 00:30:55 [notice] 1#1: start worker process 22
2021/09/02 00:30:55 [notice] 1#1: start worker process 23
2021/09/02 00:30:55 [notice] 1#1: start worker process 24
2021/09/02 00:30:55 [notice] 1#1: start worker process 25

logs 도 정상...음...일단 로깅을 모아보고 생각해야겠다.