spec:
containers:
- name: nginx
image: nginx:latest
ports:
- containerPort: 80
resources:
limits:
cpu: 500m
requests:
cpu: 200m
lifecycle:
preStop:
exec:
command: ["/bin/sleep", "30"].
.
hpa 발생시 pod의 갑작스런 종료로 pod에 연결된 사용자가 502를 받게된다.
모든 리퀘스트를 처리 후에 종료되도록 30초간의 유예를 준다 설정 변경 lifecycle preStop 를 이용하여 우아한 종료를!
블로그를 이전한지 얼마안됬기 때문에 집중모니터링 기간이다.
먼저 자원부터 본다.
k top node NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% nks-pool-1119-w-gzi 223m 5% 1265Mi 16% nks-pool-1119-w-kvi 172m 4% 1540Mi 20% k top pod NAME CPU(cores) MEMORY(bytes) php-fpm-nginx-deployment-6bc7b6df77-fbdx9 9m 138Mi storage-nfs-client-provisioner-5b88c7c55-dvtlj 2m 8Mi
자원은 얼마안쓰지만..혹시나 사용량이 늘어날까봐 scale을 늘렸다.
k scale deployment php-fpm-nginx-deployment --replicas=3 deployment.apps/php-fpm-nginx-deployment scaled
그리고 pod 를 확인했는데...
k get pod NAME READY STATUS RESTARTS AGE php-fpm-nginx-deployment-6bc7b6df77-bpf2g 2/2 Running 0 19s php-fpm-nginx-deployment-6bc7b6df77-fbdx9 2/2 Running 3 32h php-fpm-nginx-deployment-6bc7b6df77-rfpb2 0/2 ContainerCreating 0 19s storage-nfs-client-provisioner-5b88c7c55-dvtlj 1/1 Running 0 10h
생성단계에서 멈춘 pod 가 있었다. 상태를 확인해보니
Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled <unknown> default-scheduler Successfully assigned default/php-fpm-nginx-deployment-6bc7b6df77-rfpb2 to nks-pool-1119-w-gzi Normal Pulled 28s kubelet, nks-pool-1119-w-gzi Container image "linuxer-regi.kr.ncr.ntruss.com/php-fpm:12" already present on machine Normal Created 28s kubelet, nks-pool-1119-w-gzi Created container php-fpm Normal Started 28s kubelet, nks-pool-1119-w-gzi Started container php-fpm Normal Pulled 28s kubelet, nks-pool-1119-w-gzi Container image "nginx:1.21" already present on machine Normal Created 28s kubelet, nks-pool-1119-w-gzi Created container nginx Normal Started 28s kubelet, nks-pool-1119-w-gzi Started container nginx Warning FailedPostStartHook 28s kubelet, nks-pool-1119-w-gzi Exec lifecycle hook ([/bin/sh -c chmod 777 /run/php-fpm.sock]) for Container "nginx" in Pod "php-fpm-nginx-deployment-6bc7b6df77-rfpb2_default(6978da29-8045-49a0-9745-6be3cc48c364)" failed - error: command '/bin/sh -c chmod 777 /run/php-fpm.sock' exited with 1: chmod: cannot access '/run/php-fpm.sock': No such file or directory
Exec lifecycle hook ([/bin/sh -c chmod 777 /run/php-fpm.sock]) for Container "nginx" in Pod "php-fpm-nginx-deployment-6bc7b6df77-rfpb2_default(6978da29-8045-49a0-9745-6be3cc48c364)" failed - error: command '/bin/sh -c chmod 777 /run/php-fpm.sock' exited with 1: chmod: cannot access '/run/php-fpm.sock': No such file or directory
lifecycle hook 이 정상동작하지 않았다. 음...이벤트상으론 컨테이너가 생성전에 hook이 동작한건데 이건좀 확인해봐야겠다.
조금있다가 컨테이너가 자동으로 재시작되며 Runing 상태로 변경됬다.
k logs php-fpm-nginx-deployment-6bc7b6df77-rfpb2 -c nginx /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/ /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh 10-listen-on-ipv6-by-default.sh: info: can not modify /etc/nginx/conf.d/default.conf (read-only file system?) /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh /docker-entrypoint.sh: Configuration complete; ready for start up 2021/09/02 00:30:55 [notice] 1#1: using the "epoll" event method 2021/09/02 00:30:55 [notice] 1#1: nginx/1.21.1 2021/09/02 00:30:55 [notice] 1#1: built by gcc 8.3.0 (Debian 8.3.0-6) 2021/09/02 00:30:55 [notice] 1#1: OS: Linux 5.4.8-050408-generic 2021/09/02 00:30:55 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576 2021/09/02 00:30:55 [notice] 1#1: start worker processes 2021/09/02 00:30:55 [notice] 1#1: start worker process 22 2021/09/02 00:30:55 [notice] 1#1: start worker process 23 2021/09/02 00:30:55 [notice] 1#1: start worker process 24 2021/09/02 00:30:55 [notice] 1#1: start worker process 25
logs 도 정상...음...일단 로깅을 모아보고 생각해야겠다.
