[Dev Culture] 내가 경험하고 배운 Project Post-mortem 문화

오늘 동료 개발자 분께서 인프랩 기술 블로그 - <2022년 1월 100% 할인 이벤트 장애 부검>을 공유해주시면서 사내 Project post-mortem 문화를 좀 더 체계적으로 할 수 있도록 개선도 하셨다.

 

올해 '2022년에는 함께 일하기 즐거운 회사, 팀이 되기 위해 pain point에 좀 더 관심을 가지고 좀 더 팀 문화 개선에 기여하는 사람이고 싶다.'고 다짐했는데, 아직까진 매번 도움받고, 배우고, 감동받는 단계인가보다. 

 

우리 회사에는 post-mortem문화가 있다. 입사 한 지 별로 되지 않았을 때 우리 서비스에 어떤 버그가 발생했는데, 모든 개발자가 모여 다같이 부검해보고 action item을 도출해보는 시간을 가졌다. 

이 경험을 통해서 처음 post-mortem(부검)와 그 문화에 대해서 알게되었다. 

post-mortem 문화를 대하는 우리 팀의 태도: 
" 누구나 실수를 할 수 있습니다. 그래서 우리는 개인의 잘못을 따지지 않습니다. 우리는 그 실수를 통해 배울 점을 찾고, 같은 문제를 반복하지 않기 위해 노력하는 사람들입니다. 'Post-mortem' 문화를 적극적으로 활용하여 모든 팀원들과 문제 상황을 공유하고 원인과 해결책을 함께 고민합니다. "

 

요즘 여러모로 트러블슈팅한 일도 종종 있었고, 문서화의 진짜 힘을 활용하는데 익숙해지고 있는데, 이 참에 'Project post-mortem'에 대해서 배울 수 있었던 자료들을 정리해두고자 한다.

잘 정리 되어있는 자료가 많아서, 같은 내용을 또 다시 적기 보다 가장 도움이 되었던 자료의 링크와
미래의 내가 빠르게 기억을 되찾는데 도움이 될 정도의 핵심 메모를 남기는 정도로 하고자 한다 

 

Project post-mortem이란

문제가 생긴 직,간접적 원인을 사후에 총체적으로 분석하는 것이다. 

A project post-mortem is a process used to identify the causes of a project failure (or significant business-impairing downtime), and how to prevent them in the future. This is different from a Retrospective, in which both positive and negative things are reviewed for a project.

from https://en.wikipedia.org/wiki/Postmortem_documentation

 

see Postmortems vs. Retrospectives: When (and How) to Use Each Effectively for Postmortems vs. Retrospectives.

see 향로님 블로그 - 2021년 CTO 회고 , [주간 인프런 #41] 개발자의 공유 문화 이모저모 (2) 회고 문화 for retrospective example.

 

Nine steps to IT post-mortem excellence

1. Plan for post-mortem and be taken seriously.
2. Keep it close in time.
3. Record the projeect details: how big it was, how long it took, what software was used, what the objectives were, and so on....
4. Involve everyone: First, different people will have different insights about the project, and you need to collect them all to really understand what worked and didn't. Second, getting everyone involved helps prevent the post-mortem from degenerating into scapegoating.
5. Should be written down.
6. Record successes as well as failures.
7. Not punish people.
8. Create an action plan.
9. Make it available.

from https://www.zdnet.com/article/nine-steps-to-it-post-mortem-excellence

 

Postmortem Template

작성일: 2015-10-21

작성자: jennifer, martym, agoogler

Status: Complete, action items in progress

Summary: Shakespeare Search down for 66 minutes during period of very high interest in Shakespeare due to discovery of a new sonnet.

Impact: Estimated 1.21B queries lost, no revenue impact.

Root Causes: Cascading failure due to combination of exceptionally high load and a resource leak when searches failed due to terms not being in the Shakespeare corpus. The newly discovered sonnet used a word that had never before appeared in one of Shakespeare’s works, which happened to be the term users searched for. Under normal circumstances, the rate of task failures due to resource leaks is low enough to be unnoticed.

Trigger: Latent bug triggered by sudden increase in traffic.

Resolution: Directed traffic to sacrificial cluster and added 10x capacity to mitigate cascading failure. Updated index deployed, resolving interaction with latent bug. Maintaining extra capacity until surge in public interest in new sonnet passes. Resource leak identified and fix deployed.

Detection: Borgmon detected high level of HTTP 500s and paged on-call.

 

Action Items:

Action Item	Type	Owner	Bug
Update playbook with instructions for responding to cascading failure	mitigate	jennifer	n/a DONE
Use flux capacitor to balance load between clusters	prevent	martym	Bug 5554823 TODO
Schedule cascading failure test during next DiRT	process	docbrown	n/a TODO
Investigate running index MR/fusion continuously	prevent	jennifer	Bug 5554824 TODO
Plug file descriptor leak in search ranking subsystem	prevent	agoogler	Bug 5554825 DONE
Add load shedding capabilities to Shakespeare search	prevent	agoogler	Bug 5554826 TODO
Build regression tests to ensure servers respond sanely to queries of death	prevent	clarac	Bug 5554827 TODO
Deploy updated search ranking subsystem to prod	prevent	jennifer	n/a DONE
Freeze production until 2015-11-20 due to error budget exhaustion, or seek exception due to grotesque, unbelievable, bizarre, and unprecedented circumstances	other	docbrown	n/a TODO

Lessons Learned

What went well

- Monitoring quickly alerted us to high rate (reaching ~100%) of HTTP 500s

- Rapidly distributed updated Shakespeare corpus to all clusters

What went wrong

- We’re out of practice in responding to cascading failure

- We exceeded our availability error budget (by several orders of magnitude) due to the exceptional surge of traffic that essentially all resulted in failures

Where we got lucky

- Mailing list of Shakespeare aficionados had a copy of new sonnet available

- Server logs had stack traces pointing to file descriptor exhaustion as cause for crash

- Query-of-death was resolved by pushing new index containing popular search term

 

Timeline

2015-10-21 (all times UTC)

14:51 News reports that a new Shakespearean sonnet has been discovered in a Delorean’s glove compartment

14:53 Traffic to Shakespeare search increases by 88x after post to /r/shakespeare points to Shakespeare search engine as place to find new sonnet (except we don’t have the sonnet yet)

14:54 OUTAGE BEGINS — Search backends start melting down under load

14:55 docbrown receives pager storm, ManyHttp500s from all clusters

14:57 All traffic to Shakespeare search is failing: see https://monitor

14:58 docbrown starts investigating, finds backend crash rate very high

15:01 INCIDENT BEGINS docbrown declares incident #465 due to cascading failure, coordination on #shakespeare, names jennifer incident commander

....(중략)

15:36 OUTAGE MITIGATED, updated index replicated to all clusters

15:39 docbrown starts second wave of instance count increase to 10x initial capacity

...(중략)

15:55 50% of traffic balanced across nonsacrificial clusters

16:00 OUTAGE ENDS, all traffic balanced across all clusters

16:30 INCIDENT ENDS, reached exit criterion of 30 minutes’ nominal performance

 

from https://sre.google/sre-book/example-postmortem/

 

5 Whys

The 5 Whys strategy is a simple, effective tool for uncovering the root of a problem. 

1. Assemble a Team 
2. Define the Problem
3. Ask the First "Why?"
4. Ask "Why?" Four More Times
5. Know When to Stop: You'll know that you've revealed the root cause of the problem when asking "why" produces no more useful responses, and you can go no further.
6. Address the Root Cause(s)
7. Monitor Your Measures

ex 1. Single Lane

ex 2. Multiple Lanes

from https://www.mindtools.com/pages/article/newTMC_5W.htm, https://brunch.co.kr/@cliche-cliche/113

 

Post-moderm Examples

개발,디자인 등 서비스 이슈에 대한 부검 뿐만 아니라, 넷플릭스의 문화로도 유명한 퇴사 부검 등도 있다. 

• 인프랩 기술 블로그 - <2022년 1월 100% 할인 이벤트 장애 부검>
• 우아한 형제들 기술 블로그 - 퇴사 부검