RCA Workflow
The RCA process is crucial for learning from incidents and preventing recurrence. The most critical aspect of the RCA is understanding the Root Cause and, from there, creating action items to mitigate future incidents of the same cause.
Phoenix Incidents automatically creates an RCA, as a subtask of the incident, when the incident moves to "Fixing" or "Resolved."
The RCA follows its own status flow:
Data Gathering: We collect all relevant incident information. You should complete this before the analysis meeting, and participants should review the data in advance to keep the meeting focused and efficient.
Analysis Meeting: We analyze data with a team to identify root causes and create action items.
Under Review: The RCA has been documented and is being reviewed and signed off by stakeholders before it is finalized.
Finalized: The RCA process is complete.
Canceled: If the parent incident is canceled.
Components of the RCA Process
Every RCA document you create will comprise the following key sections, each serving a vital role in our continuous improvement efforts:
Incident Metrics
View and confirm key metrics here about incident type, the products affected, and key timing metrics of the Incident:
Time to Detect: This is the time it took your organization to detect that there was an outage. This is defined as the delta between Incident Start and the Creation time of the Incident ticket.
Time to Acknowledge: The time it took a team member to acknowledge the incident (either from a paging system, Slack, or Jira). This is the delta between Creation time and when the Incident moved to Assessing status.
Time to Verify: How long it took the team to verify that this is an actual incident. This is the delta between Creation time and when the Incident moved to Fixing status.
Time to Recover: The time from when the disruption started until service was restored. It does not include Action Item completion time. This is the delta between Incident Start and Incident End.
You can also modify the Incident Start and End times directly in this form. This is also where you review and refine the incident's products: the Primary Product (the main affected product, which drove paging) and the full Impacted Products list. Add or remove impacted products here as the analysis clarifies the true scope.
For the full definition of every metric, see Metrics & Definitions.
Incident Details
Impact to Customers: Clearly articulate what your customers experienced during the incident. Framing incidents by customer impact is an effective way to keep your teams focused on customer experience and the true severity of an outage. This section should detail the specific services affected, the scope of the impact (e.g., specific regions, all users), and any direct customer quotes or feedback, if available.
Resolution Summary: Explicitly describe the fix. What specific actions did your team take to resolve the issue? Being precise and detailed in this summary is important because a well-documented RCA can dramatically reduce outage length if a similar incident occurs in the future. Include technical steps, configuration changes, or code deployments.
Error Message: Copy any relevant error messages here. This is incredibly helpful for future investigations, allowing you to quickly find similar RCAs and their resolution summaries by searching for specific error patterns. Include full stack traces or logs if they are concise and directly relevant to the core issue.
Contributing Changes
Link the changes or deployments that may have caused the incident under Related Changes/Deployments. The list of available issues is controlled by your Changes/Deployments setting. Capturing these helps you see patterns across incidents and speeds up the next investigation.
Timeline
Phoenix Incidents automatically generates a partial timeline including incident start, incident transitions, and incident end times. It's good practice to meticulously fill out the full timeline with all significant events, actions taken, and key decision points. This exercise helps people recall the incident more accurately, which, in turn, helps uncover root causes and identify effective action items.
Five Whys
Drilling down into the core issue is critical for creating proper action items. Our Phoenix Incidents AI-guided Five Whys module will help you systematically uncover the fundamental root cause by iteratively asking "why" until you reach the underlying problem rather than just a symptom.
How it works
Start – The first question is "Why did the incident happen?". Write the answer and select Add Entry.
Next question – Phoenix AI suggests the next question, based on all the answers so far. You can change the question before you answer it.
Likely root cause – When an answer looks like where the whys end, Phoenix marks it with a blue flag. This is a hint only. You can still add another why.
Finish – Stop when you reach a cause your team can fix or prevent, then select I'm Done.
Edit and delete – Double-click a question or answer to edit it. Use the trash icon on a row to delete it.
Split a question into paths
Some questions have more than one answer, and each answer needs its own whys. For example, "Why did the server run out of disk space?" might have two answers: "A new debug setting wrote far more logs than usual" and "Old logs were not being cleaned up". Each answer becomes its own path.
While answering – In the entry form, select Add split to add another answer box. Each box becomes a path. The save button then reads Add 2 paths, Add 3 paths, and so on.
On an existing question – Select the branch icon on the question's row (Add split to question 2). The answer already there becomes the first path.
Numbering – The paths of question 2 are 2.a and 2.b. The whys in path 2.a are 2.a.1, 2.a.2, and so on.
Whys on each path – Phoenix AI suggests a first question for every path. Select Add next why to keep going on a path. When a path is finished, select I'm done with this path to move to the next path. On the last path, it takes you back to the main list of questions.
One shared question – Every path answers the same question. If you edit that question on one path, it changes on all of them.
Delete a path – Deletes the path and its whys. If only one path is left, the question goes back to a normal question with that path's answer and whys.
Only questions on the main list can split. A path cannot split again. The main list of questions carries on below the paths and follows from all of them. Each RCA holds up to 20 answers in total, counting every path.
RCAs written before paths were added look the same as before.
Root Cause
Discuss with your team why the root cause happened. You can select multiple root causes if the incident resulted from a combination of factors. This data powers comprehensive reporting that helps you uncover macro trends across your teams, allowing you to focus on bigger-picture systemic issues and prevent recurrence.
Action Items
These are the essential follow-up tasks that need to be addressed to ensure the incident shouldn't be repeated or its impact minimized in the future. You can assign these to any team, and all RCA stakeholders should agree on them. Action items should be specific, urgent, and completed within your defined SLA to drive effective change.
Executive Summary
In the Review phase, write a concise executive summary of the incident for stakeholders. You can draft it yourself, or use Generate to have Phoenix create a first version from the incident details that you then refine.
Export to PDF
At any point you can select Export to download the RCA as a PDF, including the metrics, timeline, Five Whys, root causes, and action items. Paths in the Five Whys show as a nested numbered list. Before it is finalized, the export is marked as a draft.