Social Security Tribunal of Canada

An evaluation of the navigator service in Canada Pension Plan disability appeals before the General Division – Income Security

2.0 Scope and objective

Objective

The evaluation set out to assess the early performance of the navigator service, in particular, by answering whether the evidence confirms that the service is on track to deliver on its intended program objectives: that navigators provide client-focussed and actionable information and guidance to appellants, appellants are properly prepared for their hearings, and they participate in a meaningful way in hearings. Answers to these questions will go toward not only improving on the navigator service today, but to inform its ongoing expansion.

Scope

To answer these research questions, the study’s methodology ensures that more than one line of evidence addresses each question. The methodology includes a review of administrative data, review of key documents, a review of online literature, and interviews and surveys of key actors in the system: navigators, members, appellants and managers. The evaluation findings and conclusions are based on the analysis and triangulation of these lines of evidence (see Annex B: Evaluation matrix).

The study draws its data from November 2019, when the service was launched, to January 31, 2021. During this 14-month period, in November 2020 the SST expanded the navigator service to unrepresented claimants before the Appeal Division. The scope of the study is limited to appeals before the GD-IS and, unless specified otherwise, all generic references to “navigator” or “appellant” are understood to mean only those in connection to the GD-IS.

3.0 Methodology

The following subsections describe the data collection methods used.

Surveys

Surveys were carried out with 2 groups, appellants and members. The SST conducts a client satisfaction survey with most appellants in the period after their hearing and before their decision. Beginning in July 2020, the survey included questions about the navigator service. A subsample of 79 navigated appellants responded to this survey. See Annex D: Client satisfaction survey. The study sent an online survey to all current members who had made a decision on a navigated appeal. The survey consisted of 11 open or closed questions, and assured members of confidentiality and anonymity. See Annex E: Survey to members. Of the 22 members who received the survey, 19 returned it completed.

Interviews

The study carried out semi-structured focus groups with navigators and one-on-one interviews with their team leaders and manager. A total of 16 key informants were interviewed. 8 former appellants were contacted but declined to be interviewed. However, the feedback of 79 navigated appellants through the client satisfaction survey provided alternative data for the evaluation.

Document review and literature review

The study benefitted from key documents about the navigator service, including its project plan, training materials, work description, and process maps. In addition, a limited review was done of online literature on the experiences of self-represented litigants (SRL) in the court system.

Administrative data

The study reviewed data from Navigator List, a database of navigators’ self-reported information on telephone calls and case status, and from Navigator View, a metadatabase linked to the SST’s case management system. To facilitate this analysis, the study created 2 comparator groups, a control group of appellants not assigned a navigator, and all unrepresented appellants who filed during the year preceding navigator and had a decision rendered. More details of these comparator groups are found in section 4.0.

The following chart illustrates the time period of appeals under evaluation.

Figure 1: Key dates in the scope of the evaluation

A chronological timeline showing key dates in the scope of the evaluation.
Text Version

Figure 1: Key dates in the scope of the evaluation

  • November 2018 to October 2019: Comparator group of pre-navigated appeals
  • November 2019: Launch of navigator and collection of navigator and control group data
  • July 2020: Launch of client satisfaction survey with navigator-specific questions
  • September 2020: Last appeal added to the control group
  • January 31, 2021: The evaluation examines only decided appeal filed from launch to this date

4.0 Findings and analysis

When the navigator service launched in November 2019, the SST could not be certain of its reception or impact among recipients. To lay a data foundation, the SST designed 2 experimental groups—a control group of appellants who didn’t receive a navigator but were otherwise eligible to have had one. On 1 day of the week (Wednesday) from launch to September 2020, Registry Operations routed all new unrepresented appellants to the control group. On all other weekdays, unrepresented appellants were streamed to a navigator, forming the treatment group. The presence of a control group of appellants exposed to the same processing conditions and variables, save for the navigator, was intended to allow comparisons to be made between navigated and control groups, and potentially attribute observed differences in outcome to the services of navigators.

To gain further comparative insight, the evaluation designed a second reference group. The “pre-navigated group” was retrospectively created consisting of all CPP-D unrepresented appellants who filed their appeal in the 12-month period leading up to the launch of navigator (November 2018 to October 2019) and had a decision rendered. Given the pandemic, however, the study proceeds with caution when making findings between the pre-navigated and navigated groups.

The following table outlines the number of appeals in the navigated group and its 2 comparator groups.

Table 1: Navigator group and 2 comparator groups
Group Time period Number of appeals
Navigated November 2019 to January 2021 807
Control November 2019 to October 2020 82
Pre-navigated November 2018 to October 2019 1,0491

1 Unrepresented appellants only

4.1 Time to become ready to proceed

What we found

The rate of navigated appellants becoming ready to proceed has trended faster since early 2020.

Why this is important

When an appellant formally notifies the SST of their readiness, they’re signalling that they have no further documents to file and are prepared to proceed to their hearing. Appellants have a maximum of 1 year to be ready to proceed. That more appellants are registering their readiness faster indicates navigator efforts are eliciting the progressive steps appellants need to take to become hearing ready. These steps may include familiarizing themselves with hearing requirements, attending appointments with needed professionals, obtaining medical documentation, and gathering witnesses. This goes to fulfilling one of the intended results of the program.

What the evidence says

Figure 3 below shows that the number of days navigated appellants took to become ready to proceed started at 202 days for the cohort who filed in November 2019, and stayed relatively high until February 2020. After February, a steady, long-term decline began until the cohort of appeals filed in October 2020 needed only 73 days to become ready. The control group also witnessed a similar trend before spiking temporarily in July 2020.

Figure 2 and 3: Days ready to proceed by monthly cohort: Navigated improved faster than control group

A line chart of navigated appeals.
Text version

Figure 2: Days ready to proceed by monthly cohort: Control group

  • November 2019: 44 days
  • December 2019: 132 days
  • January 2020: 166 days
  • February 2020: 140 days
  • March 2020: 68 days
  • April 2020: 65 days
  • May 2020: 85 days
  • June 2020: No new appeals
  • July 2020: 179 days
  • August 2020: 82 days
  • September 2020: 41 days
Line chart showing Average number of days it took for appellants to become ready to proceed
Text version

Figure 3: Days ready to proceed by monthly cohort: Navigated appeals

  • November 2019: 202 days
  • December 2019: 141 days
  • January 2020: 224 days
  • February 2020: 206 days
  • March 2020: 137 days
  • April 2020: 123 days
  • May 2020: 115 days
  • June 2020: 80 days
  • July 2020: 88 days
  • August 2020: 77 days
  • September 2020: 74 days
  • October 2020: 73 days

What is also notable in figures 2 and 3 is that each month, control group appellants spent fewer days to reach readiness than navigated appellants, except for the month of July 2020. This may be explained by the control group’s small sample size, which produces less reliable and more variable results. But analyzed over the entire year, the 2 trendlines are clear—navigated appellants were responsible for a steeper decline in the time needed to become ready than appellants in the control group.

What the data also reveal is that appellants assigned to a navigator were more likely to have become ready to proceed than appellants assigned to the control group. Figure 4 below captures the number of appellants who reached readiness as a percentage of all appellants in the monthly cohort. In all but 1 month, proportionately more navigated appellants reached readiness than control group appellants. They were also able to do this at a progressively faster rate. In short, appellants in the navigated group weren’t only much more numerous (about 5 times more) than the control group, but a larger share of them managed to become ready.

Figure 4: Percentage who reached readiness: Proportionately more navigated appeals ready than control group

A column chart showing the percentage of each monthly cohort that became ready to proceed
Text version

Figure 4: Percentage who reached readiness: Proportionately more navigated appeals ready than control group

November 2019

  • Navigated appeals: 76%
  • Control group: 50%

December 2019

  • Navigated appeals: 79%
  • Control group: 50%

January 2020

  • Navigated appeals: 72%
  • Control group: 63%

February 2020

  • Navigated appeals: 63%
  • Control group: 43%

March 2020

  • Navigated appeals: 60%
  • Control group: 40%

April 2020

  • Navigated appeals: 59%
  • Control group: 33%

May 2020

  • Navigated appeals: 55%
  • Control group: 44%

June 2020

  • Navigated appeals: 68%

July 2020

  • Navigated appeals: 73%
  • Control group: 60%

August 2020

  • Navigated appeals: 48%
  • Control group: 43%

September 2020

  • Navigated appeals: 39%
  • Control group: 50%

October 2020

  • Navigated appeals: 32%

Next, we compared navigated appellants against unrepresented appellants from the same time period 1 year earlier (figures 5 and 6 below). The data raise 2 findings. First, the cohort of appellants who filed during the first quarter of navigator operation (November 2019 through January 2020) took a relatively high average of 189 days to become hearing ready. By contrast, the pre-navigated cohort took 136 days, nearly 2 months faster. Then, starting in February to March 2020 navigated appellants began to accelerate their readiness, reaching 73 days in October 2020, an improvement of nearly 4 months. By comparison, the trajectory for pre-navigator appellants rose slightly during its year.

Arguably, 2020 saw a significantly reduced caseload compared to the year before which, combined with growing capacity in the navigator service, would have worked in favour of faster readiness times. But by the same token a smaller caseload doesn’t alone explain why readiness times suddenly accelerated in February 2020 nor why those times were largely unaffected by a pandemic-related slowdown of navigator services in the spring of 2020 (more below on the operational impact of the pandemic).

Figure 5 and 6: Days ready to proceed by monthly cohort: Navigated appellants outpaced pre-navigated

A line chart, showing average days to become ready to proceed by monthl
Text version

Figure 5: Days ready to proceed by monthly cohort: Pre-navigated appeals

  • November 2018: 149 days
  • December 2018: 139 days
  • January 2019: 122 days
  • February 2019: 118 days
  • March 2019: 136 days
  • April 2019: 125 days
  • May 2019: 141 days
  • June 2019: 120 days
  • July 2019: 137 days
  • August 2019: 183 days
  • September 2019: 137 days
  • October 2019: 151 days
A line chart of navigated appeals, showing 202 days to become ready to proceed
Text version

Figure 6: Days ready to proceed by monthly cohort: Navigated appeals

  • November 2019: 202 days
  • December 2019: 141 days
  • January 2020: 224 days
  • February 2020: 206 days
  • March 2020: 137 days
  • April 2020: 123 days
  • May 2020: 115 days
  • June 2020: 80 days
  • July 2020: 88 days
  • August 2020: 77 days
  • September 2020: 74 days
  • October 2020: 73 days

Mid-stream appellants

When the navigator service launched in November 2019, the initial caseload comprised not only new intake but appellants already in the system that had not yet reached readiness. In the first month of operation, navigators worked on 57 “mid-stream” and 23 new appellants in parallel, for a combined initial caseload of 80 appellants. The evaluation set out to identify any significant operational impact from working these 2 groups of appellants.

What the evidence suggests is that both groups together experienced bottlenecking from November 2019 to February 2020 in the form of lower readiness rates (see figure 7 below). Of the 57 mid-stream appellants, 24 reached readiness by March 2020 while 14 new appellants did so during this period.

Once the initial bulge of mid-stream and new appellants worked their way through the system, by March 2020 readiness times for new appeals quickly began their downward trend (while readiness times for the remaining mid-stream appeals rose to 387 days). Case data and interviews suggest that this trend was aided by the growing number of navigators on strength from 7 in November 2019 to 10 in March 2020. After 4 months of experience, the initial 7 would have settled into their new jobs and begun forming lasting and fruitful relationships with appellants. By the time of the June 2020 cohort, the number of days to readiness fell to its floor of 76 days where it has roughly remained since.

Figure 7: Mid-stream appellants: Days ready to proceed by monthly cohort

A scatter plot showing the number of days it took for appellants to become ready to proceed.
Text version

Figure 7: Mid-stream appellants: Days ready to proceed by monthly cohort 

Date appeal was filed Number of days to reach readiness
January 2019 501
February 2019 660
429
March 2019 422
411
656
335
April 2019 406
June 2019 431
365
349
258
July 2019 323
476
August 2019
24 appellants filed Aug. to Oct. 2019 and reached readiness in under 200 days. The remainder took 387 days.
343
372
360
428
78
106
144
September 2019 414
262
304
367
366
370
368
155
126
53
70
73
82
54
159
October 2019 271
222
318
352
365
126
54
163
120
107
55
41
41
52
127

Pandemic-related interruption

The evaluation examined the extent to which a temporary service reduction impacted productivity and readiness times. Beginning March 13, 2020, all navigators were forced to carry out duties from home. Over the course of the next 6 weeks, each navigator received a cell phone until April 28 when all navigators were fully equipped. In the month of March, phone records showed a drop of 15% in the number of inbound and outbound calls between navigators and appellants. The number of hours spent on those calls dropped sharply by 38%. However, in April the recovery began and in May the numbers reached a peak that stands to this day, indicating that navigators worked rapidly to re-establish ties with their appellants. The result of their effort is that appellants may not have noticed any substantial service reduction, as their readiness trendline continued largely unabated after the pandemic struck.

Figure 8: Pandemic slowdown saw a drop in productivity followed by rapid recovery and faster readiness times

A combination chart showing monthly cohorts of navigated appeals from November 2019 to October 2020.
Text version

 Figure 8: Pandemic slowdown saw a drop in productivity followed by rapid recovery and faster readiness times

Month Number of phone calls made Number of hours on the phone Number of days for appellants to become ready to proceed
November 2019 214 15.7 202
December 2019 408 36.5 141
January 2020 488 40.5 224
February 2020 418 36.3 206
March 2020 352 22.4 137
April 2020 470 35.1 123
May 2020 637 47.4 115
June 2020 550 44.4 80
July 2020 440 31.4 88
August 2020 352 20.2 77
September 2020 455 37.4 74
October 2020 256 18.3 73

4.2 Quality of preparation before the hearing

What we found

Most evidence points to appellants having a better understanding of the SST’s processes and making choices that advance their appeal.

Why this is important

The role of the navigator is to inform, guide and leave decisions to the appellant to make. As such, appellants demonstrating agency over their appeals is an indication that the navigator service is achieving one of its intended results.

What the evidence says

Although navigated appellants inform the SST that they’re ready earlier, whether they’re objectively ready and actually prepared compared to non-navigated appellants, is more difficult to measure. The SST’s client satisfaction survey asks all SST appellants to respond to the subjective statement: “Before your hearing you knew what to expect and how to prepare.” Results show that 80% of navigated appellants agreed with that statement, lower than the 85% among all other unrepresented appellants. For navigated and non-navigated respondents, this question persists as the lowest reported level of client satisfaction every month.

To reconcile these dissonant results, navigators and surveyed appellants alike offered possible explanations:

  • Appellants in CPP-D are more likely to face issues connected to mental health and disability that, for a small minority, affect their motivation or capacity to absorb navigator messages.
  • Appellants who do take in navigator messages have faced challenges acting on them, such as obtaining medical documentation during the pandemic.
  • Most importantly, appellants are more conscious of their situation as a consequence of navigator support. They face a heightened sense of inadequacy and unpreparedness as they’re told they should formulate arguments to defend their position that they’re disabled—doubtless, an intimidating prospect for many.

Yet members rate appellant preparedness higher than appellants themselves. From their vantage point adjudicating both navigated and non-navigated appellants, members can objectively see the downstream effects of navigator preparation. Members’ feedback speak to appellants who are more ready and grateful for the support, resulting in more streamlined hearings. A large majority (84%) of members observed positive changes, citing appellants who:

  • come to the hearing prepared to proceed from the start
  • have their evidentiary documents on hand
  • understand the process, roles of the parties, and independence of the member
  • are more at ease and focussed

Varying impact among appellants

However, the consensus among members, navigators and appellants is that improvements are far from universal and “at sea” appellants are still too common. Among appellants who do appear better prepared, the level of preparedness varies widely, indicating that the fruits of a navigator’s efforts depend heavily on appellant circumstances. One member articulated a common observation:

Overall, I've had some appellants who seem better prepared, but I've also had some who seem lost. With respect to the latter, I don't think it has anything to do with the skills and efforts put forward by the navigator. I think it's just that some appellants struggle to understand what is being said, and perhaps don’t take the time (or are unable to) read the letters sent to them.

Navigators and their team leaders echoed this observation. The vast majority of appellants welcome navigator support, but a small fraction resist offers of help or lack the capacity to accept it. The reasons vary, but the most commonly cited are lack of proficiency in one of the official languages, mental health or disability, or outright suspicion of governmental intentions.

Figure 9: Reasons for removal from navigation

A pie chart, each representing a reason why appeals were removed
Text version

Figure 9: Reasons for removal from navigation

  • Withdrawn: 46%
  • Other: 19%
  • Obtained a professional representative: 15%
  • No longer meets requirements: 9%
  • Agreement: 8%
  • Request of appellant: 2%

Up to 28%, or 57 appeals, were not navigable, spanning the “Other” and “No longer meets requirements” categories.

Navigators will make multiple attempts to reach out and connect to these appellants before deeming them no longer navigable in the standard manner. These appellants then either receive reduced or modified navigator services (such as fewer letters or phone calls) or are removed to the regular processing stream. Among all appeals removed from navigator (see figure 9), 9% were removed for “no longer meets requirements” and 19% for “other”—2 categories denoting appeals not navigable. In total, 57 appeals comprise this share and they represent 7% of all navigated appellants. However, accurate records aren’t kept as to the reasons for their removal, their demographic profile or other case characteristics—data that could inform future targeted efforts.

Recommendation 1: Navigated preparation manifests in different ways among different appellants. Most are better informed and prepared but a small minority of approximately 7% still struggle. The primary reason likely stems from appellant-specific limitations. As the navigator system matures and looks to expand to new recipients, consideration should be given to finding ways to track and address the needs of appellants still facing challenges to the fullest possible participation in their appeal.

Withdrawal rates

The converse of an informed appellant ready for their hearing is one who makes an informed decision to withdraw and terminate their appeal. The data suggest that the information and guidance navigators provide correlates with a lower withdrawal rate. Only 7% (n=57) of appellants withdrew compared to 18% (n=193) in the pre-navigated group of unrepresented CPP-D appellants (control group data is too small to compare).

When navigated appellants did decide to withdraw, they did so an average of 170 days after filing—long after receiving multiple rounds of phone calls and letters with targeted information. To contrast, pre-navigated appellants took an average of 302 days to withdraw. The influence of navigators on appellant empowerment, particularly on making an informed decision to withdraw or proceed, is underscored in figures 10 and 11 below.

Figure 10 and 11: Days to withdraw by monthly cohort: Withdrawal trendlines mirror days ready to proceed

Line chart that shows the average number of days to withdraw by month for pre-navigated appeals
Text version

Figure 10: Days to withdraw by monthly cohort: Pre-navigated appeals withdrawn - Withdrawal trendlines mirror days ready to proceed

  • November 2018: 270 days
  • December 2019: 270 days
  • January 2019: 300 days
  • February 2019: 260 days
  • March 2019: 282 days
  • April 2019: 297 days
  • May 2019: 311 days
  • June 2019: 265 days
  • July 2019: 328 days
  • August 2019: 238 days
  • September 2019: 258 days
  • October 2019: 273 days
A line chart of navigated appeals, showing 274 days to become ready to proceed
Text version

Figure 11: Days to withdraw by monthly cohort: Navigated appeals withdrawn - Withdrawal trendlines mirror days ready to proceed

  • November 2019: 274 days
  • December 2019: 308 days
  • January 2020: 300 days
  • February 2020: 200 days
  • March 2020: 141 days
  • April 2020: 132 days
  • May 2020: 97 days
  • June 2020: 88 days
  • July 2020: 104 days
  • August 2020: 93 days
  • September 2020: 109 days
  • October 2020: 93 days

On their face, the above 2 line charts nearly duplicate the previous 2 in figures 5 and 6, but whereas figures 5 and 6 measured days to readiness, the above 2 measure days to withdraw. The 2 pairs of trendlines closely mirror one another, including the same relative stability in 2019 followed by the same sudden downward acceleration around February 2020. This strongly suggests that messaging from navigators gets through to appellants, and correlates with appellants making timely choices to either withdraw their appeal or proceed to their hearing. With the help of a navigator, far fewer appellants (61% fewer) make the choice to withdraw.

Website usage

Navigators appear to effect behavioural changes in other unintended ways. For example, navigated appellants are less likely to visit the SST’s website than all other unrepresented parties. Survey research shows that parties visit the website at varying rates to research and prepare for hearings. Only 42% of navigated appellants reported visiting the website versus 50% of the control group and 60% for all other unrepresented parties. When asked, the most common explanation from surveyed appellants is that they didn’t need to thanks to their navigator.

Earlier readiness and withdrawals plus fewer web hits, therefore, would go toward satisfying the basic program output in the navigator service (see Annex A: Logic model of navigator service), of navigators delivering actionable, client-focussed information.

Adjournment rates

Figure 12: Adjournment rate fell after navigator

A column chart showing adjournment rates of navigated, control group, and pre-navigated appeals.
Text version

Figure 12: Adjournment rate fell after navigator

  • Navigated appeals: 9%
  • Control group appeals: 6%
  • Pre-navigated appeals: 16%

The evaluation set out to measure any changes to the adjournment rate. A drop in adjournments would indicate that navigated appellants, acting on time-bound information and reminders from their navigators, are less likely to seek a delay to their scheduled hearing. However, the results appear to be inconclusive. Data (see figure 12, above) show that in the year prior to navigator, the adjournment rate for unrepresented CPP-D appellants was 16%. In the year after navigator launched, that rate fell to 9% for navigated appellants, but also 6% for control group appellants. That the rate fell significantly for control group as well as navigated appellants is more difficult for the data to explain behaviourally, and suggests that factors other than navigator contribute to adjournment patterns. An analysis of appellants’ reasons for asking for an adjournment didn’t yield any clarity either. Added to this, the adjournment rate for CPP-D appellants with a professional representative exceeds all unrepresented appellants at 16 to 18% over the last 2 years, suggesting another layer of factors that drive adjournments. A separate, fulsome study would be needed to establish any correlation between adjournments and variables like navigator.

4.3 Meaningful participation in the hearing

What we found

Preliminary evidence supports the finding that most, but not all, navigated appellants are able to participate meaningfully, resulting in proceedings that are more focused and streamlined.

Why this is important

The finding is important as it goes to confirming what the navigator service was designed to achieve: self-represented appellants interacting as an engaged party in their proceedings.

What the evidence says

Figure 13: Unrepresented appellants have high levels of satisfaction with their hearing participation

column chart showing satisfaction levels for appelants
Text version

Figure 13: Unrepresented appellants have high levels of satisfaction with their hearing participation

Question 4: At your hearing, you were able to participate fully. For example, you were able to answer questions, correct any errors about the facts, or explain your case.

  • Navigated: 95%
  • Control group: 100%
  • All other unrepresented: 96%

After their hearings are over, appellants have recounted their experience through informal and systematic means, respectively post hoc feedback and the client satisfaction survey. Through direct emails or messages relayed by members, appellants frequently share feedback that is generally positive about their hearing experience. In the survey, 95% reported agreement with the statement, “At your hearing, you were able to participate fully. For example, you were able to answer questions, correct any errors about the facts, or explain your case.” Even appellants who cited a personal barrier—illiteracy, memory recall, mental health—gave positive feedback about having been able to participate and present their interests. They typically attributed their hearing performance to their navigators and flexible SST members.

The 95% rating is on par with the 100% of the control group and 96% of all other unrepresented appellants (see figure 13 above). To shed light on why navigated appellants don’t claim highest satisfaction levels among their unrepresented peers, the detailed comments from survey respondents are useful. Virtually all appellants before the SST say they participated meaningfully but those who lack a navigator depend on the affirmative steps of the member to create a patient and welcoming environment. These include longer opening preliminaries, more guidance over procedural matters, and giving the floor to appellants. In contrast, navigated appellants in general, by virtue of their pre-hearing support, are less at the mercy of the unknown. As one member described:

Navigators answer a lot of appellant's questions and concerns before the hearing that allows me to focus on getting their evidence at the hearing. Also, appellants seem to understand better what the legal test is for the hearing and what they’re expected to do at the hearing.

Prior to the hearing, members and navigators generally don’t communicate with one another about their files. One reason is that members aren’t assigned to the file until relatively late in the process when all parties are ready. When ad hoc contact happens, it ranges from the member simply reading the navigator’s call log records to arranging a pre-hearing conference to consider accommodating a distressed appellant. A few of the 11 navigators remarked that members now reach out to them more frequently to gather information and insight for upcoming hearings. Contact between the 2 groups would appear to be an emerging, if underused, best practice that can shape members’ hearing strategy.

Recommendation 2: Since navigators have built up an intimate awareness of appellants’ circumstances, the navigator system should encourage, within the limits of permissible communication with independent decision-makers, regular contact to facilitate appellant participation.

Members generally have observed other differences in how navigated appellants engage, including awareness of the role and independence of the member, demeanour and confidence, and direct answers to questioning. To be sure, not all members share these findings; one-quarter don’t report any noticeable changes. Table 2 below lists what members have most commonly observed from navigated appellants.

Table 2: Ways that navigated appellants participate differently
In what ways do navigated appellants participate better or differently, if at all, compared to pre-navigator or compared to other unrepresented appellants? (Check all that apply.)
Navigated appellants display better understanding of the process, including the roles of each participant 63%
Navigated appellants exhibit better demeanour—they’re more relaxed and confident 37%
Navigated appellants give answers that are responsive and thoughtful 26%
Navigated appellants don’t show any meaningful difference from other unrepresented appellants 11%
Not sure or not applicable 21%

Some members have indicated practical ways that hearings have unfolded differently for the better. Because navigated appellants already have at least a general sense of hearing procedures, members spend less time on explanations and can move on to substantive matters. About a third of members also saw their hearings stay focussed on the relevant issues and elicit more direct testimony. Again, however, the extent of change depends on the individual appellant, and fully one-third of members don’t report any meaningful change to how their hearings take shape.

4.4 Other findings

Navigators and management commonly viewed the navigator system as adequately resourced and delivering what it’s designed to do. A few did remark that as the caseload has been increasing, system stresses have begun to show. These have manifested in growing individual workloads which may slow navigators from following up with and taking on new appellants. Some staff have also begun to take on other duties, such as the early resolution initiative, which further constrain workload manageability. Staffing actions currently underway are anticipated to provide some relief.

One interviewee observed that navigator training was delivered in English rather than in both English and French. Although materials are translated into French for navigators’ reference, fluidly explaining complex concepts and terminology, such as minimum qualifying period, demands another skill set not addressed by the training.

Many team leaders agreed that navigators’ mental health and self-care are an issue. Navigators’ job of listening and responding to daily outpourings from emotional appellants can take a toll on well-being. The SST provides navigators with a telephone counselling service but, notably, none of the navigators interviewed in focus group sessions offered mental health as an issue for discussion. This is perhaps due to the unprivate nature of the focus group. One of the navigators privately cited this as an issue they’re conscious about.

Recommendation 3: Action should be taken to monitor caseload trends and challenges raised by personnel, particularly to anticipate or inform evolving requirements for resources, training and navigator wellbeing.

5.0 Conclusion and recommendations

Unlike frequently asked questions, letters or online guides, navigators proactively deliver access to justice. Theirs is a unique approach to mitigating unequal access and has no known analogues in the tribunal world. That has made it a challenge to not only design the program, but to benchmark and evaluate its performance 14 months after launch.

Through a mix of case data and testimonials, the evidence appears to largely confirm what one expects from a service that provides hands-on support. Appellants who receive time-bound information and guidance should demonstrate heightened preparations. Appellants who are properly prepared should signal readiness for their hearings sooner and in greater numbers. When those individuals interact with members for the first time, members should recognize an appellant better equipped to advocate their interests.

This report throughout has qualified its findings—most appellants, most hearings, most members—recognizing that not all appellants have been able to parlay navigator support into meaningful action. But the initial evidence of system performance confirm progress toward program expectations with due consideration of earlier recommended action, reproduced below.

Recommendation 1: Navigated preparation manifests in different ways among different appellants. Most are better informed and prepared but a small minority of approximately 7% still struggle. The primary reason likely stems from appellant-specific limitations. As the navigator system matures and looks to expand its effectiveness to new recipients, consideration should be given to finding ways to track and address the needs of appellants still facing challenges to the fullest possible participation in their appeal.

Recommendation 2: Since navigators have built up an intimate awareness of appellants’ circumstances, the navigator system should encourage, within the limits of permissible communication with independent decision-makers, regular contact to further facilitate appellant participation.

Recommendation 3: Action should be taken to monitor caseload trends and challenges raised by personnel, particularly to anticipate or inform evolving requirements for resources, training and navigator wellbeing.

Annex A: Logic model of navigator service

A flowchart showing the logic model of the navigator service.
Text version

Inputs:

A - Staff, training, materials, under-represented appellants

Activities:

B - Provision of navigator services

Outputs:

C - Appellants needs-assessments

D - Appellant-focused guidance or plans

Outcomes / Impact:

  • Short term:
    E - Ready to proceed: Appellants understand their obligations and requirements to be ready to proceed
  • Medium term:
    F - Meaningful participation: Appellants participate meaningfully in their hearing
  • Long term:
    G - Improved access to justice: Improved access to justice for self-represented persons

Annex B: Evaluation matrix

Evaluation question

Indicators Data collection method/source Control group comparison?
1.0 To what extent is navigator achieving its intended outputs—needs assessments, appellant-focused guidance or plans? 1.1 Evidence of achievement of needs assessments, guidance or plans 1.1.1 Navigator documents, for example call summary letters No
1.1.2 Navigator focus group No
1.1.3 Client survey and case study Yes
2.0 Is navigator resulting in appellants who understand their obligations and requirements to be ready? (short-term outcome) 2.1 Feedback from key groups on the extent that clients understood their obligations and requirements to be ready 2.1.1 Navigator focus group No
2.1.2 Client survey and case study Yes
2.1.3 Member/manager interviews Yes
3.0 Is navigator resulting in appellants able to participate meaningfully in their hearing? (medium-term outcome) 3.1 Adjournment rate + reasons 3.1.1 Atrium data Limited
3.2 Feedback from key groups on the extent of client participation in the hearing 3.2.1 Client survey and case study
3.2.2 Member/manager interviews
4.0 Is navigator enhancing access to justice for under-represented appellants? (long-term outcome) 4.1 Withdrawal rate 4.1.1 Atrium data Yes
4.2 Appellant satisfaction 4.2.1 Client survey, case study and member survey
5.0 Are there opportunities to improve the efficiency of the navigator program while continuing to meet the needs of its recipients? 5.1 Evidence of efficiency opportunities 5.1.1 Navigator focus group No
5.1.2 Manager interviews
6.0 Any unintended outcomes as a result of the navigator program? 6.1 Evidence of outcomes, negative or positive, not described in the logic model 6.1.1 All data sources Yes

Annex C: Personnel interviewed

Manager and team leaders

  • Misha Buchholz
  • Ramy Ibrahim
  • Benoit Majeau
  • François Morin
  • Mikaela Smith

Navigators

  • Emily Balice
  • Sagal Douksieh
  • Melissa Erion
  • Raymond Grenier
  • Beth Kirby
  • Ari Munisami
  • Rachel Pederson
  • Kelsey Scobie
  • Susan Scott
  • Sarah Wade
  • Karen Winsor

Annex D: Client satisfaction survey

Strongly agree Agree Disagree Strongly disagree Comments
1. You found our forms, letters, and emails easy to understand.
2. Before your hearing, you knew what to expect and how to prepare for your hearing.
3. If the SST assigned you a navigator, your navigator was helpful in getting you ready for your hearing.
4. At your hearing, you were able to participate fully. For example, you were able to answer questions, correct any errors about the facts, or explain your case.
5. You’re happy with how quickly the SST handled your appeal.
6. You’re happy with the type of hearing you had. [Possible types are teleconference, videoconference, and in person.]
7. [If hearing was held by videoconference] Would you recommend a Zoom videoconference to others?

Why or why not?

8. Have you ever visited our website?

If no, why not?

If yes, was it easy to find what you were looking for?

9. Do you have any other comments to help us improve our services?

Annex E: Survey to members

  1. To what extent do you find that navigated appellants come to their hearing more familiar with the appeal process and better prepared to present their case, particularly compared to pre-navigator or compared to other unrepresented appellants.          
    • Much better
    • Somewhat better
    • About the same
    • Somewhat worse
    • Worse
    • Not sure or not applicable
    • Please explain why:
  2. Do you agree with the statement?: “Navigated appellants are ready to proceed from the start of the hearing. I do not need to spend much time to first explain, inform, or guide.”
    • Yes
    • No
    • Not sure or not applicable
    • Please explain why:
  3. In your experience, do navigated appellants ask for an adjournment more or less often compared to pre-navigator or to other unrepresented appellants?  
    • More
    • Less
    • Not sure or not applicable
    • Please explain why
  4. In your experience, do navigated appellants ask for an adjournment for different reasons compared to pre-navigator or to other unrepresented appellants?
    • Yes
    • No
    • Not sure or not applicable
    • Please explain why
  5. Do navigated appellants arrive with documents at the hearing or submit post-hearing documents? If so, why do you think this is happening?
    • Yes
    • No
    • Not sure or not applicable.
    • If yes, why do you think this is happening?
  6. In what ways do navigated appellants participate better or differently, if at all, compared to pre-navigator or compared to other unrepresented appellants? (Check all that apply)
    • Navigated appellants submit more or better evidence to support their appeal
    • Navigated appellants give answers that are responsive and thoughtful
    • Navigated appellants exhibit better demeanour—they are more relaxed and confident
    • Navigated appellants display better understanding of the process, including the roles of each participant
    • Navigated appellants don’t show any meaningful difference from other unrepresented appellants
    • Not sure or not applicable
    • Other:                                           
  7. In what ways, if any, do your hearings unfold differently when faced with a navigated appellant? (Check all that apply)
    • I spend less time on preliminaries, such as explaining the process to the appellant and answering procedural questions
    • I can ask questions that get to the point and get answers
    • My hearings are more focussed and efficient
    • Not sure or not applicable
    • Other:
  8. In your opinion, prior to appellants having navigators, what were the barriers that unrepresented appellants faced to participating meaningfully in their hearing? (Check all that apply)
    • Lack of preparation before the hearing
    • Lack of knowledge of the SST’s mandate, process or procedures
    • Lack of confidence
    • Barriers related to appellant’s mental or physical health, education, knowledge of English or French
    • Not sure or not applicable
    • Other:
  9. Based on the appeals you have heard, to what extent have navigators begun to address those needs?
    • Significant extent
    • Somewhat
    • Very little
    • Not at all
    • Not sure or not applicable
    • Please explain:
  10. At the end of their hearing, do navigated appellants give you any impression or indication that they were satisfied with how their hearing unfolded and that they “had their day in court”?
    • Yes
    • No
    • Not sure or not applicable
    • Please explain
Date modified: