INTRODUCTION
Recent years have seen in Israel increased efforts to address the social problems that lead to crime. This was partly the result of a recognition among justice officials and policymakers of the limitations of the traditional criminal process in providing an effective response to crime. Accordingly, there has been a renewed search for creative solutions that would reduce criminality and strengthen rehabilitation pathways, alongside increasing public trust in law enforcement systems.
Against this background, a new model of community courts (CCs) was developed to provide diverse tools for coping with crime and the problems leading to it. The establishment of the Israeli CC model was led by the Joint-Ashalim Association, a partnership between the Israeli government and the Jewish Distribution Committee (a non-profit organization), with senior representatives from various players involved in the criminal process. The model is based on collaboration among the courts, law enforcement agencies, welfare authorities and community representatives. It is inspired by the CC model operating in the United States, with changes and adaptations made to the Israeli legal system and the work of local welfare and rehabilitation authorities.
During the programme’s pilot phase, we conducted formative research, and its summary report was submitted in July 2017 (Gal and Dancig-Rosenberg Reference Gal and Dancig-Rosenberg2017a) (from now on referred to as the “formative study”). The formative study described the establishment process of the CCs, analysed the unique characteristics of the programme’s legal process, identified professional positions regarding its goals and challenges, and gathered data on participants’ attitudes towards the programme and the process they went through.
In November 2017, as the project progressed and toward the completion of the pilot phase, Joint-Ashalim, in collaboration with the Courts National Administration, the Adult Probation Service, the Ministry of Justice and the Israeli Police (from now on referred to as “the partners”) issued a tender for conducting an evaluation study of the programme, designed to “examine the programme’s outcomes, the implementation of its different components, and their effectiveness”. Several applicants submitted proposals, and, eventually, we won the tender.Footnote 1
The present article aims to tell the “backstage story” of the formative and evaluation studies that examined the Israeli CC model from its inception. Specifically, it describes three modules in these studies, which represent distinct methods and research questions: the process characterization module, which was based on structured observations of court hearings during the formative study; the module focusing on subjective experiences of court participants, which were collected via in-depth interviews with male and female programme participants; and the recidivism module, which compared the repeat offending patterns of programme participants with those of mainstream court defendants. In discussing these three modules, the article aims to achieve two goals: first, to demonstrate the multiplicity of research questions relevant in evaluation studies which can produce important knowledge and the related methods applied to address these questions, and second, to discuss the challenges and dilemmas associated with formative and evaluative studies in the field of criminal justice.
Establishing the Israeli Community Courts
Initiated by Joint-Ashalim, the interagency steering committee of the new project conducted extensive groundwork, which eventually served as the basis for the project’s operational model. With the permission of the Supreme Court Presiding Justice and of the ministers of all other relevant ministries, a pilot was launched – establishing two CCs in two judicial districts. One began operating in Be’er Sheva, the largest city in the south of Israel with over 200,000 people, in December 2014. The other began its activity in Ramle, a mixed city with significant populations of Arabs and Jews in central Israel, in September 2015. Local teams were trained in each of the courts, a structured legal process with clear stages was built, and significant efforts were invested in constructing and establishing a community framework that enables immediate and effective referral of defendants to the local services they need.
Two years into the pilot project, in 2016, the Israeli government decided to reform its punishment and defendant rehabilitation policy in Israel. The government’s decision included the establishment of CCs in additional judicial districts in Israel, among other components. This decision resulted from the recommendations of the Dorner Public Committee, which was established to examine Israel’s punishment and rehabilitation policies. Following the government’s decision, an additional CC began operating in the Tel Aviv district in May 2017, which absorbed cases handled until then by the drug court operating there. In October 2019, the Nazareth CC, with jurisdiction over multiple Arab and Jewish villages in addition to the city of Nazareth itself in the north of Israel, began its operation, and in September 2020, two CCs were added in Haifa, the Northern Bay metropolis, and Jerusalem, Israel’s capital. In October 2022, two additional CCs were launched in two smaller cities, Acre in the North and Kfar Sabah in the centre of Israel.
So far, about 1,300 defendants, referred to as “participants”, have been included in the programme. Eligible participants are adults who, based on their underlying problems, are likely to reoffend and who would face incarceration of up to approximately two years if they were convicted in the mainstream criminal court in the case at hand. Participants progress through different predetermined stages, starting with Stage A – the introduction and development of the treatment plan, and ending with Stage E – programme completion (from now on referred to as “programme stages”). If participants fail to complete the programme, they are sent back to the mainstream criminal process to be sentenced, with the possibility of being incarcerated. The holistic treatment plan addresses challenges in five realms: physical and mental health; support networks; personal well-being; professional training and employment; and establishing a law-abiding routine.
Community Courts: Theory, Practice and Evaluation
CCs, presently operating in approximately 80 localities worldwide, emphasize, in addition to offender rehabilitation, the restoration and strengthening of the community suffering from crime (Karafin Reference Karafin2008; Miller, Block, and DeVault Reference Miller, Block and DeVault2020). Therefore, these courts focus on offences that affect the well-being and security of residents. The punishment policy of these courts is tailored to the community’s needs and combines community rehabilitation and empowerment with retribution and deterrence as secondary goals (Karafin Reference Karafin2008). Accordingly, sentences in CCs often include a significant component of restitution to the affected community, highlighting the responsibility of offenders towards the community in which they live. At the same time, CC sentences often rely on community resources, such as local rehabilitation programmes, welfare services and employment.
The CC model preserves the fundamental principles of the criminal justice process and recognizes the importance of formal procedures while promoting rehabilitation and community engagement in appropriate cases. CCs strive to increase the number of defendants referred to treatment programmes as an alternative to traditional punishment, expand the rehabilitative toolbox available to the justice system, and improve the system’s ability to use welfare services and community resources effectively. Indirect desired outcomes of CCs are increased public trust in law enforcement, enhanced procedural justice perceptions among defendants, and strengthened communities (Casey and Rottman Reference Casey and Rottman2005; Malkin Reference Malkin2003).
The CC model was developed in the United States to address the daily problems of neighbourhoods suffering from high crime rates that affect the quality of life and the general level of security of residents and the community (Berman and Fox Reference Berman and Fox2005; Clear and Karp Reference Clear, Karp, David and Lanham1998). Developed within the framework of problem-solving courts, CCs aim not only to rehabilitate the defendant but also to restore the community suffering from a high level of crime, strengthen the sense of security of residents, and improve their trust in law enforcement systems (Casey and Rottman Reference Casey and Rottman2005; Malkin Reference Malkin2003). CCs focus on offences that affect the quality of life and daily security of residents, such as property crimes, drug use, violence and various harassments, according to the needs of the community in which they are located. In contrast to problem-solving courts, CCs do not specialize in treating a specific problem but focus on the range of problems and needs of a specific community (Lee et al. Reference Lee, Cheesman, Rottman, Swaner, Lambson, Rempel and Curtis2013). Despite the diversity arising from the characteristics and needs of each community, all CCs share several components (Casey and Rottman Reference Casey and Rottman2005; Lang Reference Lang2011; Lee et al. Reference Lee, Cheesman, Rottman, Swaner, Lambson, Rempel and Curtis2013): collaboration with various resources in the community; community involvement; adaptation of treatment tools at the individual level; use of alternatives to imprisonment and expansion of existing means of punishment; emphasizing the responsibility of the accused; and setting success metrics (Fagan and Malkin Reference Fagan and Malkin2002; Lee et al. Reference Lee, Cheesman, Rottman, Swaner, Lambson, Rempel and Curtis2013). Alongside the court’s therapeutic and rehabilitative role, the criminal process’s formal framework is maintained, and the court continues to play a central and significant role in monitoring and tracking the participants’ progress along their treatment plan (Beinisch and Cohen Reference Beinisch and Cohen2014).
Criminological Theories of the CC Model
Several criminological theories informed the CC model; the most notable are the Broken Windows theory (Wilson and Kelling Reference Wilson and Kelling1982) and Community Justice theory (Clear and Karp Reference Clear and Karp2018). The first highlighted the importance of paying attention to local signifiers of neglect and antisocial activities; the second promoted community self-regulation and problem-solving. Other theories, however, are closely linked with the ideology and implementation of CCs. Therapeutic jurisprudence (TJ), the study of the therapeutic and anti-therapeutic effects of the law on individuals, is the underlying theory shared by all problem-solving courts (Wexler Reference Wexler, Stobbs, Bartels and Vols2019). Based on TJ, problem-solving courts, also termed therapeutic courts, utilize therapeutic practices in the judicial work, for example, the use of a behavioural contract, motivational dialogue, and “sticks” and “carrots” to maximize defendants’ collaboration and motivation (Winick Reference Winick2003). The procedural justice theory highlights the importance of individuals experiencing respect, neutrality, voice, clarity and trustworthiness during the legal process of accepting the decision (Tyler Reference Tyler1990). Problem-solving courts, including CCs, have attempted to integrate procedural justice into their daily operation in addition to their therapeutic practices (Herzog-Evans Reference Herzog-Evans, Stobbs, Bartels and Vols2019). Positive criminology promotes strengths-based and future-oriented approaches to rehabilitative practices (Ronel and Segev Reference Ronel and Segev2015). CCs incorporate positive criminology into the legal process by discussing defendants’ goals and achievements in the courtroom (Gal and Wexler Reference Gal, Wexler, Ronnel and Segev2015).
Despite the clear adherence of CCs to certain aspects of various criminology theories, these courts have also attracted significant criticism. Critiques against problem-solving courts, and CCs among them, include concerns about their lack of consistency in the referral and treatment of different defendants, potentially enhancing racial discrimination and net-widening (Hoffman Reference Hoffman1999), lack of transparency and accountability of the process, leading to violations of due-process rights (Miller Reference Miller2009), and their focus on low-level, non-violent offences (“cherry-picking”) (Collins Reference Collins2020), suggesting that CCs are unequipped to address the root causes of more complex criminal behaviour. Some critics focused on the lack of robust empirical evidence supporting the claim that CCs are more successful in reducing reoffending and rehabilitating offenders (Ross Reference Ross2015).
Do Community Courts “Work”?
Although the stated goals of CCs are to address the root causes of crime and strengthen community cohesiveness, CC programmes differ in their evaluation approaches, and many need to engage in rigorous evaluations using quantified indicators (Karafin Reference Karafin2008). Indeed, in stark comparison with the dozens of CC programmes worldwide, only a few evaluation studies provide reliable data on their success. Among the existing evaluation studies, most have focused on the effectiveness of these courts in reducing repeat offending. Others have also looked at completion rates and overall reductions in the use of incarceration. Some have additionally examined defendants’ experiences of procedural justice in the CCs.
Several studies provide evidence regarding the effectiveness of CCs in reducing recidivism rates using a valid comparison group. While the results are mixed, a few more studies found a significant reduction in recidivism rates than those without such reductions.
For example, an evaluation study of the Red Hook Community Justice Center (RHJC) in Brooklyn compared the repeat offending of defendants who were referred to magistrate courts in Brooklyn with that of defendants accused of similar crimes who were referred to the RHJC. The study found that two years after adjudications, the chances of being re-arrested were 10% lower for RHJC graduates) Lee et al. Reference Lee, Cheesman, Rottman, Swaner, Lambson, Rempel and Curtis2013). Similar significant reductions in re-arrest and reconviction rates were found among graduates of the Washington State East of the River Community Court (Westat 2012), the Vancouver Downtown Community Court (Somers et al. Reference Somers, Moniruzzaman, Rezansoff and Patterson.2014) and the Yarra Neighbourhood Justice Centre in Melbourne, Australia (Ross Reference Ross2015) – all were compared with equivalent defendants adjudicated in mainstream courts nearby. In Seattle, a study comparing recidivism rates among participants in the Seattle Municipal Community Court and those of defendants in mainstream courts did not find significant differences. Nevertheless, the CC participants committed crimes less often than defendants in mainstream courts (Nugent-Borakove Reference Nugent-Borakove2009).
In contrast, three studies found no significant reductions in reoffending, two focusing on the North Liverpool Community Justice Centre in England (Booth et al. Reference Booth, Adam Altoft, Gonçalves and Mirrlees-Black2012; Jolliffe and Farrington Reference Jolliffe, Farrington, McMurran and Howard2009) and one on defendants referred to the Indianapolis Community Court (Grommon, Hipple, and Ray Reference Grommon, Kroovand Hipple and Ray2017).
Other studies examined perceptions of court participants regarding the process fairness and found significantly higher perceptions of procedural justice and satisfaction among CC participants compared with mainstream court defendants (Custer, Cissner, and Finkelstein Reference Custer, Cissner and Finkelstein2008; Frazer Reference Frazer2006; Halsey and de Vel-Palumbo Reference Halsey and de Vel-Palumbo2018; Lee et al. Reference Lee, Cheesman, Rottman, Swaner, Lambson, Rempel and Curtis2013).
Although CCs strive to address the root causes of crime and improve community cohesiveness, more must be known about whether and to what extent these courts achieve these goals. Scholars have suggested that evaluation studies of criminal justice interventions should be more holistic in their measurements and go beyond the binary question of recidivism (Klingele Reference Klingele2019; McNeill et al. Reference McNeill, Farrall, Lightowler and Maruna2012). For example, a systematic review of mixed-methods evaluation studies in criminology and criminal justice by Wilkes et al. (Reference Wilkes, Anderson, Laura Johnson and Mae Bedell2022) found that, while there have been small increases in the prevalence of qualitative and mixed-methods research in recent years, the vast majority of studies are still solely quantitative. This is despite the valuable data that qualitative interviews can add to quantitative data in unpacking specific mechanisms at work and participant experiences in the programme. Indeed, although in other contexts, such as within the literature on desistance, there has been significant work unpacking the in-depth changes involved in the process of desisting from crime, programme evaluation studies have very seldom moved beyond the “what works” question to explain “why” and “how” programmes work (Farrall and Maruna Reference Farrall and Maruna2004, 360). Accordingly, researchers have highlighted the importance of additional measurements beyond recidivism. Berman (Reference Berman2010), for example, stressed the importance of measuring procedural justice perceptions among programme participants. Karafin (Reference Karafin2008) constructed a list of indicators to be used as measurements of success for CCs, which includes those assessing the success of the programme in helping participants resolve their underlying problems, addressing community needs, improving the public perception of the court, increasing offender accountability, and focusing on “quality of life” crime.
In the course of evaluating the Israeli CCs, we learned from the professionals involved in the establishment and operation of the programme that to measure the meaning of “success” from the individual perspective of programme participants, researchers need to inquire about their mental and physical health, employment, relationships, daily routines and programme experiences, beyond the narrow question of recidivism (Dancig-Rosenberg and Gal Reference Dancig-Rosenberg and Gal2024). Accordingly, in shaping our studies about the Israeli CC model, we sought to provide a broader set of questions about their operation. These included an examination of the actual operation of CCs, the subjective experiences of CC participants, and the ways these courts affect the circles around them. This mixed-methods and multi-questions study approach is beneficial in terms of the wide scope of insights the study can provide, but the multiplicity of research questions and methods also creates numerous methodological, ethical and logistical challenges, as described below.
Methodological and Ethical Challenges in Measuring the Effectiveness of CCs
Evaluating diversionary justice mechanisms such as CCs has methodological and ethical challenges. Each method presents its own difficulties and general challenges emerging from the collaboration between practitioners and researchers (Drawbridge, Taheri, and Frost Reference Drawbridge, Taheri and Frost2018).
The most commonly discussed methodological debate is how best to address the “million-dollar question” of whether CCs decrease crime by reducing repeat offending among those referred to them. The best way to examine whether an intervention is effective is to conduct a randomized experiment, but such studies are often very difficult to conduct due to ethical and practical barriers (Lum and Yang Reference Lum and Yang2005; Weisburd Reference Weisburd2000). A more common methodology in evaluation studies is to compare the recidivism rates of defendants referred to specialized courts with those of defendants in regular courts. This methodology requires the collaboration of law enforcement and legal authorities and raises various challenges related to selecting an adequate control group and the availability of sufficient longitudinal data beyond a one-year follow-up. For example, Murray and Blagg (Reference Murray and Blagg2018) discuss the experience of the North Liverpool Community Court, where mixed findings of the programme evaluations led to its closure in 2012.
Collecting data directly from programme participants yields important knowledge about the programme’s effectiveness in enhancing their trust toward law enforcement and willingness to obey the law, but creates additional challenges. Roberts and Indermaur (Reference Roberts and Indermaur2008), focusing on research with prisoners, described the difficulty of gaining the trust of the prisoner-interviewees and highlighted the importance of ensuring that interviewees have access to a direct line for emotional support should they need one. Such help may be necessary when their participation in the research involves answering direct, moral and emotional questions, although such participation may benefit their rehabilitation. Another risk that researchers should consider is that programme initiators or policymakers will use the research findings selectively for political reasons (Roberts and Indermaur Reference Roberts and Indermaur2008). Indeed, maintaining independence and integrity is crucial for researchers evaluating existing law enforcement programmes (Naylor Reference Naylor2015).
In narrative criminology, Maruna and Liem (Reference Maruna and Liem2021) considered existing challenges in conducting qualitative studies. These include: the risk of interviewees telling the interviewer what they want to hear and the easiness with which offenders lend the story of their criminality (or desistance); of narratives being a co-production of the narrator and the listener, with often coercion or manipulation on the part of the interviewer; and the instability of narratives – their relevance to only specific points in time. To address these challenges, researchers have called for reflexivity and transparency regarding the background and motivation for conducting the study, the settings of interviews, how the interviewers identify themselves, and their contribution to the discussion (Maruna and Liem Reference Maruna and Liem2021). It is also important to conduct the interviews in the interviewee’s natural environment, to add context, when possible, to the findings section, and to have complementary data sources for triangulation. To eliminate the risk that research participants use their stories to achieve benefits, interviews should be confidential, and anonymity must be ensured. To move beyond the standard spiel that offenders provide to social workers, researchers have used various methods in organizing semi-structured interviews with offenders, such as LAAF (life as a film), LSI (life story interview) or LHC (life history calendar) (Maruna and Liem Reference Maruna and Liem2021).
These few examples provide insights into some of the challenges assigned to specific research methods or perspectives, but there is still very little open and comprehensive discussion about what takes place “behind the scenes” of large-scale, mixed-methods projects and the ways to address the multiple challenges they present. The current article seeks to expand the literature discussing these challenges. In the context of evaluating the Israeli CCs, we sought to address multiple questions relating to “whether” the CCs are effective in achieving their goals, “how” they achieve them and “why” they are successful, to the extent they are. To address these questions, we used a range of methodologies in the two research projects that examined the Israeli CCs. The following section describes the research questions and the methodologies we undertook to address them. The rest of the paper will provide a deeper look into three modules of our examination to demonstrate the scope of the studies and the research questions, the diverse methodologies we utilized and the challenges we faced in implementing them.
METHODOLOGY
The formative study that we conducted between 2015 and 2017 had three objectives. The first was documenting the process that led to the establishment of the CCs, from the initial proposal through the construction of the operational model to the actual operation of the courts. Such documentation may assist in establishing other innovative justice models, particularly those based on complex interagency collaboration. The second objective was to analyse, characterize and theorize the dynamics in the CCs, considering the dramatic departure of the CC model from the mainstream adversarial criminal court model. The third and final objective was to capture the views of CC participants about the justice process in the CCs and the justice system as a whole.
The evaluation study lasted between 2018 and 2023 and had two main objectives. The first was to measure the programme’s effectiveness, whether and to what extent it achieves its goals, all or part of them, among the participants. The second was to identify variables and factors that affect the level of effectiveness of the programme.
The research questions in both studies were derived from their objectives; we used a mixed-methods approach combining quantitative and qualitative research techniques and approaches (Maruna Reference Maruna, Piquero and Weisburd2010) to obtain as broad a picture as possible. Our data collection methods included structured and ethnographic observations, surveys among CC participants, in-depth interviews with professionals, participants, and family members, and archival and statistical analyses of several databases. Appendix 1 describes the multiple research questions, data collection methods and the samples we used in the two studies.
Instead of providing a full description of the multiple methodologies and the findings, the present article focuses only on three modules, each addressing a different research question and representing a different methodology, each posing unique challenges and features relevant to other evaluation studies. Because the goal of the paper is not to discuss the findings of these modules but rather to examine the research process itself, the presentation of each module includes only a brief description of the method and the findings and a more detailed discussion of its contribution, challenges and dilemmas it entailed.
Module 1: Characterizing the CC’s Daily Operation
We strove to capture the court dynamics in the CCs in ways that would enable us and others to compare them to other criminal justice processes and mechanisms. In an era of multidoor criminal justice (Dancig-Rosenberg and Gal Reference Dancig-Rosenberg and Gal2019), there is a need for standardized instruments that provide detailed characterizations of various justice mechanisms. Although such an analytical instrument cannot provide answers as to the effectiveness of the programme’s various components in obtaining its goals, it should be able to give a sufficiently detailed portrayal of the legal mechanism and its daily implementation in ways that enable a structured comparison between the researched programme and others. The Criminal Law Taxonomy (CLT), developed elsewhere (Dancig-Rosenberg and Gal Reference Dancig-Rosenberg and Gal2014), provides such an analytical instrument. The CLT includes a set of 17 measurable parameters, each presented on a scale stretched between two opposing ends. The parameters provide four clusters of characteristics: process-, stakeholder-, substance- and outcome-related ones (see Appendix 2 for a full list of parameters). For example, the CLT helps to determine the level of victim involvement in the researched justice mechanism, the level of formality of the process, the extent to which laypersons other than the justice professionals are involved in it, and the extent to which the process outcomes are either rehabilitative or incapacitative. The CLT was used successfully to characterize the dynamics in restorative justice conferences held in Israel (Gal, Dancig-Rosenberg, and Enosh Reference Gal, Dancig-Rosenberg and Enosh2018). In the formative study, we sought to provide a high-resolution analysis of the dynamics of CCs, and the CLT was a suitable instrument for that goal.
First, four research assistants were trained to familiarize themselves with the CLT and its components. The CLT is not a simple grading sheet. It involves complex and potentially unclear concepts. For example, what does the “formality–flexibility” scale refer to – is it about the level of formality in legal jargon? The professionals’ attire? What is the level of the rigidness of the proceedings? Or, to give another example, what does the dual term “process as enabling/limiting” mean? We and the research assistants discussed each of the 17 parameters’ definitions thoroughly, and we used examples to minimize misconceptions. Then, the four research assistants observed the same court hearings concurrently in the two first CCs (Be’er-Sheva and Ramle) and were asked to rate each hearing, according to 13 of the 17 parameters, on a 1–7 scale. Parameters 14–17 describing outcome characteristics were excluded from data collection because most hearings did not reach the final stage of the process, and, therefore, their outcomes could not be coded. Each coder filled a separate coding sheet with the CLT for each defendant and marked the exact start time of the hearing and the defendant’s first name. Many court hearings were unsuitable for coding during the observation days because they did not involve substantial discussions. As a result, only about half of the time spent in the courtrooms ultimately resulted in usable data. Overall, we collected four coding sheets rating 13 parameters for each of the 74 court hearings in which substantive discussions occurred. As a comparison, observers also coded, using the same CLT sheets, 70 criminal court hearings in mainstream magistrate courts in Israel (Gal and Dancig-Rosenberg Reference Gal and Dancig-Rosenberg2017a, Reference Gal and Dancig-Rosenberg2020).
In the third stage, we measured inter-rater agreement (IRA) to ensure that the instrument was reliable. We used r wg (James, Demaree, and Wolf Reference James, Demaree and Wolf1993), one of the most commonly used instruments for assessing IRA and suitable for studies with two or more coders (LeBreton and Senter Reference LeBreton and Senter2008). Weak IRA (under 0.50) reflects common disparities between the observers in describing the same dynamics, thus suggesting that the parameters are unclear, vague, or subject to individual impressions. IRA levels for both the CCs and magistrate court observations were moderate (0.51 to 0.70) to very strong (0.91 to 1.00) regarding most of the parameters, demonstrating the validity of the CLT as an analytical instrument.Footnote 2 In the fourth stage, we computed the average coding of the CCs received for each process-, substance- and stakeholder-related parameter. The same analysis was conducted regarding the data collected in the magistrate court’s observations.
Table 1 presents the average coding of the court hearing dynamics in the two CCs where observations occurred.
Table 1. Be’er Sheva and Ramle Community Courts’ Average Criminal Law Taxonomy Coding a
a The range of scores is from 1 to 7.
† p < 0.1; * p < 0.05; ** p < 0.01.
A detailed analysis of the meaning of the specific findings in Table 1 exceeded the present paper’s limits and appeared in the formative research report (Gal and Dancig-Rosenberg Reference Gal and Dancig-Rosenberg2017b). However, it is worth mentioning that in most parameters, the average coding reflects the ideology and work model of the CCs: the two first CCs operated with a clear focus on the rehabilitative needs of the specific participants and exhausted the opportunities that the existing legal framework allows for ongoing judicial monitoring and supervision (parameters 7, 10 and 12). The analysis also revealed that experience improved model fidelity: the team members in Be’er Sheva CC, which began operating a year before the Ramle court, were able to better implement the CC model in a few parameters, such as maintaining a focus on the needs rather than procedural rights of participants (parameter 10), involving volunteers and family members in the discussions (parameter 5), and highlighting participants belonging to and dependence on the community around them (parameter 13). These findings, suggesting that time and experience improved the level of attainment of programme goals and values, are important in evaluating any justice mechanism. By comparing two courts that share the same programme values and goals but, in practice, differ in their personnel’s collective hands-on experience in the programme, we were able to highlight the importance of time and practice in improving the programme’s implementation.
As mentioned, we also wanted to examine how court hearings in the CCs differed from those in mainstream magistrate courts. We, therefore, compared the average CC coding with those made in four mainstream criminal courts in three cities in Israel. Table 2 demonstrates the differences between the CCs and the mainstream courts, as reflected in the dynamics of court hearings.
Table 2. Comparing Community Courts (CCs) and Magistrate Courts Using the Criminal Law Taxonomy (Gal and Dancig-Rosenberg Reference Gal and Dancig-Rosenberg2020)
a CC processes.
b Mainstream processes.
As Table 2 demonstrates, court hearings in CCs differ significantly from hearings in mainstream magistrate criminal courts in three parameters: they are significantly more oriented toward the needs, interests and rights of the individual defendants (parameter 7); the terminology used in them is significantly more needs-based (parameter 10); and the process itself is significantly more often used as a vehicle to promote the programme’s goals (parameter 12), for example through using the option for follow-up hearings to execute an effective monitoring and supervision scheme on defendants. CC hearings are also somewhat more flexible (parameter 2), more voluntary (parameter 4, although this finding is based on weak agreement among the raters), more inclusive to indirect stakeholders (parameter 8) and more open to emotional discourse (parameter 11) than hearings in mainstream magistrate courts. Nevertheless, the analysis also demonstrates that just like in mainstream courts, CCs do not involve victim–offender dialogue (parameter 1), nor are they in any way oriented toward victims (parameter 6); they are hierarchical (parameter 3), professional-centred (parameter 5) and state-managed (parameter 9), and they focus on liberal notions of autonomy and personal liberties rather than on communitarian notions of shared responsibility and mutual dependency of community members (parameter 13). These latter characteristics reflect the stark differences between state-managed courts and community-based mechanisms such as restorative justice – as was demonstrated in a separate study where we utilized the CLT as an analytical instrument to explore the characteristics of restorative justice (Gal et al. Reference Gal, Dancig-Rosenberg and Enosh2018).
Overall, the use of the CLT allowed us to capture the unique characteristics and dynamics of the CC court hearings in ways that unstructured observations would not have generated, nor would questionnaires or in-depth interviews, that are susceptible to participants’ wishing to please programme evaluators (Maruna and Liem Reference Maruna and Liem2021). Furthermore, the CLT enabled us to compare the court hearings at the CCs with those of mainstream courts and, later, with other mechanisms such as restorative justice (Gal and Dancig-Rosenberg Reference Gal and Dancig-Rosenberg2020). It also enables tracking changes in how CC hearings are conducted at the same court over time by comparing the rates of the various parameters at different points in time.
At the same time, while using the CLT, we recognized its challenges as an instrument that requires careful training and supervision of the coders and is resource-intensive due to the need to have several coders working concurrently. Also, it captures only what happens within courtrooms, whereas a significant part of the CC programme takes place in treatment sessions, one-on-one interactions between participants and team members, community events, and more. Nevertheless, while the CLT does not provide data on the outcomes of legal interventions, it is an effective tool for examining model fidelity and identifying strengths and weaknesses in new models. For example, the CLT highlighted the need to reconsider the CC model approach toward victims and the need to explore ways to involve community members more effectively in court hearings. Moreover, integrating findings from the structured observations with findings from the recidivism data and the participants’ questionnaires opens the door for further analyses of links between procedural features in the courtroom, participants’ perceptions and repeat offending.
Module 2: Exploring the Subjective Experiences of Programme Participants
As part of the evaluation study, we conducted semi-structured interviews with 12 male participants who had either graduated from the programme or were nearing completion. Interviewees’ ages ranged between 22 and 64 years (average 37.6 years of age). Nine of them were Jewish, and three were Arab. Most of them (eight) had already graduated from the programme at the time of the interview, and the other four were in their fifth and last stage. The interviewees were charged with diverse offence types: violence (four); property (five); domestic violence (two); and drug offences (four) (some of them were charged with more than one offence). Overall, we sought to hear from the interviewees their personal experiences, insights and lessons learned from the process they underwent in the CC. Specifically, the questions in the interview guide related to: the background that led the participants to the CC; their experiences with the mainstream criminal justice system before that; the different stages they went through during the CC process, the changes that occurred in their behaviour, well-being, functioning, perceptions and relationships as a result of their participation in the programme; the interactions and relationships that were formed between them and the professionals who accompanied their process in the CC; the emotional transformations that occurred in their inner world during their participation in the programme; the difficulties and challenges they faced throughout the process; and the goals and aspirations that lie ahead of them in their future, as part of their reintegration in the community. The questions were purposefully broad for two reasons. First, the interviews went beyond asking whether or not the programme helped address the concern that programme clients might want to please the researchers in evaluation studies. Second, the broad scope of the interviews was also designed to examine the “black box” of the CC process (Farrall and Maruna Reference Farrall and Maruna2004). Indeed, through the interviews, it was possible to get glimpses of the interviewees’ lives, the challenges they faced, and their experiences throughout the CC process, understood as a journey rather than a specific point in time.
Our quantitative survey addressed some of the questions that the in-depth interview guide included, such as the extent to which participants found the process fair, effective and satisfactory and the extent to which they felt trust and respect toward law enforcement and the legal systems. Nevertheless, only through in-depth interviews was it possible to comprehend why the process was fair (if it was for them), what was satisfactory in it for them (to the extent that it was), and how they reached the point of trusting the law enforcement and programme representatives (if they did). Thus, the qualitative interviews add an important layer of understanding of the researched programme, which quantitative methods cannot achieve (Maruna Reference Maruna2015). The identified themes can later be used as quantitative variables to achieve more nuanced explanations as to why a certain intervention is effective or ineffective, leading to a greater ability to improve practice and enhance theoretical understanding of the phenomenon.
We used TJ as the analysis’s theoretical framework (Winick Reference Winick2003). TJ examines the positive and negative effects that the law itself has on individuals. It expands legal and criminological analysis beyond direct rehabilitation and reduces risk to include more nuanced and indirect changes in individuals’ lives. It also considers experiences generated through legal outcomes (punishment or rehabilitation) and those emerging in the legal process itself (Wexler Reference Wexler, Stobbs, Bartels and Vols2019). Importantly, TJ is interested not only in findings that address the question of whether or not an individual will desist from crime eventually (and how) but also seeks to learn about the well-being and subjective experiences of participants during their involvement in the legal process, as a research topic with innate importance. In analysing the findings, we were informed by grounded theory principles of inductive data analysis (Strauss and Corbin Reference Strauss and Corbin1990). Starting with open coding, each interview generated short descriptor phrases collected without predetermined ideas. Axial coding integrated the codes collected from the various interviews and contributed to the organization of the various codes into categories of meaning. Finally, selective coding involved abstracting the various categories and their relationships into a theoretical model (Strauss and Corbin Reference Strauss and Corbin1990).
The in-depth interviews revealed several recurring themes. All but one interviewee described childhood and adolescent life histories as lacking guidance and supervision. Many talked about problematic environments, a void in role models, neglect, and instability that led them to a deep sense of loneliness and disconnectedness. Many began committing crimes at an early age and had recurring contacts with law enforcement. They presented themselves as “criminals” and used to talk the criminal talk and live the criminal lifestyle. Many served prison time, and most of them suffered from addictions to drugs and alcohol, which was the basis for their repetitive criminal behaviour.
Many interviewees described their reluctance to join the CC programme based on their previous experiences with the legal system. The respectful treatment, support and desire to help that they encountered from the CC personnel caught them by surprise: “As I entered the court for the first time … suddenly the prosecutor tells me: nice to meet you! and the judge tells me: welcome! You sit and are treated as a human being, not a criminal. You do not believe the place you are in.”
We asked them what they thought helped them succeed in the programme. After many years of repeated failure, some interviewees emphasized their strong motivation to change their lives dramatically. When they realized that they had been offered a second chance, they decided to grab it in both hands and use it to get out of their criminal routine: “I realized that if I did not take a hold of myself, everything would have collapsed.”
Some interviewees mentioned the support they received from their family members, mainly mothers, siblings and partners. Another ingredient which, according to the interviewees, contributed to their success was the fact that they distanced themselves from old connections. With time, they learned to identify the negative effect these contacts had on them, and they pulled apart: “I really crossed the Rubicon, they stayed where they are, and I am in a different world now, a different place.”
Most interviewees talked about the help they received from the CC team. They highlighted how the team members provided extensive tangible and emotional support:
The probation officer encouraged me and showed me a bright future … I was suddenly able to face my fear, something I had never done before. I had never been honest about my feelings. I sometimes say that all I always needed was a connection like that, a caring connection; that is all I needed.
The caring attitude of the programme team instilled a sense of belonging and urged them to succeed despite the difficulties:
I felt the warmth, and it gave me the push I needed. The warmth of the social worker, the judge who talked like a mother, laughed and talked with me directly. I had never received such love and warmth, and suddenly, I started to feel all these feelings, to feel that people wanted me to succeed – so how could I not want it? I realized that this was my chance.
A particularly important moment was when some of them failed, and despite their failure, the team did not give up on them. This new experience of having people fighting for them, not against them, and not giving up on them had a dramatic effect: “The judge decided to let me stay in the programme, to give me another chance. This was not a given … I told myself this time I am not messing things up.”
A recurring term was that of the CC as a family, a label which was particularly outstanding considering their past encounters with the justice system: “I found myself a family in the community court. Tell you the truth, the judge was like a mother to me. The probation officer and defence attorney like brothers. It is not a court, it is a family.” As a result, many of the interviewees found it hard to depart from the programme, and some of them are still involved through participation in the graduate group or through ad hoc lectures and meetings with programme newcomers.
All of the interviewees, but one, described their participation in the programme as a life-changing experience. Some of them explicitly stated that the programme saved their lives, literally: “Maybe I would have died from overdose or a cardiac event. Many of my old friends are dead. Maybe I would have been in prison again.” “I wish I got into the CC 20–30 years ago, I lost these years. But better late than never.”
We asked participants to describe changes in their lives due to their participation in the CC programme. The interviewees talked about the many realms in their lives that had been transformed, including their health, relationships, work and education, and daily routines: “I have a daily routine; I get up and eat, like normal people.” A central positive transformation took place in their relationships with family members. After many years of silence, some renewed connections with their children, parents or siblings. Others talked about their newly acquired ability to communicate with their family members openly or the positive change they saw in how their relatives looked at them: “I regained my parents’ love”; “My siblings see the real me, not the old me.”
Many interviewees described a positive change in their self-image. The experience of success they encountered made them believe in themselves in new ways: “I am proud of myself and feel that I can do things, I can achieve things.” “I earned the strength to believe in myself. I earned the essence of understanding how to deal with the world. I am proud of the process I went through.” Interviewees also described changes in how they perceived the world more generally and their place in it and said they were more optimistic and caring: “The world is not as bad as I imagined it.” “In the past, I did not care about other people … I used to steal from them. Today, I reach out and help others.”
Changed attitudes toward “the system” and the people in it were part of the gradual transformation the interviewees described: “I started to appreciate the courts and the court system. It did not happen in one day; it was a process.”
Another question that we asked the interviewees was about improvements or changes they would suggest for the CCs to be more effective in their treatment of future participants. The interviewees had several suggestions that resembled the comments we heard in interviews with CC participants who did not complete the programme (which are not addressed in this article). One interviewee commented on the extensive pressure of the treatment programme with its many requirements and mandated intervention modules:
You have to give a man the option to choose if he wants to come to the meeting or not. Me, for one … I work 12 hours a day … I almost gave up … I said I would rather go to jail with all this pressure. People break in this way.
Two interviewees mentioned the lengthy home arrest as almost a breaking point and mentioned needing “air”: “It is tough being at home knowing that you cannot go out. You cannot work; there is not much you can do.” One interviewee referred to the community events and said that these were a good idea, which could be better fulfilled to instil a sense of community and belonging among the court participants: “These events are very positive. I think there are not enough of them, or they do not have enough impact. They are trying to create a community, which is not working.” Several interviewees highlighted the need for structured support and mentoring post-graduation of the programme. They expressed an ongoing need for some contact out of fear that they would fall again into old habits: “I have to be in contact with the professionals, those who helped me or new ones, to walk hand in hand with them; I need it.”
Finally, we asked the interviewees what they would hope for the future regarding the programme or the treatment of people who broke the law. Many of them highlighted the importance of team members expressing their belief in the participants that they can succeed in the programme: “To make us feel that they believe in us.” There were also expectations of society as a whole to see people “like them” not just as criminals:
I pray that our society will understand that sometimes you need to read between the lines and not just to put people in boxes … a person does not get to do drugs with no reason … If you dig deeper, you realize you saved a person, and that person will save others.
Another interviewee expressed his wish for expanding the CC programme: “I wish they will take more people to treat them because this programme will give a chance to others.”
These findings deepen our understanding of why the programme helped our interviewees. Notwithstanding the pressing need to collect quantitative data on whether the programme had an overall positive effect on its participants that exceeds mainstream courts, we can identify at least two new insights about why the programme works concerning those who benefitted from it. First, the qualitative interviews highlight the importance of an ethic of care utilized by professionals toward the participants. They tell us that in addition to procedural justice, people might be more willing to change their behaviour when they feel the decision-makers have a caring, compassionate attitude toward them. This is an addition to the procedural justice paradigm because a respectful and fair attitude or a caring, family-like style can manifest in a distant manner. Further research should be conducted to test our suggestion that an infusion of therapeutic justice can strengthen procedural justice experiences.
The second insight arising from the subjective experiences of CC participants is that for the programme to be effective, it should consider participants’ lives as an ecosystem of concentric circles, with one realm affecting the other – from the individual person through the various support systems, to the community at large. The themes arising from the in-depth interviews highlight the mutual connections between personal experience and self-perceptions, family support, tangible and emotional support from professionals, and community attitudes toward the participants. Further research should also examine whether a multi-layered approach addressing the various interconnected circles around the individuals’ lives is key in reducing criminality.
Module 3: Measuring Reductions in Recidivism
Perhaps the question most interesting for policymakers in Israel and beyond about new justice mechanisms is their effectiveness in reducing future criminal behaviour. Accordingly, the evaluation study was set to examine whether and to what extent there was a decrease in the recidivism rate among programme participants compared with defendants who shared similar characteristics and were adjudicated in mainstream courts.
A randomized referral of eligible participants to either a CC or a mainstream court would have provided an ideal testing method for the CCs as a new rehabilitative mechanism. However, this option was categorically refused. Therefore, we used a retrospective approach in which CC participants were compared with a control group. Theoretically, relevant variables were controlled for, followed by propensity score matching (PSM) to control for possible selection effects. This approach is considered sufficiently strong in evaluation studies, although there is always a possibility for a hidden bias (O’Donnell Reference O’Donnell2020).
We first had to compose a suitable control group. Comparing programme participants and equivalent defendants in the same cities who were not referred to the CCs is problematic because the reasons that these defendants were not referred to the CC might be the reasons that led them to higher recidivism later. Comparing programme participants and equivalent defendants who were charged in the same courts but in the years before the establishment of the CC model is a better option but still vulnerable to critique about the effect of time, for example, changes in prosecutorial attitudes or global social conditions that lead to differences in recidivism rates external to the CC intervention itself. Another option is comparing programme participants and defendants who were charged in the same years but in courts in other jurisdictions. This alternative is plausible, but differences may result from the different locality, such as prosecutorial attitudes, demographic conditions and community-wide factors.
Following a lengthy process of planning,Footnote 3 we eventually set out with a plan for constructing two control groups: (a) defendants who were indicted with comparable offences and comparable criminal records during the years 2014–2020 in three magistrate courts in cities where CCs exist today; and (b) defendants with comparable characteristics who were indicted in a magistrate court in a fourth city where the programme was never operational, also between the years 2014 and 2020. The fact that the CCs began their operation gradually (Be’er Sheva in 2014, Ramle in 2015, Tel Aviv in 2017) provided a natural laboratory to examine the trajectories of those who were indicted in the same city of the programme before its commencement and during its operation in other cities. Comparing cases within the same city across time, before and after the establishment of the CC, using the first control group, allowed us to neutralize the influence of differences between cities (prosecutorial attitudes and community-wide conditions, for example). Comparing defendants who were indicted while CCs were already operational across different cities using the second control group allowed us to neutralize the possibility of changes over time. Following the construction of the control groups, we used PSM techniques so that each participant in the programme was compared to an offender who did not participate in the CC but shared similar characteristics, including criminal (type of offence), criminogenic (criminal history, previous convictions, and incarceration) and demographic (age, gender, locality).
Data were collected about programme participants in each of the first three courts (Be’er Sheva, Ramle and Tel Aviv), including their past criminal behaviour, information about the offence that brought them to the CC (the “zero point”) and the outcomes of the CC process.
In terms of the control groups, efforts to obtain the relevant data began early after the commencement of the study in 2019. However, for the reasons described below, receiving all the needed data took over three years.Footnote 4
Recidivism studies with robust comparison groups are few, often because of the complexity of obtaining access to a suitable control group (O’Donnell Reference O’Donnell2020). In the following paragraphs, we consider some obstacles we encountered along the way. In doing so, we aim not to discourage researchers from conducting evaluation studies but rather to assist them in preparing for such challenging projects. Awareness of the various complexities can help, for example, better estimate the schedule for an expected study and the necessary budget and human resources.
Ethical and Regulatory Challenges
Collecting data about people involved in the justice system requires careful consideration. Concerns about their privacy, reputation, future employment and social status are at stake (Rogers Reference Rogers2018). Accordingly, Israel, similar to most countries, strictly regulates the management of sensitive datasets. Any researcher aiming to obtain data on criminal records must become familiar with the relevant legal regulations to approach authorities with formal requests effectively. Another factor that should be considered is the dependency on the formal gatekeepers and the institutional pace, which sometimes does not fit the timeframe of the researchers or those who expect to receive the research results within a predetermined schedule.
In the present study, in addition to the general approval obtained from the academic institutional review board, we first sought the permission of the police to collect criminal record data based on a regulation authorizing the police to grant such access to a “scientific researcher” according to the Criminal Record Act. Obtaining the police approval took almost a year. We then had to deliver to the police a list of names or identification numbers of individuals whose criminal records we wanted. As for the programme participants, we had already obtained access to their files during the formative study, so they did not pose a problem. However, how do we identify potential candidates for the control group?
To begin constructing the control group, we approached the Courts Administration and requested, based on the Freedom of Information Law, a list of names, identification numbers and court file details of all those who had been indicted in four magistrate courts – the three first CCs and an additional one located in a large and heterogeneous city – between the years 2014 and 2020. However, this request was denied. The reason for the refusal was that we stated that the list would then be transferred to the police to complete it with additional data about each individual’s criminal record, essential information for conducting PSM. According to the Courts Administration representative, the law prohibits any transfer of legally obtained information to a third party. This applied to our case even though both the Courts Administration and the police were part of the project’s steering committee, and although the police hold personal data on people in contact with the criminal justice system in any case. Several months later, we reached an agreed-upon strategy with the Courts Administration. According to the new plan, one of the governmental partners to the programme would submit a formal request to obtain the data based on the Privacy Law, which allows the exchange of certain information among governmental agencies. Then, the information would go through anonymization and be delivered to us, the researchers, without identifiable information. Several more months passed before the police were able to formally submit the request, which we drafted, to the Courts Administration and before the inter-agency committee assembled and discussed it. In February 2022, the police finally received approval for the request. Three years after the beginning of the evaluation study, we were finally over the ethical and regulatory challenges.
Putting the Data Together
Ethical clearance is only the first stage. Then, there needs to be someone in the relevant agency who implements the request, obtains the data and puts everything together in a readable file. When the request is complex, like in our case, fulfilling confidentiality restrictions takes time. Three months after the formal approval, in May 2022, the data from the Courts Administration was transferred to the police. The first wave of data was transferred from the police to the research team in July 2022, but the dataset was missing many crucial variables, so there was a lengthy process of ongoing communication. As in many other studies, the agency holding the data (in this case, the police) must be equipped to generate complex datasets. To help the police construct a suitable dataset, we hired a cyber security expert with experience in integrating large and complex datasets as a consultant.
The data that the police initially received from the Courts Administration related to approximately 30,000 people and included their identity number, the name of the court and the presiding judge, the identifying number of the court file, the dates of indictment and adjudication, an indication of the finding of guilt or acquittal, and whether or not a probation officer was involved. The police then had to anonymize the data and produce three separate new datasets using the same fake ID numbers so that each person could be traced and yet remain anonymous: one dataset described the past criminal records of the people in the list (meaning, information about offences that they conducted before the case file included in the Courts Administration list; the “zero point”); a second dataset provided additional details about the court case which acted as the zero point, the offence type, the case (judge, court, dates) and the resulting sentence; and a third dataset described the “future” criminal record (meaning, information about criminal behaviour after the closure of the relevant case file acting as the zero point). The police then delivered these three datasets to our consultant, who integrated them. For each individual, now carrying a fake identity number, we chose a single “zero point”, meaning the first court case opened between 2014 and 2020 in the magistrate court included in the second control group (the fourth city in which there was no CC), and any one of the three selected magistrate courts in the first control group, as long as the zero point took place before the CC began its operation at each of these three cities (December 2014 in Be’er Sheva, September 2015 in Ramle and May 2017 in Tel Aviv). The consultant then removed all those whose offences in the zero point were related to national security (terrorism and illegal entrance), sexual offences, and international transactions because defendants accused of such crimes were not eligible to enter the CC. Individuals whose sentence at the zero point was an incarceration order longer than two years were also removed. The result was a shorter list of approximately 3,500 people, each one with only one “zero point”, and this became the unified control group.
In the meantime, we obtained similar data on the programme participants from the police. We were now ready to obtain data about the “future criminal record” of both groups – information about their criminal activity after the adjudication of their zero-point court case (be it the CC or one of the other courts) or after their release from prison, in cases where this was the resulting sentence. It should come as no surprise that constructing these data files required ongoing collaboration with the police. Similar to the experiences of others, we learned that good rapport with police representatives emerges as an important asset for researchers conducting evaluation studies relating to law enforcement (Worden, McLean, and Bonner Reference Worden, McLean and Bonner2014).
Comparing Recidivism Rates
We matched each CC participant with someone from the unified control group according to their past criminal record, offence at the zero point, gender and age, using PSM calculations.
The research hypothesis was that, compared to the matched control group, recidivism rates among programme participants would be at least slightly lower in all three time points, namely, one year, three years and five years after completing the intervention (CC or mainstream court).
Because of the structure of the control group, we have used different timelines for each individual. First, we calculated, according to the zero-point date, the number of time points for which data were available for each individual and pinned each of them. Then, we produced the data about recidivism for each identified time point.
Here, we provide only a very brief overview of the findings on recidivism. A detailed analysis of the findings in this module is underway. In a nutshell, the study found that CC participants who had completed the programme successfully committed significantly fewer repeated offences compared to the matched control group one year, three years and five years after the conclusion of the proceedings. As little as 50% of the rate of recidivism in the control group was noted after the passage of five years. In the absence of information regarding other aspects of the control group, such as motivation to avoid repeated offences and the existence of support circles, we cannot deduce that participation in the programme was the decisive factor that reduced recidivism among the programme’s successful graduates, rather than other characteristics that perhaps explain the relatively lower rates of recidivism among the treatment group in comparison to the control group. This relative decline in recidivism is more significant than the decline found in similar programmes studied internationally based on similar methods concerning the CC model (Lee et al. Reference Lee, Cheesman, Rottman, Swaner, Lambson, Rempel and Curtis2013; Nugent-Borakove Reference Nugent-Borakove2009; Ross Reference Ross2015; Somers et al. Reference Somers, Moniruzzaman, Rezansoff and Patterson.2014; Westat 2012).
Even with the significance of other research questions, the one about the effectiveness of CCs in reducing recidivism in comparison with mainstream courts is still considered the most important question, at least from the viewpoint of policymakers. Nevertheless, considering the significant reduction in recidivism detected among CC programme completers, indicators regarding other benefits related to the CCs, such as improved health, stabilized routines, strengthened relationships and greater trust in law enforcement, are key for informing criminal justice policy and practice (Rodriguez Reference Rodriguez2018).
DISCUSSION
The two research projects we were assigned to conduct on the Israeli CC model, namely, the formative and evaluation studies, were highly ambitious and wide in scope. Particularly in the evaluation study, we aimed at examining not only the effectiveness of the CCs in reducing recidivism but also questions regarding the CCs’ ability to achieve their own stated goals: addressing participants’ underlying problems which led them to criminal behaviour in the first place, such as addictions, mental health problems, extreme poverty and isolation; promoting trust in the justice system; and instilling a sense of belonging and community cohesion among individuals and families. We also sought to identify elements in the programme that promote success and those that hinder it. The formative study aimed to document and analyse the establishment of the courts, characterize their mode of operation, and identify professionals’ perceptions regarding the programme and their roles in it.
Here, our goal was to demonstrate different approaches to evaluation studies and discuss their contribution and the challenges they pose. To do that, we presented three modules out of the many we utilized in the research projects. Each module addresses a different question based on a different methodological approach. Each of them also demonstrates the challenges that researchers might stumble upon when conducting studies of this nature. The challenges relate to three realms: ethical, methodological and logistical.
The CLT was used during the formative study to characterize the dynamics of the court hearings in the CCs. The outcomes were mostly those we expected from a CC model designed to address the rehabilitative needs of defendants. However, they also highlighted some characteristics that are often overlooked in these types of studies. The CLT uncovered the complete neglect of victims in the process. It also revealed that non-professional community members were absent and active in court hearings as envisioned. These are findings that the programme leaders can consider while planning the model’s future.
Furthermore, by providing a detailed and structured portrayal of the court hearings in CCs, we were able to pinpoint some of the significant differences between them and mainstream criminal courts and, through this depiction, perhaps offer additional explanations for their outcomes. For example, the CLT highlights that CCs use processes (such as regular court hearings for judicial monitoring and encouragement) to achieve their outcomes. The CLT, then, adds a layer of understanding to the operation of CCs, which helps explain its strengths and, in some cases, its limitations.
Using the CLT as an instrument to characterize court dynamics raised several challenges within the three categories mentioned earlier. Regarding ethical challenges, the research team’s experience echoed that of other ethnographic researchers in criminal justice settings, of “spying” on research participants (Gomes and Duarte Reference Gomes and Duarte2020). Although formal ethical procedures were fully met – we followed the research design approved by the Institutional Review Board and observing open court hearings does not necessitate explicit consent (Callaghan Reference Callaghan2005) – our research team members were aware of the informal obligations that follow such confining settings and were therefore sensitive to power imbalance and reflexive on micro-relational ethics (Gomes and Duarte Reference Gomes and Duarte2020). We had four research assistants spending full days at the CC courtroom, writing down notes and observing everything that took place during these days. Surely, they did not blend with the crowd, and it was obvious that they were outsiders studying the situation. We do not know how our presence affected the participants, induced a sense of shame (or perhaps pride), or inhibited people from opening up when it was their turn to approach the bench. Indeed, the hearings in the CCs are open to the public. Typically, the courtrooms are full, mostly with other programme participants, the lawyers representing them and their supporters waiting for their hearing. The judges were informed about the study, and the programme team knew of our presence in advance. When sensitive issues arose, the judge asked everyone present, including the research team, to leave the room. However, these occurrences were rare, as most of the sensitive discussions took place in the morning staff meetings (the subject of another research module).
That leads to the second challenge, which existed across the study modules, of perceived bias (Naylor Reference Naylor2015). The evaluation study was invited by the programme’s steering committee and financed by Joint-Ashalim, the organization that took the leadership in establishing the programme. During the studies, we developed mutual trust and sometimes friendly relationships with various programme stakeholders, either from the management or the local teams. How did we affect the judges and professionals who saw us in the audience and knew exactly what we were studying? Moreover, how did our acquaintance with programme professionals affect our study design, implementation, analysis and conclusions? Has our lengthy engagement with the programme made us too involved to evaluate it objectively?
Moreover, even if we managed to keep our objectivity, we still have to face the visuals of being a research team working closely with the programme for seven years (Rantatalo et al. Reference Rantatalo, Lindberg, Kihlberg and Hällgren2018). To address these challenges, what we have is our own integrity as researchers whose compass is the scientific truth, and our motivation is to advance knowledge. We also have a wide range of methods we utilized, which offer triangulation and validity to the findings. In the context of the CLT, using numerous research assistants as observers rather than the principal investigators surely reduced the risk of mutual familiarity and engagement. However, ethical questions about the research team’s possible effects on the programme participants and professionals and dispositional questions about possible effects that the programme professionals had on us as researchers are likely to surface.
The third challenge that the CLT method raised concerned logistics, as we had to reach an adequate number of observations to enable statistical analysis. We insisted on having four coders complete the observation sheets concurrently to ensure the instrument’s validity. This method required us to coordinate times when four coders could commute to each court. The courts operated only once a week and were often occupied by other guests and delegates from Israel and abroad. This phenomenon became more common as the CCs became more known to the public and policymakers. Extended periods for collecting data due to access constraints and geographical distance are a known challenge in criminal justice research (Worden et al. Reference Worden, McLean and Bonner2014). Logistical problems can be solved with time and money, and we were fortunate to enjoy the trust of Joint-Ashalim, who accommodated our requests. However, our key lesson from this experience is to allocate approximately double the amount of time and resources initially estimated for conducting court observations and other on-the-ground data collection tasks.
The second research module presented here is the in-depth interviews with programme participants, aimed at delineating their experiences, insights and feelings concerning their participation in the CC programme. This was a qualitative method that complemented the surveys and the statistical analyses of the programme’s database. Unlike the quantitative data collection methods, the in-depth interviews could uncover the “why”, “how” and “what” happened when participants thought, said or did something during their participation in the programme. Indeed, the thematic analysis of the interviews highlighted the importance of the personal relationships that the participants felt they developed with programme personnel and the caring attitude they encountered. The interviews also identified participants’ dramatic changes in their lives, self-perceptions and attitudes toward others. These qualitative findings make a theoretical contribution to comprehending individuals’ motivations to accept judicial decisions and adhere to rules. Apart from procedural justice, an ethic of care demonstrated through compassionate expressions and genuine concern from judges and other programme personnel plays a substantial role among court participants.
Furthermore, the findings emphasize the significance of addressing the participants’ needs within multiple spheres, forming an interconnected ecosystem of circles. Changes in their self-perception have an impact on their marital life, which, in turn, affects their capacity to enhance their employment status, thereby influencing how the community perceives them – and vice versa. These ideas call for further empirical examination using quantitative methods. Furthermore, through the in-depth interviews, we also gained the participants’ insights about ways that the programme could be improved, be it the need for more flexibility or the importance of post-graduation contact. Such input has practical significance for programme leaders.
However, this module also demonstrates the three types of challenges mentioned earlier, which, in the context of this module, were particularly salient at the recruitment stage. Ethical considerations limited us from contacting programme participants or graduates directly. It is potentially harmful and untactful to initiate an unsolicited call to someone criminally accused and ask them if they are willing to talk about their experience. People want to move on; they face multiple challenges, particularly those who overcame significant difficulties relating to addictions, mental health and unemployment. There is also the concern that without a detailed explanation to potential interviewees, they might feel obligated to respond positively, fearing that a negative response would somehow endanger them as probationers and fail to realize the independence of the research team (Maruna and Liem Reference Maruna and Liem2021). To overcome these concerns, we asked the defence attorneys to contact programme graduates, invite them to participate in the study, and send them an explanation letter from the research team. The assumption was that the defence attorneys did not represent “the system” for their clients; they were used to confiding in them and would thus feel free to consult with them about the invitation to be interviewed.
Another recruitment method was to approach potential candidates during court hearings physically: our research assistants were regularly observing court hearings and implementing the surveys with participants, so a natural step for them was to ask those who were at Stage 5 if they were willing to meet for an in-depth interview. In these encounters, the participants knew that the research assistants were not part of the court team, but to be on the safe side, they regularly highlighted to the candidates that they were free to refuse the invitation and that their response would in no way affect their status in the programme. The research assistants also assured participants that the interview content would remain completely confidential (similar statements were made to research participants recruited through their defence attorneys).
Ethically speaking, we also needed to maintain the interviewees’ privacy and anonymity, and, accordingly, we expunged all identifying information and did not disclose the exact age, locality and background details when presenting the sample in the report. There is still the question of whether or not participating in the interview negatively affected the interviewees (Naylor Reference Naylor2015). This ethical question emerges in every study involving interviewees, but in the present context, the subject matter triggered content for a vulnerable population. In preparing the interview guide, we were careful not to include direct questions that might sound judgmental, focus on interviewees’ failures, or trigger trauma and shame. We also had at our disposal the collaboration of experienced therapeutic team members, who, if needed, could provide referrals without knowing who the individual under stress or duress was. Eventually, however, our impression was that the interviews had a positive rather than negative impact on most research participants. They appreciated the opportunity to reflect on their journeys and be heard, possibly making an impact on policymaking, and recall positive memories of success. We encountered more complex challenges when interviewing those who still needed to complete the programme, but this is separate from the present paper and will be discussed elsewhere.
Recruiting interviewees also presented a logistical challenge that affected the sample. Gaining access to interviewees in confined settings is extremely time-consuming (Fox, Zambrana, and Lane Reference Fox, Zambrana and Lane2011), and our experience shows that even without the confining boundaries of custody, interviewees are hard to get. Only after repeated reminders to CC defence attorneys and many visits to the courts were we able to gradually reach an adequate number of interviewees. A sample of 12 people might seem small, but it is quite acceptable in qualitative studies, especially those focusing on vulnerable populations, and very often is sufficient in reaching saturation, as we did in this case (Guest, Bunce, and Johnson Reference Guest, Bunce and Johnson2006). Although approaching active participants in court was easier than reaching them indirectly through the attorneys, many were not interested in engaging in the additional task of speaking with a stranger for 60–90 minutes. We need data about the number of CC participants and graduates who were approached, so it is impossible to determine the retention rate. The completion rate in the Israeli CCs is approximately 50%, meaning that about half of the approximately 1,300 participants who were or are treated by the CCs manage to graduate successfully. Our sample is made only of “completers”, so we conducted a separate module that included in-depth interviews with non-completers (not presented here). However, we cannot determine how the people we interviewed in the present module differ from those we did not. Those who agreed to be interviewed may have more optimistic stories, more enthusiastic impressions, or better relations with the defence attorneys. They may be the most articulate, the most stable in their lives, or the more supportive of the programme. At the same time, they may be more opinionated and more willing to be critical and state an independent opinion. These are limitations of this module, but other modules provide triangulation and validation that have been described in the full report. In the present limited context, suffice to say that many of the themes were echoed in the surveys (such as the improved relationships and enhanced trust) and in the in-depth interviews we conducted with over 50 CC professionals and 19 female participants who completed the programme (whose interviews are the subject of a separate module, in which we adopted a gender-based perspective). We (the principal investigators) also conducted separate ethnographic observations, which did not amount to a complete module but provided us with a deeper understanding of the mode of operation of the CC and the atmosphere in the courtrooms.
The dispositional challenge we faced in the context of the interviews with programme participants was the risk of the interviewees associating us mistakenly with the programme. Because the programme invited and financed the evaluation study, interviewees might have thought we were acting on behalf of the programme and expecting them to praise it. More broadly, evaluation studies are often mistakenly considered to seek a binary answer to the question “Does the programme work well?” (and the expected answer, when the founders invite the study, should better be positive – or so it might be conceived). Researchers conducting evaluation studies are expected to maintain the accuracy and truthfulness of their findings despite actual or potential pressures to present a “shiny image” of the researched programme (Duong Reference Duong2015; Naylor Reference Naylor2015). In the present study, we framed our goals as wider than addressing the yes-or-no question. We included questions that enabled us to juxtapose the findings against theories in criminology and criminal law in line with our respective research agendas. The interview guide addressed these broader questions and signified to the interviewees that we were not seeking a simple and positive answer to a simple question such as “Was the programme helpful to you?”. Hopefully, despite the undeniable context of the study being invited by the programme leaders, interviewees could openly discuss the full scope of their experiences, insights and feelings about the programme. Indeed, their insights and suggestions for improvement contributed significantly to the section in the final report, which focuses on lessons, recommendations and suggestions for the future from a critical perspective.
The third module presented here is the study comparing recidivism rates between CC completers and defendants adjudicated in mainstream magistrate courts. Without a doubt, the “recidivism module” holds great anticipation among Israeli policymakers and programme leaders, as its outcome can ultimately determine the success or failure of the CCs programme. However, from a theoretical standpoint, the remaining modules are equally, if not more, significant than the current one.
The sections above detailed the logistic, ethical and methodological difficulties we faced in this module. Although we did not meet any of the individuals included in the dataset of the control group, we obtained extensive data about them, and, therefore, we had to ensure that their anonymity was maintained. The complexity of the data required us to make painful compromises to make the integrated dataset usable. We had to exclude the various elements of punishment beyond imprisonment induced on each individual in each judicial decision because including these variables would have created an unworkably large dataset. As a result, we cannot know, for example, if a significant fine, restitution order, or a significant term of community work moderates the effect of a prison sentence or an alternative. We also had to exclude information about police investigation files against individuals in the control or intervention groups, which were closed without further legal action. This is because there are many reasons for closing a case, which the police dataset did not provide in a precise and reliable manner, thus risking putting together dramatically different situations. The result is that we cannot know whether or not CC participants, for example, are under increased and unjustified police surveillance. Another compromise we made was grouping many offence types so that the dataset only differentiates between eight categories: property; violence, harassment; fraud; traffic; narcotics; family violence; and “other”. This categorization does not differentiate, for example, between the possession of drugs for self-use and the selling of narcotics, or between tampering with a witness and stalking. These decisions are made routinely in large cohort studies, but each decision risks losing an important variable that might affect the statistical model’s efficiency and accuracy. In the present study, we were mindful of the risk of losing relevant variables and based our decisions on existing empirical knowledge and theories.
SUMMARY AND CONCLUSIONS
This is not a typical paper presenting findings from an evaluation study. It did not include a systematic description of the various methodologies utilized to address the research questions, the findings answering these questions, and a discussion linking the findings with existing knowledge. In this paper, we did not aim to bring the results of the evaluation study (or formative study) we conducted on the Israeli CCs. Instead, we focused on three research modules that these studies included, to demonstrate the range of research questions, methodologies and related challenges that evaluation studies may involve. In describing each module, we should have provided a detailed analysis of the findings correlating them with existing theoretical and empirical knowledge, but this was not our goal. Rather, what we aimed to achieve is an open discussion about the process of conducting such studies: the selection of research questions that go beyond the traditional question of repeat offending and their possible theoretical contribution; the methodological, logistical and ethical choices that are made and their underlying considerations; and the obstacles that researchers face en route.
In terms of research questions, we learned from the programme representatives themselves, in addition to contemporary criticism, that recidivism alone does not capture the whole picture. Programme graduates may recidivate but still experience meaningful interactions with law enforcement representatives, improve their self-perception, gain a stable routine, and strengthen their connections with law-abiding family and friends. These are part of the goals of CCs, so an evaluation study must examine them. The understanding of the complexity of the meaning of success for an individual participant in these courts, that we acquired throughout the implementation of this research, led us to fine-tune and adjust our research questions regarding the metrics we should utilize in evaluating the effectiveness of CCs (Dancig-Rosenberg and Gal Reference Dancig-Rosenberg and Gal2024). Our in-depth interviews with programme graduates and participants who were about to graduate provided vivid manifestations of the transformations they underwent, and of the elements in the programme that they thought were central to these transformations. Based on the CLT, our structured observations captured the characteristics of the court hearings in high resolution in ways that enabled us to compare them to those of other justice mechanisms. The analysis of the CLT provides a solid response to the question, “How are court hearings in CCs different from those in mainstream criminal courts?”.
In terms of methodologies, the three modules demonstrate completely different ways of collecting and analysing data. The CLT is a novel method that we developed as part of our research agenda to explore the phenomenon of multidoor criminal justice – the current reality in many justice systems in which there are many alternatives to mainstream courts and the resulting need to develop tools to compare, combine and optimize referral decisions among the various options (Dancig-Rosenberg and Gal Reference Dancig-Rosenberg and Gal2019). The in-depth interviews with programme participants and graduates belong to the qualitative paradigm, which many consider foreign to traditional evaluation studies or at least secondary in importance. When combined with quantitative methods, as in our research project, the qualitative approach enriches the findings and provides a deep understanding of the phenomenon, which surveys cannot offer. The third module presented in this paper, the statistical comparison of recidivism rates between matched pairs of programme participants and defendants adjudicated in mainstream courts, is the most common methodology employed in evaluation studies. Despite its difficulties, it is still the only way (assuming a randomized control trial is impossible) to give policymakers a “number”, a concrete verdict on whether or not a certain criminal justice intervention achieves the instrumental goal of reducing repeat offending.
The challenges described in the paper are common to many evaluation studies but have yet to be openly discussed. Other researchers who engaged in similar reflective writing typically focused on a single research method, such as narrative interviews (Maruna and Liem Reference Maruna and Liem2021), or provided a birds-eye view of the benefits and obstacles in researcher–practice collaboration (Rodriguez Reference Rodriguez2018). The present article is novel in its discussion of both qualitative and quantitative research methods, their distinct contributions, and the challenges each of them imposes in the context of criminal justice evaluation studies. We hope that in raising the veil over the research process, we will help other researchers pre-empt some of the problems we face or benefit from the knowledge that they are not alone.
For academic researchers, conducting an evaluation study requires the maintenance of an equilibrium between the interests of programme managers and policymakers on the one hand and those of the researchers on the other, between the expectation for a concrete answer on whether or not a certain intervention “works”, and the desire to make new theoretical or empirical discoveries, sometimes those which redefine the meaning of the term “works” in the researched context (Rantatalo et al. Reference Rantatalo, Lindberg, Kihlberg and Hällgren2018). In broadening the horizons of evaluation studies by adding new research questions, developing new measurement metrics and using a range of methodologies, researchers may find that they can achieve both ends.
Acknowledgements
The study was funded by Joint-Ashalim, No. 207083 (formative study) and No. 205085 (evaluation study).
Appendix 1. Data Collection Methods, Samples and Research Questions
CC, community court.
Appendix 2. The Criminal Law Taxonomy: Full List of Parameters
Tali Gal is a Professor of Law and Criminology at the Hebrew University of Jerusalem, Chair in Child and Youth Rights, and Academic Director of the Child and Youth Rights Programme at the Minerva Center for Human Rights. Her scholarship integrates legal, criminological and psycho-social knowledge, focusing on restorative justice, children’s rights and therapeutic jurisprudence. Previously, she was Head of the School of Criminology at the University of Haifa. Gal is an Associate Editor at the International Journal of Restorative Justice, Youth Justice, and Frontiers of Psychology, and a board member of the Israeli Society of Victimology.
Hadar Dancig-Rosenberg is a Professor of Law at the Bar-Ilan University and a Visiting Professor at Northwestern School of Law (intersession, 2023–2024). She was the Helen Diller Institute Visiting Professor at the UC Berkeley School of Law (2021–2023). She is the co-founder and co-chair of the Israeli Criminal Law Association. Her areas of expertise include the philosophy of criminal law, criminal justice reform and alternatives to incarceration, therapeutic jurisprudence, and the interface between criminal law and gender. Professor Dancig-Rosenberg is also an Affiliate at the Center for the Study of Law and Society at the Berkeley School of Law and a Fellow at the Salzburg Global Seminar.
Open access
