Methodology

This document describes the methods for collecting, analysing and displaying data in the Justice Dashboard. Its purpose is to place the data published on the Justice Dashboard in context so it can be properly understood, interpreted and replicated.

About the methodology of Justice Needs and Satisfaction surveys

Our methodology for Justice Needs and Satisfaction (JNS) surveys is based on the legal needs and justiciable events research tradition. See for an overview of the methodologies and the challenges the following papers:

  • Pleasence, Balmer and Sandefur, Paths to justice, A past, present and future roadmap, 2013
    (download here)
  • Law and Justice Foundation of New South Wales, Methodology of legal needs surveys, 2014
    (download here)

Our main new contributions to this line of data collection about user needs are the the following:

  • We developed a standard methodology that now has been applied in 15 countries. See the countries listed below, plus Fiji and Morocco that are being surveyed during the final months of 2018.
  • We added much more detailed assessment of access to justice (procedural justice, outcome justice and costs of access to justice), following a methodology that was developed at Tilburg University, see Gramatikov, A Handbook for Measuring the Costs and Quality of Access to Justice, 2010.

How we measure justice

Below we outline the main concepts and approaches underlying the JNS methodology.

JNS Survey

Justice Needs and Satisfaction (JNS) Survey is the tool that we use to measure justice from the bottom-up. Randomly selected adult (18+) respondents are interviewed about their legal problems, resolution strategies and perceptions about the costs of justice and quality of processes and outcomes.

(Gramatikov, Martin et al. 2010. A Handbook for Measuring the Costs and Quality of Access to Justice. Apeldoorn, Antwerpen, Portland: Maklu)

Access to justice

Justice is accessible when people can reach fair solutions whenever they encounter disputes, disagreements or rights violations.

(Barendrecht, Maurits et al. 2012. Towards Basic Justice Care for Everyone. Challenges and Promising Approaches. The Hague: Hague Institute for the Innovation of Law. link)

Bottom-up justice

Bottom-up justice is an approach to measuring justice from the perspective of the people. In that way bottom-up justice is focused on the needs for justice as the people encounter these needs in their lives. The alternative – top-down approach looks at justice from the perspective of the formal institutions (i.e. how many court cases, complaints to police, administrative requests, how are cases processed, resolved etc.).

(Commission on Legal Empowerment of the Poor. 2008. II Making the law work for everyone. New York: United Nations Development Programme. link

Piest, Johanna et al. 2016. Justice Needs in Uganda: Legal Problems in Daily Life. The Hague. link)

What is a legal problem

A legal problem refers to a problem that takes place in daily life – a dispute, disagreement or grievance for which there is a resolution in the (formal or informal) law. In the legal needs research the term justiciable events is also used. The resolution of the problem could be through an intervention of a third party – i.e. adjudication, administrative process, arbitration or mediation or through negotiation or reconciliation between the parties. It is not necessary that the respondent knows or recognizes its legal aspects. It is also possible that nothing has been done to resolve the problem.

 

(Gramatikov, Martin et al. 2010. A Handbook for Measuring the Costs and Quality of Access to Justice. Apeldoorn, Antwerpen, Portland: Maklu

Genn, Hazel G, and Sarah Beinart. 1999. Paths to Justice : What People Do and Think about Going to Law. Oxford, England ; Portland, Or.: Hart Pub.

Currie, Ab. 2010. The Legal Problems of Everyday Life The Nature, Extent and Consequences of Justiciable Problems Experienced by Canadians. Department of Justice, Canada. Link

OECD Guideline Document. Upcoming link)

How do we ask about legal problems?

The JNS questionnaire uses plain language to ask people about the legal problems that they encountered in daily life. People in the streets do not use legal language. They talk about divorce, accidents, losing a job or being robbed. In the questionnaire we ask:

Have you experienced serious disputes, disagreements and grievances that have resolution through a legal process as shown in Show-card 1 in the past 4 years? This means since [date adjusted to specific study]. Think of either personal or business problems in which you were involved.  It does not matter whether you did something or what you did about the situation or who was the other side – we want to hear about your experiences. Look carefully at the list (Show card 1) and report all problems which occurred to you. It is possible to list multiple problems from the same type and category.

(Gramatikov, Martin et al. 2010. A Handbook for Measuring the Costs and Quality of Access to Justice. Apeldoorn, Antwerpen, Portland: Maklu

See the methodological chapters in HiiL’s JNS reports. link)

Seriousness of legal problems

In order to avoid marginal legal problems the instrument asks about serious problems. This means that trivial issues are excluded from the scope of the JNS. Enumerators are provided with the following instructions:

We are interested in problems such as disputes, disagreements and grievances. These problems should potentially have resolution through a legal process. However, it does not matter whether the respondent sees the legal aspect of the problem or not. It also does not matter whether the individual respondent formulated legal or non-legal strategies for responding to the problem.

What is important is whether such problem or problems took place in the respondent’s life.

  • We are interested to record as many problems as possible – help the respondent to think back and report problems.
  • Display Show card 1 and go through it with the respondent.
  • Explain that we are interested to hear about serious problems but also less severe problems. Encourage the respondent to report problems.
  • Explain that it does not matter what the respondent did (or did not do) about the problem.
  • A person might experience more than one type of problem or multiple problems from the same category.

(Pleasence, Pascoe et al. 2004. Causes of Action: Civil Law and Social Justice. London: Stationery Office.

Gramatikov, Martin et al. 2010. A Handbook for Measuring the Costs and Quality of Access to Justice. Apeldoorn, Antwerpen, Portland: Maklu)

Who are the users of justice?

Users of justice are individuals who experience a legal problem and therefore need justice. From their perspective justice is the fair resolution of the specific problem.

The method only accounts for problems experienced directly by the respondent; i.e. the respondent must have been personally involved and affected by the specific problem.

(Gramatikov, Martin et al. 2010. A Handbook for Measuring the Costs and Quality of Access to Justice. Apeldoorn, Antwerpen, Portland: Maklu

Genn, Hazel G, and Sarah Beinart. 1999. Paths to Justice : What People Do and Think about Going to Law. Oxford, England ; Portland, Or.: Hart Pub.)

Who are the neutral third parties?

A neutral third party is a person/s or institution/s that has vested powers to solve the problem in an unbiased and fair manner. Neutral third parties are expected to be objective and neutral in their positions and approaches to resolving legal problems.

(Barendrecht, Maurits et al. 2012. Towards Basic Justice Care for Everyone. Challenges and Promising Approaches. The Hague: Hague Institute for the Innovation of Law. link)

Formal and informal justice

The JNS is neutral towards formal and informal justice. A justice journey can include both formal and informal dispute resolution mechanisms. The instrument asks the users of justice about their perceptions about the quality of the dispute resolution process, the quality of the outcomes and the costs of accessing justice regardless of whether the dispute resolution mechanism is formal or informal.

Which areas of law?

Our aim is to map out bottom-up justice as it happens. Therefore we ask about all types of legal problems – regardless of they belong to civil, family, criminal or administrative law.

Which categories of legal problems?

Individual legal problems are grouped into categories that combine problems based on their similarity. The problems are organized according to people’s relationships and sometimes might differ from scholastic legal classification. For instance, a consumer problem might be also seen as a contractual issue. What is important, however, for the people is that they regularly engage into interactions about purchasing goods and services and sometimes these interactions end up in a conflict. From people’s perspective what is important is that they need protection as customers. The list of problem categories can be found here.

Which specific legal problems?

Each category consists of specific instances of legal problems. For example, Family problems consist of problems like divorce, separation, custody, alimony etc. We always discuss these categories with experts from the particular country where the JNS study is being conducted. These local experts tell us if a problem has to be removed or added. They also help us to formulate and translate properly and validate the legal problems.

Multiple legal problems

It is possible that people encounter more than one legal problem during the 4-years time frame. Therefore we ask them to report all legal problems they can recall. Thus we map out the overall environment of legal problems as they occur in people’s lives.

The JNS follows in great depth one legal problem per respondent. If the particular respondent has experienced more than one problems we ask for the one that was most serious problem. The enumerators are trained to explain to the respondent that the most serious problem is that one that has the largest positive or negative impact in the life of the respondent.

Time frame

We ask about legal problems that emerged in the 4 years prior to the interview.

Justice journeys

When there is a legal problem a well functioning legal system provides one or more dispute resolution mechanisms. We call the chains of these dispute resolution mechanisms justice journeys. The concept of a justice journey recognizes that there are many different steps to resolve a problem. Most often these steps are not linear. The entirety of steps that people take to resolve a problem is a justice journey.

(Barendrecht, Maurits et al. 2012. Towards Basic Justice Care for Everyone. Challenges and Promising Approaches. The Hague: Hague Institute for the Internationalisation of Law. link

Gramatikov, Martin et al. 2010. A Handbook for Measuring the Costs and Quality of Access to Justice. Apeldoorn, Antwerpen, Portland: Maklu)

Beginning of a justice journey

The beginning of the justice journey is the moment the user first takes action. This can be the moment when the user seeks information about her possibilities, seeks legal information or advice (from relative, friend, internet, legal adviser, police, mediator, etc.), or files a petition.

(Gramatikov, Martin et al. 2010. A Handbook for Measuring the Costs and Quality of Access to Justice. Apeldoorn, Antwerpen, Portland: Maklu)

Legal information and advice

Before, during or after travelling a justice journey the users of justice might seek for information or advice about their legal problem. Such information or advice might come from various sources. We do group the sources into two categories – non-professional and professional. The former consists of people or organisations who are not providing legal information or advice as part of their professional responsibilities. The latter group provides legal information and advice in the course of delivering private or public services.

Dispute resolution mechanisms

A dispute resolution process is defined as commonly applied process that people address in order to cope with their legal problems. A court procedure is an obvious example of a dispute resolution process. However, the definition includes both formal and informal procedures. This means that a mediation procedure, or a procedure before an informal Commission or other neutral, also qualifies as a path to justice.

(Barendrecht, Maurits, Peter Kamminga, and Jin Ho Verdonschot. 2008. “Priorities for the Justice System: Responding to the Most Urgent Legal Problems of Individuals.” link)

Diversity of justice journeys

Some of these mechanisms are formal, meaning that they are regulated by law. Others are informal, based on custom, traditions, beliefs or power relationships.Justice journeys can also differ by the extent of formality. Some processes, such as adjudication or arbitration, are usually tightly structured. Others, such as self-help, involvement of a family member or reconciliation, have much less of an uniform structure.

The methodology recognizes the following broad categories of dispute resolution processes:

  • Self-action
  • Personal network
  • Professional neutral party

In the Dashboard the categories Professional neutral party is further split into two separate categories (Courts and lawyers and Police):

  • Courts and lawyers
  • Police
  • Personal network
  • Self-action
  • Other organised process

(Gramatikov, Martin, Maurits Barendrecht, and Jin Ho Verdonschot. 2008. “Measuring the Costs and Quality of Paths to Justice: Contours of a Methodology.”

Reaching Further: New Approaches to the Delivery of Legal Services: 26.)

Multiple dispute resolution processes

When multiple dispute resolution processes are used the JNS asks the respondent to report the costs, quality of procedure and quality of outcome of the process which helped the most to resolve the problem.

Quality of the process

The quality of the procedure is conceptualized as the amount of procedural (7 items), interpersonal (3 items), and informational (4 items) justice that the users of justice receive. The questions ask about the quality of a specific dispute resolution mechanism (recognized as the one that helped the most to resolve the problem).

(Klaming, Laura, and Ivo Giesen. 2008. Access to Justice: The Quality of the Procedure. link)

In the dashboard the value of the quality of the process is derived following the following steps: 1) the mean value of each sub-dimension is calculated; 2) the mean of the 3 sub-dimensions is calculated.

Quality of the outcome

The result from the last part of the justice journey is the outcome. It can be an award for damages, an agreement about future conduct, or an apology, or a combination of these. We believe the outcome has four dimensions:

  • Fair distribution (distributive justice)
  • Damage restoration (restorative justice)
  • Explanation of the outcome (motivation of outcomes)
  • Problem resolution (enforcement of the result)

(Verdonschot, Jin Ho, Maurits Barendrecht, Laura Klaming, and Peter Kamminga. 2008. Measuring Access to Justice: The Quality of Outcomes. link)

Costs of access to justice

The private costs of justice borne by the user in her pursuit to solve the legal problem. The instrument measures the following costs spent on the path to justice (not only the specific dispute resolution process):

  • Out of pocket (monetary) expenses
  • Personal time
  • Stress incurred on the path to justice
  • Negative emotions incurred on the path to justice. Three types of commonly occurring negative emotions are measured: frustration, anger and humiliation

(Currie, Ab. 2008. The Costs of Experiencing Justiciable Problem.

Gramatikov, Martin. 2009. “A Framework for Measuring the Costs of Access to Justice.” link)

End of a justice journey

The moment when a neutral makes a decision, the parties come to an agreement, or one of the parties leaves the procedure, thus ending the procedure.

(Gramatikov, Martin et al. 2010. A Handbook for Measuring the Costs and Quality of Access to Justice. Apeldoorn, Antwerpen, Portland: Maklu)

How do we collect data

Survey research

The JNS data is collected through surveys of randomly selected individuals. We select a local data collection company with experienced enumerators and supervisors to conduct fieldwork. The size of the samples of the JNS range from 2400 to more than 6000.

Sampling procedure

When implementing JNS surveys we use a full multi-stage randomized sampling procedure, which gives each adult an equal chance to be part of the sample. Clustering approach is used in order to ensure randomization at regional level. When security of respondents and enumerators is a concern we consult the selection of regions of local experts. To ensure representativeness to the population parameters we include quotas, such as gender, age and rural/urban status. These quotas ensure that the sample reflects the distribution in the population. At the level of primary sampling units (PSU), households are selected on the basis of a random step. The paths of the interviewers as well as the procedure for moving across the PSU are designed to ensure randomization. In the households, the interviewers consider the quotas and select randomly one adult individual to answer the questions. Only one individual is interviewed per household. The questionnaire asks about the legal problems of the selected respondent.

(Northrop, A., 1999. Sampling and Data Collection. In G. Miller & M. L. Whicker, eds. Handbook of research methods in public administration. New York: Marcel Decker, pp. 99–124.)

Samples

For our surveys we interview randomly selected adults (18 years or older). Until January 2019, we have collected data in the following countries:

Country Number of respondents Year
Bangladesh 6,000 2018
Indonesia (Java, West Kalimantan and South Sulawesi) 2,400 2014
Jordan 6,001 2017
Kenya 6,005 2017
Lebanon 6,000 2017
Mali 8,391 2014
Nigeria 6130 2018
The Netherlands 5,021 2013
Tunisia 6,771 2017
UAE 3,924 2016
Uganda 6,202 2016
Ukraine 6,559 2016
Yemen (Sanaa, Taiz and Adden governorates) 3,000 2014

Data collection

We select a local data collection company with experienced enumerators and supervisors to conduct fieldwork. In some countries where data collection companies are not present or are prohibitively expensive we collaborate with NGOs with national presence. Usually these NGOs are engaged in the provision of legal aid.

Training of enumerators and supervisors

The interviews are conducted face-to-face. The data is collected directly on tablets unless the circumstances require the use of pen-and-pencil questionnaires. In The Netherlands, the data was collected online using a panel of respondents provided by the data collection company.

Interviewers are carefully selected based on a number of criteria (age, literacy, experience, language fluency etc). Interviewers and supervisors undergo a comprehensive training on the subject matter, sampling procedure, interviewing and technology.

Limitations: a word of caution about the data

Despite the efforts to have a strong research design that includes quantitative survey data, qualitative interview data with users of justice, and interviews and rounds of feedback with local experts, inevitably, there are limitations to the data, just as in every study. For detail about people’s experiences with specific justice journeys, different and larger samples are needed, for example, with problem-specific surveys.

People tend to underreport specific legal problems (see also below). This situation is often confirmed by local experts, who indicate that this might be the case for certain problems like land disputes or family problems such as domestic violence and other gender-based violence problems. Family disputes, for example, are considered to be a sensitive topic, making people less likely to seek justice outside of the family.

  • Some people might not report problems due to shame and fear. For example, when people have spent time in prison or have had problems with figures of authority.
  • Cultural norms may cause people to under or over-report problems.
  • A high level of insecurity in a country can have a small effect on the survey fieldwork. In some areas the presence of our field team can make residents generally suspicious, apprehensive as well as scared about their safety. However, the presence of trained interviewers, who are also natives of the locations, can suppress the suspicion and facilitate drawback-free fieldwork.
  • We do not include sub-samples of some the most vulnerable groups, such as the disabled.

Additional data and triangulation

After the surveys are executed and the data is collected, we conduct a triangulation workshop in joint effort with our local partner organisation and with the participation of stakeholders. During these workshops we present the preliminary results of the report to the attendees, which provides the opportunity for dialogue around the data. This dialogue with stakeholders and local experts adds more depth to our understanding of the occurrence of and response to legal problems.

The local data collection company also conducts qualitative interviews with users of justice, particularly the most vulnerable groups, with the objective of getting a more in-depth account of their experiences. Their stories, in their own words, illustrate and provide in-depth clarifications of the JNS data.

Aggregation of indicators and generalizations

Some measures on the Dashboard are displayed as aggregated scores (see Process and Resolution tabs). The scheme below explains the aggregations:

Data aggregations

Quality of procedure
Mean of procedural justice (7 items), interpersonal justice (3 items) and informational justice (4 items)

Quality of outcome
Mean of distributive justice (5 items), restorative justice (3 items), informational justice (3 items) and functionality of the outcome (4 items)

Costs of justice
Out of pocket expenses (normalized distribution, rescalled to 1 to 5 scale), number of days (normalized distribution, rescalled to 1 to 5 scale) and stress and emotions (mean of stress (1 item) and negative emotions (3 items).

Generalizations towards the overall population

In order to assess the number of new problems per year, we have to take into account that people tend to forget problems that are less recent. See Pleasence, Balmer and Sandefur, Paths to justice, A past, present and future roadmap, 2013 (download here) for the following graph.

As a rule of thumb, which is on the safe side, we assume that people forget one third of problems occurring in 4 years. So the generalizations in the Justice Dashboard assume that the reported problems in the 4 years time period actually occur in three years.

The percentage of problems that have not been resolved is calculated using the following formula:

% of people who did not take action + % unresolved problems + % on-going problems = % unresolved problems

Do you need more insights?