An Assessment of Theoretical Proposals to Deal with Heterogeneous Small Samples

Summary

The studies we conducted rapidly faced a big issue. Was it possible to make reliable quantified comparisons of differences in performance between notaries and non notaries legal systems using heterogeneous small samples? The underlying issues at stake concern different levels.

The first one is related with the cost of data collection. Indeed, to be relevant the traditional quantitative methods would work on panel data but require collecting detailed data for at least thirty individuals per country. This means having sufficiently standardized the methodology, but the previous chapters of this book show that this approach may be too simplistic and that it requires a very important preliminary work to comprehensively understand the legal systems. It is time and money consuming. In addition, standardizing data collection is not an easy task. Hence, it is necessary to achieve a trade-off between two conflicting objectives: defining sufficiently harmonised data to enable comparisons to be meaningful – comparability criterion – and defining data that are sufficiently representative of local conditions – representativeness criterion –. Then comes the actual step of collecting a sample whose size is sufficient enough.

The second issue is a practical one and relates to the statistical treatment of data. Indeed, working on small samples implies that the statistical properties necessary for using the usual estimators that are based on the central limit theorem are not satisfied. Traditional inferences used to assess the magnitude of the observed effects cannot be performed rigorously. We must therefore implement alternative methodologies.

We were not the only researchers facing these challenges. They oft en arise in public policy analysis when defining targeted measures is required. Fine tuning supposes to dispose of disaggregated information at an appropriate level. The institution which has developed the most such approach is the World Bank, particularly in the context of programs destined to diminish poverty. Notably, the Millennium Declaration adopted by the member countries of the United Nations in 2000 that called for the reduction of poverty by half in a worldwide scale by the year 2015.

The methodology developed by researchers at the World Bank belongs to the so-called small area estimations methodologies. It consists in fitting statistical models in order to be able to conduct statistical analysis for areas where the sample size of the survey data is not sufficient for generating accurate estimates.