Why you care: To design and run a good online controlled experiment, you need metrics that meet certain characteristics. They must be measurable in the short term (experiment duration) and computable, as well as sufficiently sensitive and timely to be useful for experimentation. If you use multiple metrics to measure success for an experiment, ideally you may want to combine them into an Overall Evaluation Criterion (OEC), which is believed to causally impact long-term objectives. It often requires multiple iterations to adjust and refine the OEC, but as the quotation above, by Eliyahu Goldratt, highlights, it provides a clear alignment mechanism to the organization.

Why you care: Randomized controlled experiments are the gold standard for establishing causality, but sometimes running such an experiment is not possible. Given that organizations are collecting massive amounts of data, there are observational causal studies that can be used to assess causality, although with lower levels of trust. Understanding the space of possible designs and common pitfalls can be useful if an online controlled experiment is not possible.

Why you care: We begin with an end-to-end example of the design (with explicit assumptions), execution, and interpretation of an experiment to assess the importance of speed. Many examples of experiments focus on the User Interface (UI) because it is easy to show examples, but there are many breakthroughs on the back-end side, and as multiple companies discovered: speed matters a lot! Of course, faster is better, but how important is it to improve performance by a tenth of a second? Should you have a person focused on performance? Maybe a team of five? The return-on-investment (ROI) of such efforts can be quantified by running a simple slowdown experiment. In 2017, every tenth of a second improvement for Bing was worth $18 million in incremental annual revenue, enough to fund a sizable team. Based on these results and multiple replications at several companies through the years, we recommend using latency as a guardrail metric.

Why you care: For a company to move to the later phases of experimentation maturity (“Run” or “Fly”), incorporating data analysis pipelines as part of the experimentation platform can ensure that the methodology is solid, consistent, and scientifically founded, and that the implementation is trustworthy. It also helps save teams from needing to do time-consuming ad hoc analysis. If moving in this direction, understanding the common infrastructure steps for data processing, computation, and visualization can be useful.

Why you care:  Sometimes the effect that you care to measure can take months or even years to accumulate – a long-term effect. In an online world where products and services are developed quickly and iteratively in an agile fashion, trying to measure a long-term effect is challenging. While an active area of research, understanding the key challenges and current methodology is useful if you are tackling a problem of this nature.

Why you care: While experimentation is widely adopted to accelerate product innovation, how fast we innovate can be limited by how we experiment. To control the unknown risks associated with new feature launches, we recommend that experiments go through a ramp process, where we gradually increase traffic to new Treatments. If we don’t do this in a principled way, this process can introduce inefficiency and risk, decreasing product stability as experimentation scales. Ramping effectively requires balancing three key considerations: speed, quality, and risk.

Why you care: When running experiments, you also need to generate ideas to test, create, and validate metrics, and establish evidence to support broader conclusions. For these needs, there are techniques such as user experience research, focus groups, surveys, human evaluation, and observational studies that are useful to complement and augment a healthy A/B testing culture.

Why you care: As your organization moves into the “Fly” maturity phase, institutional memory, which contains a history of all experiments and changes made, becomes increasingly important. It can be used to identify patterns that generalize across experiments, to foster a culture of experimentation, to improve future innovations, and more.

Why you care? Organizations that want to measure their progress and accountability need good metrics. For example, one popular way of running an organization is to use Objectives and Key Results (OKRs), where an Objective is a long-term goal, and the Key Results are shorter-term, measurable results that move towards the goal (Doerr 2018). When using the OKR system, good metrics are key to tracking progress towards those goals. Understanding the different types of organizational metrics, the important criteria that these metrics need to meet, how to create and evaluate these metrics, and the importance of iteration over time can help generate the insights needed to make data-informed decisions, regardless of whether you also run experiments.

More than a decade has now passed since Falun Gong was suppressed by the Chinese government in July 1999. Until the suppression, Falun Gong's activities were largely confined to China itself. Any overseas presence that the movement had was, in a sense, accidental, the result of informal expansion amongst expatriate Chinese or, very rarely, amongst westerners who had come into contact with it through friends in China or overseas. The homeland of the movement was China, the cultural universe of Falun Gong doctrine was Chinese, and the language of the scriptures was Chinese. After 1999, however, this situation changed. Now based in the New York area, Falun Gong became a diasporic organization, a collection of co-religionists scattered around the globe who were prevented from returning to their homeland. This literal de-centring of the movement changed the status of the previously relatively insignificant number of overseas practitioners of Falun Gong. They have now become an international network of communities of loyal activists engaged in a range of activities such as lobbying for the Falun Gong cause with their host governments, keeping the message of the brutality of the suppression alive and at the forefront of the minds of the foreign media, as well, of course, as maintaining their cultivation practice.

Sekai Kyūsei Kyō [Church of World Messianity] (hereafter SKK) is well known as a Japanese new religion from which many offshoots have sprung. The religious organizations arising from SKK, and those religious organizations strongly affected by SKK, are often referred to as the ‘Sekai Kyūsei Kyō lineage (kei)’ organizations. The incidence of new groups branching off is common to both new and old religious organizations. To make an effective study of this process of schism it is very important to explore religions like SKK from which many offshoots have originated. This chapter will focus on SKK in South Korea, and will attempt to analyse the reasons for the formation of these offshoots from the perspective of management studies of religion.

Firstly, I will discuss ‘management studies of religion’, and give an overview of the current situation of Japanese religions in South Korea, and the development of SKK there. Then, I will explore the management system of the organization and the formation of offshoots from Kyūsei Kyō with particular reference to the ‘management of secrets’.

What is ‘management studies of religion’?

Briefly ‘management studies of religion’ in the context of this chapter is the study of the administration and operations of a religious system. I have chosen not to refer to the existing studies of religion or the sociology of religion. Rather, by fundamentally re-grasping the word, ‘management’, we can combine findings from management studies and religious studies, including how they interact with each other.

Post-war peace movements and religious groups

Religious groups expand their management and power in relation to the value consciousness of the general public. In the era when the borders of the Japanese nation state were firmly established, the strategies pursued by religious groups developed in relation to national values: either by going along with values promoted by the nation while evading state control, or by pursuing an independent course, and proclaiming their own religious values distinct from national values. This is an issue of the value strategies of religious groups. With the progress of globalization, however, value strategies once accepted in Japan are no longer accepted. This is a problem for religious groups, but it can also become an opportunity for them to develop new value strategies in the context of globalization. In the process of developing these value strategies, religious groups have addressed issues that are also now faced by secular organizations in the course of globalization. If we consider that many aspects of the management strategies of religious groups in Japan have developed ahead of similar developments in business enterprises, it is perhaps to be expected that the same can be observed with the value strategies of religious groups.

In this chapter, I will examine some characteristics of the peace movements associated with Japanese religious groups after WWII, focusing primarily on the World Conference on Religion and Peace (WCRP) as an example.

Religious organizations have long been at the forefront of the global movement of people and ideas (Rudolph and Piscatori 1997: 3). Missionary activity has preceded or accompanied trade and political domination across continents since well before the beginning of the Christian era, alongside less organized forms of dissemination of religious belief. Even today, religion is a key element in the development and intensification of globalization (Beyer 1994: 3).

The conceptualization of this process also has its own more recent intellectual history, which has become the object of academic scrutiny. The term ‘world religion’ only appeared in European writings towards the end of the nineteenth century, and initially in association with the universalizing claims of Christianity (Masuzawa 2005: 23). By the early twentieth century, the term was taking on an expanded meaning to include a number of other religions that are now widely listed under this heading.

The category of ‘world religion’ has oftentimes been conceptualized to mean large, established world religions with universal claims to relevance, in contrast to locally based religions. This assumed division has not gone unchallenged. Masuzawa (2005: 20-21) explains that this system of classification tends to operate within an Orientalist discourse, and elides specificities of locality and power relations among the so-called world religions. Another aspect of the dominant discourse of ‘world religions’ is that global relevance is seen as a marker of authenticity and status in the religious sphere.

From the second half of the nineteenth century to the end of WWII, the expansion of Japanese religions abroad depended firstly on the Japanese politics of expanding the frontiers of its empire throughout Asia and the Pacific Islands and secondly, on the emigration of labourers to places such as Hawaii, California, Brazil, Peru and others.

Analysing the diffusion of Japanese new religions in Brazil, Korea, Thailand and the United States, Shimazono (1991) focuses on two general variables: (1) the conditions of receiving societies (for example social and legal tolerance toward foreign religions, or the demand for new creeds as a result of socioeconomic change); and (2) specific features and appeal of expanding religions (for example straightforward magical practices, practical life ethics, systematic/logical statements, positive approach to religious pluralism).

In the following sections I expand Shimazono's approach to explain the missionary success of some Japanese religions abroad. I divide my argument into two large blocs based on data from the Brazilian context.