Institutional Values Influence the Design and Evaluation of Transition Knowledge in Funding Proposals at NOAA

Abstract

This article shows how institutional values influence the design and evaluation of arguments in funding proposals for research. We characterize a general argument made within proposals and several kinds of subarguments that contribute to it. We indicate that funders’ values inform the kinds of proposal documents funders require and their relative weighting of them. We illustrate these points by showing how a program office in the US National Oceanic and Atmospheric Administration (NOAA) uses its public service mission to require and heavily weigh arguments to transition new knowledge to NOAA service providers, and we suggest conceptual questions raised by the use of transition knowledge.

1. Introduction

Philosophers are drawing fresh attention to the influence of values in science, including how values shape the selection of research topics and questions (Douglas 2016, 2021). Organizations that fund science influence those selections with their institutional values, and public funders confront challenges of balancing researchers’ curiosity with work that advances the public interest (Douglas 2021, 91).

There is an open program to better articulate and examine the arguments that researchers use in funding proposals and the criteria that funders use to evaluate those arguments (O’Malley et al. 2009). Previous work often focuses on the content of proposal narratives related to methods for generating knowledge (Haufe 2013; Velarde 2018). For instance, Haufe (2013) notes that reviewers for the US National Institutes of Health (NIH) were more likely to positively score proposals that characterized planned work as testing hypotheses. Proposals are often complex arguments that include content from many documents in addition to the proposal narrative. As yet, philosophers have little characterized these complex arguments or their parts, nor have they indicated how institutional values can inform evaluations of those arguments.

In this article we advance those discussions in several ways. First, we characterize a general overarching argument of funding proposals that incorporates their plethora of documents additional to project narratives. We show how the overarching argument is composed of several subarguments, some of which are about methods for generating the planned knowledge, as discussed by Haufe, and some of which are about the research team’s use of its project knowledge in the sense of Meunier (2019). We argue that a funder’s institutional values can inform both the relative weight given to different kinds of knowledge described in proposals, and the trade-offs funders are willing to tolerate between those kinds. So, funders tailor the general proposal argument to their particular contexts.

Second, we illustrate these points with the National Oceanic and Atmospheric Administration (NOAA), a science-based US federal agency. NOAA-supported research develops and enhances services for the US public and international communities with weather forecasts, coastal management, fisheries regulation, and many other services. Taking NOAA’s service mission as an institutional value, we show that this value informs NOAA’s policies to fund research.

We further focus on NOAA’s Weather Program Office (WPO), which is at the forefront of NOAA’s efforts to develop procedures to transition the results of research projects to concrete contexts of application. WPO administers a collection of funding programs, and it requires those seeking funding to include in their proposals explicit arguments about how and why they will transition project outputs to NOAA’s service providers or other uses. ¹ Transition arguments are significant and weighty components of proposals and their review. We characterize transition arguments as instantiations of transition knowledge, and transition knowledge more generally as a kind of project knowledge specific to NOAA’s institutional value of providing public services.

Section 2 characterizes component and general arguments of funding proposals and shows how institutional values can influence the evaluation of those arguments. Section 3 overviews NOAA’s structure, functions, and mission and shows that WPO institutionalizes NOAA’s value of providing public service in the process of evaluating funding proposals. Section 4 discusses transition knowledge and arguments as weighty parts of proposal evaluations. Section 5 concludes with some topics for further study.

2. Project knowledge and the arguments of research proposals

In this section we build on Robert Meunier’s account of project knowledge and characterize a set of arguments often implicit in funding proposals. We then show with a toy example how the values of funders can influence how they weigh those arguments when selecting proposals to fund.

Meunier (2019) notes that researchers often conceptualize their work in projects, a practice funding agencies reinforce. He roughly characterizes a project as a historical episode in which a team studies a particular phenomenon, is directed by research questions and some kind of plan, and uses an assemblage of items including instruments, devices, theories, models, methods, and materials. It is an open and challenging problem in the sociology of science to characterize the prevalence, structures, and functions of research projects (e.g., Cointe 2021). We use Meunier’s rough characterization, which captures enough of the core idea for our purposes.

Meunier distinguishes two kinds of knowledge relevant to projects. The first we call focal knowledge, which is the new knowledge about particular phenomena that would address a project’s motivating research questions. ² Many think of focal knowledge as the primary or most important product or outcome of the project, often in the form of results published in peer-reviewed papers.

Project knowledge, the second kind of knowledge, is knowledge about how to design and conduct projects. More general than procedural or methodological knowledge, it includes how generally to organize, plan for, and adjust research work within the contexts of organizations and institutions such as universities, funders, and peer review. It includes knowledge of possible goals projects might achieve, how to transfer results and resources across projects, and how to tie conceptual components of a project together into a coherent account of the work and its fruitfulness. It also includes how generally to use a range of investigative tools and produce representations and publications. While project knowledge is often tacit, it can be made explicit and is a competency junior researchers often learn from their mentors.

2.1 The overarching and subarguments of funding proposals

We use the distinction between focal knowledge and project knowledge to characterize the arguments of funding proposals. Research proposals are packets of documents and they are arguments that a project should be funded. Different documents have different functions in these often complex arguments, which funders and peer reviewers evaluate according to a variety of criteria. We list four kinds of subarguments that collectively support a proposal’s overall argument that the project it describes should be funded. These four kinds of arguments do not exhaust the kinds of arguments used or evaluated in proposals, but they are operative. Each kind of subargument is often developed with several of its own constituent arguments, the conclusions of which inform the premises for the overarching subargument.

1. 1. The focal knowledge or outcome is worthy of pursuit. Constituent arguments conclude, for instance, that the proposal’s listed research questions are situated within a current and respected research agenda, that to address them requires the development of new knowledge, and that this focal knowledge is of interest to others, perhaps across various disciplines or sectors. Subarguments might also conclude that the focal knowledge could be used to ameliorate problems, to advance ongoing debates, and that the amount, quality, or generality of the focal knowledge is of sufficient caliber given the money requested. These arguments are evaluated, for instance, by criteria for Intellectual Merit and Broader Impacts at the US National Science Foundation, and as Significance and Innovation at the National Institutes of Health. The documents in funding proposals that articulate these arguments typically include abstracts, project summaries, and project narratives.

2. 2. The investigative procedures are well and appropriately designed to produce the planned focal knowledge. Constituent arguments conclude, for instance, that the proposal plans and describes relevant theories and techniques to produce and test models, methods to collect and analyze data, and procedures to control and check for sources of error in reasoning such as artifacts, noise, contaminations, motivated reasoning, and biases. The relevant documents for these arguments include project narratives.

3. 3. The researchers listed comprise an appropriate team with appropriate resources to execute the project design and produce the planned outputs. Constituent arguments conclude, for instance, that the researchers have experience or expertise with the investigative procedures, focal phenomena, and relevant research earlier described. They also indicate that the team will be reliable and ethical stewards of the money awarded and information developed, that they have access to appropriate facilities and tools to conduct their project, that they have agreements in place with required colleagues and organizations, and that they have planned an allotted appropriate time to complete the work. Relevant documents used to indicate these conclusions include researchers’ CVs, biographies, lists of recent funding awards, Gantt timelines, budgets, letters of collaboration or endorsement, data management plans, and descriptions of facilities and resources.

4. 4. The knowledge outputs will be transferred to contexts outside of the project. Constituent arguments conclude, for instance, that data will be sent to relevant repositories, results will be made available in preprints and publications, students and staff will be mentored to develop procedural and project knowledge and to advance their careers, or the team will transition the focal knowledge to patent applications or particular applications in organizations. Relevant documents that indicate these conclusions include data management plans, project narratives, publication plans, mentoring plans, and transition plans.

We note some aspects of subarguments 1–4. Subarguments 1–2 are for many funders more about focal knowledge than project knowledge. Subarguments 3–4 are more about project knowledge than focal knowledge. There are open discussions about how funders evaluate or should evaluate subarguments 1–2, especially related to methodology (e.g., O’Malley et al. 2009; Haufe 2013). While subarguments 3–4 function in the overarching arguments and evaluations of proposals, they remain little studied by philosophers.

Those subarguments mutually inform a proposal’s overarching argument. The general form of this argument is roughly as follows. Any project is well designed and will probably produce the outcomes described or outcomes of comparable value only if the conclusions of 1–4 are likely true. The proposal documents provide reasons and evidence for the truth of conclusions 1–4. Therefore, reasoning inductively (and fallibly), the project will probably produce the outcomes described or those of comparable value. If the value of those outputs is greater than those of equally well-designed projects in the pool of proposals, then this proposal should be funded.

We note a couple of points about the overarching argument. First, we characterize the first half of the argument as inductive because the satisfaction of subarguments 1–4 are necessary conditions in the evaluation of proposals, but their satisfaction isn’t always sufficient to guarantee planned outcomes or to justify selection for funding. Projects rarely proceed as planned, and many are accompanied by evaluation studies to test the empirical validity of the inference. Furthermore, particular funders often look for additional subarguments additional to 1–4. Next, the last step about funding is open to all sorts of caveats as we discuss in the following text. Finally, when investigators construct and submit proposals and their attendant arguments, they are displaying their project knowledge.

2.2. Funder values and proposal arguments

We argue that funders’ values influence many aspects of proposal design and evaluation processes beyond the evaluation of subarguments 1–2. These values influence the types and structures of documents funders require to help them make those evaluations. They also influence the criteria funders use to evaluate subarguments 1–4, including appropriate and relevant focal phenomena, methods and procedures, size and scope of projects, and so forth. For instance, a funder that values large high-energy physics projects is unlikely to consider small biomedical projects. Values also influence funders in how they weigh the importance of arguments 1–4 relative to each other.

When selecting projects to fund, funders might tolerate trade-offs between the strengths of the four kinds of subarguments, or between the strengths of focal knowledge and project knowledge. Consider a 0 to 10 scale for scoring the strengths of the four kinds of subarguments, with 0 being weakest and 10 being strongest. Consider seven proposals for which the scores of their subarguments sum to 24, as in Table 1.

Table 1. Evaluation of example proposals

	Subargument	Prop. 1	Prop. 2	Prop. 3	Prop. 4	Prop. 5	Prop. 6	Prop. 7
Focal Knowledge	1. Pursuitworthiness	6	9	3	3	9	9	3
	2. Methods	6	9	3	9	3	3	9
Project Knowledge	3. Team/Project	6	3	9	9	3	9	3
	4. Transfer	6	3	9	3	9	3	9
	Sum:	24	24	24	24	24	24	24

Note: The table shows seven proposals that each have a numerical grade of 24 points out of a possible 40. The proposals vary in their component scores for subargument categories of Pursuitworthiness, Methods, Team/Project, and Transfer. The first two categories are classed as aspects of Focal Knowledge, and the latter two as Project Knowledge.

How to choose between them? The choices depend partly on the relevant funder’s aims and values. Funders looking for steady and incremental progress may prefer Proposal 1. Funders looking to push the boundaries of knowledge may prefer Proposal 2. Funders looking to invest in strong research teams or accomplished researchers may opt for Proposals 3, 4, or 6. Funders looking to invest in high-risk/high-reward projects may target Proposals 5 or 6, such that if the focal knowledge is so worthy of pursuit (and perhaps recalcitrant in its discovery), then the funder is willing to tolerate less-proven research methods or weak arguments about how to transfer results to other contexts. Finally, funders looking to support early career researchers may choose Proposal 7, such that the researcher is likely to meet with success when starting a research program.

We next focus on WPO to illustrate how NOAA’s values inform its funding of projects. We argue that NOAA’s public service mission and related values largely preclude it from funding proposals that score comparatively low for subargument 4, and thus that mission constrains the kinds of trade-offs available to NOAA.

3. NOAA and transitions

The US federal government created NOAA in 1970 as a science-based public service agency that houses many thematically related governmental organizations. ³ NOAA’s annual appropriations for 2020–22 were $5.5–$6 billion. ⁴ NOAA is led by a White House–appointed administrator, maintains hundreds of facilities across the United States, and employs more than 12,000 people across its central office and six primary units, called line offices (e.g., National Weather Service [NWS], Office of Oceanic and Atmospheric Research [OAR]). These line offices have considerable autonomy to provide their particular services. For instance, the NWS produces weather forecasts and alerts ultimately seen ubiquitously by Americans and by many others throughout the world. Other line offices maintain and operate NOAA’s fleet of ships and aircraft, monitor and regulate US fisheries, steward and explore US coastal ecosystems, manage global environmental data and measurements from weather satellites, and run labs and research programs related to weather, water, and climate. ⁵

Due to the heterogeneity of the line offices, NOAA has a broad public service mission to benefit the US public and international communities: “To understand and predict changes in climate, weather, ocean, and coasts, to share that knowledge and information with others, and to conserve and manage coastal and marine ecosystems and resources.” ⁶

Since 2018 Congress has annually appropriated roughly $1 billion to NOAA for R&D, with a slim majority used intramurally and all research money classed as applied and not basic. NOAA differs from other federal research agencies like the NSF, for example, which in the same years neared $8 billion in R&D appropriations that were mostly extramural and basic. ⁷ All of NOAA’s line offices support intramural research or extramural project competitions. OAR supports extramural R&D partly through a variety of theme-based suboffices like the Weather Program Office (WPO).

Given its available resources, the first principle of NOAA’s R&D policy is Mission Alignment, which states that “NOAA’s R&D portfolio will be focused on NOAA’s explicitly defined mission needs,” and continues, “All participants in NOAA R&D will know the specific mission(s) they are supporting with their research or technology development and view each of their activities as directly relevant to (a) specific NOAA mission(s) need.” ⁸ That policy then states that R&D results should be transitioned to operations or other uses.

NOAA further stipulates a transition policy that begins:

[NOAA] is a science-based service agency. NOAA’s ability to meet its mission through the delivery of continually improved products and services relies on the conversion of the best available research and development (R&D) endeavors into operation and application products, commercialization, and other uses. NOAA therefore requires an integrated transition enterprise linking research, development, demonstration, and deployment that is efficient and effective in identifying and using significant new R&D products to meet NOAA’s mission needs. ⁹

So, NOAA funds R&D because it treats new knowledge as necessary to achieving its service-based missions.

The transition policy stipulates a construct of Readiness Levels (RLs) for heuristically characterizing transition processes. NASA had developed a similar construct to guide technology development given constraints of performance, schedule, and budget (Mankins 2009). NOAA expanded the construct to also include applications and focal knowledge. The nine levels together indicate a linear development trajectory (Figure 1). Projects are conceptualized to begin with focal knowledge or tools at a particular level in the trajectory. As they proceed, projects produce focal knowledge or tools that meet the qualifications required for a later level. By policy, NOAA funds projects that begin at RL 2 or higher.

Figure 1. NOAA readiness levels for transitioning R&D.

Note: The figure represents nine linear stages for the development of knowledge and tools. The first five levels represent stages of R&D, the next three represent stages of transition, and the final stage represents regular use of the new knowledge or technology by NOAA’s service providers. (From NOAA Handbook on Transitions¹⁰.)

NOAA has built significant organizational infrastructure to support transitions. OAR maintains a database of funded projects that documents their current RLs. OAR houses several capabilities that support transition processes across NOAA, including a Line Office Transition Manager, an Office of Research Transition and Application, and a community of practice among representatives from OAR’s 10 internal laboratories. Alongside program managers, these units help researchers develop new knowledge in R&D contexts and port it for use in service contexts, for instance, in the regular activities of the NWS.

NOAA also provides resources to help researchers and program managers develop and implement transition plans. Once funded, some project teams develop transition plans if they expect to satisfy conditions for RL 4 or higher. For a project’s research outputs, a transition plan details end users, a budget, and a communication route by which to ensure that these users receive the outputs. Resources include a handbook for understanding and administering the transition process ¹⁰ , transition plan templates, examples of past plans, and seminars for writing and iterating plans. NOAA treats transition plans as living documents that are reviewed and adjusted at multiple iterations by the project team, end users, program officers, and multiple levels of NOAA management.

4. Funding proposals and transition knowledge

In this section we characterize transition knowledge as a kind of project knowledge developed among NOAA-funded researchers and specific to NOAA’s institutional value of providing services to the US public. We illustrate transition knowledge using WPO and how it incorporates NOAA’s mission through funding policies, solicitations, and the review of proposals. WPO is a fruitful example because it is at the forefront of NOAA’s efforts to augment the agency’s transition processes and culture. The Weather Act of 2017 and subsequent appropriations boosted WPO’s R&D budget for projects to improve weather predictions and warning systems. ¹¹ The Weather Act further requires WPO to track the progress of its funded projects and how well outputs of those projects are transitioned to operations in other line offices, especially the NWS. Thus, statute beyond NOAA policies requires WPO to have an effective transition process.

For example, in November 2021 WPO closed a solicitation for $48.8 million for projects of three years or fewer to improve modeling and forecasting on topics ranging from fire weather to tornados and winter storms. ¹² The evaluation rubric scored proposals for 100 points: 5 for diversity and inclusion, 5 for education and outreach, 10 for project costs, 15 for applicant qualifications, 35 for technical scientific merit, and 30 for relevancy. This relevancy category asks evaluators to score proposals based on 15 questions, which cover aspects of transitions including appropriate characterization of RLs, collaborations with NWS or other end users, transition paths, and so forth. Topics about transitions also influence three of six questions for the 35-point scientific merit category, which is otherwise about methods and reasoning to produce the focal knowledge. WPO’s proposal review process requires applicants and reviewers to have and use transition knowledge especially for these two criteria of scientific merit and relevancy.

Applicants for WPO funding must learn, use, and demonstrate knowledge about how to transition the focal knowledge they propose to NOAA operations. Because it involves planning research in the context of organizations, we class this knowledge as a kind of project knowledge and for convenience label it transition knowledge. Researchers have transition knowledge when they know how to situate a piece of not-yet-conducted work and expected results in a development trajectory, advance such work or results in the trajectory, navigate resources for help, make relevant interpersonal and professional relationships, and write compelling transition arguments as components of funding proposals to demonstrate this knowledge. For WPO, these arguments must feature RLs, so applicants must understand that construct, and its operationalizations and limitations.

To write competitive proposals for WPO, researchers must invest significant amounts of resources to learn transition knowledge. This investment is compounded by variations in organizational mechanisms and priorities across line offices, funding programs, and yearly solicitations. NOAA recognizes the scale of these investments, so it provides the resources described in the previous section. Researchers must also use pregrant resources to meet with service providers, learn about their needs, and recruit their explicit participation in the transition process.

Transition arguments are weighty components of WPO proposal evaluations. For the solicitation previously mentioned, the technical scientific merit criterion is barely one-third of a proposal’s total score, and even it is influenced by reviews of transition arguments. While some funders might strongly privilege such a criterion for focal knowledge when evaluating a proposal, WPO cannot. No proposal will receive an overall high score with poor transition arguments.

Given their weight, transition arguments function centrally in the general argument of WPO proposals. These arguments provide reasons for reviewers to conclude that if WPO funds the proposed project, then the project will probably improve NOAA services and advance the public interest. Research teams often dedicate several sections of proposal narratives to give transition arguments, which are prototransition plans. They provide simplified models of organizational communication and reasons to believe that the team will be able to intervene on those channels. A good transition argument signals good faith to program managers that the team will participate in the process to develop, iterate, and follow a transition plan. Ultimately, transition arguments influence at least two of the four kinds of subarguments discussed in section 2: pursuitworthiness of focal knowledge as recognized by service providers, and transfer of outputs to those same service providers. A project could yield novel focal knowledge and efficiently send it to providers, but if they don’t need or can’t incorporate it into their protocols, it scores low on relevancy.

WPO employs NOAA staff who have their own transition knowledge to evaluate project proposals at several stages. Proposal writers first send program officers letters of intent (LOI) to submit full proposals. NOAA staff review the LOIs and return nonbinding judgments that often include feedback about the transitionability of the research described. Second, each full proposal is scored by three to five reviewers, with at least one from relevant NOAA operational offices (e.g., NWS), or other NOAA staff knowledgeable about those operations. These reviewers focus on the relevancy criterion for proposals. Academic and non-NOAA peer reviewers often have transition knowledge from previous experience with NOAA research. Ultimately, WPO ensures that the review panel includes experts with sufficient transition knowledge to evaluate proposals against the relevancy criterion.

NOAA’s mission and values limit the extent to which WPO tolerates trade-offs across focal knowledge and project knowledge. Recall that depending on their institutional aims and values, funders may tolerate trade-offs between the strengths of any of the individual four subarguments or between planned focal knowledge and project knowledge. For instance, a proposal might describe a project that if funded, is highly likely to make major advances in focal knowledge of solar winds. But if the proposal doesn’t demonstrate that its planned focal knowledge matters to any service provider or end user in NOAA’s purview, or that they know how to get it to them, then WPO is unlikely to fund it.

5. Open questions for transition knowledge

We close with some questions for further study of transition knowledge. First, how do other service-driven funders require and evaluate transition knowledge in their solicitations? Do funders develop transition processes that are causally effective, especially as the RLs provide a linear model of development known not to capture the actual R&D process (Douglas 2014)? Or do transition requirements merely add layers of red tape to the application process? How can agencies fruitfully individuate projects and transitions and conceptualize and measure success (e.g., Sen 2015)? And given that researchers need significant transition knowledge to write compelling transition arguments, are new investigators or those with fewer resources hamstrung? Do steep requirements for transition knowledge accelerate the Matthew effect, limit diversity in pools of applicants, or exclude innovative approaches and research questions that would advance NOAA’s mission? Answers to these questions will reveal the paths by which institutional values, transition knowledge, and project knowledge more generally influence the practice of science.