Development and validation of an instrument for measuring competencies on public health informatics of primary health care worker (PHIC4PHC) in Indonesia

Abstract

Because of the increasing adoption and use of technology in primary health care (PHC), public health informatics competencies (PHIC) are becoming essential for public health workers. Unfortunately, no studies have measured PHIC in resource-limited setting. This paper describes the process of developing and validating Public Health Informatics Competencies for Primary Health Care (PHIC4PHC), an instrument for measuring PHC workers’ competencies in public health informatics. Method: This study developed a questionnaire that had three stages: the Delphi technique, a pretest, and field test. Eleven academicians from a university and 13 PHC workers joined 2 rounds of group discussion in the first stage. The second stage comprised two pilot studies with 75 PHC workers in Semarang Municipality. The third stage involved validating the questionnaire with 462 PHC workers in Kendal District. This study used Pearson’s product-moment correlation for the validity check and Cronbach’s alpha coefficient for determining the internal consistency. This study used the K-means algorithm for clustering the results of the PHIC4PHC questionnaire. Results and Conclusion: PHIC4PHC is the first comprehensive PHIC questionnaire administered in a resource-limited setting, consisting of 11 indicators and 42 measurement items concerning knowledge of health information systems, skills required for health data management, ethical aspects of data sharing and health information literacy. The final results of PHIC4PHC were clustered into three classes based on the K-means algorithm. Overall, 45.7% PHC workers achieved medium competency, whereas 25.6% and 27.7% achieved low and high competency, respectively. Men had higher competency than women. The higher the worker’s level of education, the higher the PHIC level; the longer the worker’s work experience, the lower the PHIC score; and the greater the worker’s age, the lower the PHIC score. Measuring and monitoring PHIC is vital to support successful health IT adoption in PHC.

Introduction

The resolution of universal health coverage (UHC) by the UN General Assembly in December 2012 identified UHC as a central global health objective (Vega, 2013). This challenge should place health workers at the center of each country’s response, including its stock, skill mix, distribution, productivity, and quality. Particularly, in the context of low- and middle-income countries, gaining competent health workers is a foundation for accelerating the attainment of UHC (Campbell et al., 2013).

In many low- and middle-income countries, primary health care (PHC) has been chosen as the primary strategy to achieve equitable, patient- and community-oriented and comprehensive approaches to achieving UHC (Sachs, 2012). Developing the function of PHC is crucial to serving people in remote areas where PHC requires improvements in infrastructure, skilled human resources in health care, appropriate health technologies, and financial support, as well as comprehensive health program management (Hall and Taylor, 2003; De Maeseneer et al., 2007; WHO, 2014).

Studies have found that improving communication networks and internet availability have improved access to health information and furthermore could elevate digital health literacy in the community, including among PHC workers (Edejer, 2000; Berland et al., 2001; Cline and Haynes, 2001; Norman and Skinner, 2006b; Gilmour, 2007; Bujnowska-Fedak, 2015; Vâjâean and Bãban, 2015). Several studies have reported the increasing use of health information technologies in various programs in PHC, such as inpatient electronic registry, processing, and evaluation programs and management, clinical decision support systems, surveillance, and patients monitoring (Pambudi et al., 2004; Tomasi et al., 2004; Ludwick and Doucette, 2009; Tomasi et al., 2009; Denomme et al., 2011; Rachmani et al., 2013; Yazdi-Feyzabadi et al., 2015). Some countries still report poor performance despite having dedicated information systems for PHC (Belanger et al., 2012; Farahat et al., 2018). Different settings pose specific challenges during implementation, and human resources have been considered as the most critical among the factors that contribute to the success of health IT in PHC (Ludwick and Doucette, 2009).

In the 21st century, public health professionals face major challenges, particularly in terms of technological advances, and demographic changes (Hernandez et al., 2003). Public health informatics competencies (PHIC) have become critical to PHC workers because of the current trend of health IT adoption and its necessity for jobs in PHC to be performed efficiently (Alpay et al., 2000; Montague, 2014). Public health informatics (PHI) is the application of computer science and information technology systems to public health practice, research, and learning (Friede et al., 1995). It integrates public health and IT and consists of four knowledge domains; organization and management systems, public health, information system and IT (Magnuson and Fu, 2014). PHI could improve public health surveillance capacity and response, but confidentiality and security of the information systems is a concern (Hernandez et al., 2003).

Public health workers should be able to support public health decisions by facilitating the availability of timely, relevant, and high-quality information. In other words, they should always be able to provide advice on methods for achieving a public health goal faster, better, or at a lower cost by leveraging computer science, information science, or technology (Savel and Foldy, 2012). Furthermore, public health professionals in PHC need to understand many facets of health care, including public health, health promotion, health services research, and information and communication technology (Joshi and Perin, 2012).

Indonesia is a developing country committed to achieving UHC by 2019 (Simmonds and Hort, 2013). Indonesia has 9859 PHC facilities distributed across the archipelago serving an estimated 250 million people. In 2011, among all PHC facilities, 78.4% had computers and 46% had adopted PHC information systems. (Indonesia, 2011; Indonesia, 2016). The number of PHC facilities with a health information system (HIS) has increased dramatically because of the enactment of a national social security program in 2014. In this regard, measuring the PHIC of PHC workers is crucial to ensure optimal functioning of PHC activities.

A recent study described PHIC in developed countries and for the mid-tier level of health professionals (Hsu et al., 2012). As of yet, no studies have measured the PHIC of PHC workers in low- and middle-income countries despite their adoption of technology into PHC. The objective of this study was to develop an assessment instrument to measure PHIC in the domain of information systems and IT for PHC workers with limited education and resources in developing countries. The instrument was developed in three stages: first, constructing categories, indicators, and items for the questionnaire; second, conducting a pretest via two pilot tests; and third, conducting a field test with PHC workers. This study used the Delphi technique to construct the questionnaire items with the judgment of experts to generate the final set of questionnaire items.

Methods

Research setting and design

This study had three stages (Figure 1): the first stage was development of the instrument, the second stage was pretest studies, and the third stage was field testing of the questionnaire. The first and second stages were conducted in Semarang Municipality, Central Java Province of Indonesia. The third stage was conducted in Kendal District, Central Java Province of Indonesia in 25 PHC facilities. Semarang Municipality was used to obtain experts opinion and conduct the pretest because it is an urban city in which people are more exposed to technology, whereas Kendal District is a rural-urban city and can therefore represent the characteristic of Indonesia’s PHC facilities, which are located in both rural and urban area.

Figure 1. Steps in developing and validating a PHIC4PHC instrument

This study used the Delphi technique to challenge and construct the Public Health Informatics Competencies for Primary Health Care (PHIC4PHC) questionnaire. The Delphi technique is a method for structuring group communication using a series of questionnaires. The technique can ensure that the communication process is effective and that a consensus can be reached between the researcher and a group of experts on a specified topic. The technique is used when the opinions and judgment of experts are needed but precise information is unavailable (Hasson et al., 2000; Hsu and Sandford, 2007; Keller and Heiko, 2014). This study employed a modified Delphi through two steps with face-to-face meetings of experts on education and PHC. The purpose of the meetings was to achieve a consensus regarding the construction of the questionnaire. Furthermore, the expert opinions were also used in the second stage to reduce the numbers of questionnaire items. The purposive snowball sampling technique was used to identify experts who know other experts with similar characteristics such as knowledge, skills, and experiences (Biernacki and Waldorf, 1981; Palinkas et al., 2015). Experts including academicians and PHC workers evaluated a list of potential competencies derived from a literature review. Eleven academicians participated in the first round and 13 PHC practitioners joined the second round.

An instrument agreed upon in the first stage was pretested in two pilot studies in the second stage. In the first pretest, 40 public health workers filled in the questionnaire at a Gunung Pati PHC facility, and the following pretest involved 35 public health workers at Semarang Municipality Health Office.

Figure 1 shows the flow of this study and the three-stage construction of the PHI4PHC questionnaire. The participants in the first and the second stages were 24 experts and 75 public health workers, respectively. The third stage involved 462 PHC workers filling out the PHIC4PHC questionnaire in the field test.

Stage 1: instrument development

Literature review

A literature review was performed to assess recent instruments acknowledged as information and communication technology (ICT) measurement tools to obtain a comprehensive viewpoint for developing PHIC assessment tools. Previous studies have developed instruments for measuring computer literacy, computer competency, computer knowledge, computer usage, attitudes toward computers, ICT literacy, and so forth but few have specialized in health. Table 1 provides a brief description of 10 computer literacy and competency measurement tools. The table shows that the computer-email-web fluency and eHealth Literacy Scale (eHeals) tools included the internet as a component of digital technology in their constructs; however, only one instrument is concerned with health.

Table 1. Recent instruments for measuring ICT literacy

The initial structure of PHIC assessment tool in this study was based on a framework of ICT literacy, Digital Competence Assessment, and the eHeals because this study focused on information system and IT competencies specific to the health area (Panel, 2002; Norman and Skinner, 2006a; Cartelli et al., 2012). The concept of the PHIC assessment tool in this study consists of technical, ethical, and cognitive competencies and health information literacy.

Designing the main categories, indicators, and items

The process for developing the questionnaire items in this study was based on previous studies on ICT literacy in the health area (Jiang et al., 2004; Yang et al., 2004; Norman and Skinner, 2006a; Neter and Brainin, 2012; Gürdaş Topkaya and Kaya, 2015). As shown in Table 2, the PHIC4PHC has 4 main categories or domains of cognitive proficiency, technical proficiency, and ethical proficiency, and health information literacy, in addition to 12 indicators. This step created the initial version of the questionnaire (PHI4PHC v.0) consisting of 85 questions rated on a 5-point Likert scale (where 1 is strongly unimportant and 5 is strongly important) as shown in Table 3.

Table 2. Ranking of indicators in the research process

Table 3. Reduction of questionnaire items via the experts panel

Round 1: Delphi panel

In the first round, PHIC4PHC ver.0 consisting of 85 items (Table 3) was distributed to 11 academicians. They were asked to judge the importance of each question using a Likert scale ranging from 1 (strongly unimportant) to 5 (strongly important). The expert judgments of academicians were crucial to improving the content validity of the questionnaire. Items were accepted in the questionnaires construction using a mean cutoff value of 4 with standard deviation (SD) ≤ 0.75 to gain a consensus (Hsu and Sandford, 2007). In this round, 42 items with a mean score of less than 4 were removed from the questionnaire construction. Furthermore, this process also eliminated the security knowledge indicator from the ethical proficiency domain. In this round, academicians reached a consensus. Table 2 shows the mean scores and ranking from the panel round. This round resulted in PHIC4PHC ver.1 (Table 3) which was later distributed to PHC experts.

Round 2: Delphi panel

In the second round, PHIC4PHC ver.1 was distributed to 13 PHC experts. The second Delphi panel added two items and reinstated one item that was judged to be unimportant by the academicians and withdrawn in the first round. These three items related to the importance of handling data and troubleshooting errors. This study labeled all items with a mean score ≥ 4 and SD ≥ 0.75 as important. The second round resulted in the PHI4PHC v.2 questionnaire (Table 3).

Table 2 shows that the academicians and PHC experts had different views about the importance of the main categories. Academicians judged the health information literacy category as the most important, whereas for the PHC expert, the technical proficiency, ethical proficiency, and health information literacy categories were the highest ranking.

Stage 2: pretesting of the questionnaire

Pilot Study 1

The PHI4PHC v.2 questionnaire was distributed to 40 staff at the PHC in Gunung Pati, Semarang. Table 5 shows the characteristics of the respondents in the pilot study.

The PHI4PHC v.2 questionnaire comprised 46 items rated using a 5-point Likert scale ranging from 1 (strongly disagree) to 5 (strongly agree). The construction of the PHI4PHC v.2 questionnaire included both positive and negative statements to prevent any tendency in the respondents to give the same answers. Based on PHC experts’ advice, the questionnaire included three additional items asking about computer troubleshooting skills.

The results of validity and reliability testing indicated that two questions were not valid, with item-total correlation ≤ 0.263 and a Cronbach’s alpha coefficient of 0.956. The construction of the questionnaire resulting from the pilot study was discussed with the PHC experts, who still judged that two of the items were important for measuring PHIC. Based on the discussion, the sentence ‘I know the regulations concerning the protection of personal information on computer’ was revised to ‘I know the regulations concerning the protection of patient identity on computers.’ The sentence ‘I can determine the quality of health information found via the internet’ was revised to ‘I can differentiate between correct and incorrect information found via the internet.’ This step resulted in the PHIC4PHC v.3 questionnaire (Table 4).

Table 4. The Reducing Items Questionnaire on pretest

Table 5. Characteristics of the participants

Pilot Study 2

This study distributed the PHIC4PHC v.3 questionnaire to 35 PHC workers at a monthly meeting at Semarang Municipality Health Office. In this pilot study, three questions had a total item correlation ≤ 0.283 (Items 1,30,31) and a Cronbach’s alpha coefficient of 0.923. The result of the discussion with the experts revealed items that could be withdrawn from the construction because the content could be represented by other items. In this pilot study, the experts reached a consensus on the construction of the questionnaire, thus signaling it the end of the process. This second pilot study generated the Draft PHIC4PHC. Table 4 shows the the questionnaire items.

Stage 3: field testing the PHIC4PHC questionnaire

In this stage, the Draft PHIC4PHC questionnaire from Pilot Study 2 was validated to 462 PHC workers in Kendal District. The results of validity testing indicated that Item 13 (‘I do not know network and computer application that have been used in the health information system in the workplace’) was not valid with item-total correlation of 0.087, and therefore question was removed from the construct and the following process. The next reliability test for 42 items resulted in a Cronbach’s alpha coefficient of 0.946 for the entire scale ranging from 0.944 to 0.946. All of the items were valid with an item-total correlation ranging from 0.37 to 0.80 (Table 6).

Table 6. Items analysis of the final PHIC4PHC questionnaire

Data analysis

This study applied two-stage data processing. In the first stage, a statistical method was used to validate the PHIC4PHC questionnaire. Pearson’s product-moment correlation was used to determine the internal consistency; the Cronbach’s alpha coefficient was measured for each item as well as for the entire scale. The instrument has acceptable reliability if the Cronbach’s alpha coefficient is 0.70 or above. The statistical analyses were performed using the Statistical Package for Social Science 19 (Nie et al., 1975; Bryman and Cramer, 2012).

In the second stage, the results of the field test with 42 attributes were clustered into 3 categories using a data mining technique with K-means using tool Rapid Miner 8.1 (Kotu and Deshpande, 2014). Because the PHIC4PHC questionnaire is the first tool developed to measure PHIC in developing countries, no standard categories are available for judging PHIC in such cases. Therefore, this study applied K-means to categorize the PHIC standard into three unsupervised clusters. This study classified three clusters because previous studies have widely reported the results of analyzing data sets commonly containing three clusters of observations (Hill et al., 2006).

K-means is an unsupervised machine learning algorithm that clusters data, that are similar to one another into one cluster, which is then applied to unlabeled attribute. The K-means algorithm determines a set of K clusters and assigns each datum to exactly one cluster consisting of similar data. The similarity between data is based on a distance measure between them (Gan et al., 2007). The parameters set in the K-means algorithm were K (3), max run (10), measure type (Bregman divergence), max optimization steps (100), and divergence (square Euclidean distance).

Figure 2 shows K-means algorithm categorization process:

1. 1. Determine the K value (3) as the number of categories and the metric dissimilarity (distance).

2. 2. Randomize the initial centroid of each category that will be used to the cluster data.

3. 3. Allocate all data to the nearest centroid by calculating the distance from the data to the centroid. This study used Euclidean distance to find the distance, as follows:

   $$d\left( {{x_j},{c_j}} \right) = \sqrt {\sum\limits_{j = 1}^n {{{\left( {{x_j} - {c_j}} \right)}^2}} }$$

4. 4. Recalculate centroid C based on data that follows each cluster as follows:

   $${c_j} = {{1}\over{{Nk}}}\mathop {\sum} \limits_{i = 1}^{Nk} {x_{jl}}$$
   where Nk is the amount of data that incorporated in a cluster.

5. 5. Repeat Steps 3 and 4 until convergence is reached when no data switch clusters (Shidik et al., 2016).

Figure 2. The K-means algorithm

Result

Participants of the study

Delphi panel

This study recruited two groups of experts to test the initial questionnaire developed based on the literature review. The criterion for expert selection was experiences and knowledge of PHC. This stage involved 11 academicians from the Faculty of Health Science, Dian Nuswantoro University, Semarang and 13 public health practitioners from Semarang Municipality Health Office. Their average age was 40.5 years and the average work experience was 10.3 years. Table 5 shows the experts profiles.

Pretest and field test

Two pilot studies were conducted for pretest study followed by a field test. In the two pilot studies, the questionnaire was distributed to 40 and 35 PHC workers on Semarang District; in the field test, questionnaire was distributed to 462 PHC workers in Kendal District, Indonesia.

Table 5 shows the characteristics of all of the participants in this study. The proportion of women was higher than that of men among the PHC workers. The most common educational background was vocational school, particulary in PHC.

Validating PHIC4PHC

PHIC4PHC was developed through the standard process of questionnaire development, consisting of the construction of items based on the literature and expert judgment, a pilot test and field test (Boynton and Greenhalgh, 2004; Rattray and Jones, 2007). The validation process is shown in Tables 3 and 4. The final PHIC4PHC questionnaire had 42 questions, as shown in Table 6.

The final results of PHIC4PHC were then clustered into three categories, based on the K-means algorithm. The results had normal distribution with 45.7% achieveing medium competency, 25.6% achieveing low competency, and 27.7% achieving high competency.

Table 7 shows that the highest proportion of PHC workers was in the medium category of PHIC (45.7%). Table 8 shows the category distribution for the level of competencies among the PHC workers.

Table 7. Centroid attributes of PHIC4PHC cluster using K-means

Table 8. Category distribution of PHIC4PHC among PHC workers

Table 8 shows that men had higher PHIC than women, and the higher the level of education among the PHC workers, the higher their PHIC; the longer the work experience among the PHC workers, the lower their PHIC; and the older the PHC workers, the lower their PHIC4PHC score.

Discussion

This PHIC4PHC is the first comprehensive questionnaire to assess the competencies required in the digital health era for PHC workers such as computer skills, ethical skills, and health literacy skills. Health literacy has become vital in the digital era because health professionals need to harness the myriad information sources as a consequence of the implementation of ICT in health care facilities (Jackson, 2014).

The implementation of ICT in health care institutions, particularly in low- to middle-income countries such as Indonesia, still raises the concerns about confidentiality and privacy (Koo et al., 2001; Luna et al., 2014). Accordingly, this study included confidentiality and privacy as one a measurement indicator to comprehensively capture PHIC.

In the first stage, this study used a modified Delphi technique, a popular strategy that combines quantitative and qualitative method (Murphy et al., 1989; De Villiers et al., 2005; Fong et al., 2013; Keller and Heiko, 2014). This study used the Delphi technique to gather the opinions and perspectives of experts, educator, and practitioners about the construction of the questionnaire because no measurement tools are available to measure PHIC in PHC, particularly in developing countries.

The round of the Delphi process that focused on the education experts resulted in the removal of the security indicator from the ethical domain in the initial construction. This indicator was related to knowledge about computer viruses and how to handle them. The ranking of the importance of indicators differed between the two groups of experts. The education experts ranked the information domain as the most important, whereas, for PHC experts, the importance was equal among domains. This study identified 42 items regarding PHIC that are crucial for PHC workers.

Unlike in previous studies, the results of PHIC4PHC were processed using a data mining technique because it provides the ability to detect the optimal combination of precise parameter that should be assigned to each of the variables for classification according to the purpose of this study (Tufféry, 2011). This study applied cluster analysis because this method is mostly used when no a priori hypotheses are available and research remains in the exploratory phase. Cluster analysis is an exploratory data analysis tool that aims at sorting different objects into groups such that the degree of association between two objects is maximal if they belong to the same group otherwise minimal. Furthermore, this study did not assess statistical significance among clusters because, unlike many other statistical procedures, cluster analysis is a ‘collection’ of different algorithms that ‘place objects into clusters’ according to similarity rules. Hence, statistical significance testing is not appropriate in cluster analysis (Hill et al., 2006).

PHIC are crucial for Indonesia’s public health workers because the specific geography of the thousands of islands of Indonesia pose a challenge for PHC service, particularly in rural areas of the country. ICT is a solution for improving effectiveness and efficiency in PHC service with the implication that public health professionals should be the earliest adopters of computers and other information technologies. PHIC will generate innovative ways to promote public health using information science and technology (Yasnoff et al., 2000).

PHIC4PHC revealed that women likely have lower PHIC than men in PHC, which is consistent with the results of previous studies that a gender issue remains in ICT implementation, particularly in developing countries, despite continual claims that IT is gender-neutral (Hafkin and Taggart, 2001; Hafkin and Huyer, 2007; Flynn-Dapaah and Tareq Rashid, 2010). The implementation of a HIS in PHC should contemplate gender issues at the early stages of ICT adoption to allow women to participate fully in using the HIS, particularly in the rural areas.

This study showed that the longer the work experience of public health practitioners, the lower their PHIC. This finding differs from that of previous studies because ICT literacy requires a certain amount of experience (Usluel, 2007). However, the result is unsurprising because although the duration of using ICT was related to ICT literacy, work experience was related to the age of public health practitioners and older public health practitioners have longer work experience. Older people generally have lower ICT competencies than younger people (Tijdens and Steijn, 2005), which is consistent with the finding of this study that the older public health practitioners had lower competency.

PHIC4PHC could fill the gap in the tools available for measuring the readiness of human resources in PHC institutions to adopt HISs because it can evaluate the PHIC of public health workers. The evaluation results can determine the work necessary to promote the competency of PHC workers in ICT, such as training for existing health workers and developing a curriculum, gender-specific training, education and work experience, and so forth (Hagdrup et al., 1999).

Conclusion

This paper describes the research method for measuring PHIC in the form of a questionnaire comprising 7 indicators and 42 items. The primary indicators were cognitive proficiency, technical proficiency, ethical proficiency, and health information literacy.

Previous studies have measured PHIC in developed countries and typically for PHC workers in higher education. This PHIC4PHC is valid and reliable in measuring PHIC in urban and rural PHC facilities. The final version of the assessment tool developed in this study is expected to be used in the future study of PHI in PHC, particularly in developing countries and resource-limited settings to elevate the success of implementing ICT in health care service.