Development and evaluation of a questionnaire to capture environmental and occupational inhalational exposures in adults with fibrotic interstitial lung disease

Abstract

Background

Identification of exposures in patients with interstitial lung diseases (ILDs) is essential for diagnosis and management and can be facilitated through the use of exposure questionnaires. However, for most ILDs, a patient-focused questionnaire is lacking. Cognitive interviewing is a methodology used to evaluate sources of understanding and misunderstanding in a questionnaire and to provide evidence of content validity. We developed and refined a new exposure questionnaire for patients with fibrotic ILDs by using cognitive interviewing to establish its understandability and content validity.

Methods

An exposure assessment questionnaire was developed by a multidisciplinary team. Cognitive interviews with 24 patients with fibrosing ILDs were conducted by trained interviewers over the phone or Zoom using a semi-structured interview guide. The questionnaire was amended based on the interviewers’ interpretation of sources of misunderstanding. The revised questionnaire was tested in a second round of cognitive interviews with a different group of 24 patients.

Results

Among the 48 patients who completed interviews, mean age was 61 years, 58.3% were male and 75.0% were white. Based on the first round of cognitive interviews, the multidisciplinary team modified the questions, organization, and instructions of the questionnaire to facilitate recall, adjust for exposures that were frequently misunderstood or required clarification, and focus on clinically relevant exposures. The revised questionnaire performed well in the second round of interviews.

Conclusion

An exposure questionnaire, developed with input from patients, can be used to assess clinically relevant exposures in adults with fibrosing ILDs. This is the first questionnaire for all types of fibrosing ILD to have undergone content validation.

Background

Fibrotic interstitial lung diseases (ILDs) encompass a group of parenchymal lung disorders that can lead to progressive lung function decline and early death [1]. While fibrotic ILDs vary in etiology, they share common mechanisms, including epithelial cell injury [2]. A potential source of epithelial injury is inhalational exposures to organic and inorganic antigens, which have classically been associated with development of hypersensitivity pneumonitis (HP) and pneumoconioses, respectively. A high prevalence of occupational and environmental exposures has been reported in patients with many types of fibrotic ILD [3,4,5,6]. However, the burden of occupational ILDs remains under-recognized, both due to long latency periods between an exposure and the development of disease and the difficulty in obtaining an exposure history.

The identification of relevant exposures in patients with ILDs is essential not only for diagnosis, but also to enable prevention and mitigation strategies that may ultimately improve clinical outcomes [7,8,9]. Eliciting a comprehensive and relevant exposure history in patients with ILD is challenging but can be facilitated by the use of standardized exposure assessment tools [8,9,10,11,12,13,14]. Exposure questionnaires have predominantly focused on exposures associated with HP [9,10,11,12] and one HP-focused questionnaire has undergone evaluation for content validity [11]. Exposures applicable to a broader range of ILDs are included in several questionnaires used to aid differential diagnosis of ILD in clinical practice, such as those developed by the American College of Chest Physicians [15], National Jewish Health [16], University of California San Francisco Medical Center (UCSF) [17], and German Respiratory Society [18], However, many of these exposure questionnaires have not undergone extensive content validation with patients. Thus for most ILDs, a user-friendly and validated questionnaire is lacking.

While questionnaires have the advantage of being systematic, the items they include may not be interpreted by patients in the way that the clinician or researcher intended. Cognitive interviewing is a qualitative methodology used to evaluate sources of understanding and misunderstanding in a questionnaire and to provide evidence of content validity [19]. Content validity evaluates the extent to which a questionnaire captures the most relevant and important aspects of the concepts being assessed [20,21,22]. Cognitive interviewing generally occurs over multiple rounds, with refinement of the questionnaire between rounds to address any issues identified. The objective of this study was to develop and refine a new exposure questionnaire for patients with fibrotic ILDs for use in research by using cognitive interviewing to establish its understandability and content validity.

Methods

Questionnaire development

An exposure assessment questionnaire was developed by a multidisciplinary team with expertise in occupational medicine, ILD, and survey development [MG, CAR, ACS, LDS, MM, BBR]. A PubMed search was conducted in April 2022 with the search terms “interstitial lung disease” plus “inhalational exposures”, “environmental exposures”, “occupational exposures” and “exposures”. The initial questions were developed based on a review of the literature, as well as exposures included in existing questionnaires [11, 23]. The initial questionnaire comprised 33 questions covering smoking history, occupations, military service, and regular exposures to organic/inorganic materials at work, at home, or associated with a hobby. A “regular” exposure was defined as one that occurred at least twice a week for at least six months. This definition was selected to be inclusive of many potentially relevant exposures while excluding incidental minor exposures. Questions to assess timing and total duration of exposures and job title / industry were also included.

Cognitive interview patient recruitment

This study enrolled adults with fibrosing ILDs who were being followed in the ILD clinic at Duke University Health System. The study included both patients who were already established in the ILD clinic and patients who were newly evaluated. Patients were excluded if they had clinically significant cognitive impairment, were not fluent in English, or were currently hospitalized. The research team reviewed clinical charts to ensure that a variety of exposures at home and work was represented. The goal was to interview a broad group of patients with a range of exposures, to optimize questionnaire performance for any patient with ILD. The contact information of consented participants was shared with collaborators in the Center for Health Measurement at Duke University School of Medicine, who scheduled and conducted cognitive interviews and were not aware of the reported exposures in the medical record. Purposive sampling was used to ensure diversity of age, sex, race and ethnicity, and level of education (Additional file 1: Table S1). This study was approved by the Duke Institutional Review Board (Pro00110042).

Cognitive interview procedures

The cognitive interviews were conducted by three trained interviewers who had a total of 12 years’ cognitive interview experience (LM, LW, MM) using a semi-structured interview guide. Interviews were conducted over the phone or via Zoom between June 2022 and April 2023. Interviews were conducted in English. Each lasted approximately 60 min.

The interview started with a concept elicitation phase, during which participants described their symptoms and how ILD impacted their quality of life and functional status. Participants were then asked to describe domestic and occupational exposures that they believed may have contributed to development of their ILD. This information was used to determine the extent to which the questionnaire captured the range of exposures spontaneously described by participants.

The second phase of the interview was the cognitive interview phase, during which the interviewers administered the questionnaire and probed the participants on their understanding of the items and recollection of their exposures. The interviewers used a concurrent probing approach and asked participants to “think aloud” as they responded to the questions. Participants were asked to discuss how the concepts were relevant and whether items needed to be added or changed to reflect their exposures. The interviewers probed question content with a focus on: (1) comprehension (level of understanding), (2) clarity (level of straightforward meaning), (3) knowledge and memory (ease of recall of information needed to respond), and (4) judgment (ease of fitting personal experiences to the survey options). Interviewers noted hesitations, spontaneous comments about the questions, and vocal pauses that may indicate meaningful reactions to the measure. The questionnaire instructions, recall period, and response choices were evaluated.

Interviews were audio recorded. Interviewers completed interview debriefing forms, which included participants’ demographic data and survey responses, and interviewer notes, in the study’s Research Electronic Data Capture (REDCap) database. Changes to the questionnaire after the first round of interviews were tested during the second round of interviews. Items were defined as being “significantly revised” if their revision involved (1) adding or removing word(s) that changed the meaning of a phrase; (2) word substitutions that in the judgment of the investigators were more than a semantic simplification; or (3) significant changes to the response options (e.g., changing from a severity to frequency scale).

Analyses

After the first round, the interviewers summarized the results at the level of the individual items in the questionnaire. These summaries included the interviewers’ interpretation of sources of participant misunderstanding, such as overall comprehension, recall of exposures, and issues around wording. From these results, the interviewers made recommendations about modifications to the questionnaire. These recommendations and the participant responses were reviewed by the multidisciplinary team, which met biweekly. Options to modify the survey were discussed. Decisions for changes were based on consensus among all team members.

The same procedures were followed for round two of the interviews, which were performed in a different group of patients. At the end of all the interviews, the changes made to the questionnaire, with the reasons for the changes, were documented in an Item Tracking Matrix. This documented the source of the issue (e.g. overall comprehension, difficulty recalling exposure), what change was made after the first round of interviews, and whether that change had worked based on the second round of interviews.

Comparison with chart review

On intake to the Duke ILD clinic as part of routine clinical care, patients complete an adapted version of the UCSF questionnaire [17] that includes an assessment of environmental and occupational exposures. The exposure-related modifications to the UCSF questionnaire are first, that exposures are not limited to those that occurred within three years of onset of respiratory symptoms, and second, that a free text field is provided for reporting additional exposures or occupations. The responses to the adapted UCSF questionnaire, as well as additional exposures elicited by clinicians, were ascertained through a chart review and compared with responses to the exposure questionnaire. We evaluated whether the new exposure questionnaire and adapted UCSF questionnaire identified a comparable number of potentially relevant exposures and occupations.

Results

Cohort

Fifty-two participants were enrolled, of whom 48 completed interviews (n = 24 in each round). Among the participants who completed interviews, average age was 61 years, 58.3% were male and 75.0% were white (Table 1). Over one-third (37.5%) did not complete education beyond high school; 31.3% were currently employed (full- or part-time). The most frequent ILD diagnoses were IPF (27.1%), connective tissue disease-associated ILDs (22.9%), and HP (16.7%).

Table 1 Demographics of interview participants

Concept elicitation findings

During the concept elicitation phase of the interview, the symptoms most commonly mentioned by participants were shortness of breath, fatigue, and cough. Participants frequently reported delays in diagnoses and misdiagnoses. When participants were asked to describe exposures that they believed may have caused their ILD, they most frequently named their smoking history and exposures from work.

Cognitive interview findings

Based on responses in the first round of interviews, the multidisciplinary team changed the questions, organization, and instructions in the questionnaire to facilitate recall, comprehension, and focus on clinically relevant exposures (Table 2 and Additional file 1: Table S2). For example, some participants were unable to recall occupational exposures until they described their job history, which, in the initial version of the questionnaire, was only reviewed towards the end. To improve recall, employment and military history were moved earlier in the questionnaire, prior to questions about occupational or agricultural exposures. Given that some participants had difficulty recalling exposure dates and duration, the questionnaire administrator’s instructions were adjusted to provide specific prompts. The questionnaire was adjusted to provide multiple ways to record exposure duration. For exposures that were frequently misunderstood or required clarification, the phrasing was adjusted. For example, several participants requested a definition of the term “solvents”. However, certain examples of solvents, such as 1,1,1 trichloroethane, caused confusion and so were removed. The term “man-made vitreous fibers” caused confusion and was changed to “insulating materials”.

Table 2 Examples of changes made to the exposure questionnaire based on cognitive interview findings

The team amended questions on specific exposures to maximize the clinical relevance of the exposures ascertained. For example, the phrase “long-term water leaks” was added to the question about water damage or mold in the home to differentiate relevant versus minor exposure events. Many participants endorsed exposure to general household dust, which the study team did not consider clinically significant, so the question probing vapors, dust, gases, and fumes exposure at home was removed. Some participants denied exposure to “metal dust” from welding or machining because no particles were visible in the air, so the term “dust” was removed and the question was changed to ask about exposure to “metal or metal alloys”. Other common household exposures were de-emphasized; specifically, questions about dogs and cats were removed and the question about hair dye exposure was limited to the occupational context. Further, given that work settings typically have higher levels of exposure than household exposures [24], the questions were reordered and the instructions adjusted to emphasize occupational exposures and hobbies. Certain exposures that were brought up by participants as potentially contributing to their ILD, such as plastics manufacturing, were added to the questionnaire.

The changes made to the questionnaire were tested during the second round of cognitive interviews and performed well. Specifically, participants were probed on items that had been changed from the first round, and no potential misunderstandings were identified. While conducting the second round of interviews, the study team made minor changes to the questionnaire to capture exposures to epoxy/resins, petroleum and other combustion products, and metal casting. Cognitive interviews were concluded after the second round because no significant changes were needed to the questionnaire. The final questionnaire is provided in Additional file 2.

Comparison with chart-reported exposures and occupations

In general, the exposure questionnaire interviews identified more exposures than the chart review (Table 3). This was particularly notable for occupational exposures. For example, 11 participants reported exposure to machine oil, lubricants, or metal working fluid in the interviews, whereas only four of these participants had this exposure documented in their clinical record. The number of occupational exposures identified in the interviews decreased when affirmative responses were limited to individuals with at least 10 years of a particular exposure (Table 3). The domestic exposures identified in the interviews and chart review were more comparable, but the interview identified more exposures in categories such as musical wind instruments, humidifiers, bird feather products, and wall air conditioner units. The only instances in which exposures were noted in the clinical record but not in the interviews pertained to electronic cigarette use and smokeless tobacco products. Notably interviews focused on “regular” exposures whereas any affirmative response was captured in the chart review. Ascertainment of job history in the interviews captured more potentially relevant occupations than were documented in the clinical record (Additional file 1: Table S3).

Table 3 Summary of exposures identified through exposure questionnaire versus clinical record

Discussion

This qualitative study developed and refined a new exposure questionnaire for patients with fibrotic ILDs by utilizing cognitive interviewing methodology. Direct patient input into the development of questionnaires is critical to evaluate whether respondents’ understanding of each item corresponds to the developer’s intention [25, 26]. Through cognitive interviewing, we identified opportunities to improve the clarity of the items in our questionnaire, facilitate recall, focus on clinically relevant exposures, and incorporate the patient’s perspective. The cognitive interview process also provided opportunities to test the training provided to questionnaire administrators, which may be important in the research context.

An exposure questionnaire related specifically to HP, designed to aid diagnosis and antigen identification, was previously tested through cognitive interviewing in 24 patients [11]. However, to our knowledge, our new exposure questionnaire is the first questionnaire for all types of fibrosing ILD to have undergone content validation. As highlighted in a consensus statement on exposure assessment tools for HP issued by the American Thoracic Society [13], it cannot be assumed that all exposures reported on questionnaires are clinically significant. To maximize clinical relevance and minimize misclassification of exposures, it is necessary to evaluate how patients interpret questions about them. We found that some questions had unintended interpretations and that slight changes in the phrasing resulted in substantial enhancements in their interpretation by patients. The inclusion of pulmonary occupational specialists enabled the research team to modify or eliminate questions that detected exposures that were not clinically relevant.

Many of the issues identified during the first round of cognitive interviews have been acknowledged as potential challenges in exposure ascertainment [11]. For example, recollection of remote exposures and occupations was difficult for some participants. We found that asking about employment and military history in chronologic order facilitated recall of job titles and occupational exposures as well as the timing and duration of exposures. Beyond prompting memory, an occupational history could identify exposure settings that are overlooked by patients. Occupational history may also serve to differentiate the presence of a higher “dose” of an exposure than present in a domestic or environmental setting. A job title obtained from an occupational history (ideally with industry and dates) can enable researchers and practitioners, with the use of a job exposure matrix, to estimate that person’s levels of exposure to specific substances while employed at that job. Job exposure matrices have been used to assess the risk of occupational exposures in the field of chronic obstructive pulmonary disease [27].

Compared with review of the clinical record, the new exposure questionnaire elicited a similar number of domestic exposures, but a greater number of occupational exposures. There are several potential explanations for this. Adjustments made during the cognitive interview process resulted in a comprehensive and understandable questionnaire that identified more relevant exposures than documented in the clinical record. Our exposure questionnaire recorded duration, which may help identify exposures likely to be clinically relevant in future studies. Interestingly, limiting exposures to those of at least 10 years’ duration resulted in better comparability in occupational exposures between the questionnaire and clinic documentation. We also note that the approach of eliciting information via interview with trained staff differs to a patient filling out a questionnaire on his/her own and that clinician discretion is applied when noting exposures in a patient’s medical record.

An additional benefit of content validation through cognitive interviews is the creation of a document that educates those administering the questionnaire. Our final questionnaire included clear instructions as well as prompts for the person conducting the interview. While the questionnaire was designed for use in research, it is possible that its use could be extended to the clinical setting. While each cognitive interview lasted approximately 60 min, this was inclusive of the concept elicitation phase and cognitive interview probing for question clarity, and is not reflective of the time needed to administer the survey. Currently, we utilize this questionnaire in a research setting and the average time for completion is about 20 min.

This study has several limitations. First, it is important to note that this questionnaire was administered as a Zoom- or phone-based survey conducted by research staff and that further evaluation would be needed to validate the questionnaire for use without the support of an administrator. Second, all the cognitive interviews were performed with English-speaking individuals living in the Southeastern United States and there may be geographical differences in the type/frequency of exposures and in the interpretation of questions on exposures. Third, this survey was not directed toward a specific ILD, but we recognize that the associations of certain ILDs with exposures may bias a patient’s reporting. Finally, the selection of which exposures to include in the survey, as well as the threshold of a clinically relevant exposure frequency is challenging. We aimed to be inclusive in exposure selection, so that future studies can assess the clinical relevance of a broad range of exposures. The inclusion of an occupational history (job title, industry), in addition to specific exposures, may identify exposures linked to high-risk jobs, even if that exposure is not specifically asked about. A recent scoping review of exposure questionnaires in ILD noted that qualification of exposure duration is highly variable across surveys, and often uses phrasing that is open to interpretation [28]. The definition of a “regular exposure” in this survey, as well as the recording of duration of each exposure, will allow future studies to evaluate clinical relevance across standardized definitions and time frames. As more information linking the duration, magnitude, and timing of exposures with the risk of ILDs is identified, the survey should continue to be iterated upon.

Conclusions

A comprehensive and patient-centered exposure questionnaire facilitates the ascertainment of workplace and home exposures relevant to fibrotic ILDs. It is essential that the content validity and understandability of such tools are established before they are used in clinical studies. We used cognitive interviewing to identify issues in item interpretation and recall for an exposure questionnaire for patients with fibrotic ILDs. In future, this questionnaire will be used to investigate whether particular exposures are associated with clinical outcomes. Given the high morbidity associated with fibrotic ILDs and the modifiable nature of exposures, this topic is a priority for future research.