- Open Access
Validation of a brief telephone battery for neurocognitive assessment of patients with pulmonary arterial hypertension
Respiratory Researchvolume 6, Article number: 39 (2005)
The effects of pulmonary arterial hypertension on brain function are not understood, despite patients' frequent complaints of cognitive difficulties. Using clinical instruments normally administered during standard in-person assessment of neurocognitive function in adults, we assembled a battery of tests designed for administration over the telephone. The purpose was to improve patient participation, facilitate repeated test administration, and reduce the cost of research on the neuropsychological consequences of acute and chronic cardiorespiratory diseases. We undertook this study to validate telephone administration of the tests.
23 adults with pulmonary arterial hypertension underwent neurocognitive assessment using both standard in-person and telephone test administration, and the results of the two methods compared using interclass correlations.
For most of the tests in the battery, scores from the telephone assessment correlated strongly with those obtained by in-person administration of the same tests. Interclass correlations between 0.5 and 0.8 were observed for tests that assessed attention, memory, concentration/working memory, reasoning, and language/crystallized intelligence (p ≤ 0.05 for each). Interclass correlations for the Hayling Sentence Completion test of executive function approached significance (p = 0.09). All telephone tests were completed within one hour.
Administration of this neurocognitive test battery by telephone should facilitate assessment of neuropsychological deficits among patients with pulmonary arterial hypertension living across broad geographical areas, and may be useful for monitoring changes in neurocognitive function in response to PAH-specific therapy or disease progression.
Pulmonary arterial hypertension (PAH) is a devastating disease characterized by progressive shortness of breath and the eventual development of life-threatening heart failure [1–4]. While its effects on cardiovascular function have been well documented, little is known about effects of this disease on other organ systems, notably the brain. Patients frequently complain of changes in memory, concentration and judgment in association with the development of cardiopulmonary symptoms. Objective assessments of cognitive function, however, have not been performed.
The "gold standard" for comprehensive assessment of neurocognitive function is a comprehensive battery of individually validated tests that are administered in-person by an experienced interviewer. Comprehensive, standard testing often takes four hours or longer and may require two or more separately scheduled sessions. The experience can be stressful and fatiguing, particularly for chronically ill or physically disabled patients. The stress of additional travel to a testing site adds to the burden. Consequently, many patients decline to participate in clinical research that uses formal in-person neurocognitive testing, especially if the protocol includes repeated testing over time [6, 7].
To address these concerns, we developed a focused battery of neurocognitive tests for administration over the telephone. The battery is comprised of individually validated neurocognitive test instruments appropriate for administration to adults with cardiopulmonary disease who are fluent in English and capable of communicating verbally. Telephone administration of the test battery has been evaluated for feasibility and validity in survivors of the acute respiratory distress syndrome . The present study establishes the validity of telephone administration of the test battery by comparing the results of telephone testing against "gold standard" in-person administration in adult ambulatory patients with moderately to severely symptomatic PAH.
Consecutive patients diagnosed with pulmonary arterial hypertension according to standard criteria [2, 9] were prospectively recruited for neurocognitive testing from the Pulmonary Hypertension Clinic at LDS Hospital. Written informed consent was obtained from the patients for both the in-person and telephone neuropsychological testing. Inclusion criteria were age 18 years or older and the ability to give informed consent. Exclusion criteria included non-fluency in English, a history of major psychiatric illness (e.g. schizophrenia, schizoaffective disorder, bipolar disorder, psychoses requiring medication and/or hospitalization, and major depression requiring hospitalization), known learning disability, prior traumatic brain injury, diagnosis of dementia, cerebral vascular accident, neurologic disorder (e.g. multiple sclerosis, Huntington's Disease, etc.), prior cardiac surgery, or current alcohol or drug abuse.
Patients were recruited for neurocognitive assessment from a group who had consented to in-person testing. Sixty-seven patients were screened. Seventeen patients declined, two were excluded due to non-fluency in English, and two were medically unstable at the time of evaluation and died during the recruitment period. Of the 46 patients who consented to in-person neurocognitive testing, 25 consented to additional testing by telephone.
Patient demographic, medical and laboratory data were collected for all enrolled patients. This study was approved by the University of Pennsylvania and LDS Hospital Institutional Review Boards and conformed to institutional and federal guidelines for the protection of human subjects.
An interdisciplinary team of neuropsychology, traumatic brain injury, rehabilitation medicine, and pulmonary disease specialists selected a battery of standardized neurocognitive tests amenable to both in-person and telephone administration. Tests were also chosen on the basis of established sensitivity in detecting impairment in patients with cardiopulmonary disorders and concomitant hypoxemia [10–13]. The cognitive domains assessed and the tests included in the battery are listed in Table 1. All neurocognitive tests included in the battery have been empirically validated and standardized [14–17] with established reliability, internal and external validity[12, 14, 16, 18, 19]. The neurocognitive tests were administered in a random sequence to minimize order effects. However, as a delay is required between the Wechsler Memory Scale-III Logical Memory I and II tests, Logical memory I (immediate recall) was the first and Logical Memory II (delay recall) the last test administered in each session.
The in-person assessment was carried out in a private office at LDS Hospital. The identical tests were administered subsequently by telephone at a prearranged time when patients were at home and free from distraction. To minimize potential learning effects, telephone testing was performed at least 2.5 months following in-person assessment, except with one patient who was tested 57 days after in-person evaluation. A Ph.D. neuropsychologist (ROH) administered the in-person tests and a neuropsychology doctoral student administered the telephone tests with no knowledge of the results of the previous in-person testing. During both the in-person and telephone tests, patients were instructed not to write down information and to answer questions without assistance. The in-person and telephone assessments were both conducted in single sessions, and each took 35 to 45 minutes to complete.
All neuropsychological tests were scored according to the published guidelines. Each test yields a raw score that was converted into a scaled score (mean = 10; SD = 3), which was used for statistical analyses, except for Logical Memory where the raw scores are used.
Descriptive statistics were carried out for demographic and medical data. The neuropsychological test scores from the in-person administration were compared to telephone test scores using interclass correlations. To facilitate interpretation of significant correlations (p ≤ 0.05) and because traditional significance levels for correlations coefficients are influenced by factors such as group size, range of scores, and multiple comparisons, we used the following conservative classification: fair correlation with coefficients between 0.21 and 0.40; moderate correlation 0.41 to 0.60; substantial correlation 0.61 to 0.80; and almost perfect correlation 0.81 to 1.00 .
To assess potential learning effects, systematic differences between first and second administrations for each of the tests were assessed using paired sample t-tests. The differences between the in-person and telephone test results are expressed as standardized effects sizes (T2-T1 differences divided by T1 standard deviation) .
Twenty-five patients with pulmonary arterial hypertension were enrolled for neurocognitive evaluation using both in-person and telephone testing. All 25 patients completed in-person testing. Telephone testing could not be completed on one subject due to a non-functioning telephone line, and one subject died of progressive right heart failure. All of the remaining 23 subjects completed both the in-person and telephone assessments and were included in the validation group. Eighty-three percent (n = 19) of these subjects were women. The mean ± SD age was 49.7 ± 13.9 years (range 20 to 60 years) and the mean education level was 13.6 ± 3.0 years (range 6 to 20 years). The mean number of days between in-person and telephone testing was 121.6 (range 57 to 200 days). The etiology of PAH was: idiopathic ("primary") PAH in ten patients (43%), associated with anorexigen use in six (26%), collagen vascular disease in four (17%), congenital heart disease in two (9%) and one with portopulmonary hypertension (4%). The mean (± SD) right atrial pressure was 5.1 ± 1.8 mmHg, mean pulmonary artery pressure 52.1 ± 16.9 and pulmonary capillary wedge pressure 12.1 ± 6.2. The mean cardiac output was 5.1 ± 1.8 L/min. Demographic and medical data are shown in Table 2.
The results of the telephone and in-person neurocognitive assessments are shown in Table 3. The correlation coefficients for the comparison between in-person and telephone testing are presented in Table 4. Interclass correlation coefficients of at least 0.54 (p < 0.05 to < 0.0001) were found for the agreement of telephone and in-person scores on tests assessing the cognitive domains of attention, memory, concentration / working memory, reasoning, and language / crystallized intelligence. An almost perfect correlation was observed in the assessment of reasoning (Similarities). Substantial correlations were found for the Digit Span, Similarities, and Vocabulary tests (.61 to .80) and moderate correlations (.41 to .60) were found for each Logical Memory immediate and delay recall. A moderate correlation (0.56) was seen with the in-person and telephone administration of the Digit-Span-backward (concentration / working memory). Only a fair correlation (0.28; p= 0.09) was found between the in-person and telephone administration of the Hayling Sentence Completion test of executive function. For Letter-Number Sequence test (concentration/working memory) scores were not correlated for the in-person and telephone tests.
Stability over time was greatest for the Similarities and Vocabulary tests. The effects of learning showed that test scores tended to increase between the in-person and telephone tests, with the most improvement for verbal memory (e.g. logical memory immediate and delayed recall).
We found the battery of well-established neurocognitive tests to be amenable to administration by telephone and valid for the identification of neurocognitive deficits in patients with PAH. Testing was readily completed in a single, 30–60 minute session, and required neither specialized testing facilities nor travel by physically debilitated patients spread across a broad geographic area.
Our study was designed to validate the administration by telephone of a battery of neurocognitive tests against the in-person ("gold standard") performance of these same assessments. Each of the tests in the battery has been previously validated for the identification of neurocognitive deficits in various populations, including those with cardiopulmonary disease-. As such, our subjects' scores during in-person testing served as matched controls for comparison with the results obtained upon application of these same tests over the telephone.
The scores from telephone and in-person assessments correlated strongly for the majority of tests. Overall, the strengths of the correlations with in-person testing found here are comparable to those that we reported previously for the same test battery applied to ARDS survivors  and the correlations reported for other telephone neurocognitive test batteries [21–25]. Two items in the test battery did not correlate as well as the others. The interclass correlations for the Hayling Sentence Completion test only approached significance (p = 0.09). The in-person and telephone administrations of the Letter-Number-Sequencing component of the WMS III (a test of concentration/working memory) did not correlate well. Some subjects appeared to have difficulty discriminating phonetically similar sounds (e.g. the letters 'm' and 'n') when presented during telephone sessions; visual cues may have alleviated such issues at in-person sessions. In contrast, another component of the WMS III (Digits Backward) evaluating the same cognitive domain (concentration/working memory) had substantial correlations.
Although the correlations we found between in-person testing and subsequent telephone administration of the same test battery were moderate or higher, they were not perfect. The effects of learning or practice suggest that test scores increase between the in-person and telephone tests, with the most improvement in verbal memory. Thus, the improvement in test scores on the telephone administration of the test likely reflects practice effects. An alternative explanation for the tendency of subjects to perform better on the telephone test battery may be environmental factors. For example subjects scored higher on certain tasks when assessed at home as compared to similar tasks performed in a clinic setting . Improved orientation to time and place have been found when patients were tested in their own residence . Further, patients report less anxiety and prefer telephone testing compared to in-person evaluation . In addition to the pragmatic advantages, telephone testing may provide a better assessment of patients' cognitive function in their normal environment. Finally, it is possible that neurocognitive function improved during the interval between the two test sessions (mean 122 days). The reason for a potential improvement in neurocognitive performance will be important in future studies that use repeated test administration to determine the effect of drugs for PAH or other interventions on neurocognitive function.
An important limitation of our study was our inability to reverse the order of administration (in-person and telephone) . Our subjects were enrolled in another ongoing study, which required in-person assessments prior to enrollment in this study of telephone testing . An alternative would be to repeat in-person and telephone assessments in random order following the initial interview. Such additional testing, however, might have increased the potential for learning or practice affects, and the further time and travel commitments for patients likely impacting study participation. Future studies should counterbalance the order of in-person and telephone administration. Due to pragmatic limitations the time interval between in-person and telephone testing was somewhat longer than the minimum time necessary to minimize recall and learning effects.
Telephone testing has been used successfully in the assessment of neruocognitive impairment in other patient populations. Further, the ease in application and relative low cost of telephone testing have enabled assessments in several large clinical studies of cognitive function: screening of 4,932 elderly patients for Alzheimer's Disease using the modified Mini-Mental State Examination for telephone administration ; cognitive function in 4,023 patients with cardiovascular risk factors ; 466 patients with coronary artery bypass graft surgery , and in a self-referred ARDS patient group . In addition to neurocognitive testing, telephone-based assessments have provided accurate determinations of quality of life, medication usage, 24-hour physical activity and dietary recall [33–35].
Cognitive function has not been studied in pulmonary arterial hypertension, despite the frequent reports of problems with memory and concentration . Cognitive impairments are important complications of other chronic and life-threatening illnesses, and are associated with a significantly worse prognosis [36–38]. Cognitive impairments can also profoundly reduce quality of life [39–42]. Therapies that improve physical function in PAH may have important (but as yet unknown) effects on neurocognitive function, positive or negative. For these reasons, research on neurocognitive function is warranted. The brief neurocognitive telephone test battery is valid for assessment of cognitive function in this population and provides the means to pursue further, larger studies to assess the frequency and risk factors of cognitive sequelae in patients with PAH.
This study has demonstrated that scores on a battery of neurocognitive tests obtained by telephone administration correlated well with in-person testing in patients with pulmonary arterial hypertension. The strong correlations observed are comparable to previous studies that assessed in-person and telephone versions of neurocognitive tests. With minor modification, the telephone neuropsychological test battery described here provides an economical and reliable method for assessing cognitive function in patients with pulmonary arterial hypertension.
Pulmonary Arterial Hypertension
Acute Respiratory Distress Syndrome
Wechsler Memory Scale
Wechsler Adult Intelligence Scale
Rubin LJ: Primary pulmonary hypertension. N Engl J Med 1997,336(2):111–7.
Barst RJ, et al.: Diagnosis and differential assessment of pulmonary arterial hypertension. J Am Coll Cardiol 2004,43(12 Suppl S):40S-47S.
Rich S, et al.: Primary pulmonary hypertension. A national prospective study. Ann Intern Med 1987,107(2):216–23.
D'Alonzo GE, et al.: Survival in patients with primary pulmonary hypertension. Results from a national prospective registry. Ann Intern Med 1991,115(5):343–9.
Hopkins RO, et al.: Cognitive dysfunction in patients with pulmonary arterial hypertension. Am J Respir Cell Mol Biol 2003,167(7):A273.
Rothenhausler HB, et al.: The relationship between cognitive performance and employment and health status in long-term survivors of the acute respiratory distress syndrome: results of an exploratory study. Gen Hosp Psychiatry 2001,23(2):90–6.
Jackson JC, et al.: Six-month neuropsychological outcome of medical intensive care unit patients. Crit Care Med 2003,31(4):1226–34.
Christie J, et al.: Long-term cognitive, mood, and quality of life impairments in a select population of ARDS survivors from an internet-based ARDS support center. Am J Respir Cell Mol Biol 2002,165(8):A220.
McGoon M, et al.: Screening, early detection, and diagnosis of pulmonary arterial hypertension: ACCP evidence-based clinical practice guidelines. Chest 2004,126(1 Suppl):14S-34S.
Gale SD, Hopkins RO: Effects of hypoxia on the brain: neuroimaging and neuropsychological findings following carbon monoxide poisoning and obstructive sleep apnea. J Int Neuropsychol Soc 2004,10(1):60–71.
Gale SD, Hopkins RO, Weaver LK, Walker JM, Bigler ED, Cloward TV: Hippocampal atrophy following sleep apnea and carbon monoxide poisoning: simialrities and differences. J Int Neuropsychol Soc 2000, 6:154.
Hopkins RO, et al.: Neuropsychological sequelae and impaired health status in survivors of severe acute respiratory distress syndrome. Am J Respir Crit Care Med 1999,160(1):50–6.
Weaver LK, et al.: Hyperbaric oxygen for acute carbon monoxide poisoning. N Engl J Med 2002,347(14):1057–67.
Wechsler D: Wechsler Memory Scale. Volume 3. San Antonio: The Psychological Corporation; 1997.
Wechsler D: Wechsler Adult Intelligence Scale. San Antonio: The Psychological Corporation; 1997.
Kiernan RJ, et al.: The Neurobehavioral Cognitive Status Examination: a brief but quantitative approach to cognitive assessment. Ann Intern Med 1987,107(4):481–5.
Justice AC, Covinsky KE, Berlin JA: Assessing the generalizability of prognostic information. Ann Intern Med 1999,130(6):515–24.
Engelhart C, Eisenstein N, Meininger J: Psychometric properties of the neurobehavioral cognitive status exam. Clin Neuropsychol 1994,8(4):405–415.
Mitrushina M, Abara J, Blumenfeld A: Aspects of validity and reliability of the Neurobehavioral Cognitive Status Examination (NCSE) in assessment of psychiatric patients. J Psychiatr Res 1994,28(1):85–95.
Kukull WA, et al.: Interrater reliability of Alzheimer's disease diagnosis. Neurology 1990,40(2):257–60.
Prince MJMA, Richards N, Quarishi 5, Horn I: The development and initial validation of a telephone-administered cognitive test battery (TACT). International journal of methods in psychiatric research 1999, 8:49–57.
Roccaforte WH, et al.: Validation of a telephone version of the mini-mental state examination. J Am Geriatr Soc 1992,40(7):697–702.
Ferrucci L, et al.: Is the telephone interview for cognitive status a valid alternative in persons who cannot be evaluated by the Mini Mental State Examination? Aging (Milano) Ferrucci 1998,10(4):332–8.
Desmond DW, Tatemichi TK, Hanzawa L: The telephone interview for cognitive status (TICS): reliability and validity in stroke sample. International journal of geriatric psychiatry 1994, 9:803–807.
Debanne SM, et al.: Validation of a Telephone Cognitive Assessment Battery. J Am Geriatr Soc 1997,45(11):1352–9.
Ward HW: Cognitive function testing in comprehensive geriatric assessment. A comparison of cognitive test performance in residential and clinic settings. J Am Geriatr Soc 1997,38(10):1088–92.
Kent J, Plomin R: Testing specific cognitive abilities by telephone and mail. Intelligence 1987, 11:391–400.
Robins LN: Reflections on testing validity of psychiatric interviews. Arch Gen Psychiatry 1985, 42:918–924.
White J, et al.: Relationship between cognitive and Emotional Function, asnd Quality of Life in Patients with Pulmoanry Arterial Hypertension (PAH). Am J Respir Crit Care Med 2004,169(7):A174.
Breitner JC, et al.: APOE-epsilon4 count predicts age when prevalence of AD increases, then declines: the Cache County Study. Neurology 1999,53(2):321–31.
Garrett KD, et al.: The relationship between cardiovascular risk factors and cognitive decline. J Int Neuropsychol Soc 2003, 9:243.
Potter GG, et al.: The effects of coronary artery bypass graft on cognitive status change among elderly male twins. J Int Neuropsychol Soc 2003, 9:243–244.
Glasso KD, et al.: Relative validity of multiple telephone versus face-to-face 24-hour dietary recalls. Ann Epidemiol 1994, 4:332–336.
Matthews CE, et al.: Evaluation of computerized 24 hour physical activity recall. Med Sci Sports Exerc 2002,34(suppl S):236.
Matthews CE, et al.: Comparing physical activity assessment methods in the Seasonal Variation of Blood Cholesterol Study. Med Sci Sports Exerc 2000,32(5):976–84.
Cohen RA, et al.: Neurocognitive functioning and improvement in quality of life following participation in cardiac rehabilitation. Am J Cardiol 1999,83(9):1374–8.
Andersen K, et al.: Cognitive impairment and mortality among nonagenarians: the Danish 1905 cohort survey. Dement Geriatr Cogn Disord 2002,13(3):156–63.
Zuccala G, et al.: The effects of cognitive impairment on mortality among hospitalized patients with heart failure. Am J Med 2003,115(2):97–103.
Tozzi V, et al.: Neurocognitive performance and quality of life in patients with HIV infection. AIDS Res Hum Retroviruses 2003,19(8):643–52.
Tozzi V, et al.: Neurocognitive impairment influences quality of life in HIV-infected patients receiving HAART. Int J STD AIDS 2004,15(4):254–9.
Weitzner MA: Psychosocial and neuropsychiatric aspects of patients with primary brain tumors. Cancer Invest 1999,17(4):285–91. discussion 296–7
Harder H, et al.: Cognitive functioning and quality of life in long-term adult survivors of bone marrow transplantation. Cancer 2002,95(1):183–92.
These studies were supported by a Development Partner's Junior Faculty Award from GSK Pharmaceuticals (DBT). ROH is a recipient of grant support from the National Institute of Mental Health (R01 MH065406-01A1).
The author(s) declare that they have no competing interests.
DBT, JHF, HIP and ROH designed and SK coordinated this study. RP and JC designed the neurocognitive testing battery. JM, JW and ROH performed and interpreted the neuropsychological assessments and ROH the statistical analysis. CGE recruited patients and assessed results. The manuscript was written by DBT, JHF and ROH and has been read and approved by all authors.