Content validity and test-retest reliability of patient perception of intensity of urgency scale (PPIUS) for overactive bladder

Background The Patient Perception of Intensity of Urgency Scale (PPIUS) is a patient-reported outcome instrument intended to measure the intensity of urgency associated with each urinary or incontinence episode. The objectives of this study were to assess the content validity, test-retest reliability, and acclimation effect of the PPIUS in overactive bladder (OAB) patients. Methods Patients undergoing treatment for OAB were recruited to participate in a non-interventional study by completing a three-day micturition diary including the PPIUS for three consecutive weeks. Following completion of the three-week study, participants from two select sites also completed a cognitive interview to assess their comprehension of the PPIUS. Results Thirty-nine participants successfully completed the three-week test-retest study; twelve of these participants completed the cognitive interview. Test-retest reliability was high based on intra-class correlation coefficient of 0.95. Among stable patients, the difference between the mean ratings of any two weeks was non-significant. Among the twelve interview participants, nine found it simple to choose a PPIUS rating for each of their micturition episodes and most found the urgency rating definitions consistent with their urgency experiences. Conclusions The results demonstrated content validity based on qualitative interviews, and excellent test-retest reliability among stable patients. In addition, no acclimation effect was observed among stable patients. These findings support the use of the PPIUS as a reliable measure of urgency in both clinical trial and real life settings. The validity of PPIUS could be further established with future studies investigating the relationship between discretely graded urgency and incontinence continuum.


Background
Overactive bladder syndrome (OAB), according to the Standardization Subcommittee of the International Continence Society (ICS), is a symptom-defined condition characterized by urinary urgency, usually with urinary frequency and nocturia, and may or may not be accompanied by urge incontinence [1]. OAB has been shown to cause significant symptomatic burden and disrupt daily activities, sleep, and personal relationships [2]. It is common in clinical practice to ask patients to use a diary to record urinary episodes of their OAB symptoms. However, few existing questionnaires ask about the intensity of the urgency, one of the key symptoms of OAB, which can have as much of an impact as the frequency of urinary urgency [2].
The Patient Perception of Intensity and Urgency Scale (PPIUS) has been recommended when assessing the degree of urgency felt by patients at each micturition [3]. The PPIUS is a 5-point scale designed for measurement of both urinary urgency and urge incontinence. It has been incorporated into a micturition diary and has been used in OAB clinical trials [4][5][6]. Cartwright, Srikrishna, Cardazo, & Robinson [7] found that the PPIUS has good reliability, excellent known groups validity and convergent validity, and high responsiveness when used in a clinical trial study with adult women with OAB. The test-retest reliability of the PPIUS incorporated into a seven-week diary completed before and after a one-week interval has been reported in a group of asymptomatic women [8]. However, content validity has yet to be supported with qualitative research in patients with OAB. Furthermore, the possibility of an acclimation effect has not been examined among patients with OAB. Acclimation effects exist when data initially collected (i.e., the first week) are significantly different from data collected after patients become familiar with the diary, rendering the initial data unreliable.
The objective of this study was to assess the content validity of the PPIUS, the test-retest reliability of the mean PPIUS urgency ratings in men and women with symptoms of OAB, and whether OAB patients demonstrate an acclimation effect when completing the PPIUS for three weeks.

Study design
Males and females over the age of 18 years old with clinical records that described symptoms of OAB for at least three months and on continuous treatment for OAB for at least three months (based on clinician-report/prescription) were eligible for this study. These symptoms included frequency, urgency, nocturia, and/or urge incontinence. The patients were not required to undergo the urodynamic testing (UDS) for this study. Exclusion criteria were as follows: pregnant or breastfeeding women; significant stress incontinence or mixed stress/ urge incontinence where stress is the predominant factor as determined by site study staff; self-catheterization; diabetic neuropathy; urinary tract infection (confirmed by dipstick test); non-drug treatment, including electrostimulation therapy; any clinically significant condition or visual or cognitive impairment which in the opinion of the investigator makes the subject unsuitable for the study. Patients were recruited from five clinical sites in the different states in the United States (US): Oklahoma, California, Utah, Ohio, and Massachusetts. Recruitment was on a continuous basis and lasted approximately two months until recruitment goals were met. The study protocol was approved by Ethical Review Committee, Inc. (Independence, MO) (ERC ID #485-07-09). In addition, all recruitment procedures met Health Insurance Portability and Accountability Act of 1996 requirements and approval standards, and all applicable state and federal laws and regulations. An estimated sample size of 36 stable participants was required in order to achieve 90% statistical power at a 2-sided 5% significance test for an intra-class correlation coefficient (ICC) of 0.80 versus 0.50 [9].
Patients were first recruited to participate in a twovisit non-interventional study to assess the test-retest reliability of the PPIUS. Following completion of the twovisit study, successful completers of the test-retest study from two geographically diverse and high-recruiting sites were asked to participate in a third visit, consisting of a one-on-one cognitive interview to evaluate their comprehension and understanding of the PPIUS, and provide evidence of content validity. Participants from the two selected sites who agreed to participate in the cognitive interview were scheduled 1-2 weeks after they completed the test-retest study.
At the first clinic visit, participants were trained by site study staff on how to complete the PPIUS as a part of a three-day diary at home over the following three consecutive weeks. Participants mailed back the Week 1 and Week 2 diaries to the clinical site immediately upon completion, and returned the Week 3 PPIUS diary to the site at the second clinic visit (3 weeks + 5 days from the first clinic visit). Following the completion of the second visit, two select sites also recruited participants who successfully completed the three-week study to participate in a cognitive interview to assess their comprehension of the PPIUS. This cognitive interview was designated as the third visit.

Study questionnaires Patient perception of intensity of urgency questionnaire (PPIUS)
The PPIUS is part of a daily micturition diary to be completed for three days in a given week. Participants were instructed to record for each urinary episode throughout the day: the time of urination, whether the episode was an urination (defined as passed urine into the toilet) or incontinence (defined as involuntary release of urine), and the "degree of associated urgency according to the definitions provided" (Figure 1). The PPIUS uses a 5point scale for measurement of both urinary urgency and urge incontinence. Responses range from "No urgency" to "Mild urgency," "Moderate urgency," "Severe urgency," and "Urge incontinence." The definitions provided for the urgency ratings are listed in Table 1.

Patient-and clinician-completed overall treatment effect
The Overall Treatment Effect (OTE-Patient and OTE-Clinician) assesses patients' change in symptoms since the previous study visit. The questions ask the patient/ clinician to indicate whether OAB symptoms have improved, remained the same, or worsened since the last study visit. [10][11][12] The OTE-Patient and OTE-Clinician provided the information for classification of stable patients at the end of the three-week study in order to assess test-retest reliability.

RAND 36-item short form health questionnaire
The validated RAND 36-item short form questionnaire is a commonly used generic instrument designed to measure health-related quality of life [13]. The questionnaire consists of 36 items pertaining to the following eight health categories: physical functioning, social functioning, role limitations due to physical health problems, role limitations due to emotional problems, pain, mental health, vitality, and general health perceptions. The RAND-36 was administered at the first visit to characterize the initial characteristics of the study participants.

Quantitative analysis Descriptive statistics
Descriptive statistics were used to summarize demographic information collected from the participant and clinical information collected by the clinician at the first visit, as well as the number of micturition per 24 hours, number of incontinence episodes per 24 hours, and the PPIUS mean scores. Categorical variables derived from the OTE-Patient and the OTE-Clinician were also summarized. A micturition was defined as an entry in the diary with a checked "urinated" box. An incontinence episode was defined as an entry in the diary with checked "incontinence" box. If no "incontinence" box was checked for a given day, the number of incontinence episodes was set as zero for that day.
The daily PPIUS score was the mean of all reported urgency ratings on that day (including daytime and nighttime-a diary day starts when the subject gets up with the intention of staying awake and ends on the following day when the subject gets up). The mean level of urgency for a given week was the mean of the non-missing PPIUS daily scores over the three-day micturition diary period of the corresponding week.

Test-retest reliability
Data from participants whose OAB health state was classified as stable between the first and second clinic visits were used for assessment of reproducibility. Two definitions of stable patients were used and two separate analyses of reproducibility were performed. First, a stable patient was defined as those who responded "about the same," "almost the same, hardly better at all," or "almost the same, hardly worse at all" on the OTE-Patient [10][11][12]. Second, a stable patient was defined those who were classified as "about the same," "almost the same, hardly better at all," or "almost the same, hardly worse at all" based on the OTE-Clinician.
The assessment of reproducibility was conducted by comparing the stable patients' PPIUS mean level of urgency during Week 2 and Week 3. The test-retest reliability was assessed through paired t-tests, Spearman correlation, and ICC, and it was assumed that the ICC should be greater than 0.80 which represented good to excellent test-retest reliability [14].

Acclimation effect
Acclimation effect of the PPIUS was assessed by comparing the mean level of urgency at Weeks 1, 2, and 3 of the stable patients defined based on the OTE-Patient. Paired t-tests were used to assess whether the PPIUS mean level of urgency was significantly different between Week 1 and Week 2, between Week 2 and Week 3, and between Week 1 and Week 3.

Qualitative analysis
Cognitive interviews were conducted to gather feedback from the participants as an assessment of the content validity of the PPIUS. Participants were interviewed by a trained interviewer from United BioSource Corporation (UBC) using a standardized semi-structured interview guide with a think-aloud approach. The interview guide provided instructions for conducting the interview and for questioning the patients on their understanding of the PPIUS content and how they selected the PPIUS ratings. The interview guide included questions on the daily diary instructions, item content, and response scales. All interviews were audio recorded and transcribed. Content analysis and descriptive statistics were used to analyze the responses collected during the cognitive interviews. Qualitative data was analyzed using ATLAS. ti software [15]. Using ATLAS.ti, qualitative data (patient quotes from interviews) were systematically analyzed, coded, and compared. "Open coding" was performed to fracture the data (i.e., passages from transcripts and notes) into smaller units and identifying concepts, themes, or recurring regularities that appeared within each interview. Axial coding was then used to connect categories and subcategories of the data. Specifically, each of the detailed codes or categories in open coding was connected based on their emergent theoretical linkages so that they represented specific instances of more general, abstract phenomena or processes.

Patient disposition
A total of 81 respondents were screened for eligibility, and a total of 44 participants were eligible and consented for the study. Of the 44 consented participants, 43 participants completed the first clinic visit and 41 participants completed the second clinic visit. Forty-one participants completed three-day diaries for Week 1 and Week 2, however only 39 participants completed both clinical visits and all three weeks of the three-day diaries. These 39 participants were defined as the completers. Reasons for discontinuation after consent included screen failure due to positive urinary tract infection (UTI) dipstick test (n = 1), change in medication (n = 1), non-compliance with diary completion (n = 1), and loss to follow up (n = 2).
Twelve of the 39 successful completers of the testretest study also completed the cognitive interview (third visit).

Sociodemographic and clinical characteristics
Demographic and clinical characteristics reported by patients are summarized in Table 2 for the 39 participants who completed the test-retest study and the 12 successful completers who also completed the cognitive interviews. The mean (standard deviation [SD]) age of the 39 participants was 59.1 (15.23) years and most participants were Caucasian (n = 27, 69.2%), female (n = 32, 82.1%), and the most common patient-reported comorbid conditions were hypertension (n = 20, 51.3%) and arthritis (n = 17, 43.6%). Norm-based scores of RAND-36 were computed to a 0-100 score scale, where the average health of the general population is around 50 with a SD of 10 [16]. In general, the participants reported below-average physical health on the RAND-36, but average mental health ( Table 2).
The mean (SD) number of micturition per 24-hours was 10.0 (5.26) at Week 1, 10.3 (5.40) at Week 2, and 10.6 (5.70) at Week 3. The mean number of micturition per 24-hours was stable over the three weeks.
The mean number of incontinence episodes per 24hour period was 2.2 (3.99) at Week 1, 2.0 (3.46) at Week 2, and 1.9 (4.16) at Week 3. Again, the mean number of incontinence episodes per 24-hours was stable over the three weeks.

Test-retest reliability
Thirty-four participants (87.2%) were classified as stable based on the OTE-Patient and 35 participants (89.7%) were classified as stable based on the OTE-Clinician. Test-retest reliability of the PPIUS three-day mean score in stable patients as classified based on the OTE-Patient from Week 2 to Week 3 is shown in Table 3, with an ICC of 0.95 and Spearman's correlation of 0.89 (p < 0.0001). The difference in the mean values between Weeks 2 and 3 was not statistically significant (p = 0.3043). Similarly, test-retest reliability of the PPIUS mean level of urgency in stable patients defined by the OTE-Clinician from Week 2 to Week 3 was also high (Table 3), with an ICC of 0.95, and Spearman's correlation of 0.89 (p < 0.0001). The difference in the mean values between Weeks 2 and 3 was not statistically significant (p = 0.2677).

Acclimation effect
No statistically significant differences were found in the three-day mean PPIUS scores of stable patients between Week 1 and Week 2 (difference = -0.05, t = -0.66, p = 0.5173), Week 2 and Week 3 (difference = 0.01, t = 0.08, p = 0.9348), or Week 1 and Week 3 (difference = 0.05, t = 1.04, p = 0.3043). These results suggested that the PPIUS did not demonstrate an acclimation effect in this study.

Qualitative results from the cognitive interview (n = 12)
During the cognitive interviews, participants described their thought process for completing the three-day diary including the PPIUS. Participants were asked for their interpretation of the diary instructions and to describe their experience completing the three-day diary. Participants were asked about their understanding of the definitions of the urgency ratings and whether they were consistent with their own experiences and interpretation of the rating terms. Finally, the participants were asked about the formatting and feasibility of completing the diary on a daily basis for an extended period of time.

Diary instructions
Participants were asked to summarize the diary instructions for diary completion in their own words. Nine participants (75.0%) were able to accurately summarize the instructions. Half of participants (n = 6, 50.0%) felt that the instructions were clear and did not need any changes. Suggestions for modifying the instructions included clarifying the 12-hour clock as opposed to military time (n = 2, 16.7%) and providing more instructions about when to record episodes as "daytime" or "nighttime" (n = 3, 25.0%).

Overall experience
Participants were also asked to speak about their overall experience with the three-day diary. Ten participants (83.0%) said they had no difficulty completing the diary. Six participants (50.0%) expressed a positive opinion of the diary. Others (n = 2, 16.7%) reported that they had trouble remembering to complete the diary, found the diary inconvenient (n = 1, 8.3%), or were bothered by having to record urinations at night (n = 1, 8.3%).

Definition of urgency ratings
Participants were asked if the definitions of the urgency ratings were clear. Nine participants (75.0%) reported that the definition provided for "No urgency" was consistent with their interpretation of the phrase. Seven participants (58.3%) indicated that the definition of "Mild urgency" was consistent with the way they think of the phrase. Eight participants (66.7%) and nine participants (75.0%), respectively, found the definitions of "Moderate urgency" and "Severe urgency" to be consistent with their interpretation of the phrases. When asked about the definition provided for "Urge incontinence," nine participants (75.0%) felt that the definition was consistent with their interpretation of the phrase. Notable suggestions came from two participants for simplifying the definition by changing the word "postpone" to "delay;" and revising the definition of "Moderate urgency" from "I could postpone voiding . . ..without fear of wetting myself" to "I could postpone voiding for a longer period."

Formatting and feasibility
Because the participants rated their PPIUS rating on a micturition diary, they were also asked about the diary itself. Seven participants (58.3%) responded positively to the diary formatting and described it as "wonderful," "organized," "good," "self explanatory," "great," and "well done." Participants were then asked to describe a typical day when they completed the diary. Seven participants (58.3%) said they kept the diary by their bed at night. Six participants (50.0%) said they completed their diary entry after each episode, while three participants recorded the episode on a separate sheet of paper after each episode and transferred the information to the diary at a later time.

Discussion
In order to assess content validity and test-retest reliability of the PPIUS, this study recruited OAB patients to complete a three-day micturition diary including the PPIUS for three consecutive weeks followed by cognitive interviews with participants at selected sites. During the cognitive interview, participants were debriefed on the entire diary including the instructions, the urgency rating scale of the PPIUS, and their overall experience with the diary and completing it for three weeks. The interviews ascertained that participants interpreted the PPIUS as intended, with most participants reporting that they understood the urgency rating definitions, did not have difficulty rating their urgency grade for each micturition, and found the definitions to be consistent with their interpretation of the rating terms. Participants also reported that they liked and referred to the full urgency rating definitions that were included in the diary instruction packet.
Test-retest reliability of the PPIUS was assessed in 39 patients with OAB receiving pharmacological treatment for their symptoms. Reproducibility was excellent, as evidenced by high ICCs and Spearman's correlations. In addition, paired t-tests found no difference of PPIUS three-day mean scores between any two weeks supporting that the PPIUS did not demonstrate an acclimation effect.
Although the PPIUS was based on the definition recommend by ICS Standardization of Terminology of Lower Urinary Tract Function Report [1] and the Committee for Proprietary Medical Products Standardization Sub-Committee [3], and has been used in other clinical studies [4][5][6], this is the first attempt in investigating the content validity of the PPIUS from the perspective of OAB patients. While cognitive interview participants were recruited from only two sites, the participants consisted of a good representation of age range, education level, comorbid conditions, and races. In addition, participants also were interviewed shortly after completing the diary for three consecutive weeks, to ensure participants could accurately recall their experience with the diary. The study results are representative and reliable.
Several limitations should be kept in mind when interpreting these results. First, previous studies have indicated that OAB was almost equally prevalent among men and women [17][18][19][20]. However, a relatively small percentage (18%) of men participated in the study, while target enrollment for men in OAB clinical trials is typically about 30%. The results could be more confidently generalized to the male patient population if more male patients were interviewed. Another limitation was that the urgency and incontinence continuum was not fully explored, which remained a challenge in assessing urinary urgency. One participant described experiencing incontinence at all level of urgency. Further studies are needed as Starkman et al [21]. pointed out "While some argue that urgency is episodic and maximal, we believe it can be subjectively graded. For example, some episodes of urgency can be suppressed, some persistently require immediate action, and some are so overwhelming that urge incontinence results. Although evidence for such a continuum is lacking, it remains conceptually intuitive." PPIUS has been included in two completed, one withdrawn, and one ongoing clinical trial as shown in the ClinicalTrial.gov website.

Conclusions
Participants in general demonstrated good understanding of the PPIUS instructions and found the ratings of urgency severity to be consistent with their interpretation of the ratings. Results of the cognitive interviews demonstrated evidence of content validity of the PPIUS, as participants in general agreed with the definition of the urgency ratings, finding the definitions consistent with their experiences. The PPIUS demonstrated excellent testretest reliability as evidenced by high ICC in stable patients classified by both the OTE-Patient (ICC = 0.95) and the OTE-Clinician (ICC = 0.95). Acclimation effect was not observed among stable patients completing the PPIUS for three consecutive weeks. These findings support the use of the PPIUS as a reliable measure of urgency in both clinical trial and real life settings. The validity of PPIUS may be further established with future studies investigating the relationship between discretely graded urgency and incontinence continuum.