- Research article
- Open Access
Defining competency in flexible cystoscopy: a novel approach using cumulative Sum analysis
BMC Urology volume 16, Article number: 31 (2016)
Flexible cystoscopy (FC) is one of the most frequently performed urological intervention. Cumulative sum analysis (CUSUM) allows objective assessment of a proceduralist’s performance to ensure acceptable outcomes. This study investigated the application of CUSUM to assess a trainee’s learning curve and maintenance of competence in performing FC.
A single urology trainee, with no previous experience of FC, performed FCs between August 2013 and February 2014. For assessment FC was divided into 5 steps. Each step was assigned a CUSUM completion score. The primary outcome measure was successful performance of a complete FC. Prospective data were collected and analysed using CUSUM.
In total, 419 FCs were performed. Acceptable performance of FC was achieved by the 122nd procedure. Complete assessment of the ureteric orifices and trigone was the most difficult step of FC to achieve consistently. Competence for complete FC was achieved following 289 procedures.
CUSUM analysis objectively assesses acquisition of competence in flexible cystoscopy. Recommended indicative numbers may underestimate the number of FCs trainees require to achieve, and maintain, competency. Validation of CUSUM method in a larger cohort of trainees should be considered.
Flexible cystoscopy (FC) is a vital diagnostic and therapeutic urological procedure, which enables immediate visual assessment of both the urethra and bladder.
FC comprises multiple steps, each of which require varying degrees of endoscopic skill. It is often assumed that competence in performing FC is achieved in the early years of urological training or within a limited number of procedures. Guidance informing assessment of competency in performing FC is limited. In 2000, a working party of the British Association of Urological Surgeons (BAUS) recommended that a minimum of 60 FCs should be performed under supervision to achieve technical competence . This number has not been validated. It is now recognised that individuals training to perform a procedure, acquire skills at different rates . Indicative numbers are a weak method of assessing competence and fail to identify or aid struggling trainees. There is a need for a more objective method to assess competency and guide training.
Cumulative Sum (CUSUM) analysis is a statistical tool that can be used to evaluate the development of competence in defined tasks . CUSUM analysis has previously been used to chart learning curves and maintaining competency in surgical techniques, but not in FC, in vivo .
The aim of this study was to prospectively define the learning curve in FC of a surgical trainee with no previous FC experience and to evaluate the role of CUSUM as an objective measure of achieving and maintaining competency.
From August 2013 to February 2014 a Trust Grade in Urology, 2 years post qualification, with no previous experience in FC or endoscopy maintained a prospective database of all FCs performed at Weston General Hospital, North Somerset. The trainee intended to pursue a career in Urology and observed ten FCs prior to starting to perform FC. All FCs were performed using a flexible video cystoscope (Karl Storz, Germany). A senior Specialty Trainee competent in performing FC, or a consultant, provided supervision during each procedure.
FC was deconstructed into five key components for the purpose of assessment; these were defined prior to starting the project. The components were based on FC steps recommended by the Intercollegiate Surgical Curriculum Programme (ISCP) and the British Association of Urology Nurses for training and assessment (Table 1) [4–6].
CUSUM analysis requires an acceptable and unacceptable failure rate to define success and competence. Consultant Urologists in the South West of England were emailed a link to an online questionnaire describing the study. In the questionnaire consultants were asked to allocate acceptable failure rates for each flexible cystoscopy step. Of 65 invited to participate, 13 consultants completed the questionnaire. Each step has a different acceptable and unacceptable failure rate to take into account more challenging steps of the procedure (Table 1).
The Type I and Type II error rates for CUSUM analysis were set at 0.10, as a standard. [7, 8] The acceptable and unacceptable failure rates varied with each step, however, were critical for the derivation of a CUSUM score. The CUSUM score (S) was calculated for each step (Additional file 1). For each success, a decrement of (S) is applied and each failure, an increase of (1 – S) is applied. The total score is cumulative and the total score is plotted as a continual plotted line.
Horizontal control lines are plotted at regular intervals on the y-axis. The spacing between control lines is calculated using a standardised equation (Additional file 1).
The CUSUM value is plotted on the y-axis against the number of procedures on the x-axis. The CUSUM plotted line is a running sum of increments (1- S) and decrements (S).
Therefore, if the plotted line crosses the control line in an upward trend, performance is deemed unacceptable. If the control line is crossed in a downward trend then performance is deemed acceptable. If performance is maintained between two control lines then acceptable performance is being maintained. Competence is declared when 2 consecutive control lines are crossed in a downward fashion [9, 10].
Four hundred nineteen FCs were performed under local anaesthesia during the study period in 251 Males and 168 Females, median age 70 years (range 16–96). Indications for FC are detailed in Table 2.
Each of the five steps has been evaluated individually and is presented in the order FC is performed. For each step, a number of procedures were excluded due to pathology preventing complete examination. The pathologies excluded were: urethral stricture, gross haematuria, obscuring malignancy and bladder stones.
Atraumatic passage of cystoscope via urethra
Figure 1 shows the CUSUM score plotted against the number of procedures. It can be seen that the unacceptable control line [crossing the horizontal control line on the y-axis in an upwards fashion (positive gradient)] is first crossed at attempt 19 and again at attempt 47. During the first 51 attempts (Maximal point) there was a failure rate of 12 %. Performance begins to improve (no further unacceptable control lines crossed) and acceptable performance can be first concluded following attempt 117 when the acceptable control line is crossed [crossing the horizontal grid line from above to below (negative gradient)] . Performance continues to improve crossing two further control lines following attempt 224 and 332. This component successfully completed by 117 procedures and competence following 224 procedures.
Assessment of Body and Dome of Bladder
Figure 2 shows the unacceptable control line (horizontal line crossed in an upwards fashion) is first crossed following attempt 3. The maximal point is achieved following 4 procedures and no further unacceptable control lines are crossed. Acceptable performance can be concluded following attempt 59 when the acceptable control line is crossed (crossing the horizontal line in a downwards fashion). No further failures occur and performance continues to improve, crossing two further control lines following attempt 221 and 330.
This component successfully completed by 59 procedures and competence following 221 procedures.
Identification of Trigone and Ureteric Orifices
Figure 3 shows the unacceptable control line (horizontal line crossed in an upwards fashion) is first crossed following the 3rd attempt with a further 15 unacceptable control lines continually crossed. Although the first acceptable control line is crossed following the 74th attempt, the overall trend continues upwards until attempt 136. From attempt 136 to 257 satisfactory performance is evident as no further unacceptable control lines are crossed. Three acceptable control lines are crossed following 258th to 301st procedure, although the unacceptable control line is crossed following 302nd procedure. No further unacceptable control lines are crossed in the final 81 procedures. Competence achieved following 279 procedures.
Due to the large number of acceptable and unacceptable control lines being crossed during CUSUM score, the average failure rate was calculated with the number of procedures being divided into thirds. The initial failure rate for the first 128 attempts was 34 %. Between the 128th and 254th procedures the failure rate improved to 9 % then continued to improve further with a failure rate of 5 % between the 254th and 383rd procedure.
Examination of the bladder neck
Figure 4 shows the unacceptable control line (horizontal line crossed in an upwards fashion) is first crossed following attempt 2 and again following attempt 4, 11 and 23. The maximal point is reached following the 23rd procedure. Acceptable performance can be concluded following the 65th procedure when the acceptable control line is crossed (crossing the horizontal line in a downwards fashion). However, failure following the 103rd procedure leads to an unacceptable control line being crossed. No further unacceptable control lines are crossed following this procedure, with the acceptable control line being crossed following the 121st, 229th and 338th procedure.
This component successfully completed by 103 procedures and competence following 229 procedures.
Performance of full procedure
Figure 5 shows the unacceptable control line first crossed following the 2nd attempt with a further 19 unacceptable control lines continually crossed. Acceptable performance is first achieved when the control line is crossed following the 118th procedure. However, acceptable performance is not maintained with multiple unacceptable control lines crossed. The maximal point was reached following 257th procedure with a sustained period of acceptable performance following this, crossing 2 consequence acceptable control lines following the 286th procedure. Competence achieved following 286 procedures.
Due to the large number of acceptable and unacceptable control lines being crossed, the average failure rate was calculated with the number of procedures being divided into thirds. The failure rate of the first 128 procedures was 38 %. The failure rate between 128th and 256th procedure was 13 %. Improvement continued with a failure rate of 6 % between 257th and 383rd procedure.
This study describes the learning curve to competency in FC of a trainee with no previous experience in endoscopy under supervision. In an era of quality assurance and credentialing the results provide further evidence that CUSUM analysis is an objective technique, which can be used to evaluate progression to competence.
The prospective data in the current study demonstrate that CUSUM is a relatively simple and sensitive method to apply practically to self assessment in surgical training. CUSUM was able to highlight areas for improvement, guiding further training in addition to defining competence. In this study, the trainee took longer to achieve, and maintain, competency than suggested indicative numbers .
The utility of CUSUM was that it was able to define the specific aspects of the procedure which the trainee found most difficult. For three out of five of the components, acceptable performance was maintained following the 122nd procedure. Competence in examination of the ureteric orifices and trigone was a FC step which took substantially longer to acquire, being achieved by the 280th procedure. CUSUM highlighted this as an area for targeted tuition in this trainee.
Even if it is assumed that performing a minimum number of procedures will result in competence the number of FCs required to attain competence has never been validated. BAUS recommended a minimum of 60 procedures in 2000 . In 2014 the Speciality Advisory committee in Urology (SAC) stipulated that the indicative number of FCs which must be performed for the award of a Certificate of Completed Training (CCT) in Urology should be 300. Prior to this, a review of logbook data from trainees applying for CCT between 2010 and 2012 revealed that only 42 % had recorded flexible cystoscopy activity . This wide range of recommendations and trainee activity highlights the need for an alternative method, such as CUSUM, to be introduced as a more robust modality for determining competence.
CUSUM analysis has not previously been used to assess skill acquisition in FC. Studies, using virtual reality simulators to assess skill acquisition for FC, have developed a five-point Global Rating Score (GRS). Although this has been of value in evaluating technical and nontechnical skills, it may be limited by inter assessor variability [12, 13]. Such variability does not occur with CUSUM analysis due to each defined task having a binary outcome. In addition CUSUM has the advantage of being suitable for both self-assessment and supervisor assessment. As a result, CUSUM analysis has the potential to be incorporated into the trainee’s curriculum with the plotted graphs providing readily visualised, accurate, comparable evidence of progression and competence rather than the current implicit logbook approach.
While ideally CUSUM could be used as a tool for assessing skill acquisition, and its maintenance, in trainees and consultants a significant issue is that the statistical calculations are detailed and time intensive. This may be one of the reasons why, to date, CUSUM has not been widely adopted. A possible development would be creation of software to facilitate the data entry and analysis.
The present study has limitations; CUSUM was demonstrated to be an objective technique however the authors acknowledge that only one trainee’s performance was assessed. Maguire et al. used CUSUM analysis for the evaluation of a group of trainees performing retropubic mid-urethral sling procedures . In keeping with that study, our study found the number of procedures required to acquire, and maintain, competence in performing the procedure was significantly more than expected . Furthermore, Maguire at al. identified considerable variability in the number of procedures each trainee needed to achieve competence . For CUSUM to be used for FC assessment routinely its applicability would have to be evaluated in a larger trainee cohort where similar inter-trainee variability is likely be identified. The very least that such a study would achieve would be a more accurate estimate of the range of indicative numbers which a trainee requires to achieve, and maintain, competence.
A key element in CUSUM analysis is the determination of the acceptable and unacceptable failure rates of the FC procedure. Another possible limitation of this study is that these rates were based on a relatively small sample of thirteen consultants. Despite the sample being representative of the centres involved in training in the South West of England it would be desirable to increase this sample size in future studies.
Patient experience and complication rates are important outcome measures, which are integral to true competence. Currently these measures are not incorporated into UK urology trainees’ assessments. These factors were not assessed in this study as part of the CUSUM analysis because the focus was on FC performance. It would be appropriate in future studies to incorporate patient experience into CUSUM and a parallel audit to accurately capture complications.
This study demonstrates the successful use of CUSUM analysis in the assessment of surgical competence for FC. The method is one, which could be used to assess, and monitor competence, in surgical trainees, however, validation of the process, using a larger trainee cohort, is required.
BAUS, British Association of Urological Surgeons; CCT, certificate of completed training; CUSUM, cumulative sum; FC, flexible cystoscopy; GRS, global rating score; ISCP, Intercollegiate Surgical Curriculum Programme; MRC, medical research council; NHS, National Health Service; SAC, Surgical Advisory Committee
Shah J, Darzi A. Validation of a flexible cystoscopy course. BJU Int. 2002;90(9):833–5.
Vassiliou MC, Kaneva PA, Poulouse BK, et al. How should we establish the clinical case numbers required to achieve proficiency in flexible endoscopy? Am J Surg. 2010;199:121–5.
Sharp JF, Cozens N, Robinson I. Assessment of surgical competence in parotid surgery using a CUSUM assessment tool. Clin Otolaryngol Allied Sci. 2003;28(3):248–51.
Flexible Cystoscopy: Training and Assessment Guideline. British Association of Urological Nurses (BAUN) and British Association Urological Surgeons (BAUS). http://www.baus.org.uk/professionals/baus_business/publications/15/flexible_cystoscopy_guidelines__assessment. Accessed 3 Aug 2013.
Flexible Cystoscopy: Performance Criteria, Training & Assessment Logbook. British Association of Urological Nurses (BAUN) and British Association Urological Surgeons (BAUS). www.baus.org.uk/Updates/publications-new/flexi-cystoscpy. Accessed 3 Aug 2013.
Flexible Cystoscopy (Local Anaesthetic/Male and Female), Intercollegiate Surgical Curriculum Programme (ISCP). http://www.iscp.ac.uk. Accessed 3 Aug 2013.
Bolsin S, Colson M. The use of the CUSUM technique in the assessment of trainee competence in new procedures. Int J Qual Health Care. 2000;12(5):433–8.
Waller HM, Connor SJ. Cumulative sum (CUSUM) analysis provides an objective measure of competency during training in endoscopic retrograde cholangio-pancreatography (ERCP). HPB (Oxford). 2009;11(7):565–9.
Weerasinghe S, Mirghani H, Revel A, Abu-Sidan FM. Cumulative Sum (CUSUM) analysis in the assessment of trainee competence in fetal biometry measurement. Ultrasound Obstet Gynecol. 2006;28(2):199–203.
Naik VN, Devito I, Halpern SH. Cusum analysis is a useful tool to assess resident proficiency at insertion of labour epidurals. Can J Anaesth. 2003;50(7):694–8.
Robinson R, O’Flynn K. Indicative operative numbers in urology training in the UK and Ireland. J Clin Urol. 2015;8(3):188–95.
Schout BM, Muijtjens AM, Hendrix AJ, et al. Acquisition of flexible cystoscopy skills on a virtual reality simulator by experts and novices. BJU Int. 2010;105(2):234–9.
Schout B, Ananias H, Bemelmans B, et al. Transfer of cysto-urethroscopy skills from a virtual-reality simulator to the operating room: a randomized controlled trial. BJU Int. 2010;106(2):226–31.
Maguire T, Mayne CJ, Terry T, Tincello DG. Analysis of the surgical learning curve using the cumulative sum (CUSUM) method. Neurourol Urodyn. 2013;32(7):964–7.
We would like to thank Professor Kenneth MacKenzie for his assistance and guidance in this research.
No funding obtained.
Availability of data and materials
KM researched the literature, conceived the study, obtained and analysed the data. KM & JA wrote the first draft. Both authors reviewed and edited the manuscript and approved the final draft of the manuscript.
KM MBChB, MRCS (Ed), Core Surgical Trainee Year 2, Northern Deanery. JA BMBS, BMedSci, DM, FRCS (Urol), Consultant Urologist and Associate clinical lecturer, Newcastle Upton Tyne NHS Trust and University of Newcastle.
The authors declare no conflict of interest
Consent for publication
Ethics approval and consent to participate
This study did not require ethical committee review, or patient consent, as it fulfilled the criteria for audit and quality assurance by the National Health Service (NHS) Health Research Authority and Medical research Council (MRC) (Additional file 2).
An erratum to this article can be found at http://dx.doi.org/10.1186/s12894-016-0175-1.
CUSUM Equation. Additional file 1 is the mathematical equation used to calculate the value to increase or decrease the CUSUM score along with the equation used to establish the interval between control lines for each graph. (PDF 56 kb)
About this article
Cite this article
MacKenzie, K.R., Aning, J. Defining competency in flexible cystoscopy: a novel approach using cumulative Sum analysis. BMC Urol 16, 31 (2016). https://doi.org/10.1186/s12894-016-0143-9
- Flexible cystoscopy
- Cumulative sum
- CUSUM analysis
- Learning curve