Evaluation of Measurement Performance of Routine Chemistry Analytes by Application of Sigma Metrics, Method Evaluation Decision (MEDx) Charts and Quality Goal Index (QGI)

Objective: To evaluate performance of routine chemistry analytes in a tertiary care hospital laboratory by the application of sigma metrics.

Introduction: Six sigma (6σ) is a popular Quality Management System (QMS) tool. Laboratories are increasingly using the six sigma method for the objective assessment and comparison of the analytical methods and instrument performance. Six sigma is about measuring or counting the number of defects. It quantifies the performance of a process as a rate of Defects-Per-Million-Opportunities (DPMO or DPM). The aim is to assess the performance and to eliminate or reduce the variation in a process.

Materials and Methods: This prospective study was conducted over a period of six months duration. Sigma metrics were calculated using coefficient of variation (%CV), %Bias and total allowable error (%TEa). For %CV we used Internal Quality Control (IQC) samples, at two levels L1 and L2. Daily IQC results of L1 and L2 for 25 routine chemistry analytes were recorded in an excel sheet and %CV was calculated for each analyte for the period of study. For each analyte %Bias was calculated based on values obtained from monthly RIQAS-EQA program data. The total allowable error (%TEa) values for each analyte were extracted from various sources like Clinical Laboratories Improvement Amendment act (CLIA), Canadian Fixed Limits from the College of Physicians and Surgeons of Saskatchewan (CFX) and Spanish Society of Clinical Chemistry and Molecular Pathology (SEQC) table of Desirable Quality Specifications based on Biological Variation (BV) criteria for acceptable performance. Sigma values were calculated. The minimal acceptable performance criteria was considered as 3 sigma. Normalized MEDx charts were used to plot sigma metrics to visually present the performance. Quality Goal Index (QGI) analysis was carried out as a part of Root Cause Analysis (RCA).

Results: Highest sigma value of 16.7 was noted for HDL-C and the lowest of 2.08 for chloride at level L1. Many analytes like ALP, Amylase, AST, CK, GGT, HDL-C, Magnesium and Uric Acid attained world class quality performance at both levels L1 and L2 with sigma levels of >6. Many other analytes showed satisfactory performance with sigma levels of >3. Sodium, potassium, chloride and urea did not show a satisfactory performance.

Conclusion: Sigma metrics evaluation of analytical performance of our laboratory showed an acceptable performance for a wide range of analytes in patient samples.

The total testing process of a laboratory is a complex process that includes three phases: pre-analytical phase, analytical phase and the post-analytical phase.

Defects may occur at any point during the process. For pre-analytical phase these may include haemolyzed samples, insufficient samples, incorrect label, clotted sample, leaking tube, and broken or wrong container. For post-analytical phase these events may include errors like failure to report a test result, delay in reporting, incorrect calculation, wrong patient results or critical values not reported in time.

Different strategies have been employed to identify and assess the errors and reduce their rate of occurrence in the three phases of the testing process. Six sigma is a popular QMS tool. Sigma metrics can be applied to quantify the performance of a laboratory in these phases of the testing process on sigma scale based on counting or measuring the total number of defects/errors and the total number of opportunities and then calculating the rate of Defects-Per-Million-Opportunities (DPMOs or DPM). The sigma metrics is compared with the expected defect rate to quantitatively assess the process performance [1,2] (Table 1).

Laboratory staff can exercise full control only on the analytical phase of the total testing process [3]. During the analytical phase, however, the defects do not occur as discrete events but rather as a degree of variation from the actual/true value, therefore detecting and then counting a defect is more difficult. When a single test report for a particular analyte is generated on a patient sample it is not possible to know what the true value would be or how much it is away from the true value of the analyte in the sample [4]. By measuring %CV and %Bias, the traditional parameters of variation, we can have an idea of the magnitude of variation in a testing process. However, by using an additional parameter i.e. total allowable error (TEa) obtained from a reliable source sigma metrics can be calculated and compared to the expected values for assessment of the method performance [5]. Allowable total error (TEa) sets a limit for combined imprecision (random error) and bias (systematic error or inaccuracy) and refers to the degree of change that needs to be detected in an analyte for a clinically important decision to be made with regard to further investigation or treatment [6].

In a clinical chemistry laboratory quality control material is routinely used as a part of the daily IQC program. These controls have different analytes with known values. With multiple serial IQC results for an analyte over a period of time e.g. 20 days or more, %CV can be measured which reflects variation of the testing method affecting patient sample results.

%Bias of an analytical testing process can be calculated by comparing results generated by a test method with a reference method or by linear regression in a method comparison study or by assessing the results of a test method in an external quality assurance (EQA) program.

Once %CV, %Bias and %TEa for an analyte of interest are known then sigma metrics for it can easily be calculated to evaluate the performance of a method and to decide about its suitability as a testing method.

Tables are a usual way of data presentation but have limited ability to summarize and compare a large amount of data at the same time and therefore it is often difficult to generally observe and objectively judge method performance. Westgard developed a normalized Sigma Method Evaluation Decision (MEDx) chart that converts all the sigma metrics into a simple visual graph with the %CV along the X-axis and %Bias along the Y-axis (Figures 1,2). Superimposed on this graph are the sigma metrics zones depicting sigma performance levels like Six Sigma zone (World class quality) followed by a Five Sigma zone (Excellent), Four Sigma zone (Good), Three Sigma zone (Acceptable), Two Sigma zone (Poor), and then below the Two Sigma (Unacceptable). A higher sigma value for an analyte means fewer defects are being generated. As method’s performance decreases the sigma value also decreases implying more defects, adding more noise to the patient’s signal i.e. patient result, and thus confusing the clinicians instead of helping them to rule in or rule out a diagnosis [4]. In clinical laboratories an arbitrary value of <3 sigma is considered unacceptable.

The Quality Goal Index (QGI) is a part of Root Cause Analysis (RCA) to find the reason for a lower sigma level for an analyte i.e. if the problem is due to imprecision or inaccuracy or both [7] (Table 2). The target QGI score is 0.0.

The aim of this study is to evaluate the performance of our laboratory by quantifying errors in the analytical phase of laboratory on a sigma scale by calculating the sigma metrics/values for routine chemistry analytes. We use autoanalyzer Cobas 6000-c501-e601 (Hitachi/Roche, Germany) for routine chemistry. The calculated sigma metrics were plotted on the sigma MEDx chart. The idea is to upscale the laboratory performance to provide accurate and reliable patient test results to the customers’ satisfaction [4,8,9].

This prospective study was carried out over a period of 6 months from Jan to June 2022 in the Department of Laboratory and Blood Bank of an accredited tertiary care hospital.

The study was conducted to evaluate the sigma metrics performance of 25 routine chemistry analytes including serum Albumin (ALB), Alkaline phosphatase (ALP), Alanine aminotransferase (ALT), Amylase (AMY), Aspartate aminotransferase (AST), Direct bilirubin (D BILI), Total bilirubin (T BILI), Calcium (Ca), Chloride (Cl), Total cholesterol( T CHOL), Creatine Kinase (CK), Creatinine (CREAT), Gamma Glutamyl Transferase (GGT), Glucose (GLU), HDL cholesterol (HDL-C), Iron (Fe), Lactate Dehydrogenase (LDH), Magnesium (Mg), Phosphorus (PO4), Total protein (TP), Sodium (Na), Potassium (K), Triglyceride (TG), Urea (UREA) and Uric acid (UA).

For each analyte % Bias and %CV were calculated and using %TEa extracted from reliable resources sigma metrics for each parameter was calculated and plotted on MEDx charts. The minimal acceptable performance criteria was considered as 3 sigma. Quality Goal Index (QGI) analysis was carried out as a part of Root Cause Analysis (RCA).

Internal Quality Control (IQC)

There are three working shifts per day in the laboratory. The bulk of the work load is handled in the morning shift. As per policy of the laboratory after daily equipment maintenance, calibration of any analyte if indicated is performed and then two levels of QC material are run i.e. normal L1 and pathological L2 from Roche Germany, daily in the morning shift and on ‘as and when required’ basis during evening and the night shifts. This QC material, from Roche Germany, is in lyophilized form and is reconstituted by adding 5 ml of distilled water. The analyzer software generates an L-J chart for the two levels of controls. At the time of study original Westgard rules applied for the interpretation of QC results included 13s, 22s, R4s, 41s and 10x as rejection and 12s as warning sign for the current QC run. Although in manual applications, 12S rule should be used as a warning to trigger application of the other rules, however the computer applications of Westgard Sigma Rules exclude the 12S as a warning rule to minimize waste and reduce the cost.

Calculation of %CV

%CV value for each analyte was calculated using daily IQC results of L1 and L2 recorded in an excel sheet for the period of study.

Calculation of %Bias

Monthly %Bias for an analyte is estimated by the RIQAS-EQA program based on the difference between the monthly EQA result obtained by the laboratory’s method and the peer group mean for comparison calculated from results of all the participating laboratories from all over the world. RIQAS-EQA program reports this monthly %Bias as %Deviation (%Dev). Average of these monthly %Deviations for a period of 6 months was taken to estimate Average %Bias.

Avg. %Bias = (%Dev. Jan+……+ %Dev. Jun)/6 Eq. 1

Calculation of Sigma metrics: Sigma metrics for each analyte was calculated using the following equation.

Sigma metric= (%TEa – %Bias)/%CV ………..……….Eq. 2

Bias may be positive or negative. However while subtracting it from TEa, + or – sign of %Bias should be ignored.

Calculation of QGI: QGI for an analyte is calculated as follows:

QGI= %Bias/(1.5 x %CV)……..………….……………………………….….......Eq. 3

Total Allowable Error Values: %TEa values for each analyte were extracted from sources like CLIA, CFX and BV criteria for acceptable performance.

Plotting of Normalized Method Evaluation Decision Charts (MEDx Charts): Normalized sigma MEDx Charts were plotted for L1 and L2 for visual assessment of an analyte’s method performance on sigma scale. For this purpose normalized %CV and normalized %Bias were calculated as percentage of the %TEa as follows.

Normalized %CV for normalized MEDx Chart= (%CV /%TEa) *100….………………. Eq. 4

Normalized %Bias for normalized MEDx Chart= (%Bias /%TEa) *100……………..…. Eq. 5

Data Analysis: Outliers were excluded and data was analysed to calculate %CVs, %Bias, Sigma metrics and QGIs using formulae mentioned above.

Using %TEa, %CVs and %Bias, the sigma metrics for 25 analytes at two levels of IQC i.e. L1 and L2 were calculated (Table 3).

Normalized %CV and Normalized %Bias (Equations 4,5) were calculated for each analyte for the two levels L1 and L2 (Table 4) and then plotted on normalized Method Evaluation Decision (MEDx) chart for a visual assessment (Figures 1,2).

Figure 1: Normalized MEDx Chart for Level 1.

Figure 2: Normalized MEDx Chart for Level 2.

A number of analytes including ALP, amylase, AST, CK, GGT, HDL-C, magnesium and uric acid attained sigma metrics of >6 at both L1 and L2. ALT and triglycerides showed sigma levels of 5-6 at L1 and L2. Cholesterol, glucose and calcium showed sigma metrics of 4-5 at L1 and L2. Albumin, direct bilirubin, total bilirubin, LDH, total protein and urea showed sigma metrics of 3-4 at both L1 and L2.

Electrolytes sodium, potassium and chloride each showed sigma level of less than 3 (Figures 1,2 & table 5).

QGIs for the analytes showing sigma metrics below 3 were calculated for the presence of imprecision, inaccuracy or both (Table 6).

We performed sigma metric analysis of 25 analytes. It was noted that the highest sigma metrics of 16.7 was attained by HDL-C whereas chloride showed the lowest sigma level of 2.08.

A number of analytes evaluated showed a world class performance with sigma levels of >6 like ALP, Amylase, AST, CK, GGT, HDL-C, Magnesium, and Uric Acid at both IQC levels, L1 and L2. At any given point of time the sigma metrics obtained is called short term sigma. A high short term sigma metrics is very comforting and assuring for the stability of the long term performance of these analytes.

Considering the practical challenges, it has been agreed that one may expect, in the long run, a shift of the mean value by 1.5σ on either side of the mean and that will still be considered as an acceptable long term performance/capability for an analyte with a sigma metrics of >6 [10]. Thus in order to attain a long term process performance of 4.5σ, we need to ensure a short term capability of 6σ (6σ- 1.5σ = 4.5σ). Many of our analytes showed sigma levels of >5. The expected long term performance of the laboratory at 4.5σ will be rated as good, with confidence, by a laboratorian/ healthcare provider for the satisfactory patient management.

It was noted that analytes like enzymes ALP, amylase, CK and AST with high TEa values of 30, 30, 30 and 20 respectively showed high sigma levels of 10.42, 14.4, 13.4 and 8.0 at L1 and 11.99, 12.7, 14.4 and 7.83 at L2 respectively. Liberal TEa values, however, lead to the possibility of missing out errors while stringent values give rise to false rejections [11].

The performance depicted by calculated sigma metrics may not solely reflect the analyzer’s performance as some random error is always added during the preparation of lyophilized controls by the laboratory staff. From the sigma metrics equation (Equation 2), it is apparent that the presence of any bias will always shrink the allowable error [4]. This is evident in case of albumin and total protein with low TEa values of 10 each and bias values of 2.48 and 4.65 respectively resulting in low sigma scores. Therefore the contribution to the total error in the form of bias should be minimized by a better maintenance and better calibration of the analyzer. Thus if the bias is minimum then a large portion of the allowable error would be available to accommodate the random error. This will allow six standard deviations of the process to be contained/accommodated within the tolerance specifications and the goal of less than 3.4 defects per million results will become achievable.

Like many laboratories the main problem appeared to be with the electrolytes Na, K and Cl which showed sigma levels of 2.31, 2.40 and 2.08 respectively at L1 and sigma levels of 2.33, 2.60 and 2.11 respectively at L2. Different sources, like CLIA, CAP, CFX, BV, RilliBAK from Germany, RCPA Australia, Ricos from Spain and Turkish Medical Authorities quote different TEa values. In order to establish TEa values, usually clinicians are surveyed to determine their expectations of analytical quality required for confident management of their patients using standard diagnostic methods. The goal is to reach a clinical consensus that would reduce the likelihood of alpha error (False positive) or beta error (False negative) as a result of analytical error. There is no standardization nor any harmonization of the existing resources. These, however, still represent a vast majority of goals that are implemented and used throughout the world. Owing to the low biological variation of sodium, potassium and chloride (within-subject: 0.6, 4.6, 1.2 and between-subject: 0.7, 5.6, 1.5 respectively) the TEa limits have been set very low. A few authorities have proposed to raise the allowable limits to make it possible to strive for some reasonable and achievable goals. Turkish medical authorities have proposed to set TEa values at 9% for Na, K and Chloride each [3]. Even with these TEa values it is difficult to achieve high sigma levels without regular maintenance of the equipment, high quality type I water, well trained laboratory staff and a very tight IQC.

By using Turkish TEa values our sigma metrics for Na, K and Cl improved considerably and sigma levels rose beyond 3 at both L1 and L2 (Table 7).

To construct Levey-Jennings control charts (LJ charts) to apply Westgard rules for a new lot of IQC material, laboratories usually use the mean, SD/CV values quoted by the manufacturer till sufficient number of readings is available to calculate in-house mean, SD and CV. These manufacturer’s control ranges are usually very wide and, if used, will lead to less frequent alerts and less rejections of the daily QC runs based on Westgard rules. This will take away the drive for a tighter control. This may result in most of the daily IQC results to be acceptable even with large day to day random variations. So daily IQC results for an analyte, if approved based on these control limits will end up in a large %CV for an analyte of interest. Therefore the analytes with high %CVs are likely to fall below 3 sigma (Equation 2). It is therefore recommended that, when using a new lot of quality control material, a laboratory should preferably calculate its own mean, and SD/CV values according to revised guidelines of Clinical and Laboratory Standards Institute (CLSI) 2016 to construct L-J charts for daily IQC monitoring [12].

Although minimal acceptable sigma level for manufacturing industries is 3 but it may be different for the clinical chemistry laboratory [13]. It is to be noted, however, that a normalized method decision (MEDx) chart used for analytical methods in a clinical laboratory, rates a sigma metrics of less than 3 as a ‘poor’ performance but as ‘unacceptable’ only when the sigma metrics is less than 2.

Sigma metrics is a useful tool for all parts of the QC design process. It allows laboratory to easily visualize performance on MEDx charts, choose suitable Westgard rules, the level of QC and the number of QC runs as per sigma performance as well as schedule the frequency of running these controls. Prior to this study same set of Westgard rules were being applied for all analytes. However, after this study we have a different attitude towards those analytes which have sigma levels of >6. We are now a little lenient with analytes like ALP, AST, amylase, CK, GGT, HDL-C, Mg and UA and these are now judged by simple QC rule with low false rejection rate like 13S at N = 2 and R = 1 where N denotes number of controls in a run and R denotes number of runs in a day. For other analytes with smaller sigma levels the QC becomes more difficult and requires Continued Professional Development (CPD) of staff to improve their analytical skills, use of Westgard multirules and more control measurements per run. The control rules can be chosen according to the observed quality and can be customized to match each test. Thus tests with 5 sigma require three rules 13S, 22S and R4S at N = 2 and R = 1 and with 4 sigma requires 13S, 22S, R4S, and 41S at N = 2 and R = 2. Sigma metrics <4 requires Westgard multirules procedure that includes 13S, 22S and R4S, 41S and 8X at N = 2 and R = 4. A stable performance may require IQC to be run once a day like in the morning shift. However you may require IQC to be run twice or three times a day. As sigma metrics is helpful in deciding the number of runs of QC per day so after this study we have redesigned the QC policy for electrolytes and now we run QC for electrolytes Na, K and Cl in every shift and on ‘as and when required’ basis for a stricter control. For low sigma metrics (<3σ) reducing both bias and CV is the key to improve the quality [14].

Sigma metrics is affected by imprecision and inaccuracy. QGI hinted at the type of error; random or systematic. The quality goal indices (QGIs) for the electrolytes Na, K and Cl are below 0.800 indicating a component of random error leading to imprecise results. It was noted that although average bias for each electrolyte remained low but because of relatively large CVs the overall sigma performance was found unsatisfactory.

It is important to perform RCA for the analytes with sigma levels <3 as these are expected to produce 6.7% clinically unacceptable results and are therefore generally considered unacceptable for routine operations [15]. The Cause-and-effect diagram (fishbone diagram) was used, every time, to identify the potential causes of a specific event (the effect) leading to inaccuracy (bias) or imprecision. Random error leading to imprecision resulting in high CVs may be due to improper thawing of the IQC material, inadequately mixed IQC material after thawing, fluctuating incubation/reaction chamber temperature, a small clot in the sample pipettor or sample probe and of course operator to operator variation in following steps of the procedure. It was noted that the daily IQC in the morning shift was being performed, each day, by different lab staff as this lab also serves as a teaching lab for the trainee lab technicians of the local university. This could be one of the reasons of random error as shown by the QGI index. Section supervisor was instructed to monitor the junior staff closely during morning shift IQC procedures.

Inaccuracy (Bias) can be reduced by ensuring integrity of cold chain during transportation, lot to lot comparison, regular maintenance of the equipment (daily, weekly and monthly), replacement of parts of autoanalyzer like halogen lamp, ensuring the quality of feed water by timely replacing the filters of RO water units and recalibration of the instrument before its use. Micropipettes used for reconstitution of lyophilized calibrators and control materials were recalibrated by a service provider. Although laboratory staff can exercise their control on analytical phase only, however a thorough RCA of the total testing process is necessary to detect errors using IQC and EQA, minimize variability through appropriate corrective and preventive actions, and generate accurate laboratory reports to help physicians make appropriate clinical decisions.

Sigma metrics allow us to assess performance and find solutions. Evaluation of analytical performance of our laboratory showed an acceptable performance for a wide range of analytes in the patient samples.

A more elaborate study over a period of one year including other analytes like hormones, tumor markers, drugs and special chemistry tests should be carried out to fully understand and assess the performance of the chemistry section. It is also suggested that the sigma metrics analysis should be applied to other departments of the laboratory and all the key performance indices (KPIs) should be evaluated on sigma scale in all phases of the laboratory process.

We are thankful to our Laboratory Director Mr Ahmed Alsharan for his support in making this study possible.

Statement of Ethics

This study protocol was reviewed and approved by the Research Ethics Committee Qurayyat Health Affairs Registered with NCBE Reg No: H-13-S-071, approval No. 098.

Conflict of interest statement

The authors have no conflicts of interest to declare.

Funding sources

This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.

Authors contributions

Tariq Mahmood Awan, Azza Abdelbaki Albayoumi, Muhammad Haseeb Malik, Faisal Khulaif Al-Ruwaili, Rawan Abdelrahman, Ahlam Zohair Al-Johani

Data availability statement

The data that support the findings of this study are available with the corresponding author.

1. Nevalainen D, Berte L, Kraft C, Leigh E, Picaso L, Morgan T. Evaluating laboratory performance on quality indicators with the six sigma scale. Arch Pathol Lab Med. 2000 Apr;124(4):516-9. doi: 10.5858/2000-124-0516-ELPOQI. PMID: 10747306.
2. Harry MJ. Six Sigma: a breakthrough strategy for profitability. Quality Progress. 1998; 31(5):60–64.
3. Keleş M. Evaluation of the clinical chemistry tests analytical performance with Sigma Metric by using different quality specifications - Comparison of analyser actual performance with manufacturer data. Biochem Med (Zagreb). 2022 Feb 15;32(1):010703. doi: 10.11613/BM.2022.010703. Epub 2021 Dec 15. PMID: 34955671; PMCID: PMC8672391.
4. Westgard S, Bayat H, Westgard JO. Analytical Sigma metrics: A review of Six Sigma implementation tools for medical laboratories. Biochem Med (Zagreb). 2018 Jun 15;28(2):020502. doi: 10.11613/BM.2018.020502. PMID: 30022879; PMCID: PMC6039161.
5. CLIA proficiency testing criteria for acceptable analytical performance: Federal Register 1992 Feb 28;57(40):7002-186.
6. Tietz–Westgard JO and Klee GG. Quality Management. In: Burtis CA, Ashwood ER, Bruns DE, eds. Tietz Textbook of Clinical Chemistry and Molecular Diagnostics. 4th ed. Philadelphia: Elsevier Saunders, 2006:488-9
7. Parry DM. Quality Goal Index.
8. Westgard JO, Westgard SA. Quality control review: implementing a scientifically based quality control system. Ann Clin Biochem. 2016 Jan;53(Pt 1):32-50. doi: 10.1177/0004563215597248. Epub 2015 Jul 6. PMID: 26150675.
9. Westgard JO, Westgard SA. Establishing Evidence-Based Statistical Quality Control Practices. Am J Clin Pathol. 2019 Mar 1;151(4):364-370. doi: 10.1093/ajcp/aqy158. PMID: 30517600.
10. Westgard JO. Statistical quality control procedures. Clin Lab Med 2013;33(1):111-24
11. Sharma LK, Datta RR, Sharma N. Sigma metrics evaluation of drugs in a clinical laboratory: Importance of choosing appropriate total allowable error and a troubleshooting roadmap. J Lab Physicians. 2021 Mar;13(1):44-49. doi: 10.1055/s-0041-1726572. Epub 2021 May 21. PMID: 34103878; PMCID: PMC8159660.
12. C24 A4: Statistical quality control for quantitative measurement procedures: Principles and Definitions, 4th Edition. CLSI., Wayne, PA, 2016.
13. Aggarwal K, et al. Application of six sigma metrics and method decision charts in improvising clinical Chemistry laboratory performance enhancement. Int J Adv Med. 2019 Oct;6(5):1524-1530.
14. Cooper G, DeJonge N, Ehrmeyer S, Yundt-Pacheco J, Jansen R, Ricós C, Plebani M. Collective opinion paper on findings of the 2010 convocation of experts on laboratory quality. Clin Chem Lab Med. 2011 May;49(5):793-802. doi: 10.1515/CCLM.2011.149. Epub 2011 Mar 3. PMID: 21366504.
15. Shah S, Saini R, Singh SB, Aggarwal O, Goel AK. Six Sigma Metrics and Quality Control in clinical laboratory.  Int J Med Res Rev. 2014;2(2):140-149.