Quick™ Ultra Low II Test Kit: Verification Report

August 2003 Environmental Technology Verification Report INDUSTRIAL TEST SYSTEMS, INC. QUICK™ ULTRA LOW II TEST KIT Prepared by Battelle Under a cooperative agreement with U.S. Environmental Protection Agency August 2003 Environmental Technology Verification Report ETV Advanced Monitoring Systems Center Industrial Test Systems, Inc. Quick™ Ultra Low II Test Kit by Tim Kaufman Patty White Amy Dindal Zachary Willenberg Karen Riggs Battelle Columbus, Ohio 43201 ii Notice The U.S. Environmental Protection Agency (EPA), through its Office of Research and Development, has financially supported and collaborated in the extramural program described here. This document has been peer reviewed by the Agency and recommended for public release. Mention of trade names or commercial products does not constitute endorsement or recommendation by the EPA for use. iii Foreword The U.S. Environmental Protection Agency (EPA) is charged by Congress with protecting the nation’s air, water, and land resources. Under a mandate of national environmental laws, the Agency strives to formulate and implement actions leading to a compatible balance between human activities and the ability of natural systems to support and nurture life. To meet this mandate, the EPA’s Office of Research and Development provides data and science support that can be used to solve environmental problems and to build the scientific knowledge base needed to manage our ecological resources wisely, to understand how pollutants affect our health, and to prevent or reduce environmental risks. The Environmental Technology Verification (ETV) Program has been established by the EPA to verify the performance characteristics of innovative environmental technology across all media and to report this objective information to permitters, buyers, and users of the technology, thus substantially accelerating the entrance of new environmental technologies into the marketplace. Verification organizations oversee and report verification activities based on testing and quality assurance protocols developed with input from major stakeholders and customer groups associated with the technology area. ETV consists of seven environmental technology centers. Information about each of these centers can be found on the Internet at http://www.epa.gov/etv/. Effective verifications of monitoring technologies are needed to assess environmental quality and to supply cost and performance data to select the most appropriate technology for that assessment. In 1997, through a competitive cooperative agreement, Battelle was awarded EPA funding and support to plan, coordinate, and conduct such verification tests for “Advanced Monitoring Systems for Air, Water, and Soil” and report the results to the community at large. Information concerning this specific environmental technology area can be found on the Internet at http://www.epa.gov/etv/centers/center1.html. iv Acknowledgments The authors wish to acknowledge the support of all those who helped plan and conduct the verification test, analyze the data, and prepare this report. In particular we would like to thank Rosanna Buhl, Adam Abbgy, and Bea Weaver of Battelle, and Mike Madigan and Rick Linde of the Ayer, Massachusetts, Department of Public Works Water Division. We also acknowledge the assistance of Jeff Adams of EPA and AMS Center stakeholders Vito Minei and Marty Link, who reviewed the verification reports. v Contents Page Notice ........................................................................................................................................... ii Foreword ..................................................................................................................................... iii Acknowledgments ....................................................................................................................... iv List of Abbreviations ................................................................................................................. viii 1 Background............................................................................................................................. 1 2 Technology Description.......................................................................................................... 2 3 Test Design and Procedures.................................................................................................... 4 3.1 Introduction ................................................................................................................... 4 3.2 Test Design .................................................................................................................... 4 3.3 Test Samples.................................................................................................................. 5 3.3.1 QC Samples ....................................................................................................... 5 3.3.2 PT Samples ........................................................................................................ 7 3.3.3 Environmental Samples...................................................................................... 7 3.4 Reference Analysis ........................................................................................................ 8 3.5 Verification Schedule..................................................................................................... 8 4 Quality Assurance/Quality Control....................................................................................... 10 4.1 Laboratory QC for Reference Method ......................................................................... 10 4.2 Audits .......................................................................................................................... 13 4.2.1 Performance Evaluation Audit ......................................................................... 13 4.2.2 Technical Systems Audit.................................................................................. 13 4.2.3 Data Quality Audit ........................................................................................... 13 4.3 QA/QC Reporting ........................................................................................................ 14 4.4 Data Review................................................................................................................. 14 5 Statistical Methods................................................................................................................ 16 5.1 Accuracy...................................................................................................................... 16 5.2 Precision ...................................................................................................................... 17 5.3 Linearity....................................................................................................................... 17 5.4 Method Detection Limit............................................................................................... 17 5.5 Matrix Interference Effects .......................................................................................... 18 5.6 Operator Bias ............................................................................................................... 18 5.7 Inter-Unit Reproducibility............................................................................................ 18 5.8 Rate of False Positives/False Negatives ....................................................................... 18 vi 6 Test Results .......................................................................................................................... 19 6.1 QC Samples ................................................................................................................. 19 6.2 PT and Environmental Samples ................................................................................... 23 6.2.1 Accuracy .......................................................................................................... 23 6.2.2 Precision........................................................................................................... 25 6.2.3 Linearity........................................................................................................... 31 6.2.4 Method Detection Limit ................................................................................... 34 6.2.5 Matrix Interference Effects .............................................................................. 34 6.2.6 Operator Bias ................................................................................................... 35 6.2.7 Inter-Unit Reproducibility................................................................................ 35 6.2.8 Rate of False Positives/False Negatives ........................................................... 37 6.3 Other Factors ............................................................................................................... 37 6.3.1 Ease of Use ...................................................................................................... 37 6.3.2 Analysis Time .................................................................................................. 41 6.3.3 Reliability......................................................................................................... 41 6.3.4 Waste Material ................................................................................................. 41 6.3.5 Cost .................................................................................................................. 41 7 Performance Summary ......................................................................................................... 42 8 References ............................................................................................................................ 45 Figures Figure 2-1. Industrial Test Systems, Inc., Quick™ Ultra Low II Test Kit .................................. 2 Figure 2-2. Quick™ Ultra Low II Color Chart ........................................................................... 3 Figure 6-1. Linearity of Quick™ Ultra Low II Color Chart Results ......................................... 33 Figure 6-2. Linearity of Quick™ Ultra Low II Quick™ Arsenic Scan Results ........................ 33 Figure 6-3. Linearity of Quick™ Ultra Low II Compu-Scan Results ....................................... 34 Figure 6-4. Comparison of Quick™ Ultra Low II Test Results for Technical and Non-Technical Operators................................................................ 36 Figure 6-5. Comparison of Quick™ Ultra Low II Test Results for Quick™ Arsenic Scan and Compu-Scan Units.................................................................................. 36 vii Tables Table 3-1. Test Samples for Verification of the Quick™ Ultra Low II Test Kit ........................... 6 Table 3-2. Schedule of Verification Test Days ............................................................................. 9 Table 4-1. Reference Method QCS Analysis Results.................................................................. 11 Table 4-2. Reference Method LFM Sample Results ................................................................... 12 Table 4-3. Reference Method Duplicate Analysis Results .......................................................... 12 Table 4-4. Reference Method PE Audit Results ......................................................................... 13 Table 4-5. Summary of Data Recording Process ........................................................................ 15 Table 6-1a. RB Sample Results for the Technical Operator........................................................ 20 Table 6-1b. RB Sample Results for the Non-Technical Operator ............................................... 20 Table 6-2a. QCS Results for the Technical Operator.................................................................. 21 Table 6-2b. QCS Results for the Non-Technical Operator.......................................................... 22 Table 6-3a. LFM Sample Results for the Technical Operator..................................................... 24 Table 6-3b. LFM Sample Results for the Non-Technical Operator ............................................ 25 Table 6-4. Quick™ Ultra Low II Test Kit and Reference Sample Results.................................. 26 Table 6-5. Quantitative Evaluation of Accuracy for Quick™ Ultra Low II Test Kits................. 29 Table 6-6. Qualitative Evaluation of Agreement for Quick™ Ultra Low II Test Kits ................ 30 Table 6-7. Precision Results for Quick™ Ultra Low II Test Kits ............................................... 32 Table 6-8. Detection Limit Results for Quick™ Ultra Low II Test Kit....................................... 35 Table 6-9. Rate of False Positives for Quick™ Ultra Low II Test Kits....................................... 38 Table 6-10. Rate of False Negatives for Quick™ Ultra Low II Test Kits ................................... 40 Table 7-1. Summary of Linear Regression Equations for Test Kit and Reference Results ......... 43 viii List of Abbreviations AMS Advanced Monitoring Systems EPA U.S. Environmental Protection Agency ETV Environmental Technology Verification HDPE high-density polyethylene ICPMS inductively coupled plasma mass spectrometry LFM laboratory-fortified matrix MDL method detection limit MSDS Material Safety Data Sheets NIST National Institute of Standards and Technology ppb parts per billion ppm parts per million PE performance evaluation PT performance test QA quality assurance QA/QC quality assurance/quality control QC quality control QCS quality control standard QMP Quality Management Plan RB reagent blank RPD relative percent difference RSD relative standard deviation TSA technical systems audit 1 Chapter 1 Background The U.S. Environmental Protection Agency (EPA) supports the Environmental Technology Verification (ETV) Program to facilitate the deployment of innovative environmental tech-nologies through performance verification and dissemination of information. The goal of the ETV Program is to further environmental protection by accelerating the acceptance and use of improved and cost-effective technologies. ETV seeks to achieve this goal by providing high-quality, peer-reviewed data on technology performance to those involved in the design, distribution, financing, permitting, purchase, and use of environmental technologies. ETV works in partnership with recognized testing organizations; with stakeholder groups consisting of buyers, vendor organizations, and permitters; and with the full participation of individual technology developers. The program evaluates the performance of innovative technologies by developing test plans that are responsive to the needs of stakeholders, conducting field or laboratory tests (as appropriate), collecting and analyzing data, and preparing peer-reviewed reports. All evaluations are conducted in accordance with rigorous quality assurance (QA) protocols to ensure that data of known and adequate quality are generated and that the results are defensible. The EPA’s National Exposure Research Laboratory and its verification organization partner, Battelle, operate the Advanced Monitoring Systems (AMS) Center under ETV. The AMS Center recently evaluated the performance of the Industrial Test Systems, Inc., Quick™ Ultra Low II test kit for measuring arsenic in water. 2 Chapter 2 Technology Description The objective of the ETV AMS Center is to verify the performance characteristics of environmental monitoring technologies for air, water, and soil. This verification report provides results for the verification testing of the Quick™ Ultra Low II test kit for arsenic in water. Following is a description of the test kit, based on information provided by the vendor. The information provided below was not verified in this test. The test kit components are shown in Figure 2-1. To perform arsenic analyses with the Quick™ Ultra Low II test kit, the water sample to be tested is mixed in the supplied reaction vessel with reagent #1 (tartaric acid with rate enhancers) to acidify the water sample. Reagent #2, an oxidizer (potassium peroxymonosulfate), is added to remove hydrogen sulfide interference. The test tolerates up to 2 parts per million (ppm) hydrogen sulfide without interference. Zinc powder, reagent #3, is added to reduce inorganic arsenic compounds (As+3 and As+5) to arsine gas. As arsine gas is generated and comes in contact with the test strip, the mercuric bromide indicator on the test strip changes color from white to shades of yellow or brown. Material Safety Data Sheets (MSDSs) for all reagents and test strips are provided with each test kit. The MSDSs include information on how to safely handle the reagents and test strips, including instructions for exposure controls and personal protection. Once the reaction is completed, the test strip is removed and visually compared to a color chart to obtain a semi-quantitative measure of the arsenic concentration in the tested sample. The color chart consists of a series of color blocks (Figure 2-2). The color blocks correspond to concentrations ranging from 0.4 parts per billion (ppb) to >25 ppb. If the color on the test strip is between two color blocks, then the operator may estimate the concentration as between the two values associated with the color blocks on either side. Figure 2-1. Industrial Test Systems, Inc., Quick™ Ultra Low II Test Kit 3 The test strip may also be read with the Quick™ Arsenic Scan hand-held instrument, which operates on the same principle as a colorimeter and provides a quantitative result. The Quick™ Arsenic Scan is calibrated weekly using a card provided by the manufacturer. Quantitative results may also be obtained from the test strip with a portable Compu-Scan scanner and laptop computer system. The scanned test strip image is converted to an arsenic concentration using the Home Port Computer System Arsenic Program Revision 5b software program. The scanner is calibrated by the manufacturer. The Quick™ Arsenic Scan and Compu-Scan are not provided with the Quick™ Ultra Low II test kit as a standard feature. The standard test kit with the color chart was the subject of the verification test; however, results for the Quick™ Arsenic Scan and Compu-Scan are also provided. The optimal detection range for the Quick™ Ultra Low II test kit is below 4 ppb. Dilution instructions are provided for samples with arsenic levels above 4 ppb. The recommended temperature range for sample analysis is 24°C to 30°C. A modified testing protocol that specifies longer reaction times (up to 30 minutes longer for samples between 5°-15°C) is available for sample temperatures below this range. The Quick™ Ultra Low II test kit is available in sets of 50 tests. The typical shelf life of the kits is 24 months. Figure 2-2. Quick™ Ultra Low II Color Chart 4 Chapter 3 Test Design and Procedures 3.1 Introduction This verification test was conducted according to procedures specified in the Test/QA Plan for Verification of Portable Analyzers.(1) The verification was based on comparing the arsenic results from the Quick™ Ultra Low II test kit to those from a laboratory-based reference method. The reference method for arsenic analysis was inductively coupled plasma mass spectrometry (ICPMS) performed according to EPA Method 200.8(2) The Quick™ Ultra Low II test kit relies on comparisons to a color chart provided with the test kit to allow semi-quantitative measurements of arsenic concentrations. Quantitative results were also obtained from a Quick™ Arsenic Scan instrument and Compu-Scan system. The test kit performance was verified by analyzing laboratory-prepared performance test samples, treated and untreated drinking water, and fresh surface water. All samples were tested using both the test kit and the reference method. Both semi-quantitative and quantitative analyses were performed by the technical and non-technical operators. The test design and procedures are described below. 3.2 Test Design The Quick™ Ultra Low II test kit was verified by evaluating the following parameters: ? Accuracy ? Precision ? Linearity ? Method detection limit (MDL) ? Matrix interference effects ? Operator bias ? Inter-unit reproducibility ? Rate of false positives/false negatives. All sample preparation and analyses were performed according to the manufacturer’s recommended procedures. All samples were warmed to 24°C prior to analysis using a hot water bath, which is at the lower end of the optimal temperature range listed in the test kit instructions. Color chart, Quick™ Arsenic Scan and Compu-Scan results were recorded manually. The results from the Quick™ Ultra Low II test kits were compared to those from the reference method to assess accuracy and linearity. Multiple aliquots of performance test samples, drinking water 5 samples, and surface water samples were analyzed to assess precision. Multiple aliquots of a low-level performance test sample were analyzed to assess the detection limit of the method. Potential matrix interference effects were assessed by challenging the test kit with performance test samples of known arsenic concentrations that contained both low levels and high levels of interfering substances. Identical sets of samples were analyzed independently by a technical and a non-technical operator. The technical operator was a technician at Battelle with three years of field and laboratory experience and a B.A. degree. The non-technical operator was a part-time temporary helper enrolled in undergraduate studies. Because the reagents of the Quick™ Ultra Low II test kits were consumed in use, it was not feasible for the two operators to use the same kits; however, each operator used multiple kits in order to analyze all the samples and it is assumed that kit-to-kit variability was similar for both operators. Results of all analyses were statistically compared to evaluate operator bias. The technical operator analyzed all samples using two different Quick™ Arsenic Scan units and two different Compu-Scan units to assess inter-unit reproducibility. The rate of false positive and false negative results were evaluated relative to the 10-ppb maximum contaminant level for arsenic in drinking water.(4) Other factors that were qualitatively assessed during the test included time required for sample analysis, ease of use, and reliability. 3.3 Test Samples Three types of samples were analyzed in the verification test, as shown in Table 3-1: quality control (QC) samples, performance test (PT) samples, and environmental water samples. The QC and PT samples were prepared from National Institute of Standards and Technology (NIST) traceable standards purchased from a commercial supplier and subject only to dilution as appropriate. Under the Safe Drinking Water Act, the EPA lowered the maximum contaminant level for arsenic from 50 ppb to 10 ppb, effective January 2001; public water systems must comply with this standard by June 2006.(4) Therefore, the QC sample concentrations targeted the 10 ppb arsenic level. The PT samples ranged from 10% to 1,000% of the 10 ppb level (i.e., from 1 ppb to 100 ppb). The environmental water samples were collected from various drinking water and surface freshwater sources. Each sample was assigned a unique sample identification number when prepared in the laboratory or collected in the field. The PT and environmental samples were submitted blind to the technical and non-technical operators and were analyzed randomly to the degree possible. 3.3.1 QC Samples QC samples included laboratory reagent blank (RB) samples, quality control samples (QCS), and laboratory-fortified matrix (LFM) samples (Table 3-1). The RB samples consisted of the same ASTM Type I water used to prepare all other samples and were subjected to the same handling 6 Table 3-1. Test Samples for Verification of the Quick™ Ultra Low II Test Kit Type of Sample Sample Characteristics Arsenic Concentration (a) No. of Replicates Reagent Blank (RB) ~ 0 ppb 10% of all Quality Control Sample (QCS) 10 ppb 10% of all Quality Control Laboratory Fortified Matrix (LFM) 10 ppb above native level 1 per site Prepared arsenic solution 1 ppb 4 Prepared arsenic solution 3 ppb 4 Prepared arsenic solution 10 ppb 4 Prepared arsenic solution 30 ppb 4 Prepared arsenic solution 100 ppb 4 Prepared arsenic solution for detection limit determination 3 ppb 7 Prepared arsenic solution spiked with low levels of interfering substances 10 ppb 4 Performance Test Prepared arsenic solution spiked spiked with high levels of interfering substances 10 ppb 4 Battelle drinking water <0.5 ppb 4 Ayer untreated water 64.8 ppb 4 Ayer treated water 1.39 ppb 4 Falmouth Pond water <0.5 ppb 4 Environmental Taunton River water 1.31 ppb 4 (a) Performance Test sample concentrations are target levels; environmental sample concentrations are actual (average of four replicate measurements). and analysis procedures as the other samples. The RB samples were used to verify that no arsenic contamination was introduced during sample handling and analysis. RB samples were analyzed at a frequency of 10%. The QCS consisted of Milli-Q water spiked in the lab to a concentration of 10 ppb arsenic with a NIST-traceable standard. QCS were used as calibration checks to verify that the Quick™ Ultra Low II test kit was operating properly. QCS were analyzed at the beginning and end of each testing period, as well as after every tenth sample. Because the test kit utilized a color chart that could not be calibrated, no performance criteria were specified for the QCS. 7 The LFM samples consisted of aliquots of environmental samples that were spiked in the field to increase the arsenic concentration by 10 ppb. The spike solution used for the LFM samples was prepared in the laboratory and brought to the field site. One LFM sample was prepared from each environmental sample. 3.3.2 PT Samples Three types of PT samples used in this verification test (Table 3-1): spiked samples ranging from 1 ppb to 100 ppb arsenic, a low-level spiked sample for evaluation of the test kit’s detection limit, and matrix interference samples that were spiked with potential interfering substances. All PT samples were prepared in the laboratory using Milli-Q water and NIST-traceable standards. Five PT samples containing arsenic at concentrations from 1 ppb to 100 ppb were prepared to evaluate Quick™ Ultra Low II test kit accuracy and linearity. Four aliquots of each of these samples were analyzed to assess precision. To determine the detection limit of the Quick™ Ultra Low II test kit, a PT sample was prepared with an arsenic concentration approximately five times the manufacturer’s estimated detection level. Seven non-consecutive replicates of this 3 ppb arsenic sample were analyzed to provide precision data with which to estimate the method detection limit (MDL). The matrix interference samples were spiked with 10 ppb arsenic as well as potentially interfering substances commonly found in natural water samples. One sample contained low levels of interfering substances that consisted of 1 ppm iron, 3 ppm sodium chloride, and 0.1 ppm sulfide. The second sample contained high levels of interfering compounds at the following concentrations: 10 ppm iron, 30 ppm sodium chloride, and 1.0 ppm sulfide. Four replicates of each of these samples were analyzed. 3.3.3 Environmental Samples The environmental samples listed in Table 3-1 included three drinking water samples and two surface water samples. All environmental samples were collected in 20-L high density polyethylene (HDPE) carboys. The Battelle drinking water sample was collected directly from a tap without purging. Untreated and treated groundwater samples from the Ayer, Massachusetts Department of Public Works Water Treatment Plant were collected directly from spigots, also without purging. Four aliquots of each sample were analyzed using the Quick™ Ultra Low II test kit in the Battelle laboratory as soon as possible after collection. One aliquot of each sample was preserved with nitric acid and submitted to the reference laboratory for reference analysis. One surface water sample was collected from a pond in Falmouth, Massachusetts and another was collected from the Taunton River near Bridgewater, Massachusetts. These samples were collected near the shoreline by submerging a 2-L HDPE sample container no more than one inch below the surface of the water, and decanting the water into a 20-L HDPE carboy until full. Each water body was sampled at one accessible location. These samples could not be analyzed at the field location as planned because of persistent, severe winter weather conditions. Therefore, the samples were returned to a storage shed at the Battelle laboratory, which was heated but not 8 serviced by running water. The storage shed was intended to simulate realistic field conditions under which the test kits might be used. Four aliquots of each surface water sample were analyzed in the storage shed as soon as possible after collection. One aliquot of each sample was preserved with nitric acid and submitted to the reference laboratory for reference analysis. 3.4 Reference Analysis The reference arsenic analyses were performed in a Battelle laboratory using a Perkin Elmer Sciex Elan 6000 ICPMS according to EPA Method 200.8, Revision 5.5.(2) The sample was introduced through a peristaltic pump by pneumatic nebulization into a radiofrequency plasma where energy transfer processes caused desolvation, atomization, and ionization. The ions were extracted from the plasma through a pumped vacuum interface and separated on the basis of their mass-to-charge ratio by a quadrupole mass spectrometer. The ions transmitted through the quadrupole were registered by a continuous dynode electron multiplier, and the ion information was processed by a data handling system. The ICPMS was tuned, optimized, and calibrated daily. The calibration was performed using a minimum of five calibration standards at concentrations ranging between 0.5 and 250 ppb, and a required correlation coefficient of a minimum of 0.999. Internal standards were used to correct for instrument drift and physical interferences. These standards were introduced in line through the peristaltic pump and analyzed with all blanks, standards, and samples. 3.5 Verification Schedule The verification test took place from January 29 through February 24, 2003. Table 3-2 shows the daily activities that were conducted during this period. The reference analyses were performed on March 7 and March 14, 2003, five to six weeks after sample collection. 9 Table 3-2. Schedule of Verification Test Days Sample Analysis Date Sample Collection Date Tech. Op. Non-tech. Op. Testing Location Activity 1/29/03-2/10/03 1/29/03-2/10/03 1/29/03-2/10/03 Battelle Laboratory Preparation and analysis of PT and associated QC samples. 2/12/03 2/14/03 2/14/03 Battelle Laboratory Collection and analysis of Ayer untreated and treated water and associated QC samples. 2/17/03 2/18/03 2/17/03 Battelle Laboratory Collection and analysis of Battelle drinking water and associated QC samples. 2/21/03 2/21/03 2/21/03 Battelle Storage Shed Collection and analysis of Falmouth Pond water and associated QC samples. 2/23/03 2/24/03 2/24/03 Battelle Storage Shed Collection and analysis of Taunton River water and associated QC samples. 10 Chapter 4 Quality Assurance/Quality Control Quality assurance/quality control (QA/QC) procedures were performed in accordance with the quality management plan (QMP) for the AMS Center(3) and the test/QA plan for this verification test.(1) QA/QC procedures and results are described below. 4.1 Laboratory QC for Reference Method Reference analyses were conducted on March 7 and March 14, 2003. Laboratory QC for the reference method included the analysis of RB, QCS, LFM, and analytical duplicate samples. Laboratory RB samples were analyzed to ensure that no contamination was introduced by the sample preparation and analysis process. The test/QA plan stated that if arsenic was detected in a RB sample above the MDL for the reference instrument, then the contamination source would be identified and removed and proper blank readings achieved before proceeding with the reference analyses. All of the laboratory RB samples analyzed were below the reporting limit for arsenic (i.e., below the concentration of the lowest calibration standard) except for several blanks that were analyzed at the end of the day on March 7. Three of the six test samples that were associated with these RB samples were re-analyzed on March 14, with acceptable RB sample results. The other three test samples had arsenic concentrations that were approximately twenty times higher than the RB sample concentrations; therefore, no action was taken. On March 7, the instrument used for the reference method was calibrated using nine calibration standards, with concentrations ranging from 0.5 to 250 ppb arsenic. On March 14, it was calibrated using eight standards ranging in concentration from 0.1 to 25 ppb arsenic for more accurate analysis of low level samples. The accuracy of the calibration was verified after the analysis of every ten samples by analyzing a QCS of a known concentration. The percent recovery of the QCS was calculated from the following equation: 100??sCR s where Cs is the measured concentration of the QCS and s is the spike concentration. If the QCS analysis differed by more than 10% from the true value of the standard, the instrument was recalibrated before continuing the test. As shown in Table 4-1, all QCS analyses were within the required range. (1) 11 Table 4-1. Reference Method QCS Analysis Results Sample ID Analysis Date Measured (ppb) Actual (ppb) Percent Recovery CCV 25 3/7/2003 24.96 25.00 100% QCS 25 3/7/2003 26.81 25.00 107% CCV 25 3/7/2003 24.50 25.00 98% CCV 25 3/7/2003 25.39 25.00 102% CCV 25 3/7/2003 25.73 25.00 103% CCV 25 3/7/2003 25.81 25.00 103% CCV 25 3/7/2003 25.64 25.00 103% CCV 25 3/7/2003 25.30 25.00 101% CCV 25 3/7/2003 24.90 25.00 100% CCV 25 3/7/2003 22.67 25.00 91% QCS 25 3/14/2003 24.90 25.00 100% CCV 2.5 3/14/2003 2.74 2.50 110% QCS 2.5 3/14/2003 2.70 2.50 108% CCV 2.5 3/14/2003 2.58 2.50 103% CCV 2.5 3/14/2003 2.65 2.50 106% CCV 2.5 3/14/2003 2.66 2.50 106% CCV 2.5 3/14/2003 2.61 2.50 104% CCV 2.5 3/14/2003 2.60 2.50 104% LFM samples were analyzed to assess whether matrix effects influenced the reference method results. The LFM percent recovery (R) was calculated from the following equation: 100???sCCR s where Cs is the measured concentration of the spiked sample, C is the measured concentration of the unspiked sample, and s is the spike concentration. If the percent recovery of an LFM sample fell outside the range from 85 to 115%, a matrix effect was suspected. As shown in Table 4-2, all of the LFM sample results were within this range. Duplicate samples were analyzed to assess the precision of the reference analysis. The relative percent difference (RPD) of the duplicate sample analysis was calculated from the following equation: 1002/)()(????DDCCCCRPD Where C is the concentration of the sample analysis, and CD is the concentration of the duplicate sample analysis. If the RPD was greater than 10%, the instrument was recalibrated before (3) (2) 12 Table 4-2. Reference Method LFM Sample Results Sample ID Matrix Analysis Date Unspiked (ppb) Spiked (ppb) Amount Spiked (ppb) Percent Recovery CAA-22 ASTM Type I water 3/7/2003 11.02 37.20 25.00 105% CAA-25 R4 ASTM Type I water 3/7/2003 0.95 22.76 25.00 87% CAA-28 R2 ASTM Type I water 3/7/2003 3.45 30.64 25.00 109% CAA-29 R4 ASTM Type I water 3/7/2003 34.98 60.37 25.00 102% CAA-37 R4 Drinking water 3/7/2003 0.52 28.20 25.00 111% CAA-41 R4 Drinking water 3/7/2003 1.24 28.88 25.00 111% CAA-48 Surface water 3/7/2003 12.26 39.40 25.00 109% CAA-47 R4 Surface water 3/7/2003 1.07 28.41 25.00 109% CAA-27 R1 ASTM Type I water 3/14/2003 2.56 4.73 2.50 87% CAA-37 R3 Drinking water 3/14/2003 0.45 3.11 2.50 107% CAA-47 R1 Surface water 3/14/2003 1.36 4.16 2.50 112% CAA-88 R3 Drinking water 3/14/2003 0.43 3.16 2.50 109% CAA-88 R4 Drinking water 3/14/2003 0.42 3.18 2.50 111% Table 4-3. Reference Method Duplicate Analysis Results Sample ID Analysis Date Sample Concentration (ppb) Duplicate Concentration (ppb) Relative Percent Difference CAA-4 3/7/2003 9.33 9.20 1.4% CAA-70 3/7/2003 10.93 10.82 1.0% CAA-26 R1 3/7/2003 1.14 1.13 1.4% CAA-28 R3 3/7/2003 3.49 3.45 1.1% CAA-31 R1 3/7/2003 111.89 112.20 0.3% CAA-38 3/7/2003 11.96 11.90 0.5% CAA-42 3/7/2003 13.02 13.06 0.3% CAA-48 3/7/2003 12.26 12.22 0.4% CAA-23 3/14/2003 3.03 2.99 1.3% CAA-27 R2 3/14/2003 2.64 2.61 0.9% CAA-37 R4 3/14/2003 0.44 0.43 2.3% CAA-47 R2 3/14/2003 1.31 1.32 0.2% CAA-88 R4 3/14/2003 0.42 0.38 9.5% 13 continuing the test. As shown in Table 4-3, the RPDs for the duplicate analysis were all less than 10%. The RPD for one duplicate pair was 9.5%; however, the reported concentrations were below the reporting limit for the reference method (i.e., below the concentration of the lowest calibration standard). 4.2 Audits Three types of audits were performed during the verification test: a performance evaluation (PE) audit of the reference method, a technical systems audit of the verification test performance, and a data quality audit. Audit procedures are described further below. 4.2.1 Performance Evaluation Audit A PE audit was conducted to assess the quality of the reference measurements made in this verification test. For the PE audit, an independent, NIST-traceable, reference material was obtained from a different commercial supplier than the calibration standards and the standard used to prepare the PT and QCS samples. Accuracy of the reference method was verified by comparing the arsenic concentration measured using the calibration standards to those obtained using the independently-certified PE standard. Relative percent difference as calculated by Equation 3 was used to quantify the accuracy of the results. Agreement of the standard within 10% was required for the measurements to be considered acceptable. As shown in Table 4-4, the PE sample analysis was within the required range. Table 4-4. Reference Method PE Audit Results Sample ID Date of Analysis Measured Arsenic Concentration (ppb) Actual Arsenic Concentration (ppb) Percent Difference PE-1 3/24/03 9.63 10.0 4 4.2.2 Technical Systems Audit An independent Battelle Quality staff conducted a technical systems audit (TSA) on February 6 to ensure that the verification test was being conducted in accordance with the test/QA plan(1) and the AMS Center QMP.(3) A TSA of the reference method performance was conducted by the Battelle Quality Manager on March 5, 2003, when the reference analyses were initiated. As part of the TSA, test procedures were compared to those specified in the test/QA plan, data acquisition and handling procedures were reviewed, and the reference standards and method were reviewed. Observations and findings from the TSA were documented and submitted to the Battelle Verification Test Coordinator for response. None of the findings of the TSA required corrective action. TSA records are permanently stored with the Battelle Quality Manager. 14 4.2.3 Data Quality Audit At least 10% of the data acquired during the verification test was audited. The Battelle Quality Manager traced the data from the initial acquisition, through reduction and statistical analysis, to final reporting to ensure the integrity of the reported results. All calculations performed on the data undergoing the audit were checked. 4.3 QA/QC Reporting Each audit was documented in accordance with Sections 3.3.4 and 3.3.5 of the QMP for the ETV AMS Center.(3) Once the audit reports were prepared, the Battelle Verification Test Coordinator ensured that a response was provided for each adverse finding or potential problem and imple-mented any necessary follow-up corrective action. The Battelle Quality Manager ensured that follow-up corrective action was taken. The results of the TSA and the data quality audit were submitted to the EPA. 4.4 Data Review Records generated in the verification test received a one-over-one review before these records were used to calculate, evaluate, or report verification results. Table 4-5 summarizes the types of data recorded and reviewed. All data were recorded by Battelle staff. Data were reviewed by a Battelle technical staff member involved in the verification test, but not the staff member that originally generated the record. The person performing the review added his/her initials and the date to a hard copy of the record being reviewed. Review of some of the test data sheets occurred outside of the two week period specified in the test/QA plan. 15 Table 4-5. Summary of Data Recording Process Data to be Recorded Where Recorded How Often Recorded Disposition of Data(a) Dates, times of test events ETV field data sheets Start/end of test event Used to organize/check test results; manually incorporated in data spreadsheets as necessary Test parameters (temperature, analyte/ interferant identities, and all Quick™ Ultra Low II test kit results for color chart, Quick™ Arsenic Scan and Compu-Scan ETV field data sheets When set or changed, or as needed to document test Used to organize/check test results, manually incorporated in data spreadsheets as necessary Reference method sample analysis, chain of custody, and results Laboratory record books, data sheets, or data acquisition system, as appropriate Throughout sample handling and analysis process Transferred to spreadsheets (a) All activities subsequent to data recording were carried out by Battelle. 16 Chapter 5 Statistical Methods The statistical methods used to evaluate the performance factors listed in Section 3.2 are presented in this chapter. Qualitative observations were also used to evaluate verification test data. 5.1 Accuracy All samples were analyzed by both the Quick™ Ultra Low II test kit and reference methods. For each sample, accuracy was expressed in terms of a relative bias (B) as calculated from the following equation: 100xCdBR? where d is the average difference between the reading from the Quick™ Ultra Low II test kit and those from the reference method, and RC is the average of the reference measurements. An additional assessment of accuracy was conducted for the color chart results because of the semi-quantitative nature of the visual comparisons. Each color in the chart represents a concentration range. Performance was assessed by determining whether the result falls within the expected concentration range as measured by the reference analysis. Overall agreement was assessed by calculating the percent of results that fell within the correct range, calculated from the following equation: 100??nYA where A is the percent of measurements in agreement, Y is the number of measurements within the expected color range, and n is the total number of measurements. Readings below the vendor-stated detection limit of the test kit (i.e., <0.4 ppb) were judged to be in agreement with the reference result if the reference value was in the specified “less than” range. (4) (5) 17 5.2 Precision When possible, the standard deviation (S) of the results for the replicate samples was calculated and used as a measure of Quick™ Ultra Low II test kit precision at each concentration. Standard deviation was calculated from the following equation: 2112)(11????????? ??nkk CCnS where n is the number of replicate samples, Ck is the concentration measured for the kth sample, and C is the average concentration of the replicate samples. Precision was reported in terms of the relative standard deviation (RSD) as follows: 100??CSRSD 5.3 Linearity Linearity was assessed by performing a linear regression of Quick™ Ultra Low II test kit results against the reference results, with linearity characterized by the slope, intercept, and correlation coefficient (R). Linearity was tested using the five PT samples over the range 1 to 100 ppb arsenic and the detection limit study sample. Samples with results below the vendor-stated test kit detection limit were not included. Color chart results, Quick™ Arsenic Scan and Compu-Scan results were plotted against the corresponding reference concentrations and separate regressions were performed. 5.4 Method Detection Limit The MDL for the Quick™ Ultra Low II test kit was assessed using results from seven replicate analyses of a sample spiked with approximately 3 ppb arsenic. The standard deviation of the seven replicate samples was calculated using Equation (6). The MDL was calculated using the following equation: StMDL ?? where t is the Student’s t value for a 99% confidence level and S is the standard deviation of the seven replicate samples. (6) (7) (8) 18 5.5 Matrix Interference Effects The potential effect of interfering substances on the sensitivity of the Quick™ Ultra Low II test kit was evaluated by the calculating accuracy (expressed as bias) using Equation 4. These results were qualitatively compared with accuracy results for PT samples containing only arsenic to assess whether there was a positive or negative effect due to matrix interferences. 5.6 Operator Bias Potential operator bias for the Quick™ Ultra Low II test kit was assessed by performing a linear regression of sample results above the detection limit generated by the technical and non-technical operator. Color chart, Quick™ Arsenic Scan, and Compu-Scan results were evaluated. The slope, intercept, and correlation coefficient were used to evaluate the degree of operator bias. A paired t-test was also conducted to evaluate whether the two sets of sample results were significantly different at a 95% confidence level. 5.7 Inter-Unit Reproducibility Inter-unit reproducibility for the Quick™ Arsenic Scan and the Compu-Scan devices was assessed by performing a linear regression of sample results generated by the two units used by the technical operator. The slope, intercept, and correlation coefficient were used to evaluate the degree of inter-unit reproducibility. A paired t-test was also conducted to evaluated whether the two sets of sample results were significantly different at a 95% confidence level. 5.8 Rate of False Positives/False Negatives The rates of false positives and false negatives produced by the Quick™ II test kit were assessed relative to the 10-ppb target arsenic level. A false positive result is defined as any result reported to be greater than the guidance level (10 ppb) and greater than 125% of the reference value, when the reference value is less than or equal to the guidance level. Similarly, a false negative result is defined as any result reported below the guidance level and less than 75% of the reference value, when the reference value is equal to or greater than the guidance level. The rates of false positives and false negatives were expressed as a percentage of total samples analyzed for each type of sample. 19 Chapter 6 Test Results The results of the verification test of the Quick™ Ultra Low II test kits are presented in this section. 6.1 QC Samples As described in Section 3.3.1, the QC samples analyzed with the Quick™ Ultra Low II test kit included RB, QCS, and LFM samples (these QC samples were different than those analyzed in conjunction with the reference method). The RB samples were analyzed at a frequency of 10% and results were used to verify that no arsenic contamination was introduced during sample handling and analysis. QCS were analyzed at the beginning and end of each test period, and after every tenth sample. The QCS results were used to verify that the test kit was operating properly. One LFM sample was prepared from each environmental sample to evaluate potential matrix interferences. Acceptance criteria for test kit QC samples were not specified in the test/QA plan because modifications to the technology would not be made during testing. RB sample results for the technical and non-technical operators are presented in Tables 6-1a and 6-1b, respectively. Unique sample identification codes were assigned to each container of ASTM Type I water that was used. The RB samples were analyzed at the required frequency. The technical and non-technical operators recorded all RB sample results as below the detection limit for the color chart and the Quick™ Arsenic Scan. The Compu-Scan units always returned a detected value for RB samples except for one sample analyzed by the non-technical operator. Because all color chart and Quick™ Arsenic Scan results for the RB samples were below detection, it appeared that arsenic contamination resulting from sample handling and analysis had not occurred. QCS results for the technical and non-technical operators are presented in Tables 6-2a and 6-2b, respectively. The QCS were analyzed at the required frequency except on the first day of testing, when the technical operator inadvertently omitted one of these samples. The percent recovery of the QCS was calculated using Equation 1 (Section 4.1). The QCS percent recovery for the technical operator ranged from 40% to 140% for the color chart, from 0% to 125% for the Quick™ Arsenic Scan, and from 15% to 265% for the Compu-Scan. The QCS percent recovery for the non-technical operator ranged from 22% to 140% for the color chart, from 10% to 103% for the Quick™ Arsenic Scan and from 11% to 175% for the Compu-Scan. On average, QCS recoveries for the color chart and Compu-Scan were within approximately 20% of the true value, 20 Table 6-1a. RB Sample Results for the Technical Operator Sample ID Replicate Analysis Date Color Chart (ppb) Quick™ Arsenic Scan #1 (ppb) Quick™ Arsenic Scan #2 (ppb) Compu-Scan #1 (ppb) Compu-Scan #2 (ppb) CAA-3 1 1/29/2003 <0.4 <0.2 <0.2 1.5 2 CAA-8 1 1/31/2003 <0.4 <0.2 <0.2 1.1 1 CAA-10 1 2/3/2003 <0.4 <0.2 <0.2 0.7 1.3 CAA-13 1 2/5/2003 <0.4 <0.2 <0.2 0.5 0.8 CAA-49 1 2/7/2003 <0.4 <0.2 <0.2 0.5 0.7 CAA-51 1 2/10/2003 <0.4 <0.2 <0.2 0.4 0.4 CAA-52 1 2/10/2003 <0.4 <0.2 <0.2 0.5 0.6 CAA-54 1 2/14/2003 <0.4 <0.2 <0.2 0.5 1 CAA-55 1 2/14/2003 <0.4 <0.2 <0.2 0.9 1.5 CAA-56 1 2/18/2003 <0.4 <0.2 <0.2 0.6 0.3 CAA-58 1 2/21/2003 <0.4 <0.2 <0.2 0.5 1 CAA-59 1 2/24/2003 <0.4 <0.2 <0.2 0.7 0.7 Table 6-1b. RB Sample Results for the Non-Technical Operator Sample ID Replicate Analysis Date Color Chart (ppb) Quick™ Arsenic Scan #1 (ppb) Compu-Scan #1 (ppb) CAA-5 1 1/30/2003 <0.4 <0.2 0.2 CAA-7 1 1/31/2003 <0.4 <0.2 0.5 CAA-11 1 2/4/2003 <0.4 <0.2 0.06 CAA-13 1 2/5/2003 <0.4 <0.2 0.4 CAA-49 1 2/7/2003 <0.4 <0.2 ND( CAA-51 1 2/10/2003 <0.4 <0.2 0.06 CAA-52 1 2/10/2003 <0.4 <0.2 0.6 CAA-54 1 2/14/2003 <0.4 <0.2 0.3 CAA-55 1 2/14/2003 <0.4 <0.2 0.5 CAA-56 1 2/17/2003 <0.4 <0.2 0.4 CAA-58 1 2/21/2003 <0.4 <0.2 0.6 CAA-59 1 2/24/2003 <0.4 <0.2 0.9 (a) Compu-Scan reading was 0.0. 21 Table 6-2a. QCS Results for the Technical Operator(a) Sample ID Replicate Analysis Date Color Chart (ppb) Quick™ Arsenic Scan #1 (ppb) Quick™ Arsenic Scan #2 (ppb) Compu-Scan #1 (ppb) Compu-Scan #2 (ppb) Amount Spiked (ppb) Percent Recovery Color Chart Percent Recovery Quick™ Arsenic Scan #1 Percent Recovery Quick™ Arsenic Scan #2 Percent Recovery Compu-Scan #1 Percent Recovery Compu-Scan #2 CAA-2 1 1/29/2003 10 (c) 11.5 (c) 11.5 (c) 9.5 (c) 14.5 (c) 10.0 100% 115% 115% 95% 145% CAA-15 1 1/31/2003 9 (c) 5.25 (c) 5.25 (c) 15 (c) 21 (c) 10.0 90% 53% 53% 150% 210% CAA-15 2 1/31/2003 11 (c) 9.9 (c) 10.75 (c) 11.5 (c) 13.5 (c) 10.0 110% 99% 108% 115% 135% CAA-16 1 2/3/2003 11 (c) 9.5 (c) 9.5 (c) 7.5 (c) 10 (c) 10.0 110% 95% 95% 75% 100% CAA-16 2 2/3/2003 10 (c) 9 (c) 8.5 (c) 13 (c) 19 (c) 10.0 100% 90% 85% 130% 190% CAA-17 1 2/5/2003 9 (c) 6.15 (c) 6.9 (c) 10.5 (c) 15 (c) 10.0 90% 62% 69% 105% 150% CAA-17 2 2/5/2003 14 (c) 12.5 (c) 11.5 (c) 19.5 (c) 26.5 (c) 10.0 140% 125% 115% 195% 265% CAA-17 3 2/7/2003 11 (c) 10.5 (c) 12 (c) 14.5 (c) 21 (c) 10.0 110% 105% 120% 145% 210% CAA-17 4 2/7/2003 7.5 (c) 5.25 (c) 5.25 (c) 5.5 (c) 10 (c) 10.0 75% 53% 53% 55% 100% CAA-19 1 2/10/2003 14 (c) 12 (c) 12 (c) 10 (c) 15 (c) 10.0 140% 120% 120% 100% 150% CAA-19 2 2/10/2003 10 (c) 4.9 (c) 5.75 (c) 3 (c) 5 (c) 10.0 100% 49% 58% 30% 50% CAA-20 1 2/14/2003 7.5 (c) 2.35 (c) 2.35 (c) 3 (c) 4.5 (c) 10.0 75% 24% 24% 30% 45% CAA-20 2 2/14/2003 4 (c) <0.2 (c) 1 (c) 3 (c) 3 (c) 10.0 40% 0% 10% 30% 30% CAA-21 1 2/18/2003 6 (b) 4.4 (b) 4.4 (b) 3.4 (b) 6 (b) 10.0 60% 44% 44% 34% 60% CAA-21 2 2/18/2003 4.4 (b) 2.2 (b) 2.2 (b) 3.8 (b) 6 (b) 10.0 44% 22% 22% 38% 60% CAA-71 1 2/21/2003 6 (c) 1 (c) 1 (c) 1.5 (c) 2 (c) 10.0 60% 10% 10% 15% 20% CAA-71 2 2/21/2003 5 (c) 2 (c) 1.5 (c) 3.5 (c) 3.5 (c) 10.0 50% 20% 15% 35% 35% CAA-72 1 2/24/2003 9 (c) 7.7 (c) 7.7 (c) 14.5 (c) 9.5 (c) 10.0 90% 77% 77% 145% 95% CAA-72 2 2/24/2003 14 (c) 11.5 (c) 10.5 (c) 13 (c) 12.5 (c) 10.0 140% 115% 105% 130% 125% (a) = The technical operator inadvertently omitted one of the QCS samples on the first day of testing (1/29/03); (b) = diluted 1:2; (c) = diluted 1:5. 22 Table 6-2b. QCS Results for the Non-Technical Operator Sample ID Replicate Analysis Date Color Chart (ppb) Quick™ Arsenic Scan #1 (ppb) Compu-Scan #1 (ppb) Amount Spiked (ppb) Percent Recovery Color Chart Percent Recovery Quick™ Arsenic Scan #1 Percent Recovery Compu-Scan #1 CAA-4 1 1/30/2003 11 8.8 8 10.0 110% 88% 80% CAA-4 2 1/30/2003 11 8.8 6.6 10.0 110% 88% 66% CAA-15 1 1/31/2003 6 5.2 5.5 10.0 60% 52% 55% CAA-15 2 1/31/2003 6 8.8 6 10.0 60% 88% 60% CAA-16 1 2/4/2003 6 5 7.2 10.0 60% 50% 72% CAA-16 2 2/4/2003 6 5.2 5.4 10.0 60% 52% 54% CAA-17 1 2/5/2003 9 7.2 10 10.0 90% 72% 100% CAA-17 2 2/5/2003 11 (a) 8.25 (a) 17.5 (a) 10.0 110% 83% 175% CAA-17 3 2/7/2003 9 (a) 4.6 (a) 3.5 (a) 10.0 90% 46% 35% CAA-17 4 2/7/2003 9 (a) 8.25 (a) 10.5 (a) 10.0 90% 83% 105% CAA-19 1 2/10/2003 6 6.3 6.3 10.0 60% 63% 63% CAA-19 2 2/10/2003 11 (a) 6.15 (a) 5.5 (a) 10.0 110% 62% 55% CAA-20 1 2/14/2003 9 6.9 8.2 10.0 90% 69% 82% CAA-20 2 2/14/2003 11 (a) 6.5 (a) 17 (a) 10.0 110% 65% 170% CAA-21 1 2/17/2003 2.2 1.23 1.1 10.0 22% 12% 11% CAA-21 2 2/17/2003 2.2 0.98 1.7 10.0 22% 10% 17% CAA-70 1 2/21/2003 9 (a) 6.5 (a) 5.5 (a) 10.0 90% 65% 55% CAA-70 2 2/21/2003 9 (a) 1 (a) 4 (a) 10.0 90% 10% 40% CAA-72 1 2/24/2003 11 (a) 6.5 (a) 10.5 (a) 10.0 110% 65% 105% CAA-72 2 2/24/2003 14 10.25 16.5 10.0 140% 103% 165% (a) = diluted 1:5. 23 indicating that these methods were operating as expected. The average recoveries measured by the Quick™ Arsenic Scan were more than 30% lower than the true value, which was lower than expected. The LFM sample results for the technical and non-technical operators are presented in Tables 6-3a and 6-3b. The percent recovery associated with each LFM sample was calculated using Equation 2 (Section 4.1). No evidence of matrix interferences is clearly indicated by these results, with the possible exception of the Falmouth Pond water LFM sample. The low recoveries measured by both operators for this sample indicate that a matrix interference may be affecting the recovery of arsenic. Consequently, test kit results for this sample may be biased low. 6.2 PT and Environmental Samples Table 6-4 presents the sample results for the PT and environmental samples. The table includes the Quick™ Ultra Low II test kit results and the reference method results. The Quick™ Ultra Low II test kit results are shown for both the technical and non-technical operators, the Quick™ Arsenic Scan Units #1 and #2, and the Compu-Scan Units #1 and #2. Some Quick™ Ultra Low II test kit results were below the detection limit and were assigned a value of <0.4 ppb for the color chart and <0.2 ppb for the Quick™ Arsenic Scan. The reporting limit for the reference analyses was 0.5 ppb, which corresponds to the lowest calibration standard used. Results for each performance factor are presented below. 6.2.1 Accuracy Table 6-5 presents the accuracy results for the Quick™ Ultra Low II test kit, expressed as percent bias as calculated by Equation 4 (Section 5.1). Percent bias was not calculated for results below the detection limit. The four replicate analyses for each sample were averaged in the calculation of bias. The relative bias for the color chart ranged from -78% to 18% for the tech-nical operator and -87% to 45% for the non-technical operator. The relative bias for the Quick™ Arsenic Scan ranged from -91% to 22% for the technical operator and -95% to 16% for the non-technical operator. The relative bias for the Compu-Scan ranged from -80% to 161% for the technical operator and -92% to 70% for the non-technical operator. The reference method results for the Falmouth Pond water sample were below the reporting limit; therefore, the apparent matrix effect observed in the Falmouth Pond water LFM sample could not be verified. Table 6-6 presents accuracy results for each PT and environmental replicate sample according to whether the color chart result agreed with the reference value for that sample. Each color block on the color chart represents a range of concentrations. The reference sample result was assigned to the correct corresponding color block. A test kit result was considered to agree with the reference method result if it fell within the range of plus or minus one color block (i.e., the concentration range spanning three adjacent color blocks). If the color chart test result for a given sample was within this range, then a “Y” was reported in Table 6-6. If the color chart result was 24 Table 6-3a. LFM Sample Results for the Technical Operator Description Unspiked(a) (ppb) Spiked (ppb) Amount spiked (ppb) Percent Recovery Battelle drinking water LFM Color Chart <0.4 6 10 60% Quick™ Arsenic Scan #1 <0.2 3.4 10 34% Quick™ Arsenic Scan #2 <0.2 3.4 10 34% Compu-Scan #1 0.4 5.6 10 52% Compu-Scan #2 0.45 4.6 10 42% Reference <0.5 11.96 10 120% Ayer untreated water LFM Color Chart 14 20 10 60% Quick™ Arsenic Scan #1 7 10 10 30% Quick™ Arsenic Scan #2 7 11.75 10 49% Compu-Scan #1 15 22.5 10 78% Compu-Scan #2 18 20 10 18% Reference 64.82 69.74 10 49% Ayer treated water LFM Color Chart <0.4 14 10 140% Quick™ Arsenic Scan #1 <0.2 10.25 10 103% Quick™ Arsenic Scan #2 <0.2 11 10 110% Compu-Scan #1 0.3 11.5 10 112% Compu-Scan #2 0.5 18.5 10 180% Reference 1.39 13.02 10 116% Falmouth Pond water LFM Color Chart <0.4 3 10 30% Quick™ Arsenic Scan #1 <0.2 1 10 10% Quick™ Arsenic Scan #2 <0.2 1.5 10 15% Compu-Scan #1 0.6 3 10 24% Compu-Scan #2 0.6 3.5 10 29% Reference <0.5 11.50 10 115% Taunton River water LFM Color Chart <0.4 11 10 110% Quick™ Arsenic Scan #1 <0.2 8.5 10 85% Quick™ Arsenic Scan #2 <0.2 8.5 10 85% Compu-Scan #1 0.6 7.5 10 69% Compu-Scan #2 0.7 11.5 10 108% Reference 1.31 12.26 10 109% (a) Average of four replicates. Non-detects were assigned a value of zero. 25 Table 6-3b. LFM Sample Results for the Non-Technical Operator Description Unspiked(a) (ppb) Spiked (ppb) Amount spiked (ppb) Percent Recovery Battelle drinking water LFM Color Chart <0.4 2.2 10 22% Quick™ Arsenic Scan #1 <0.2 1.3 10 13% Compu-Scan #1 0.5 1.2 10 7% Reference <0.5 11.96 10 120% Ayer untreated water LFM Color Chart 14 30 10 161% Quick™ Arsenic Scan #1 1.6 <0.2 10 -16% Compu-Scan #1 8.2 25 10 168% Reference 64.82 69.74 10 49% Ayer treated water LFM Color Chart <0.4 2.2 10 22% Quick™ Arsenic Scan #1 <0.2 1.38 10 14% Compu-Scan #1 0.6 3.5 10 29% Reference 1.39 13.02 10 116% Falmouth Pond water LFM Color Chart <0.4 1.5 10 15% Quick™ Arsenic Scan #1 <0.2 0.58 10 6% Compu-Scan #1 0.6 0.9 10 3% Reference <0.5 11.50 10 115% Taunton River water LFM Color Chart <0.4 11 10 110% Quick™ Arsenic Scan #1 <0.2 6.5 10 65% Compu-Scan #1 1.0 15 10 140% Reference 1.31 12.26 10 109% (a) Average of four replicates. Non-detects were assigned a value of zero. outside this range, then an “N” was reported. Overall agreement was determined by calculating the total percent of results in agreement for the technical and non-technical operators. The total percent agreement using this method was 70% for the technical operator and 57% for the non-technical operator. 6.2.2 Precision Precision results for the Quick™ Ultra Low II test kit are presented in Table 6-7. The RSD was determined according to Equation 7 (Section 5.2). The RSD was not calculated if any of the results for a set of replicates were below the detection limit (i.e., <0.4 ppb for the color chart or <0.2 ppb for the Quick™ Arsenic Scan). For the technical operator, RSDs ranged from 0% to 55% for the color chart, 2% to 51% for the Quick™ Arsenic Scan, and 6% to 85% for the 26 Table 6-4. Quick™ Ultra Low II Test Kit and Reference Sample Results Description Sample ID Replicate Technical Operator Color Chart (ppb) Technical Operator Quick™ Arsenic Scan #1 (ppb) Technical Operator Quick™ Arsenic Scan #2 (ppb) Technical Operator Compu-Scan #1 (ppb) Technical Operator Compu-Scan #2 (ppb) Non-Technical Operator Color Chart (ppb) Non-Technical Operator Quick™ Arsenic Scan #1 (ppb) Non-Technical Operator Compu-Scan #1 (ppb) Reference (ppb) CAA-26 1 1.2 1.1 1.1 1.3 2.2 1.5 15 0.7 0.91 CAA-26 2 1.2 1.1 0.9 1.8 2.6 1.5 0.98 0.8 0.86 CAA-26 3 1.5 1.1 1 2.4 1.3 1.5 1.05 0.9 0.90 PT - 1 ppb As CAA-26 4 1.5 1.1 1.1 0.6 0.8 1.5 0.98 0.6 0.86 CAA-28 1 3 2.6 2.5 4.2 4.9 5 3.4 6.1 2.56 CAA-28 2 2.8 2.1 2.1 3.8 4.5 2.2 1.8 4.2 2.64 CAA-28 3 2.2 1.7 1.7 1.6 2.8 2.2 1.23 2.6 2.50 PT - 3 ppb As CAA-28 4 3 2.8 2.7 4.4 7 5 3.1 4.3 2.71 CAA-1 1 10 (b) 9 (b) 8.5 (b) 9.5 (b) 12.5 (b) 9 6.9 6.1 9.09 CAA-1 2 11 (b) 11.5 (b) 11 (b) 11 (b) 17 (b) 9 17.1 7.9 8.95 CAA-1 3 11 (b) 13 (b) 12 (b) 11.5 (b) 15 (b) 9 9.8 7.7 8.83 PT - 10 ppb As CAA-1 4 10 (b) 10.3 (b) 9.5 (b) 7 (b) 15 (b) 9 7.9 7 8.99 CAA-29 1 30 (b) 29.5 (b) 29.5 (b) 36.5 (b) 58 (b) 56 (d) 34 (d) 80 (d) 33.96 CAA-29 2 30 (b) 29.5 (b) 25 (b) 39 (b) 48 (b) 56 (d) 33 (d) 80 (d) 34.39 CAA-29 3 30 (b) 32.5 (b) 31.5 (b) 36 (b) 60 (b) 44 (d) 24.6 (d) 26 (d) 34.51 PT - 30 ppb As CAA-29 4 30 (b) 27.5 (b) 25 (b) 34 (b) 48.5 (b) 44 (d) 34 (d) 48 (d) 34.98 CAA-31 1 100 (d) 80 (d) 80 (d) 126 (d) 230 (d) 120 (d) 74 (d) 144 (d) 111.89 CAA-31 2 100 (d) 126 (d) 94 (d) 138 (d) 198 (d) 120 (d) 66 (d) 130 (d) 115.57 CAA-31 3 120 (d) 112 (d) 104 (d) 136 (d) 220 (d) 80 (d) 80 (d) 104 (d) 114.65 PT - 100 ppb As CAA-31 4 100 (d) 100 (d) 94 (d) 122 (d) 228 (d) 80 (d) 104 (d) 90 (d) 113.83 CAA-23 1 2.8 1.98 2.1 3 4 0.6 0.2 0.2 CAA-23 2 4 3.05 3.1 4.9 5.4 1 0.2 0.5 CAA-23 3 3 2.7 2.82 4.6 5.3 2.8 1.1 4.2 CAA-23 4 2.8 2.3 2.4 2.6 3.3 2.8 2.2 1.8 CAA-23 5 1.8 1.05 1.15 1.9 1.9 2.8 1.98 4.6 CAA-23 6 1.2 0.58 0.7 0.7 1.3 2.8 2.1 3.1 Detection limit CAA-23 7 3 2.7 2.6 3.6 4 2.8 1.9 1.9 3.03 27 Table 6-4. Quick™ Ultra Low II Test Kit and Reference Sample Results (continued) Description Sample ID Replicate Technical Operator Color Chart (ppb) Technical Operator Quick™ Arsenic Scan #1 (ppb) Technical Operator Quick™ Arsenic Scan #2 (ppb) Technical Operator Compu-Scan #1 (ppb) Technical Operator Compu-Scan #2 (ppb) Non-Technical Operator Color Chart (ppb) Non-Technical Operator Quick™ Arsenic Scan #1 (ppb) Non-Technical Operator Compu-Scan #1 (ppb) Reference (ppb) CAA-33 1 14 (b) 11.5 (b) 11 (b) 17 (b) 28.5 (b) 11 (b) 11.5 (b) 18.5 (b) CAA-33 2 11 (b) 11.5 (b) 11.5 (b) 17.5 (b) 29 (b) 11 (b) 6.9 (b) 15 (b) CAA-33 3 7.5 (b) 5.8 (b) 6.5 (b) 15 (b) 17 (b) 11 (b) 9 (b) 14 (b) PT - 10 ppb As + low level interferents CAA-33 4 11 (b) 12 (b) 12 (b) 19.5 (b) 29 (b) 11 (b) 5.5 (b) 3.5 (b) 9.90 CAA-35 1 14 (b) 10.3 (b) 10.5 (b) 19.5 (b) 28.5 (b) 5 (b) 1 (b) 3 (b) CAA-35 2 11 (b) 8.5 (b) 9.5 (b) 13.5 (b) 24.5 (b) 11 (b) 10.25 (b) 4.5 (b) CAA-35 3 12.5 (b) 9 (b) 10.3 (b) 19.5 (b) 27 (b) 11 (b) 6.9 (b) 6 (b) PT - 10 ppb As + high level interferents CAA-35 4 12 (b) 9.5 (b) 10.5 (b) 15.5 (b) 19.5 (b) 11 (b) 6.9 (b) 7 (b) 11.59 CAA-37 1 <0.4 <0.2 <0.2 0.0 0.3 <0.4 <0.2 0.5 <0.5 CAA-37 2 <0.4 <0.2 <0.2 0.5 0.5 <0.4 <0.2 0.5 <0.5 CAA-37 3 <0.4 <0.2 <0.2 0.5 0.6 <0.4 <0.2 0.5 <0.5 Battelle drinking water CAA-37 4 <0.4 <0.2 <0.2 0.5 0.4 <0.4 <0.2 0.4 <0.5 Battelle drinking water LFM CAA-38 1 6 (a) 3.4 (a) 3.4 (a) 5.6 (a) 4.6 (a) 2.2 1.3 1.2 11.96 CAA-39 1 20 (c) 11 (c) 10.5 (c) 20 (c) 19 (c) 12 (c) 3 (c) 1.7 (c) 65.61 CAA-39 2 12 (c) 4 (c) 4.7 (c) 5 (c) 8 (c) 12 (c) 3 (c) 5 (c) 62.73 CAA-39 3 4 (c) <0.2 (c) <0.2 (c) 5 (c) 6 (c) 30 (f) <0.2 f 25 f 67.47 Ayer untreated water CAA-39 4 20 (c) 13 (c) 12.3 (c) 29 (c) 40 (c) 1.8 0.2 1 63.48 Ayer untreated water LFM CAA-40 1 20 (e) 10 (e) 11.75 (e) 22.5 (e) 20 (e) 30 (f) <0.2 (f) 25 (f) 69.74 CAA-41 1 <0.4 <0.2 <0.2 0.1 0.5 <0.4 <0.2 0.5 1.36 CAA-41 2 <0.4 <0.2 <0.2 0.1 0.4 <0.4 <0.2 0.6 1.45 CAA-41 3 0.4 <0.2 <0.2 0.5 0.8 <0.4 <0.2 0.6 1.44 Ayer treated water CAA-41 4 <0.4 <0.2 <0.2 0.4 0.3 <0.4 <0.2 0.7 1.32 Ayer treated water LFM CAA-42 1 14 (b) 10.25 (b) 11 (b) 11.5 (b) 18.5 (b) 2.2 1.38 3.5 13.02 28 Table 6-4. Quick™ Ultra Low II Test Kit and Reference Sample Results (continued) Description Sample ID Replicate Technical Operator Color Chart (ppb) Technical Operator Quick™ Arsenic Scan #1 (ppb) Technical Operator Quick™ Arsenic Scan #2 (ppb) Technical Operator Compu-Scan #1 (ppb) Technical Operator Compu-Scan #2 (ppb) Non-Technical Operator Color Chart (ppb) Non-Technical Operator Quick™ Arsenic Scan #1 (ppb) Non-Technical Operator Compu-Scan #1 (ppb) Reference (ppb) CAA-43 1 <0.4 <0.2 <0.2 0.8 0.8 <0.4 <0.2 0.4 <0.5 CAA-43 2 <0.4 <0.2 <0.2 0.5 0.4 <0.4 <0.2 0.7 <0.5 CAA-43 3 <0.4 <0.2 <0.2 0.6 0.8 <0.4 <0.2 0.6 <0.5 Falmouth Pond water CAA-43 4 <0.4 <0.2 <0.2 0.5 0.5 <0.4 <0.2 0.7 <0.5 Falmouth Pond water LFM CAA-46 1 3 (b) 1 (b) 1.5 (b) 3 (b) 3.5 (b) 1.5 0.58 0.9 11.50 CAA-47 1 <0.4 <0.2 <0.2 0.7 0.7 <0.4 <0.2 1.1 1.36 CAA-47 2 <0.4 <0.2 <0.2 0.7 0.6 <0.4 <0.2 0.7 1.31 CAA-47 3 <0.4 <0.2 <0.2 0.5 0.7 <0.4 <0.2 1.6 1.31 Taunton River water CAA-47 4 <0.4 <0.2 <0.2 0.5 0.8 <0.4 <0.2 0.7 1.26 Taunton River water LFM CAA-48 1 11 (b) 8.5 (b) 8.5 (b) 7.5 (b) 11.5 (b) 11 (b) 6.5 (b) 15 (b) 12.26 (a) = 1:2 dilution; (b) = 1:5 dilution; (c) = 1:10 dilution; (d) = 1:20 dilution; (e) = 1:25 dilution; (f) = 1:50 dilution. 29 Table 6-5. Quantitative Evaluation of Accuracy for Quick™ Ultra Low II Test Kits Bias Description Technical Operator Color Chart Technical Operator Quick™ Arsenic Scan #1 Technical Operator Quick™ Arsenic Scan #2 Technical Operator Compu-Scan #1 Technical Operator Compu-Scan #2 Non-Technical Operator Color Chart Non-Technical Operator Quick™ Arsenic Scan #1 Non-Technical Operator Compu-Scan #1 Performance Test Samples 1 ppb As 18% -7% -12% 33% 51% 31% -11% -35% 3 ppb As -24% -36% -38% -3% 33% 0% -34% 19% 10 ppb As 17% 22% 14% 9% 66% 0% 16% -20% 30 ppb As -13% -14% -19% 6% 56% 45% -9% 70% 100 ppb As -8% -8% -18% 14% 92% -12% -29% 3% 10 ppb As + low level interferents 10% 3% 4% 74% 161% 11% -17% 29% 10 ppb As + high level interferents 7% -20% -12% 47% 115% -18% -46% -56% Environmental Samples Battelle drinking water NA NA NA NA NA NA NA NA Battelle drinking water LFM -50% -72% -72% -53% -62% -82% -89% -90% Ayer untreated water -78% NA NA -77% -72% -78% NA -87% Ayer untreated water LFM -71% -86% -83% -68% -71% -57% NA -64% Ayer treated water NA NA NA -80% -64% NA NA -57% Ayer treated water LFM 8% -21% -15% -12% 42% -83% -89% -73% Falmouth Pond water NA NA NA NA NA NA NA NA Falmouth Pond water LFM -74% -91% -87% -74% -70% -87% -95% -92% Taunton River water NA NA NA -54% -47% NA NA -22% Taunton River water LFM -10% -31% -31% -39% -6% -10% -47% 22% Percent bias calculated according to Equation 4, Section 5.1. NA: one or more replicates below detection limit. 30 Table 6-6. Qualitative Evaluation of Agreement for Quick™ Ultra Low II Test Kits Description Sample ID Replicate Within Range (Y/N) Technical Operator Color Chart Within Range (Y/N) Non-Technical Operator Color Chart Performance Test Samples CAA-26 1 Y Y CAA-26 2 Y Y CAA-26 3 Y Y 1 ppb As CAA-26 4 Y Y CAA-28 1 Y Y CAA-28 2 Y N CAA-28 3 N N 3 ppb As CAA-28 4 Y Y CAA-1 1 Y Y CAA-1 2 Y Y CAA-1 3 Y Y 10 ppb As CAA-1 4 Y Y CAA-29 1 Y N CAA-29 2 Y N CAA-29 3 Y Y 30 ppb As CAA-29 4 Y Y CAA-31 1 Y Y CAA-31 2 Y Y CAA-31 3 Y N 100 ppb As CAA-31 4 Y N CAA-33 1 Y Y CAA-33 2 Y Y CAA-33 3 Y Y 10 ppb As + low level interferents CAA-33 4 Y Y CAA-35 1 Y N CAA-35 2 Y Y CAA-35 3 Y Y 10 ppb As + high level interferents CAA-35 4 Y Y Environmental Samples CAA-37 1 Y Y CAA-37 2 Y Y CAA-37 3 Y Y Battelle drinking water CAA-37 4 Y Y Battelle drinking water LFM CAA-38 1 N N CAA-39 1 N N CAA-39 2 N N CAA-39 3 N N Ayer untreated water CAA-39 4 N N Ayer untreated water LFM CAA-40 1 N N 31 Table 6-6. Qualitative Evaluation of Agreement for Quick™ Ultra Low II Test Kits (continued) Description Sample ID Replicate Within Range (Y/N) Technical Operator Color Chart Within Range (Y/N) Non-Technical Operator Color chart CAA-41 1 N N CAA-41 2 N N CAA-41 3 N N Ayer treated water CAA-41 4 N N Ayer treated water LFM CAA-42 1 Y N CAA-43 1 Y Y CAA-43 2 Y Y CAA-43 3 Y Y Falmouth Pond water CAA-43 4 Y Y Falmouth Pond water LFM CAA-46 1 N N CAA-47 1 N N CAA-47 2 N N CAA-47 3 N N Taunton River water CAA-47 4 N N Taunton River water LFM CAA-48 1 Y Y Percent Agreement 70% 57% Compu-Scan. For the non-technical operator, RSDs ranged from 0% to 84% for the color chart, 4% to 78% for the Quick™ Arsenic Scan, and from 11% to 139% for the Compu-Scan. For the reference measurements, RSDs were a maximum of 4%. 6.2.3 Linearity The linearity of the Quick™ Ultra Low II test kit readings was assessed by performing a linear regression of the test kit results against the reference method results for the five PT samples ranging from 1 ppb to 100 ppb arsenic. In these regressions, results reported as below detection limits by the Quick™ Ultra Low II test kit were not used. Figures 6-1, 6-2 and 6-3 present the results of the linear regressions for the color chart, Quick™ Arsenic Scan and Compu-Scan results, respectively. The slope, intercept, and correlation coefficient for each regression equation are shown on the charts. For the color chart and Quick™ Arsenic Scan, the results for the technical operator were more linear and more closely corresponded to the reference method than the results for the non-technical operator. For the Compu-Scan, the results for the technical operator were more linear than those for the non-technical operator, and the results for Unit #1 for both operators corresponded more closely to reference than the results for Unit #2. 32 Table 6-7. Precision Results for Quick™ Ultra Low II Test Kits RSD Description Technical Operator Color Chart Technical Operator Quick™ Arsenic Scan #1 Technical Operator Quick™ Arsenic Scan #2 Technical Operator Compu-Scan #1 Technical Operator Compu-Scan #2 Non-Technical Operator Color Chart Non-Technical Operator Quick™ Arsenic Scan #1 Non-Technical Operator Compu-Scan #1 Reference Method Performance Test Samples 1 ppb As 13% 2% 8% 50% 48% 0% 4% 17% 3% 3 ppb As 14% 22% 20% 37% 36% 45% 43% 33% 4% 10 ppb As 5% 16% 15% 21% 12% 0% 44% 11% 1% 30 ppb As 0% 7% 12% 6% 12% 14% 15% 45% 1% 100 ppb As 10% 19% 11% 6% 7% 23% 20% 21% 1% Environmental Samples Battelle drinking water NA NA NA 50% 29% NA NA 11% NA Ayer untreated water 55% 51% 43% 80% 85% 84% 78% 139% 3% Ayer treated water NA NA NA 75% 43% NA NA 14% 4% Falmouth Pond water NA NA NA 24% 33% NA NA 24% NA Taunton River water NA NA NA 19% 12% NA NA 42% 3% NA: one or more replicates below detection limit. 33 y = 0.92x + 0.22R = 0.9948y = 0.87x + 4.45R = 0.94980204060801001201400 20 40 60 80 100 120 140Reference concentration (ppb)Color chart concentration (ppb)Technical OperatorNon-technical OperatorLinear (Technical Operator)Linear (Non-technical Operator)1:1 line Figure 6-1. Linearity of Quick™ Ultra Low II Color Chart Results y = 0.91x + 0.04R = 0.9830y = 0.81x + 0.55R = 0.9934y = 0.70x + 2.51R = 0.97000204060801001201400 20 40 60 80 100 120 140Reference concentration (ppb)Quick Arsenic Scan concentration (ppb)Technical Operator Unit #1Technical Operator Unit #2Non-techncial operator Unit #1Linear (Technical Operator Unit #1)Linear (Technical Operator Unit #2)Linear (Non-techncial operator Unit #1)1:1 line Figure 6-2. Linearity of Quick™ Ultra Low II Quick™ Arsenic Scan Results 34 y = 1.15x - 0.88R = 0.9980y = 1.93x - 3.82R = 0.9946y = 1.03x + 3.99R = 0.93220501001502002500 20 40 60 80 100 120 140Reference concentration (ppb)Compu-Scan concentration (ppb)Technical Operator Unit #1Technical Operator Unit #2Non-technical Operator Unit #1Linear (Technical Operator Unit #1)Linear (Technical Operator Unit #2)Linear (Non-technical Operator Unit #1)1:1 line Figure 6-3. Linearity of Quick™ Ultra Low II Compu-Scan Results 6.2.4 Method Detection Limit The MDL was assessed by analyzing seven replicates of a sample spiked at approximately 3 ppb arsenic. Table 6-8 provides the standard deviation for the seven replicate samples for the technical and non-technical operator on the color chart, Quick ™ Arsenic Scan and Compu-Scan results, and the calculated MDLs. 6.2.5 Matrix Interference Effects Matrix interference effects were assessed by comparing the calculated bias for the samples containing low-level and high-level concentrations of interfering substances with the bias reported for the other PT samples containing arsenic only (Table 6-5). The biases for the samples with low and high concentrations of interfering substances were similar to those for the PT samples that contained arsenic only, indicating no apparent effect due to the presence of the interferents. The biases associated with the Compu-Scan Unit #2 results for the low and high interferent samples were 161% and 115%, respectively; however, these high biases were not observed in the samples analyzed by the color chart, Quick™ Arsenic Scan, or Compu-Scan Unit #1 results. 35 Table 6-8. Detection Limit Results for Quick™ Ultra Low II Test Kit Sample ID Technical Operator Color Chart (ppb) Technical Operator Quick™ Arsenic Scan #1 (ppb) Technical Operator Quick™ Arsenic Scan #2 (ppb) Technical Operator Compu-Scan #1 (ppb) Technical Operator Compu-Scan #2 (ppb) Non-Technical Operator Color Chart (ppb) Non-Technical Operator Quick™ Arsenic Scan #1 (ppb) Non-Technical Operator Compu-Scan #1 (ppb) CAA-23 Rep 1 2.8 1.98 2.1 3 4 0.6 0.2 0.2 CAA-23 Rep 2 4 3.05 3.1 4.9 5.4 1 0.2 0.5 CAA-23 Rep 3 3 2.7 2.82 4.6 5.3 2.8 1.1 4.2 CAA-23 Rep 4 2.8 2.3 2.4 2.6 3.3 2.8 2.2 1.8 CAA-23 Rep 5 1.8 1.05 1.15 1.9 1.9 2.8 1.98 4.6 CAA-23 Rep 6 1.2 0.58 0.7 0.7 1.3 2.8 2.1 3.1 CAA-23 Rep 7 3 2.7 2.6 3.6 4 2.8 1.9 1.9 Standard Deviation 0.91 0.92 0.89 1.48 1.57 0.98 0.88 1.71 Method Detection Limit (ppb) 2.9 2.9 2.8 4.7 4.9 3.1 2.8 5.4 6.2.6 Operator Bias Operator bias was evaluated by comparing the color chart, Quick™ Arsenic Scan Unit #1, and Compu-Scan Unit #1 results above the detection limit for all PT and environmental samples produced by the technical and non-technical operators (the non-technical operator did not use the Quick™ Arsenic Scan Unit #2 or the Compu-Scan Unit #2). Linear regression results are shown in Figure 6-4. The regression results suggest that there is little difference in the color chart and Compu-Scan results for the technical and non-technical operator. The Quick™ Arsenic Scan results tended to be higher for the technical operator than for the non-technical operator. A paired t-test of each data set indicated that the results were not significantly different at a 5% signifi-cance level for the color chart and Compu-Scan, but they were significantly different for the Quick™ Arsenic Scan. 6.2.7 Inter-Unit Reproducibility Inter-unit reproducibility was evaluated by comparing the data for the two Quick™ Arsenic Scan units and the two Compu-Scan systems used by the technical operator. Only results above the detection limit were included in the analysis. Linear regressions of the two sets of data are shown in Figure 6-5. The results for the Quick™ Arsenic Scan closely corresponded, indicating that the performance of the two units was very similar. The Compu-Scan results indicated that Unit #2 tended to return higher readings than Unit #1. A paired t-test of the two sets of data indicated that the results were not significantly different at a 5% significance level for the Quick™ Arsenic Scan, but were significantly different for the Compu-Scan. 36 y = 0.95x + 1.73R = 0.9204y = 0.76x + 0.73R = 0.9435y = 0.94x + 2.22R = 0.93130204060801001201401600 20 40 60 80 100 120 140 160Technical OperatorNon-technical OperatorColor ChartQuick Arsenic Scan Unit #1Compu-Scan Unit #1Linear (Color Chart)Linear (Quick Arsenic Scan Unit #1)Linear (Compu-Scan Unit #1)1:1 line Figure 6-4. Comparison of Quick™ Ultra Low II Test Results for Technical and Non-Technical Operators y = 0.87x + 1.10R = 0.9918y = 1.66x - 1.99R = 0.99120501001502002500 20 40 60 80 100 120 140 160Unit #1Unit #2 Quick Arsenic Scan Unit #1Compu-Scan Unit #1Linear (Quick Arsenic Scan Unit #1)Linear (Compu-Scan Unit #1)1:1 line Figure 6-5. Comparison of Quick™ Ultra Low II Test Results for Quick™ Arsenic Scan and Compu-Scan Units 37 6.2.8 Rate of False Positives/False Negatives Tables 6-9 and 6-10 show the data and results for the rates of false positives and false negatives, respectively, obtained from the Quick™ Ultra Low II test kit. All PT and environmental samples were included in this evaluation. As shown in Table 6-9, 32 samples had an arsenic concentration below 10 ppb as measured by the reference analysis. For these samples, none of the test kit color chart results were >10 ppb and greater than 125% of the reference measurement for the technical and non-technical operators, yielding false positive rates of 0%. The rates of false positives for the Quick™ Arsenic Scan units were 6% and 3% for the technical operator (Units #1 and #2, respectively) and 3% for the non-technical operator (Unit #1). The rates of false positives for the Compu-Scan units were 16% and 25% for the technical operator (Units #1 and #2) and 9% for the non-technical operator (Unit #1). Twenty-one samples had arsenic concentrations above 10 ppb as measured by the reference analysis (Table 6-10). For these samples, the test kit color chart results were <10 ppb and less than 75% of the reference measurement for three samples for the technical operator and five samples for the non-technical operator, yielding false negative rates of 14% and 24%, respect-tively. The rates of false negatives for the Quick™ Arsenic Scan units were 29% and 24% for the technical operator (units #1 and #2) and 57% for the non-technical operator (unit #1). The rates of false negatives for the Compu-Scan units were 24% and 19% for the technical operator (units #1 and #2) and 48% for the non-technical operator (unit #1). 6.3 Other Factors During testing activities, the technical and non-technical operators were instructed to keep a record of their comments on ease of use, reliability, portability, and generation of waste materials. This section summarizes these observations and other comments pertaining to any problems encountered during testing. Cost information is also presented. 6.3.1 Ease of Use The technical and non-technical operator both reported that the Quick™ Ultra Low II test kit was very easy to use. The test kit instructions were clear and easy to follow. Although the manu-facturer provided instructions for diluting samples above the 4 ppb arsenic level, the non-technical operator sometimes had difficulty successfully performing dilutions and correctly converting the results to a final concentration. The three scoops used to sequentially add reagents were color coordinated, which facilitated the efficient operation of the test kit. The sample bottles were relatively easy to handle, although a relatively large sample volume was required for analysis (600 mL). Extra care had to be taken to ensure that the caps to the reaction vessels were completely dry before proceeding with further analyses. Dilution of samples with arsenic concentrations exceeding the optimal detection range may be a source of error and reduce the 38 Table 6-9. Rate of False Positives for Quick™ Ultra Low II Test Kits False Positive (Y/N) Description Sample ID Replicate Technical Operator Color Chart Technical Operator Quick™ Arsenic Scan #1 Technical Operator Quick™ Arsenic Scan #2 Technical Operator Compu-Scan #1 Technical Operator Compu-Scan #2 Non-Technical Operator Color Chart Non-Technical Operator Quick™ Arsenic Scan #1 Non-Technical Operator Compu-Scan #1 CAA-26 1 N N N N N N N N CAA-26 2 N N N N N N N N CAA-26 3 N N N N N N N N 1 ppb As CAA-26 4 N N N N N N N N CAA-28 1 N N N N N N N N CAA-28 2 N N N N N N N N CAA-28 3 N N N N N N N N 3 ppb As CAA-28 4 N N N N N N N N CAA-1 1 N N N N Y N N N CAA-1 2 N Y N N Y N Y N CAA-1 3 N Y Y Y Y N N N 10 ppb As CAA-1 4 N N N N Y N N N CAA-37 1 N N N N N N N N CAA-37 2 N N N N N N N N CAA-37 3 N N N N N N N N Battelle drinking water CAA-37 4 N N N N N N N N CAA-41 1 N N N N N N N N CAA-41 2 N N N N N N N N CAA-41 3 N N N N N N N N Ayer treated water CAA-41 4 N N N N N N N N CAA-43 1 N N N N N N N N CAA-43 2 N N N N N N N N CAA-43 3 N N N N N N N N Falmouth Pond water CAA-43 4 N N N N N N N N 39 Table 6-9. Rate of False Positives for Quick™ Ultra Low II Test Kits (continued) False Positive (Y/N) Description Sample ID Replicate Technical Operator Color Chart Technical Operator Quick™ Arsenic Scan #1 Technical Operator Quick™ Arsenic Scan #2 Technical Operator Compu-Scan #1 Technical Operator Compu-Scan #2 Non-Technical Operator Color Chart Non-Technical Operator Quick™ Arsenic Scan #1 Non-Technical Operator Compu-Scan #1 CAA-47 1 N N N N N N N N CAA-47 2 N N N N N N N N CAA-47 3 N N N N N N N N Taunton River water CAA-47 4 N N N N N N N N CAA-33 1 N N N Y Y N N Y CAA-33 2 N N N Y Y N N Y CAA-33 3 N N N Y Y N N Y 10 ppb As + low level interferents CAA-33 4 N N N Y Y N N N Total number of samples 32 32 32 32 32 32 32 32 Total false positives 0 2 1 5 8 0 1 3 Percent false positive 0% 6% 3% 16% 25% 0% 3% 9% 40 Table 6-10. Rate of False Negatives for Quick™ Ultra Low II Test Kits False Negative (Y/N) Description Sample ID Replicate Technical Operator Color Chart Technical Operator Quick™ Arsenic Scan #1 Technical Operator Quick™ Arsenic Scan #2 Technical Operator Compu-Scan #1 Technical Operator Compu-Scan #2 Non-Technical Operator Color Chart Non-Technical Operator Quick™ Arsenic Scan #1 Non-Technical Operator Compu-Scan #1 CAA-29 1 N N N N N N N N CAA-29 2 N N N N N N N N CAA-29 3 N N N N N N N N 30 ppb As CAA-29 4 N N N N N N N N CAA-31 1 N N N N N N N N CAA-31 2 N N N N N N N N CAA-31 3 N N N N N N N N 100 ppb As CAA-31 4 N N N N N N N N Battelle drinking water LFM CAA-38 1 Y Y Y Y Y Y Y Y CAA-39 1 N N N N N N Y Y CAA-39 2 N Y Y Y Y N Y Y CAA-39 3 Y Y Y Y Y N Y N Ayer untreated water CAA-39 4 N N N N N Y Y Y Ayer untreated water LFM CAA-40 1 N N N N N N Y N Ayer treated water LFM CAA-42 1 N N N N N Y Y Y Falmouth Pond water LFM CAA-46 1 Y Y Y Y Y Y Y Y Taunton River water LFM CAA-48 1 N Y Y Y N N Y N CAA-35 1 N N N N N Y Y Y CAA-35 2 N Y N N N N N Y CAA-35 3 N N N N N N Y Y 10 ppb As + high level interferents CAA-35 4 N N N N N N Y Y Total number of samples 21 21 21 21 21 21 21 21 Total number of false negatives 3 6 5 5 4 5 12 10 Percent of false negatives 14% 29% 24% 24% 19% 24% 57% 48% 41 accuracy and precision of the associated results because of the difficulty in performing accurate dilution in a field setting. The test kit materials were readily transported to the Battelle storage shed where the environmental samples were tested. 6.3.2 Analysis Time The average total analysis time for a sample was about fifteen minutes at a sample temperature of 24°C. The manufacturer provided a modified protocol that specified increased reaction times for samples below 24°C. Two samples could be run concurrently without any confusion. 6.3.3 Reliability The Quick™ Ultra Low II test kits operated reliably throughout the period of the test. 6.3.4 Waste Material The waste generated by the Quick™ Ultra Low II test kit was manageable. The vendor’s instructions provide a warning that hydrogen and arsine are generated during the test and recommend that testing be conducted in a well-ventilated area away from open flames and other sources. MSDSs should be reviewed before handling any chemicals. Instructions for the disposal of residual materials were clear and complete. The residual liquid in the reaction vessel was allowed to settle before disposal in order to let particulates accumulate on the bottom. A dilute hydrochloric acid solution was used to clean the reaction vessel prior to subsequent analyses. Disposal of this waste in an appropriate manner must be taken into consideration. 6.3.5 Cost The listed price for a Quick™ Ultra Low II test kit with color chart for analysis of 25 samples is $299.99. Replacement reagents and supplies are not available; kits are provided as a complete set because reagents, test strips, and color charts are made to perform optimally with each other. The Quick™ Arsenic Scan and Compu-Scan are available as options for an additional cost of $1,599.99 each. 42 Chapter 7 Performance Summary The Quick™ Ultra Low II test kit was verified by evaluating the following parameters: ? Accuracy ? Precision ? Linearity ? MDL ? Matrix interference effects ? Operator bias ? Inter-unit reprodicibility ? Rate of false positives/false negatives. The quantitative assessment of accuracy indicated that the relative bias for the color chart ranged from -78% to 18% for the technical operator and -87% to 45% for the non-technical operator. The relative bias for the Quick™ Arsenic Scan ranged from -91% to 22% for the technical operator and -95% to 16% for the non-technical operator. The relative bias for the Compu-Scan ranged from -80% to 161% for the technical operator and -92% to 70% for the non-technical operator. The overall agreement for the color chart results based on an assessment of whether the result was assigned to the correct color block indicated that the total percent agreement was 70% for the technical operator and 57% for the non-technical operator. Precision was assessed by analyzing four replicates of each sample. For the technical operator, RSDs ranged from 0% to 55% for the color chart, 2% to 51% for the Quick™ Arsenic Scan, and 6% to 85% for the Compu-Scan. For the non-technical operator, RSDs ranged from 0% to 84% for the color chart, 4% to 78% for the Quick™ Arsenic Scan, and from 11% to 139% for the Compu-Scan. The linearity of response was evaluated by plotting the test kit results against the reference analysis results for the PT samples. The equations for the linear regressions that were performed to evaluate linearity are summarized in Table 7-1. The slope, y-intercept, and correlation coefficient corresponding to a linear response that exactly matched reference concentrations would be 1, 0, and 1, respectively. The MDL was assessed by analyzing seven replicates of a sample spiked at approximately 3 ppb. The MDLs calculated using the precision data from these replicates ranged from 2.9 ppb to 3.1 ppb for the color charts, 2.8 ppb to 2.9 ppb for the Quick™ Arsenic Scan, and 4.7 ppb to 5.4 ppb for the Compu-Scan. 43 Table 7-1. Summary of Linear Regression Equations for Test Kit and Reference Results Description Slope Intercept Correlation Coefficient (R) Color chart, technical operator 0.92 0.22 0.9948 Color chart, non-technical operator 0.87 4.45 0.9498 Quick™ Arsenic Scan #1, technical operator 0.91 0.04 0.9830 Quick™ Arsenic Scan #2, technical operator 0.81 0.55 0.9934 Quick™ Arsenic Scan #1, non-technical operator 0.70 2.51 0.9700 Compu-Scan #1, technical operator 1.15 -0.88 0.9980 Compu-Scan #2, technical operator 1.93 -3.82 0.9946 Compu-Scan #1, non-technical operator 1.03 3.99 0.9322 Results for samples containing low and high levels of interfering substances indicated that low and high levels of interferents did appear to affect the detection of arsenic. Biases for these samples were similar to those calculated for PT samples containing arsenic only. An evaluation of Quick™ Ultra Low II test kit results for the technical and non-technical operators indicated no apparent difference in the color chart and Compu-Scan results. Measurements for the Quick™ Arsenic Scan done by the technical operator tended to be higher than for the non-technical operator. A paired t-test of each data set indicated that the results were not significantly different at a 5% significance level for the color chart and Compu-Scan, but were significantly different for the Quick™ Arsenic Scan. Inter-unit reproducibility was evaluated by comparing the data for the two Quick™ Arsenic Scan units and the two Compu-Scan systems used by the technical operator. The results for the Quick™ Arsenic Scan closely corresponded; however, the results for Compu-Scan Unit #2 tended to be higher than the results for Unit #1. Paired t-tests of the two sets of data indicated that the Quick™ Arsenic Scan results were not significantly different at a 5% significance level. The Compu-Scan results for the two systems were significantly different. A false positive was defined as a test kit result that was greater than 10 ppb and greater than 125% of the reference concentration, when the reference concentration is less than or equal to 10 ppb. The rates of false positives for the technical and non-technical operators using the color charts were 0% for both operators. The rates of false positives for the Quick™ Arsenic Scan units were 6% and 3% for the technical operator (Units #1 and #2, respectively) and 3% for the non-technical operator (Unit #1). The rates of false positives for the Compu-Scan units were 16% and 25% for the technical operator (Units #1 and #2) and 9% for the non-technical operator (Unit #1). A false negative was defined as a test kit result that was equal to or below 10 ppb and less than 75% of the reference concentration, when the reference concentration was greater than 10 ppb. The false negative rates for the technical and non-technical operators using the color charts were 14% and 24%, respectively. The rates of false negatives for the Quick™ Arsenic 44 Scan units were 29% and 24% for the technical operator (units #1 and #2) and 57% for the non-technical operator (Unit #1). The rates of false negatives for the Compu-Scan units were 24% and 19% for the technical operator (Units #1 and #2) and 48% for the non-technical operator (Unit #1). The Quick™ Ultra Low II test kits were easy to use and readily transportable to the field. The time to analyze one sample is approximately 15 minutes at a temperature range of 24°C to 30°C; longer reaction times are required for samples below this range. Two samples can be run con-currently without difficulty. The sample bottles were relatively easy to handle. Dilution of samples with arsenic concentrations exceeding the optimal detection range may be a source of error and reduce the accuracy and precision of the associated results because of the difficulty in performing accurate dilution in a field setting. The cost for a 25-sample test kit with color chart is listed as $299.99. Replacement reagents and supplies are not available; kits are provided as a complete set because reagents, test strips, and color charts are made to perform optimally with each other, according to the vendor. The Quick™ Arsenic Scan and Compu-Scan are available as options for an additional cost of $1,599.99 each. 45 Chapter 8 References 1. Test/QA Plan for Verification of Portable Analyzers, Battelle, Columbus, Ohio, Version 2. December 8, 2000. 2. U.S. EPA Method 200.8, Determination of Trace Elements in Waters and Wastes by Inductively Coupled Plasma Mass Spectrometry, Revision 5.5, October 1999. 3. Quality Management Plan (QMP) for the ETV Advanced Monitoring Systems Pilot, Version 4, U.S. EPA Environmental Technology Verification Program, Battelle, Columbus, Ohio, December, 2002. 4. U.S. Code of Federal Regulations, Title 40, Part 136, Appendix B.