13  Qualitative Analysis

13.1 Analysis Objectives

Qualitative analysis evaluates the degree of agreement between the candidate method and the reference method in positive/negative determinations, corresponding to CLSI EP12. ivdtools provides raw_to_table() to build a four-fold table from raw paired data, counts_to_table() to build one directly from existing TP/FP/TN/FN counts, and three accompanying analysis methods: diagnostics() (sensitivity, specificity, PPV, NPV, likelihood ratios, etc.), kappa() (Kappa and PABAK), and mcnemar() (paired-difference test). Quantitative results can be converted to qualitative with continuous_to_binary().

This document uses three examples to demonstrate the complete analysis for the three data sources. The example data are deterministic teaching data; the positive direction and threshold should be determined according to the protocol.

Code 
library(ivdtools)
library(readr)

13.2 Overview of Functions

Function Main purpose Key input or output
raw_to_table() Build a four-fold table from raw paired data candidate, reference, positive, na.rm
counts_to_table() Build a four-fold table from TP/FP/TN/FN candidate/reference naming
continuous_to_binary() Convert quantitative to qualitative cutoff (≥ cutoff is 1)
describe() Description of raw data only available for raw_to_table() sources
diagnostics() Diagnostic accuracy metrics ci.method, prevalence
kappa() Kappa / PABAK prevalence switch
mcnemar() Paired-difference test uses exact test when discordant pairs <25

13.3 Example 1: Complete Analysis of Raw Paired Data

13.3.1 Reading and Validating Data

Code 
qual <- read_csv("./data/qualitative-raw.csv", show_col_types = FALSE)
qual <- as.data.frame(qual)
str(qual)
'data.frame':   60 obs. of  5 variables:
 $ id    : chr  "S001" "S002" "S003" "S004" ...
 $ new   : chr  "positive" "negative" "negative" "negative" ...
 $ gold  : chr  "positive" "negative" "negative" "negative" ...
 $ age   : num  46.1 44.7 65.7 60 64.6 32.7 47 40.9 38.4 48.1 ...
 $ gender: chr  "M" "F" "M" "F" ...
Code 
table(qual$gold)

negative positive 
      36       24
Code 
table(qual$new)

negative positive 
      36       24

13.3.2 Building the Four-Fold Table

Code 
qa <- raw_to_table(
  qual,
  candidate = "new",
  reference = "gold",
  id = "id",
  positive = "positive"
)
qa

Fourfold Table
  Source            : raw data
  Total n           : 60
  Duplicate IDs     : none
  Candidate         : new
  Reference         : gold

  2x2 Contingency Table
  ---------------------------------------- 
              gold                       
                  positive  negative  Sum       
  ---------------------------------------- 
  new   positive  23        1         24        
        negative  1         35        36        
        sum       24        36        60        
  ----------------------------------------

Parameter notes: raw_to_table() requires that both the candidate and reference columns have exactly 2 levels; positive specifies the positive level (used for level mapping of factor/character columns); na.rm controls whether missing rows are deleted. id is optional and used for duplicate-sample detection.

13.3.3 Describing the Raw Data

Code 
qa <- describe(qa)

Data Summary
  Rows:    60
  Columns: 5
  ID:      60 unique, no duplicates

  new
    negative        36  ( 60.0%)
    positive        24  ( 40.0%)
  gold
    negative        36  ( 60.0%)
    positive        24  ( 40.0%)
  age
    Mean (SD):       51.05 (13.45)
    Median (Q1-Q3):  48.90 (42.15 - 57.28)
    Range:           24.30 - 90.10
  gender
    F               32  ( 53.3%)
    M               28  ( 46.7%)

describe() is only available for raw_to_table() sources; it outputs the row count, ID duplicates, per-column summaries, and missingness.

13.3.4 Diagnostic Accuracy Metrics

Code 
qa <- diagnostics(qa, ci.method = "wilson")

Diagnostic Performance Summary
  Sensitivity        0.958  (0.798, 0.993) 
  Specificity        0.972  (0.858, 0.995) 
  PPV                0.958  (0.798, 0.993) 
  NPV                0.972  (0.858, 0.995) 
  Accuracy           0.967  (0.886, 0.991) 
  Prevalence         0.400  (0.286, 0.526) 
  LR+                34.500  (4.986, 238.724) 
  LR-                0.043  (0.006, 0.292) 
  Odds Ratio         805.000  (47.921, 13522.837) 

  # PPV & NPV are based on data prevalence of 0.400.
  # Confidence intervals for proportions use 'wilson' method.
Code 
qa$diagnostics[c("sensitivity", "specificity", "ppv", "npv", "accuracy")]
$sensitivity
      est     lower     upper 
0.9583333 0.7975819 0.9926065 

$specificity
      est     lower     upper 
0.9722222 0.8583028 0.9950796 

$ppv
      est     lower     upper 
0.9583333 0.7975819 0.9926065 

$npv
      est     lower     upper 
0.9722222 0.8583028 0.9950796 

$accuracy
      est     lower     upper 
0.9666667 0.8863623 0.9908107

In this example, the sensitivity is 0.958 and the specificity is 0.972. If the prevalence in the target population is known, use prevalence to recompute PPV/NPV:

Code 
qa <- diagnostics(qa, prevalence = 0.4)

Diagnostic Performance Summary
  Sensitivity        0.958  (0.798, 0.993) 
  Specificity        0.972  (0.858, 0.995) 
  PPV                0.958  (0.790, 0.993) 
  NPV                0.972  (0.864, 0.995) 
  Accuracy           0.967  (0.886, 0.991) 
  Prevalence         0.400 
  LR+                34.500  (4.986, 238.724) 
  LR-                0.043  (0.006, 0.292) 
  Odds Ratio         805.000  (47.921, 13522.837) 

  # PPV & NPV are based on user-specified prevalence of 0.400.
  # Confidence intervals for proportions use 'wilson' method.
Code 
qa$diagnostics[c("ppv", "npv")]
$ppv
      est     lower     upper 
0.9583333 0.7895852 0.9926193 

$npv
      est     lower     upper 
0.9722222 0.8641373 0.9950711

Parameter notes: The ci.method of diagnostics() accepts 7 proportion confidence-interval methods (default "wilson"); when prevalence is given, PPV/NPV are recomputed by the Bayes formula; otherwise they are based on the observed prevalence.

13.3.5 Kappa Agreement

Code 
qa <- kappa(qa)

Cohen's Kappa
  new  vs  gold
  n = 60

  Kappa                 0.9306
  SE                    0.0483
  95% CI                (0.8359, 1.0000)
  Observed agreement    0.9667
  Expected agreement    0.5200
Code 
qa <- kappa(qa, prevalence = 0.5)

PABAK
  new  vs  gold
  n = 60

  Kappa                 0.9333
  SE                    0.0463
  95% CI                (0.8425, 1.0000)
  Observed agreement    0.9667
  Expected agreement    0.5000
  # PABAK (prevalence = 0.500): 2 * p_obs - 1, assumes expected agreement = 0.5

Parameter notes: kappa() gives Cohen’s Kappa; when prevalence is given, it instead computes PABAK (2·p_obs − 1). The two handle prevalence bias differently; note the method in the report.

13.3.6 McNemar Test

Code 
qa <- mcnemar(qa)

McNemar Test
  new  vs  gold
  Discordant pairs:  b (pos, neg) = 1,  c (neg, pos) = 1
  n (b + c)        = 2

  Method: Exact binomial test
  P-value:          1.0000
  Conclusion: Not significant at 95% confidence level

The discordant pairs b+c = 2 < 25, so the function automatically uses the exact binomial test.

13.3.7 Summary

Code 
summary(qa)

Qualitative Analysis - summary
-------------------------------------------------

Fourfold Table
  Source            : raw data
  Total n           : 60
  Duplicate IDs     : none
  Candidate         : new
  Reference         : gold

  2x2 Contingency Table
  ---------------------------------------- 
              gold                       
                  positive  negative  Sum       
  ---------------------------------------- 
  new   positive  23        1         24        
        negative  1         35        36        
        sum       24        36        60        
  ---------------------------------------- 

Analyses performed:

  [x] Describe
  [x] Diagnostics
  [x] Kappa
  [x] McNemar

Data Summary
  Rows:    60
  Columns: 5
  ID:      60 unique, no duplicates

  new
    negative        36  ( 60.0%)
    positive        24  ( 40.0%)
  gold
    negative        36  ( 60.0%)
    positive        24  ( 40.0%)
  age
    Mean (SD):       51.05 (13.45)
    Median (Q1-Q3):  48.90 (42.15 - 57.28)
    Range:           24.30 - 90.10
  gender
    F               32  ( 53.3%)
    M               28  ( 46.7%)

Diagnostic Performance Summary
  Sensitivity        0.958  (0.798, 0.993) 
  Specificity        0.972  (0.858, 0.995) 
  PPV                0.958  (0.790, 0.993) 
  NPV                0.972  (0.864, 0.995) 
  Accuracy           0.967  (0.886, 0.991) 
  Prevalence         0.400 
  LR+                34.500  (4.986, 238.724) 
  LR-                0.043  (0.006, 0.292) 
  Odds Ratio         805.000  (47.921, 13522.837) 

  # PPV & NPV are based on user-specified prevalence of 0.400.
  # Confidence intervals for proportions use 'wilson' method.

PABAK
  new  vs  gold
  n = 60

  Kappa                 0.9333
  SE                    0.0463
  95% CI                (0.8425, 1.0000)
  Observed agreement    0.9667
  Expected agreement    0.5000
  # PABAK (prevalence = 0.500): 2 * p_obs - 1, assumes expected agreement = 0.5

McNemar Test
  new  vs  gold
  Discordant pairs:  b (pos, neg) = 1,  c (neg, pos) = 1
  n (b + c)        = 2

  Method: Exact binomial test
  P-value:          1.0000
  Conclusion: Not significant at 95% confidence level

13.4 Example 2: Building Directly from TP/FP/TN/FN

When counts are already available, no raw data are needed:

Code 
tab <- counts_to_table(
  tp = 80, fp = 5, tn = 120, fn = 3,
  candidate = "new method",
  reference = "gold standard"
)
tab

Fourfold Table
  Source            : counts
  Total n           : 208
  Candidate         : new method
  Reference         : gold standard

  2x2 Contingency Table
  ----------------------------------------------- 
                     gold standard              
                         positive  negative  Sum       
  ----------------------------------------------- 
  new method   positive  80        5         85        
               negative  3         120       123       
               sum       83        125       208       
  -----------------------------------------------
Code 
tab <- diagnostics(tab)

Diagnostic Performance Summary
  Sensitivity        0.964  (0.899, 0.988) 
  Specificity        0.960  (0.910, 0.983) 
  PPV                0.941  (0.870, 0.975) 
  NPV                0.976  (0.931, 0.992) 
  Accuracy           0.962  (0.926, 0.980) 
  Prevalence         0.399  (0.335, 0.467) 
  LR+                24.096  (10.198, 56.934) 
  LR-                0.038  (0.012, 0.114) 
  Odds Ratio         640.000  (148.773, 2753.180) 

  # PPV & NPV are based on data prevalence of 0.399.
  # Confidence intervals for proportions use 'wilson' method.
Code 
tab <- kappa(tab)

Cohen's Kappa
  new method  vs  gold standard
  n = 208

  Kappa                 0.9201
  SE                    0.0277
  95% CI                (0.8659, 0.9744)
  Observed agreement    0.9615
  Expected agreement    0.5184
Code 
tab <- mcnemar(tab)

McNemar Test
  new method  vs  gold standard
  Discordant pairs:  b (pos, neg) = 5,  c (neg, pos) = 3
  n (b + c)        = 8

  Method: Exact binomial test
  P-value:          0.7266
  Conclusion: Not significant at 95% confidence level
Code 
summary(tab)

Qualitative Analysis - summary
-------------------------------------------------

Fourfold Table
  Source            : counts
  Total n           : 208
  Candidate         : new method
  Reference         : gold standard

  2x2 Contingency Table
  ----------------------------------------------- 
                     gold standard              
                         positive  negative  Sum       
  ----------------------------------------------- 
  new method   positive  80        5         85        
               negative  3         120       123       
               sum       83        125       208       
  ----------------------------------------------- 

Analyses performed:

  [ ] Describe
  [x] Diagnostics
  [x] Kappa
  [x] McNemar

Diagnostic Performance Summary
  Sensitivity        0.964  (0.899, 0.988) 
  Specificity        0.960  (0.910, 0.983) 
  PPV                0.941  (0.870, 0.975) 
  NPV                0.976  (0.931, 0.992) 
  Accuracy           0.962  (0.926, 0.980) 
  Prevalence         0.399  (0.335, 0.467) 
  LR+                24.096  (10.198, 56.934) 
  LR-                0.038  (0.012, 0.114) 
  Odds Ratio         640.000  (148.773, 2753.180) 

  # PPV & NPV are based on data prevalence of 0.399.
  # Confidence intervals for proportions use 'wilson' method.

Cohen's Kappa
  new method  vs  gold standard
  n = 208

  Kappa                 0.9201
  SE                    0.0277
  95% CI                (0.8659, 0.9744)
  Observed agreement    0.9615
  Expected agreement    0.5184

McNemar Test
  new method  vs  gold standard
  Discordant pairs:  b (pos, neg) = 5,  c (neg, pos) = 3
  n (b + c)        = 8

  Method: Exact binomial test
  P-value:          0.7266
  Conclusion: Not significant at 95% confidence level
Code 
describe(tab)   # counts sources have no raw data; describe() is unavailable

Objects built by counts_to_table() have no raw data, so describe() is unavailable; diagnostics(), kappa(), and mcnemar() all work normally.

13.5 Example 3: Converting Quantitative Data to Qualitative

13.5.1 Reading Data

The candidate method is a quantitative result result and the reference is qualitative gold:

Code 
cont <- read_csv("./data/qualitative-continuous.csv", show_col_types = FALSE)
cont <- as.data.frame(cont)
str(cont)
'data.frame':   50 obs. of  3 variables:
 $ id    : chr  "S001" "S002" "S003" "S004" ...
 $ result: num  10.11 13.06 1.53 7.78 4.98 ...
 $ gold  : chr  "positive" "positive" "negative" "negative" ...

13.5.2 Converting to Binary

Convert to binary at cutoff = 8:

Code 
cont_bin <- continuous_to_binary(cont, cols = "result", cutoff = 8)
head(cont_bin)

continuous_to_binary() generates a result_binary column (≥ cutoff is 1, otherwise 0). To satisfy the positive mapping requirement of raw_to_table(), first relabel it with labels consistent with the reference:

Code 
cont_bin$result_binary_lab <- ifelse(cont_bin$result_binary == 1, "positive", "negative")

13.5.3 Building the Four-Fold Table and Analyzing

Code 
qa3 <- raw_to_table(
  cont_bin,
  candidate = "result_binary_lab",
  reference = "gold",
  id = "id",
  positive = "positive"
)
qa3 <- diagnostics(qa3)

Diagnostic Performance Summary
  Sensitivity        0.952  (0.773, 0.992) 
  Specificity        0.897  (0.736, 0.964) 
  PPV                0.870  (0.679, 0.955) 
  NPV                0.963  (0.817, 0.993) 
  Accuracy           0.920  (0.812, 0.968) 
  Prevalence         0.420  (0.294, 0.558) 
  LR+                9.206  (3.140, 26.994) 
  LR-                0.053  (0.008, 0.361) 
  Odds Ratio         173.333  (16.746, 1794.100) 

  # PPV & NPV are based on data prevalence of 0.420.
  # Confidence intervals for proportions use 'wilson' method.
Code 
qa3$diagnostics[c("sensitivity", "specificity", "accuracy")]
$sensitivity
      est     lower     upper 
0.9523810 0.7733064 0.9915440 

$specificity
      est     lower     upper 
0.8965517 0.7361492 0.9641851 

$accuracy
      est     lower     upper 
0.9200000 0.8116175 0.9684505

Note: The level specified by positive must exist in both columns, so first relabel the binary column as positive/negative. If positive is not specified, the function may reverse the positive/negative direction according to the sorted levels, flipping TP/FP; be sure to check.

13.6 Key Points for Interpreting Results

  1. Direction consistent: Confirm that the positive-direction definitions of the candidate and reference are consistent, to avoid flipping the whole four-fold table.
  2. Levels unique: Both the candidate and reference columns must have exactly 2 levels; multi-level or continuous columns must first be converted to binary.
  3. Prevalence: PPV/NPV depend on prevalence; report the observed prevalence or specify the target-population prevalence.
  4. Method labeling: Kappa and PABAK have different meanings; McNemar uses an exact test for small numbers of discordant pairs; report the method.
  5. Threshold recorded: Converting quantitative to qualitative must record the cutoff and positive direction; the results hold only under that threshold.