Inter-observer Reliability of Indicators and Decision of Pattern Identification Using Diagnostic Flowchart with Traditional Korean Medicine

Objective

To evaluate the consistency of patterns and pattern identification (PI), and the improvement in the diagnosis of blood stasis after the standard operating procedures (SOPs) were enhanced by using a diagnostic flflowchart. MethodsA total of 358 subjects who were admitted by the Jaseng Hospital of Korean Medicine between June 2013 and December 2014 were included in the study. Each patient was independently diagnosed by two experts from the same department. In 2014, the SOPs included the use of a diagnostic flflowchart for each indicator. Inter-observer consistency was assessed by simple percent agreement and by the kappa and AC1 statistics.

Results

The improvements in inter-observer agreement for the indicators (for all patients) and PI were greater in 2014 than in 2013. In 2013, the j-value measure of agreement between the two experts ranged from "poor" (κ=–0.051) to "good" (κ=0.968). The AC1 measure of agreement between the experts was generally high for the indicators and ranged from–0.010 to 0.978. In most cases, agreement was considerably lower when it was assessed by the j-values compared with the AC1 values. In 2014, the j-value measure of agreement on the indicators (for the subjects) generally ranged from poor to good, with the values ranging from–0.007 to 0.994. Moreover, the AC1 measure of agreement between the experts was generally high for all of the indicators and ranged from "moderate" (AC1=0.408) to "excellent" (AC1=0.996).

Conclusions

In four examinations, there was moderate agreement between the clinicians on the PI indicators of blood stasis. To improve clinician consistency (e.g., in the diagnostic criteria used), it is necessary to analyse the reasons for inconsistency and to improve clinician training. (Trial registration No. KCT0000916)