Using this process could accelerate or standardize IRR practices in qualitative studies. Through this process, three coders were able to consistently get 80-90% IRR on 95% of the codes. Codes for the three researchers were compared using our IRR method described above.
A total of 64 codes were developed through an initial pass through the data, then three coders analyzed the remaining responses independently. In this study, participants’ responses to open-ended survey questions were coded by three researchers using inductive, open coding. Our coding and IRR methods were employed on a dataset from a survey that was taken by undergraduate students at five different universities (n=154). We calculated overall IRR (between all three coders) as well as IRR between each set of coders. IRR was calculated as the proportion of agreed codes over the total number of codes in the document. We compared codes and phrases to determine coder agreement for each participant and then calculated IRR. The table was then moved into Excel to enable comparison of codes between individual coders. A macro (a customizable function that combines many commands into a single process) was then used to extract these comments to a table in a separate document. First, the interview transcripts were coded in Word, and codes were inserted in the appropriate locations as comments in the document. The process discussed in this paper uses Microsoft Word® (Word) and Excel® (Excel). The authors provide recommendations, or “tricks of the trade” for researchers performing qualitative coding who may be seeking ideas about how to calculate IRR without specialized software.
This paper summarizes one approach to establishing IRR for studies where common word processing software is used.
This leads to a variety of methods for calculating IRR. Methods of coding without software vary greatly and include using spreadsheet software, word processing software, or even hard copies with different colored highlighters. However, the process of manually determining IRR is not always clear, especially if specialized qualitative coding software that calculates the reliability automatically is not being used. When using qualitative coding techniques, establishing inter-rater reliability (IRR) is a recognized process of determining the trustworthiness of the study.
0 kommentar(er)
