Nominal dichotomous yes/no data: Krippendorff alpha inter-rater reliability
Вставка
- Опубліковано 2 бер 2017
- The video shows how I perform the statistical test Krippendorff alpha to check the reliability of a variable with nominal/dichotomous data. (Windows PC & SPSS.)
Step 1. Search for and download the Kalpha macro. I use the keywords download, macro, kalpha, spss to find it online.
Sometimes, you will find the file named "kalpha.sps". And sometimes, you will find a compressed, zipped file called "kalpha.zip." Either way, download this to your own computer.
Step 2. If it is the uncompressed file "kalpha.sps", leave it as it is. If it is the zip file, which is a compressed folder, you must extract it. Right-click on it, and extract. Remember where you save it because you need it later.
Step 3. Start SPSS and open a blank, empty page.
Step 4. File, Open, Syntax
Step 5. Search your computer, select and open the file "kalpha.sps". The macro.
As you can see, it is just a script. Do not change anything. Just leave it.
Step 6. Execute this macro by Run, All.
The window that pops up is normal. This is just a confirmation that all is OK. You can close it and the script if you want.
Step 7. Open your data. Make sure that your data is correct. Numeric type and nominal level of measure.
Step 8. It is time to run the statistical test. Normally, you select a test from a menu. But, since this test is not included inside SPSS, we need to execute it ourselves.
File, New, Syntax, and Type:
kalpha judges = obs1 obs2 obs3 obs4 obs5/level = 1/detail = 0/boot = 10000
I will explain what it means later.
Then: Run, All
This is how I would write my report:
Method
Krippendorff's alpha test was used (Hayes & Krippendorff, 2007) to estimate the inter-coder reliability, and these alpha (α) values are reported in the results below.
Results
The results show that the inter-coder reliability was relatively high (α = 0. 7831), i.e., the five coders agreed.
Discussion
(Here, you interpret and discuss plausible reasons why you think that the coders did (or did not) agree. It is beyond the scope of this example, where I only focus on the statistical test.)
References
Hayes, A. F., & Krippendorff, K. (2007). Answering the call for a standard reliability measure for coding data. Communication Methods and Measures 1(1), 77-89.
you are a legend, i seriously cannot put in words how much i am thankful for your tutorial. i wish you all the ease and peace life has to offer
Very helpful and clear guide on using kalpha, and I loved the little moments of humour! Tack -Thank you.
THANK YOU SO MUCH. I kept reading articles that explain but nothing was as helpful!!!!
Thank you so much....I actually got to clear about Krippendorff alpha from this video. Thanks!
This walkthrough was so clear and simple. Amazing!
Thanks for your feedback!
What a great way to teach. Thanks a lot
Good Work Mr Kent . You have opened my eyes on SPSS . Love from Kenya
Wonderful!
Really a great video..We can also use Fleiss' kappa with a data on a nominal scale (i.e., either an ordinal or nominal variable) ..and use it for any categorical data to measure the reliability of agreement between a fixed number of raters!
Thanks so much for this video training, Well done, Kent!
Glad you enjoyed it, it was a complicated video to make, but it turned out OK.
thank you so much sir for making this task smooth
Thank you for this video. It helped me a lot.
Thank you thank you thank you! Very helpful!
I don't know you but I already consider you very much. :)
thank you professor!
Thank you very much for sharing! Just done it at the last minute.
Fantastic!
Greetings from a new subscriber in Indonesia. Thank you very much for this very informative and clearly presented video.
All the best! Glad you liked the video.
Thanks for the video! I greatly benefit from it. And btw is that your cat? Very cute - s/he must have developed his/her own Catistics :)
THANK YOU SIR
Thank you so much
Thank you too!
you rock!
Hi, thanks for the video. One question. In my case the alpha of cronbach have values over 0.8 but in the alpha of Krippendorff have values below 0.33. This is normal? (sorry for my english.)
Hola gracias por el videos. Una pregunta. En mi caso el alpha de cronbach tuvo valores sobre 0.8 pero en el alpha de Krippendorff tuvo valores bajo los 0.33 ¿Es normal esto?
Thank you for the great video. Quick question: Why do you use 0/boot and not 1/boot like Hayes and Krippendorff 2007? Best Regards Sören Bärsch
Thank you for the video. I would like to know can we use both cronbach alpha and Krippendorff alpha inter-rater in the same study? since i have nominal, dichotomous and likert scale questions in my questionnaire. Should i only use Krippendorff alpha inter-rater for the three types of items? or use Krippendorff alpha inter-rater for nominal and dichotomous items and cronbach alpha for the likert scale items? Waiting for your reply. Thanks
Dear Prof. Löfgren, I have 94% agreement (187/199 = yes, based on percentage count values) between two coders, for 199 items with codes 1= yes, 2=no, on these 199 items. however, K-alpha value calculated as shown above is negative (-0.0285), wondering how is this possible? Please let me know SOS
thank you
Thank for the video! I'm a question. The variables must be dichotomous or may be nominal?
can be both.
Is it possible to apply the macro in an already existent data sheet in spss?? Thank you for the video!
it is helping
I followed all instructions however syntax shows a messages as " kalpha The first word in the line is not recognized as an SPSS Statistics command." what should I do?
Thank you so much, I just got one question. I am doing a meta-analysis, and working on inter-rater reliability. I have followed your video step by step, for some reason my alpha was -.3, which is really small, despite the majority of the rating from two raters was consistent.
What you describe is quite common, I am afraid. You look at you data and the differences between the group. You think to yourself that there differences are large --- but then the Krippendorff statistical analysis only show these differences as non-significant.
please I want a video how to perform binary logistic regression with SPSS
Really thanks, I understad everything... but I have Macbook and I can't see Run
i CANNOT FIND THE MACRO, PLEASE SOMEONE help me!
Thanks so much - does the same method in the video still work for nominal variables that aren't dichotomous? for example, eye colour (brown, blue, green)?
Yes, I'm quite certain that it works.
@@kentlofgren Thanks for your response, I've tried it and it does work. Your video is very clear and super helpful Kent =]
Thank you for the video but what is the reason behind the distracting music and animations?
Oh, just an attempt to be creative, nothing more ;-)
In any case, thank you for the video it really helped me and your approach is really great. But i have to say it again, the music and the images are really distracting :) thanks again
Can the same approach be used for any nominal data, not just nominal dichotomous data?
/Users/maravandooren/Desktop/Schermafbeelding 2017-07-31 om 11.50.23.png
I will get the following error code, what dit I do wrong??
Error: input reliability Data Matrix Exhibits No Variation
can we use Krippendorff Alpha for an intra-rater reliability?
What is 10.000 mean?
i hate music, we can not concentrate