Data Coding Lab

Whenever we have an open-ended measure, like a verbal protocol, videotaped observation or narrative record, we need some way to reduce the data to manageable size.  One way that we do this is by coding the data based on specific aspects of interest.  In this way, the large amount of data that we have obtained can be placed into a set of categories.

 

While it is helpful to be able to code data in this way, there are two central problems:  definition and reliability.  Definition refers to the need to operationalize the behavioral categories that interest us in a clear and unambiguous manner.  The clearer and more precise the definition, the easier it will be to apply that definition to the data (i.e., to code the data).  Reliability refers to the fact that coding data is a subjective task, and not all people will necessarily code the same event in the same way.  Even if the definition of the coding categories are clear, there still may be cases in the data that appear to have aspects of more than one category.  Thus, it is crucial to make sure that more than one rater familiar with the coding scheme would place a given event in the same category.
 

In this lab, we will look at a video (on the computer) from a study on communication and conflict and code a segment of the tape.  Some students may find this video troubling.  After watching the video, we will come up with a set of categories for behaviors that we want to code (some suggestions are listed below).  Then, we will code the tape individually.  Then, pairs of you will get together and calculate inter rater reliability.  Then, as a class we will develop a clear way of operationalizing (i.e., defining) the categories.  Then, we will recode the video and see if inter rater agreement changed.  If necessary, you will attempt to resolve differences in coding.  Finally, as an assignment, you will answer the questions below.
 

Behavioral categories (i.e., constructs) and some initial definitions
 

1. Mood changes: How many mood changes (either person) occurred during the video?
 

2. Insanity: How many insane episodes (either person) occurred during the video?
 

3. Marge Factor: How many times was the word ?Marge? uttered?

 

4. Some other behavior?
 

 
 

An assignment

 
 

1. Who was your lab partner?  Did your ratings agree?  Did your agreement change between the first and second run?  Why?  How did you resolve disagreement?
 

2.  What were the operational definitions of the constructs?

 

3. Were the definitions easy to apply?  Why or why not?
 

4. How could you improve your coding scheme?