Maximizing Precision/Reliability

• An apparatus for measurement should be as accurate as possible relative to what we want to measure.
• Measuring response times:  Don't use a sweep second hand on a clock
• There is a point of diminishing returns
• Olympic Skiing
• Our operational definitions should be specific
• Our measures should be repeatable
• There is often a tradeoff between precision and ease of measurement
• Do the results replicate under similar circumstances?
• e.g., a weight scale + or - 2 pounds
• e.g., Bush over Gore 51 to 46 + or - 2 points

Accuracy

• Easier to assess when precision is high
• Bias-systematically high or low

Operational Definitions

• In order to make precise measurements, we must know what we are measuring
• Constructs
• Abstract (hypothetical) factor
• Cannot be observed directly, but is inferred from observations
• Constructs generally cohere
• Generally assumed to have an underlying cause
• Examples
• Depression
• Learned Helplessness
• Constructs are made concrete via operational definitions.

Validity

• Operational definitions
• Must be clear and precise
• Milgram:  Defined "disobedience"
• The point of rupture is the act of disobedience.  A quantitative value is assigned to the subject's performance based on the maximum intensity shock he is willing to administer before he refuses to participate further.
• Our definitions must match what we think we are testing
• What is a good measure of communication?
• Why wouldn't we count sneezes?
• Olympic skiing (again)
• Downhill skiing:  Time
• Ski jumping:  Distance
• Freestyle skiing:  Aesthetic value

Types of Validity (External Validity)

• Face Validity
• e.g., asking math questions to assess math ability
• Content Validity
• Using representative material
• e.g., typing test for a typist
•  Criterion-related validity
• Can one infer a value of some other measure?
• e.g., two intelligence tests correlate
• SAT and grades
• Construct validity
• Does the measure really capture the underlying construct?
• e.g., low validity: measuring intelligence by feeling for lumps on the head.

Practice with Operationilization (indulge me)

• People who live in clean houses get sick less often than people who live in dirty ones.
• What is a clean house?  A dirty house?
• People with high mathematical ability can play chess better than people with low mathematical ability.
• Physically attractive individuals make better salespeople than physically unattractive people.

Problems with Dependent Measures

• Range effects
• Scores crammed to the top or bottom of the scale
• Floor and ceiling effects
• Sensitivity
• Measuring fear
• Screaming vs. blood pressure

Maximizing Objectivity

• There is always a danger of observer bias
• Expectancy
• The experimenter's actions may influence the outcome of an experiment
• Clever Horse
• Bias can affect observations
• Observer may be more sensitive to behaviors that support the hypothesis than to those that disconfirm it.
• Observers involved in a situation may become emotionally involved

Objectivity (cont)

• Ensuring objectivity
• Naive raters
• Don't know the point of the study
• Blind raters
• Don't know the experimental condition of the data they are scoring
• Subjective measures require a check on inter-rater reliability
• double blind: neither subject nor experimenter know condition
• e.g., using a placebo

Issues

• Observation requires losing information
• When will that information be lost
• Some measurements preserve a lot of information
• Video-tape recordings
• Some measurements lose a lot of information
• Response and latency measurements
• How can we deal with these kinds of observations?

Classes of Measures

• Narrative records
• Videotape, audiotape, written descriptions
• Requires massive data reduction
• Coding of behavior
• What events should be recorded?
• Sampling of record
• Frequency method
• How often does a particular behavior occur?
• Must define the event...
• Intervals Method
• only look at certain set times
• Must define the event...
• Duration Method
• How long does a particular behavior last?
• Must define the event...