## Description

STAT 477/577 – Homework Assignment 7

General homework guidelines: All homework assignments should be submitted using Canvas.

Please submit your answers separately, for each of the problems, as set-up in the Canvas submission portal.

You are allowed to either type in your answers, as well as submit graphs directly, within each submission

item. You are also allowed to submit a scanned copy of your answer, as long as the answers are submitted

separately for each question, as instructed. You can either scan your answers (ask if you don’t have access to

a scanner or do not know how to use your phone to do so), or just submit a picture of your answer. Please

note that if we can’t read your answer, we won’t be able to award any (partial) credit.

You have one attempt to submit your answers. If technical issues appear and your submission portal

has closed for some reason, please email Prof. Caragea explaining the situation and requesting permission

to resubmit. Please note that such requests must be made before the deadline.

For full credit, please make sure you submit your HW by the deadline of March 29. A late submission is

possible, with a 20% penalty, as long as it is turned in before the end of the day on Thursday following the

due day (for this HW, it is March 31-st). No submissions will be accepted past this date.

Homework problems.

1. In lecture, we discussed results from one of the surveys in the baby pictures study that looked at the

effect of Dr. Nettleton’s son wearing a hat in the picture. In the problem below, we will analyze data

from another survey in this study. In this survey, judges (students) were asked to identify which baby

was Dr. Froelich’s twice, once when she was pictured as an adult and and then when she was pictured

as a baby. Similar to the survey we studied in class, judges were not allowed to revisit their response

to the previous question. The pictures used in the study are located in the file babypictures.pdf.

The babies selected by the 123 judges on the two questions is given in the table below. You will recall

that Baby C is the correct baby in the analysis.

Baby Selected With

Baby Parent Picture

Baby Selected With

Adult Parent Picture A B C D Total

A 7 1 4 0 12

B 18 12 8 5 43

C 17 7 22 8 54

D 3 2 2 7 14

Total 45 22 36 20 123

(a) Using McNemar’s test, determine whether there is a difference in the proportion of correct responses between the two questions. Do the judges perform differently based on which picture is

given? Hint: Collapse the table above to a 2 x 2 table in order to answer the question.

(b) Obtain a 95% confidence interval for the difference in the proportion of correct responses between

the two questions. Interpret this confidence interval in the context of the problem.

(c) Using the extension of McNemar’s test, conduct a test of marginal homogeneity for the baby

selected. Clearly write down the null and alternative hypotheses, the value of the test statistic

and the p-value (which you should obtain using R), and a conclusion within the context of the

problem.

(d) Describe the differences you find in the marginal distributions between the baby selected when

the parent was pictured as an adult versus when the parent was pictured as a baby.

1

2. In educational assessment, open-ended questions (called free-response questions) tend to provide more

information about learning than multiple choice questions. However, these open-ended questions are

more difficult and time consuming to grade than multiple choice questions. In an on-going project,

researchers at a large university are studying the accuracy of computer-scored open-ended responses

by comparing them to person-scored responses. In this example, the computer and a person scored

the same 1,011 student responses to the same open-ended question. Each question was scored as

either a 1 = minimal understanding of concept, 2 = moderate understanding of concept, or 3 = full

understanding of concept. The table below gives the cross-classification table of scores and the full

data set with all 1,011 rows can be found in the file Scores.csv.

Computer-scored

Person-scored 1 2 3

1 439 26 20

2 48 113 33

3 38 29 265

(a) Calculate Cohen’s kappa for these data.

(b) Calculate weighted Cohen’s kappa using the squared weight function for these data.

(c) Based on your answers to part (a) and (b), does there seem to be agreement between the computer

scored and person scored responses? Explain your answer.

3. During the 1970’s, 80’s and 90’s, Gene Siskel and Roger Ebert worked as film critics for the Chicago

Tribune and Chicago Sun Times, respectively. In their syndicated TV show, At the Movies, they

presented their reviews of recently released movies using a Thumbs Up/Thumbs Down system. This

system also allowed them to give a mixed review. Their reviews for 160 movies from April 1995 through

September 1996 are given in the data file movies.csv.

(a) Use R to calculate the contingency table for Siskel and Ebert’s reviews.

(b) On what proportion of these movies did Siskel and Ebert agree on their review?

(c) Calculate Cohen’s kappa for these data.

(d) Calculate weighted Cohen’s kappa for these data using the squared weight function.

(e) Based on your answers to parts (c) and (d), do Siskel and Ebert seem to agree on their ratings

for these movies? Explain your answer.

2