đź“š MC Test Analysis
Jump to: Try Online | Installation | Usage (app | report) | Data Format | Built With
Many educators design multiple-choice question examination. How do we know that these tests are valid and reliable? How can we improve upon the test by way of modifying, revising and deleting items based on student responses?
In a paper in the highly regarded Journal of Engineering Education, Jorion, et al (2016) developed “an analytical framework for evaluating the validity of concept inventory claims”. We believe that we can use this framework to help educators design their multiple-choice tests as well, especially, if they are designed as the final mastery examination in a course. An open source software to analyze a multiple-choice question examination would be encouraging to educators who have minimal programming experience and promising to contributors who would enhance the program.
This R package provides useful interfaces and functions to assist with the analysis of data from a typical multiple-choice test. The user needs only to provide an answer key that optionally identifies the concept or topic of each question and a table of responses given by each student to the questions in the test. MCTestAnalysis provides a Shiny web app interface and an automatic report-generation tool featuring concepts from Classical Test Theory (CTT), Item Response Theory (IRT) and structural analysis. We regard this package to be a work-in-progress and encourage contributions. At this time CTT results include item difficulty, item discrimination, Cronbach’s alpha, and alpha-with-item-deleted. Item response functions include model estimation and item characteristic curves. Tetrachoric and scree plots are included with an introductory exploratory factor analysis.
Try Online
You can try out the Shiny web interface online at https://apps.garrickadenbuie.com/mctestanalysis/, which demonstrates the test results explorer interface but cannot produce the PDF or HTML reports. For the complete features, you can install the package and use the interface locally (without have to upload your data) by following the instructions below.
Installation
For more detailed installation requirements and usage instructions, see Get Started.
In general, the package can be installed using devtools via
install.packages("devtools")
devtools::install_github("gadenbuie/mctestanalysis")Usage
Open RStudio or R on the command line or GUI and run the following commands:
library(MCTestAnalysis)
# Launch test exploration interface
explore()
# Create a test analysis report
report()When running explore() a browser window will open with the MC Test Analysis exploration application. The report() function will launch a window within RStudio to guide the user through the creation of a PDF report.
Explore: Shiny App for Interactive Multiple-Choice Test Analysis
The explore() function starts a local Shiny app. From this app, the user uploads their answer key and test data and then can view and explore the results of common test analysis methods.
Click on the image thumbnails below to view examples of the various screens of the MCTestAnalysis explore() app.
Load Data
Classic Test Theory
Item Response Theory
Factor Analysis
Distractor Analysis
Export Report
Report: A Custom Multiple-Choice Test Analysis Report
The report() function opens a window in RStudio that allows the user to import their answer key and test results and quickly generate a custom multiple-choice test analysis as a PDF or HTML file. This report includes all of the information in the explore() app, in addition to helpful background information on the various statistics and visualizations in the report.
Click here to view an example report (from synthetic data) or click the thumbnails below to view the report() app screens.
Data format
The MCTestAnalysis package requires both an Answer Key and a table of student responses, in other words the Test Data. An example of each table is provided, in the preferred format, in the inst/extdata folder of the repo, or from links on the “Import” tab of the explore() or report() interfaces.
A detailed overview of the required data format is available at http://www.eng.usf.edu/~kaw/MCTestAnalysis/MCTestAnalysis_input.pdf. Additional example test results and answer key CSV files are also available.
Answer Key Format
The answer key is a .csv file with columns Question and Answer, with optional columns Title and Concept. The table below shows an example answer key with 5 questions.
| Question | Answer | Title | Concept |
|---|---|---|---|
| Q1 | 3 | Question 1 | A |
| Q2 | 1 | Question 2 | B |
| Q3 | 3 | Question 3 | C |
| Q4 | 4 | Question 4 | D |
| Q5 | 4 | Question 5 | A |
Test Data Format
The test data is also a .csv file, where the first column is optionally the student ID and each column afterwards records the student’s answer to the test questions. The columns after the student ID need to be in the same order as the answer key. Here are the test responses for the first 5 students to the above example answer key.
| id | Q1 | Q2 | Q3 | Q4 | Q5 |
|---|---|---|---|---|---|
| Student 1 | 3 | 2 | 3 | 4 | 1 |
| Student 2 | 3 | 2 | 3 | 4 | 1 |
| Student 3 | 3 | 1 | 3 | 2 | 4 |
| Student 4 | 4 | 2 | 4 | 4 | 4 |
| Student 5 | 3 | 1 | 3 | 3 | 4 |
Built on These Great Packages
MCTestAnalysis was built by Garrick Aden-Buie and lead by Dr. Autar Kaw. The package was built to be used with R and RStudio, using Shiny for the interactive apps, R Markdown for the custom reports, and ggplot2 for plots. Many test analysis functions used in this package were made available by other packages and R developers, including the following packages: psych, psychometric, ltm, shiny, dplyr, ggplot2, reshape2, rmarkdown, DT, tibble, gridExtra, miniUI.