Susan Voigt and Jason Conway
Program Evaluation and Data Specialists
Capital Area Intermediate Unit
Enola, PA
http://www.caiu.org/services/curriculum/program-grant-eval.aspx
Background:
Collecting and analyzing data from a variety of assessments, surveys, and evaluations usually under very tight deadlines and specific reporting requirements. For one current grant evaluation (Title IIB), CAIU PAGES conducts a large scale student science assessment twice a year and also a pre/post assessment of a teacher summer academy for school districts in Appalachia Intermediate Unit 8 in Altoona, PA (www.iu08.org). CAIU PAGES has been the independent evaluator for two different three-year grant cycles.
Challenge:
Collecting and analyzing data from a variety of assessments, surveys, and evaluations usually under very tight deadlines and specific reporting requirements. For one current grant evaluation (Title IIB), CAIU PAGES conducts a large scale student science assessment twice a year and also a pre/post assessment of a teacher summer academy for school districts in Appalachia Intermediate Unit 8 in Altoona, PA (www.iu08.org). CAIU PAGES has been the independent evaluator for two different three-year grant cycles.
Solution:
Twice a year, CAIU PAGES conducts a large scale student science assessment. Depending on the year, they have about 1,700 answer sheets per test administration. CAIU PAGES created a plain paper answer sheet in MS Word®. The assessments can have 20, 40 or 50 questions depending on grade level. For ease of large scale copying, one paper document was created but three different Remark Office OMR scanning templates were created. When they receive the completed assessments, they select the correct template based on the grade level prior to scanning.
The scanning goes very quickly and any difficulties are typically student or participant error. With Remark Office OMR, CAIU PAGES is able to scan and analyze the 1,700 answer sheets within 2-3 days, even with interruptions, for the pre-test and post-test assessment forms. Susan says, “I can't imagine how we would be able to complete this without software that would scan the answers, grade them, and allow us to run basic statistical information on the results.”
Another piece of this project is a pre/post teacher assessment for a two week summer academy. About 70 teachers are given an assessment on their first day. They spend the next two weeks attending classes and enhancing their skills. On the last day of the two week academy they are given the post assessment. The CAIU team members create customized answer sheets in MS Word to match each test. They travel onsite with a laptop and portable Fujitsu® scanner. In about two hours they scan all the assessments and provide a preliminary report. The preliminary reports are exported and inserted into PowerPoint slides to show the pre- and post-assessment comparison and overview. Final reports are created later and included in the grant reporting documents.
For both parts of this project, Susan uses the Condensed Item Analysis and Test Statistics Reports in Remark Quick Stats®, the built-in analysis component. The Condensed Item Analysis Report shows each question, the answer choices, the frequency of each answer choice and corresponding percent, and a bar chart of the frequency. The Test Statistics Report provides basic statistics on the test as a whole and is useful for gaining a quick look at performance and basic statistical analysis. Susan especially likes the statistics for test reliability that are included, such as the Kuder-Richardson and Coefficient Alpha statistics. These reliability statistics are required for grant reporting requirements. Multiple reports are generated from the assessments with various sorting options, such as by school or grade. Reports are provided to Appalachia IU 8, who shares this with the district. The report information is eventually transferred to an online form provided by the US Department of Education for grant reporting requirements.
CAIU loves the image deskew feature in version 7 and Remark’s customer support team. Susan says, "The persistence of the support staff to find an answer to the problem and the quick response time beats out every other customer service helpline I have dealt with (even with other unrelated products).&qu
About CAIU:
The CAIU is one of 29 regional educational service agencies in Pennsylvania. CAIU serves Cumberland, Dauphin, Perry, and Northern York Counties in south central PA. CAIU focuses our efforts on our 24 member school districts, 2 area vocational technical schools, over 60 non-public schools, several charter schools and cyber charter schools. In addition, CAIU operates the Capital Area School for the Arts (CASA), Capital Area Adventures (CAA), and the Capital Region Partnership for Career Development (PCD). CAIU also works very closely with partners in higher education.
Several statewide projects are administered by the CAIU, including Classrooms for the Future, Distinguished Educator, eStrategic Plan, and the PA Academic Standards Revision Project.
Institutional Research - Check out how Savannah College of Art and Design handles an internally developed survey given annually for 2500 respondents using Remark Office OMR and Remark Web Survey.
Institutional Effectiveness - Find out how Susan Evans was able to collect over 2000 responses and analyze data over a 9 day period using Remark Office OMR and Remark Web Survey.
Quality Assurance Program - St. Patrick’s College was undergoing an important quality assurance program involving both internal and external evaluators. Learn how they used Remark Office OMR to process their data.
Using Remark Office OMR for Course and Instructor Evaluations- Find out how Tiffany Martinez at the University of New Haven uses Remark Office OMR to process course evaluations in a timely, cost-effective manner.
Croatian University Uses Remark to Grade Thousands of Tests - Find out how the University of Rjeka corrects 50,000 tests every three weeks using Remark Office OMR.
Evaluate Impact of Professional Development Programs - Learn how Rebekah Nix was able to replace using transparent answer keys and reduce costs with Remark Office OMR.
Grading Biology Exams at a Large State University - See how Michael Wagenheim used Remark Office OMR to scan over 12,000 multiple choice biology tests per year and save money.
No such thing as a perfect form - Margaret Rutherford at Victoria ISD using Remark Office OMR to collect data for a grant.
Grading Assessments for Title IIB Grants - Learn how Capital Area Intermediate Assessments using Remark Office OMR to collect and analyze data from a variety of assessments, surveys, and evaluations under very tight deadlines.
Collect and Analyze Test Data for Title I - See how Tom Hays reduced the amount of time to collect data for Title I using Remark Office OMR.
Remarkable Changes to Instruction - After Tom Hays proved the feasibility and usability of Remark Office OMR for collecting data, his next challenge was to improve instruction in the elementary schools.
Remarkable Addition to Data Collection - Tom Hays discusses the remarkable addition to the data collection efforts: Evaluation of the kindergarten program through two assessments.
The Beginning of a Remarkable Relationship - The beginning of Tom Hays with Remark Office OMR.
Scanning Technology Conquers Common Testing Dilemmas - See how Dr. Bill Palya, Professor of Psychology at Jacksonville State University used Remark Classic OMR to enhance teaching and learning by enabling more frequent testing, improving test result analysis, and decreasing a student's ability to cheat.
How a Leading Hospital Uses Remark Office OMR® for Regulatory Compliance - Stephanie P provides overall management for the Patient Care Services Performance Improvement program that includes seven Quality Indicators. Patient data is analyzed and benchmarked against other hospitals. Reports are then provided to internal management and outside regulatory agencies.
Gravic's Contribution to Volunteerism - The challenge was to collect data from thousands of evaluation forms completed by participants in the Presidential Summit on Volunteerism held in Philadelphia.
Little League Assessments - Creating fair and balanced little league teams is a very complex process. Glenmoore Eagle Little League strives to create teams with a 50/50 win-to-loss ratio, making the season enjoyable for all the children and parents. In the past this goal was achieved by manually entering talent ratings of every child into a spreadsheet. This was a painstakingly slow process, so three years ago they started using Remark Office OMR® to scan and process player evaluation forms saving an enormous amount of time and effort for the volunteers who perform these functions.
Football Pool - The End Zone Bar & Grill is a sports bar that runs both a weekly and an annual NFL football pool. Employees had to manually check the weekly sheets, then sort and enter data from these sheets to keep a tally for the yearly pool. This process was enormously time-consuming, as there were 500-600 sheets per week to be processed.
