Tom Hays
Data Coordinator for Title I
(Name withheld) Public Schools
Much has been said about the new accountability requirements since the No Child Left Behind law was enacted in January of 2001. However, school systems have had accountability as part of their requirements long before the law was enacted. In 1996, when I worked in a Maryland school system as the data specialist in the Offices of Accountability and Compensatory Education; we were well aware of the need for accountability for the programs the school system implemented.
Our superintendent charged our office with determining the efficacy of intervention programs to ensure that the money being spent on the programs was getting an adequate return in student performance. One of the first tasks I embarked upon was finding suitable means for collecting data on student performance for students involved in a variety of programs. I knew the best method of collecting and analyzing the data would be accomplished with an electronic data collection system. I attended several conferences from the Association for Supervision and Curriculum Development (ASCD). I browsed the vendor fairs at these conferences looking for an electronic means of collecting data. I found several, but the price for the hardware and software was well above the budget for our offices. At one conference, I talked with a Gravic representative about Remark Office OMR®. He showed me the Remark program and how it worked with a small document scanner connected to his laptop computer. The representative ran a few different forms through the scanner and the results of the scan appeared on the screen. In my discussions with Gravic, I asked about the portability of the data - could I export the scanned data results into the different programs I used for data analysis? The answer was that I could export the data into just about any program for data analysis as well as a standard ASCII format that would allow me to move the data anywhere. That made my day!
But, what about creating the scanning documents? Did I need a special computer program to create the documents that could be scanned? No, any word processing program would do just fine. Ok, how do I make those bubbles for the users to fill in for my multiple choice items? No problem, I can use the font that installs with the software or just a capital letter “O” in the standard Arial font. I needed to collect information on each student. How can I make a sheet for every student that has a unique identifier for each student? The identifier can not take up much space on the paper since that limits the amount of space needed for the evaluation data. Again, Remark Office OMR had the answer. Use a barcode like on the cans of food in the grocery store. It takes only about a half inch by one inch space. That barcode can be the student’s identification number. What if I have more data to collect than will fit on one side of the page, can I use the back of the paper as well? Yes, that too is possible.
I saw the program run on a standard Windows computer, so I knew our office already had some of the equipment needed to run the program. Now, what about the cost of the software, could we afford it? The cost of the program was less than the cost of my data analysis program, so it was affordable. But, our office did not have a scanner. What kind of scanner did I need? The requirements for the scanner were relatively simple. I would need an image scanner that had a document feeder and was TWAIN compliant. The speed of the scanner and other features of the scanner would be whatever was needed to meet my efficiency requirements.
The time had come for a trial run of the program for suitability for collecting data for our office. We purchased a scanner and the Remark software. We were ready to go. Well, almost ready.
I had to create the documents for data collection. How to design the data collection documents was my next challenge. I read through the manual that came with the program, paying attention to the document design notes. That didn’t seem too difficult. I knew I could make the documents I needed.
The first project was to evaluate an intervention for early reading. This intervention targeted students in 1st and 2nd grades who were behind in their reading skills. The process for data analysis is the same as most intervention programs. Pre-test the student to determine the beginning level of skill and then at the end of the intervention, post-test the students to determine the exit level of skill. Compare the results of the post-test with the initial results of the pre-test to compute the gain in skill that the program produced.
Because students in the program do not yet read, or at least do not read well, the testing of the students both in the pre- and post-test require teacher evaluation and recording of the individual student results on a scannable sheet. The most recent version of the sheet is available for download at the beginning of this article.
The scan form is two pages, a front and back of one sheet of paper. The barcode is the link to the student database containing all demographic data on each student. So the teacher knows which sheet to use with each student, the identifying text is added in the barcode corner. This assessment is used to evaluate student reading ability through the use of seven specific tests: Running record, writing vocabulary, dictation assessment, comprehensive word test, phonemic awareness, letter id and concepts of print. Some of the specific tests have sub-test components. It is through the combination of the scores of the tests and the sub-tests that a derived score is calculated, thus determining the pre- and post-test achievement level of the student. On the post-test form, other important components of the evaluation are completed. The pre- and post-test book level and the number of lessons and partial lessons delivered are important to assess student growth and program delivery. Also completed only on the post-test form are explanations for why some lessons may not have been delivered. The early dismissal and late entrance marks identify these unusual circumstances.
This project of data collection to determine the value of an intervention program proved to be very useful. Although the anecdotal data from the school staff, principals and teachers indicated the program was beneficial to the engaged students, that was not enough to meet the requirements. The central office needed proof the intervention worked. The data collected using Remark Office OMR provided that proof. It was shown over the ensuing years that the students involved with the intervention improved their ability to read, even enough that the students needed no further intervention. Through other tests given in the later years (grades 3, 4, and 5) of the students in the intervention, these students maintained their level of ability or improved through these grades. This was the proof needed to show the superintendent and the central office that this intervention was worth the time and money spent. Student academic performance improved, and the data documented this improvement. It was through the use of Remark Office OMR to collect the initial and ensuing data that the Office of Accountability was able to provide this data on the individual students.
In my next article, I will discuss a “Remarkable Addition to Our Data Collection” efforts.
Thomas Hays is a retired educator who served a 50,000 student school district in Maryland as assistant supervisor for accountability and data coordinator for compensatory education from 1996 through June 2007.
Download the sample form used in this article.
Institutional Research - Check out how Savannah College of Art and Design handles an internally developed survey given annually for 2500 respondents using Remark Office OMR and Remark Web Survey.
Institutional Effectiveness - Find out how Susan Evans was able to collect over 2000 responses and analyze data over a 9 day period using Remark Office OMR and Remark Web Survey.
Quality Assurance Program - St. Patrick’s College was undergoing an important quality assurance program involving both internal and external evaluators. Learn how they used Remark Office OMR to process their data.
Using Remark Office OMR for Course and Instructor Evaluations- Find out how Tiffany Martinez at the University of New Haven uses Remark Office OMR to process course evaluations in a timely, cost-effective manner.
Croatian University Uses Remark to Grade Thousands of Tests - Find out how the University of Rjeka corrects 50,000 tests every three weeks using Remark Office OMR.
Evaluate Impact of Professional Development Programs - Learn how Rebekah Nix was able to replace using transparent answer keys and reduce costs with Remark Office OMR.
Grading Biology Exams at a Large State University - See how Michael Wagenheim used Remark Office OMR to scan over 12,000 multiple choice biology tests per year and save money.
No such thing as a perfect form - Margaret Rutherford at Victoria ISD using Remark Office OMR to collect data for a grant.
Grading Assessments for Title IIB Grants - Learn how Capital Area Intermediate Assessments using Remark Office OMR to collect and analyze data from a variety of assessments, surveys, and evaluations under very tight deadlines.
Collect and Analyze Test Data for Title I - See how Tom Hays reduced the amount of time to collect data for Title I using Remark Office OMR.
Remarkable Changes to Instruction - After Tom Hays proved the feasibility and usability of Remark Office OMR for collecting data, his next challenge was to improve instruction in the elementary schools.
Remarkable Addition to Data Collection - Tom Hays discusses the remarkable addition to the data collection efforts: Evaluation of the kindergarten program through two assessments.
The Beginning of a Remarkable Relationship - The beginning of Tom Hays with Remark Office OMR.
Scanning Technology Conquers Common Testing Dilemmas - See how Dr. Bill Palya, Professor of Psychology at Jacksonville State University used Remark Classic OMR to enhance teaching and learning by enabling more frequent testing, improving test result analysis, and decreasing a student's ability to cheat.
How a Leading Hospital Uses Remark Office OMR® for Regulatory Compliance - Stephanie P provides overall management for the Patient Care Services Performance Improvement program that includes seven Quality Indicators. Patient data is analyzed and benchmarked against other hospitals. Reports are then provided to internal management and outside regulatory agencies.
Gravic's Contribution to Volunteerism - The challenge was to collect data from thousands of evaluation forms completed by participants in the Presidential Summit on Volunteerism held in Philadelphia.
Little League Assessments - Creating fair and balanced little league teams is a very complex process. Glenmoore Eagle Little League strives to create teams with a 50/50 win-to-loss ratio, making the season enjoyable for all the children and parents. In the past this goal was achieved by manually entering talent ratings of every child into a spreadsheet. This was a painstakingly slow process, so three years ago they started using Remark Office OMR® to scan and process player evaluation forms saving an enormous amount of time and effort for the volunteers who perform these functions.
Football Pool - The End Zone Bar & Grill is a sports bar that runs both a weekly and an annual NFL football pool. Employees had to manually check the weekly sheets, then sort and enter data from these sheets to keep a tally for the yearly pool. This process was enormously time-consuming, as there were 500-600 sheets per week to be processed.
