Do our teachers implement our curriculum with fidelity?
With changes in the Head Start Performance Standards and increased focus on implementing a programs chosen curriculum with fidelity, ABC Head Start decided to start by assessing how faithfully teachers were using the curriculum. Since Creative Curriculum has a tool designed to do just this, they sent their education support staff to check on curriculum fidelity. While aggregating the data from this monitoring activity, it became clear there was a problem. While some classrooms showed a very strong fidelity in implementing the curriculum, others showed extremely low fidelity that seemed inconsistent with what they knew about their classrooms.
Upon further examination, they determined that the majority of the classrooms, 11 of 16 had not been properly monitored. For some classrooms, the observer did not complete whole sections of the tool. For other classrooms, the observer only partially answered the questions.
Analysis
They created an Excel spreadsheet to look at the fidelity scores of individual classrooms as well as averages of the whole program divided by program type (Head Start and Early Head Start). They also compared the scores of subsections. In this way, they hoped to identify both overall fidelity and specific areas of curriculum implementation that might benefit from additional training. This were also hoping to identify if specific classrooms needed additional support.
For examining and aggregating the data, they created a worksheet in Excel for each classroom to enter the fidelity checklist information. They used formulas in Excel to calculate the fidelity scores as indicated in the checklist instructions. Then scores were aggregated across worksheets to graph both classroom scores and agency wide averages in one graph .
When they looked at classroom level data, it was striking to see that some classrooms were implementing whole sections of the data with 0% fidelity. This pattern of low scores across many classrooms in part I of the tool led them to question the validity of the data. The highlighted cells in the chart below are show the scores which caused them to look more closely at the accuracy of our data.
After talking with those who conducted the observations, they learned that the majority of observers didn’t answer whole sections of the tool and so when the data was entered, the data entry staff were told to mark blank sections as no, causing abnormally low scores across most classrooms and the agency as a whole.
Classroom level and Agency average scores
Result
While ABC Head Start hoped to find out how the teachers were implementing the curriculum across the agency, they found something quite different. Their implementation of the monitoring activity and the gathering of data was poorly executed. Several of the observers were not shown how to use the tool and mistakenly thought only some sections needed to be completed. They also found the whole process of gathering the data on a paper tool by one person and then having another staff dedicated to entering the data increased the likelihood of confusion and error.
To address these issues they created an Excel file where observers can directly enter the answers from their observations. The sheets are designed to calculate all scoring for the tool based on the yes or no answers selected by the observer. The file has been shared with those who will be doing the observations and training has been provided to them on how to use the tool. A section of the tool is displayed below and the full template file can be downloaded here.
The scores and graphs automatically calculate based on the answers selected for each question. The answers are limited to y or n from a drop down menu and only cells that are designed to be answered, are unlocked. The program will begin implementing this new method of monitoring curriculum fidelity in October. They hope it will give them a better idea of how teachers are implementing the curriculum, and thus provide essential data for providing support and training to their teachers.
Additional Details
Audience: Leadership, management
Level of Analysis: Classroom, grantee
Difficulty: Moderate
Content Area: Program management, education and child development
Data Sources: Teaching Strategies Gold, Excel
Technical Appendix
Download the Excel template here.
Step 1: For classroom level data, ABC Head Start used the insert graph feature in Excel. They highlighted the pertinent cells (in blue and lavender below) and used percentages and fidelity ratings for data call outs.
Step 2: For agency level data, they created a graph in the same way but used data from each classroom as well as average value across classrooms to compare side by side.
Questions about this use case? Email them to analytics@nhsa.org and we’ll get in touch with the author for you!