Introduction
Dear Reader,
We are certainly glad you have chosen to peruse and make use of the Data Analytics Playbook. Data, which is just another word for information, is so vitally important in today’s Head Start world. With the charge that we are to be engaged in continuous quality improvement and making data-informed decisions sometimes we even wonder where we should begin. It is truly our hope and intention that the material we have gathered and placed in the Playbook will be helpful to you in that effort.
Certainly, we see this Playbook as just the beginning to a constantly evolving and ever learning effort to help you, your program, and the greater Head Start community make sense of, and use, the important and oft overwhelming volume of information that may be available to you. Whether you are trying to improve child attendance, help more of your families become engaged and achieve self-sufficiency, find out what foods your three-year olds will or won’t eat, or make decisions in regard to what curriculum works best for your population, this resource has been created to help and inspire you with those efforts.
We hope that you will take the time to explore the Playbook and see what your peers are doing and working on in their exploration of this data universe. To that end we have included a variety of project examples, explanations, and rationales, along with the respective program contact info. Since we are a national community and all in this together, we are confident the example programs would welcome a dialogue with you as we all continue to learn and improve together. Not only do we hope you will use this resource, but as you use data in your program and develop your expertise, we encourage you to consider contributing to this resource as well.
So together, let us boldly embark upon this ongoing mission of exploring data and harnessing its power to inform us in making better decisions about our efforts to meaningfully impact and improve the lives of, and outcomes for, the children and families we are missioned to serve.
- The Data Analytics Playbook Team
What are the characteristics of a data culture in Head Start?
The most fundamental characteristic of a data-driven culture is the realization that it is not developed overnight and thus requires patience and perseverance. The work of a Head Start/Early Head Start program is a complex enterprise, which includes many variables and very few well-defined outcomes. It can be a real challenge to focus in on the most important data and then to determine how to appropriately interpret and use it.
A second characteristic is the realization that data is only useful if it is foundational to the organization’s approach to decision-making and their priorities. Data is certainly useful to show evidence of the organization’s present success, but an equally important use is to help inform and guide the organization’s plans and processes for the future. Leaders need to be able to determine priorities and to identify the related data needed to support those priorities. Data is everywhere in the organization. It can be stored on a central server, in the cloud, or on a staff member’s laptop. Estimates are that likely around 85-90% of data in a typical organization is not analyzed at all. Since it would be quite impossible to try to analyze all of this data, an important characteristic is the ability to focus in on the most important work of the Program first.
Organizations with positive data-cultures value access to data and give appropriate access to all employees throughout the organization who need to see that data. From the CEO to the cook, there is data that can be gathered and analyzed to help employees make better decisions about their work. For example, the agency CEO might review overall enrollment data to determine if the Head Start program is successful in its recruiting efforts. A cook at that same program might gather data on food not eaten by the children to better inform future menu choices.
A final and most important characteristic is that the organization has a culture of data literacy development that includes training to promote data literacy throughout the organization. Although an organization may have flashy graphs and data dashboards, if the staff, management, policy council, and board do not understand what they are looking at, then the organization missed the mark. Ongoing training in this area has to be a priority, especially since the field of data science is constantly changing as knowledge, processes, and computing ability rapidly expands.
Lastly, to fully utilize data as a decision-making tool in our continuous improvement process, we must employee staff who have the technical capacity to help us understand and interpret the data. For example, if we have introduced a new curriculum into a classroom as a pilot project and the children’s child development scores improved for that year, we should review the data to determine if it is due to the curriculum, a chance variation, or potentially another variable that may have contributed significantly to the improved scores. It may be very helpful to employ a data scientist to assist with the data review and interpretation.
What are the characteristics of a data leader in Head Start?
In order to be a leader in the use of data in Head Start, a leader should embrace the concepts outlined in the previous section about a data culture and lead with a mindset that uses data for continuous improvement. If the leader is not on board with these concepts and constructs, and does not actively support and promote the use of data, then it is very difficult for the Pprogram to effectively move forward in the use of data.
Certainly, a leader does not have to be an expert in data use and analysis. However, it is critical that she/he work to provide the resources necessary to support the efforts and ensure success. As a program begins this work of developing a data-driven culture, the leader must be willing to make mistakes and to use these learning moments to push forward and to expand the team’s understanding of the data and the implications of this data for their program.
A foundational component of being a data-driven organization is to ask questions, and even a leader who is not an expert in data analysis and visualization can ask questions and seek answers that drive progress and open the doors wide for other staffers to ask questions too.
What is the Data Utilization Cycle?
Presently, there are various approaches to data utilization, management, and analysis, often referred to as the data life cycle. They can vary somewhat by industry or sector, but the following is typically relevant to our field.
Step 1. Plan—The program should consider what questions or issues they wish to address such as child attendance and student performance, teacher attendance and student performance, or curriculum effectiveness to name a few. Based upon those discussions the program will then decide what data is to be gathered, how it will be gathered, where it will come from, how it will be managed, and who will have access to it.
Step 2. Collect—The data will then be collected by a person or a software program in an accessible manner. Often that will be the program’s software program that contains the child and family data, attendance data, child assessment, and/or other data.
Step 3. Inspect—The data needs to be examined to ensure the quality of the data. For example, taking a look at samples of the data, did you find that it was collected according to the data dictionary and data glossary guidelines mentioned in the previous section about data culture? Assuming it was, one can move to step 4. However, if it was not, then it is necessary to clean the data up or to begin again. Decisions are usually worthless if they are made based upon poor, incorrect, or old data.
Step 4. Preserve—Make sure the data is securely stored on a local server, in the cloud, or in a data center.
Step 5. Review—If possible, consider data from different sources that relate to the question, issue, or topic. For example, one set of data related to child assessment may come from teacher observations, another might come from parent reports, and yet another may come from a formal assessment instrument. Does it make sense to try to put those together into one measure or is it best to keep the different data separate?
Step 6. Analyze—As the term suggests, analyze the data to reveal information that can be used in making decisions. For example, data from one subset of classrooms using a particular curriculum might be compared to data from another subset of classrooms using a different curriculum. The first curriculum may seem to produce better results than the second. However, in analyzing the data it is important to make sure there are no significant beginning or demographic differences in the classes that may have contributed to the results. Some obvious differences that might skew the results could be if the ages of the children in the two groups are significantly different, or if one subset of classes has mostly new and inexperienced teachers while the other group has highly rated and experienced teaching staff. Also, one group may be implementing the curriculum with fidelity, whereas the other group has not. In all of these situations, these other factors not specifically related to the curriculum, may be causing the difference in scores. The goal in analyzing is to determine what caused any differences in outcomes to occur. If the program goes through this analyzation process and determines that both subgroups were comparatively similar in the beginning, and the staff were comparatively experienced, and both implemented the different curriculums with fidelity, then they can conclude with some reasonable certainty that the differences in child progress was due to the different curriculums.
Step 7. Decide—As shared in the scenario in step 6, the program can then make a decision based upon good data. They can be reasonably assured that one curriculum is producing better results than the other. To further verify this they can begin anew with the Planning phase as outlined in Step 1.
How do we use data to make decisions?
Since data is simply information, it may be helpful to restate the question as “How do we make decisions using information?” This will serve to take some of the mystery out of the process. With that in mind, the first thing to consider in reviewing data, as well as all types of information, is to ask:
Is it reliable?
Is it valid?
Does it appear to be relevant?
Just as information can be erroneous or bad, data can also be erroneous, bad, or completely irrelevant. In order for us to have the greater likelihood of making good decisions based upon the data we are collecting and analyzing, we must never be afraid of asking questions related to its reliability, validity, and relevance. For example, if the data is said to provide us information about how parents feel about their child’s teacher and we discover the data was collected by having the parent complete a paper survey and then hand it to the very staff person the survey was about, one is certainly justified in raising the issue of whether there may be a problem with the validity of the survey information given the obvious weaknesses in such a collection process.
Unfortunately, there are no hard and fast rules or formulas governing how much data or information is necessary to make a good decision. A general guideline is that if the situation being considered is rather straightforward, then a limited amount of data may suffice, whereas if it is more complex, likely more data is necessary. In all instances, the goal for the management team is striving to understand the situation enough so they can determine what levers can be pulled, or what variables must be manipulated to produce the desired outcome or change. To attain this level of understanding, someone on the team must be the type of person who analyzes situations and tries to understand the possible interactions between the variables or levers. Also, some experience with the subject matter at hand is always a plus and, in many situations, there is a certain amount of trial and error involved in figuring out what elements to examine.
For example, a program may be gathering data to better understand attendance patterns with the goal of improving child attendance. The program may begin by looking at their program-wide attendance percentages. Although that information is likely, valid, reliable, and relevant, it typically does not provide enough information to inform very specific plans of action. A relevant additional question to ask could be, do all sites have the same level of attendance or is does the attendance vary by site? In this case, even after site-level analysis, the program may drill down even further and focus particularly on those children who are most chronically absent. After examining that information and developing a plans of action, they may decide to drill even deeper and look at child attendance by center, by day of the week, and by child. A specific decision to drill to a particular depth in the data often does not follow an exact formula. In reality, it may be the result of someone’s hunch or educated guess about the relationship between elements of the situation. However, the result of continuing to explore and poke around inside the levels of the data, and not being content with the answers found at the previous level, may eventually lead the data explorer to discover some new and very actionable information.
In many respects, using data to make informed decisions is very similar to working on solving a mystery or prospecting for gold. Although the entire process is not guided by a specific formula, it does have some general guidelines and it usually works best when you:
Have engaged and competent staff
Work together to discover tantalizing bits of information
Collaborate to understand how and why these pieces of information may, or may not, relate to each other
In Head Start, we use our data (including essential data from our various monitoring efforts) to improve quality. This is all in service of improving outcomes for children and families. As we continue on our journey of developing, analyzing, and putting data to use, a bit of a warning and caution is in order here. If we, in the Head Start community, truly wish to improve, we must be relentless in our focus upon mining reliable, valid, and relevant data. In order to produce this gem-like quality of data, our staff must completely believe and know we will not use this data in a punitive manner. If our staff do not, or cannot, have this level of trust in the management team they will not support our efforts to gather accurate data, (especially data that may not always look so good on the program in general, or them in particular). Without trust, we will be saddled with a dysfunctional state data collection and use, resulting in management team decisions based upon this invalid, unreliable, and irrelevant data. Such a situation will not help us to improve in the ways we want, and will not produce the improved outcomes for our families and children we desire.
Let us, as a nationwide Head Start community, strive to be transparent. Let us not only celebrate our successes, but also strive to better understand and learn from those situations where we have sometimes fallen a bit short of the mark we were striving for.