To assess how well each Solution Option you are considering meets your stated Evaluation Criteria, you will need common measures to evaluate the options against each criterion. For example, if the criterion relates to improving performance on a standardized test, the evaluation measure might be “score on standardized test” or “gain in score on standardized test since last Academic Year.”

DecisionMaker® suggests ways to evaluate Solution Options against many Evaluation Criteria here » but, if you need additional ideas, the resources listed below may be helpful in finding existing data or collecting your own data.

Some examples of existing information you can gather to evaluate Solution Options include:

• Efficacy studies or process evaluations published in peer-reviewed journals or other publications
• Information from the program vendor
• Data collected on outcomes in places where the Solution Options have been implemented previously, either in your own context or in other contexts

You may also choose to collect your own information by:

• Piloting Solution Options and surveying teachers or students for feedback
• Collecting data on outcomes
• Holding focus groups with stakeholders to understand buy-in

In some cases, decision-makers may want to use available evidence to evaluate Solution Options against Evaluation Criteria. However, decision-makers may struggle to determine whether the available evidence is relevant or even appropriate to use in their context.

Additionally, in the particular case of assessing evidence of effectiveness, prior studies may exist, but these studies are likely to vary in quality or credibility. Decision-makers may want to consider the results of a less rigorous study to assess whether a Solution Option is effective, but may want to account for the fact that the study could be over- or under- estimating the size of the impact.

To account for these issues, DecisionMaker® provides two optional indices to evaluate existing evidence: a Relevance Index and a Credibility Index.

The Relevance Index helps decision-makers determine how well a study’s findings apply to their own setting by assessing the extent to which various characteristics of the study population and context are similar to the decision-maker’s own context. The Credibility Index helps decision-makers assess how seriously to take the findings of a study by scoring it on the quality of study design. Click the Relevance and Credibility Indices in your preferred format below.

• RI/CI MS Word Template
• RI/CI PDF Template

The RI and CI template is also available as an Excel workbook that calculates the relevance and credibility indices automatically. For a more detailed explanation of the calculations, technical guidance is available. Click below to download the Excel template.

• RI/CI Excel Template
• RI/CI Excel Technical Documentation

We have created a four-part video tutorial to help practitioners use the Relevance and Credibility Indices. See the Relevance and Credibility Index Tutorials page in DecisionMaker® to learn more.

Once you have identified the measures you will use to assess each Solution Option against each Evaluation Criterion and have the data you need to do so in hand, you will need to assign values on each measure for each Solution Option and enter these values into DecisionMaker®.

Data must be collected in a numerical form in order for DecisionMaker® to calculate a utility value, e.g., points on a 0-10 scale, percentage points, hours per week, days per year, or rubric scores. If you have qualitative data, for example, rubric ratings of “Meets grammar standards to the highest expected level,” “Meets most grammar standards at an acceptable level,” etc., you will need to assign a numerical score to each rubric rating.

DecisionMaker® uses your expectations about likely best and worst case scenarios with respect to each evaluation measure to establish the high and low bounds of utility. The best case scenario you provide is set at 10 out of 10 for utility, assuming that you will be totally satisfied if a Solution Option performs at this level. Similarly, the worst case scenario is set at 0 out of 10 for utility, assuming you will be very dissatisfied with a Solution Option that performs at the lowest likely level. For example, if the evaluation measure is a test of reading comprehension scored between 0 and 40, you would enter the lowest score you think a student in the decision context might feasibly earn, and the highest score. If the population you are trying to serve is struggling readers, you might enter a likely lowest score of 10 and a likely highest score of 25. If they are your advanced readers, the range might be 30-40. These ranges might be based on your past experiences reviewing data from this test, or based on benchmarks provided by the test’s developer/vendor.

For some evaluation measures such as test scores, graduation rate, or time on task, you are likely to prefer higher values. For others, such as days of absence, dropout rate, or suspensions, you are likely to prefer lower values. DecisionMaker® asks you to indicate whether higher values are better for each Evaluation Criterion in order to establish whether utility increases for you and your stakeholders as the values go up or down.

The free, online resources listed below can be used to help find existing evidence on educational programs, strategies and tools, or to help you produce new evidence to evaluate your Solution Options. Using Evidence to Strengthen Education Investments » is a useful introduction to thinking about evaluating educational programs and strategies.

We use the ingredients method to estimate the costs of educational programs, strategies or tools (see Levin, 1975; Levin, 1983; Levin & McEwan, 2001; or Levin, McEwan, Belfield, Bowden, & Shand, 2018, for full details on this method). This method defines costs as the resource requirements for implementing a program, strategy or tool (i.e., a Solution Option), regardless of how they are budgeted or financed. These may include personnel time such as teacher and school administrator time, training, facilities, materials, technology and other equipment, services, and other resources.

You may also consider enumerating costs by only considering new expenditures. However, such an analysis may neglect accounting for important resource demands such as the reallocation of teacher and administrator time.

For example, expenditures for a new program might include items such as the initial costs of training staff, and any curriculum or additional materials you need to purchase in order to implement the program. However, to understand the full costs of implementing the program, you should consider how teacher and administrator time is being used now, and how that will change when implementing the new program. If one program is very time-consuming for teachers to implement, and another program is not, that may be an important consideration for you, in addition to expenditures. However, if primary concerns for key decision-makers are the additional start-up costs and new demands on the school/district budget, then considering expenditures only may be useful.

*Note that if you are only focused on expenditures, then you will be producing expenditure-utility ratios rather than cost-utility ratios in the analysis.

Example:

Below is a table comparing implementation costs vs. expenditures for a school that is considering adopting a social-emotional learning (SEL) program. The school considered two SEL curricula, RULER and Character First. A third Solution Option considered was maintaining the status quo, which involved students attending advisory sessions each day.

As you can see from the table, maintaining the status quo required no new expenditures, while Character First required over $1,300 for training and curriculum materials, and RULER required almost $11,000, primarily because the training is more intensive and involves travel.

While the expenditure amounts for the two SEL programs were not large, the amount of personnel time involved in implementing an SEL curriculum is quite considerable, especially in the first year. As shown in the table, this amounts to over $100,000 in personnel time (i.e., salary and fringe benefits for the number of hours collectively spent by personnel on this activity). SEL curricula tend to encourage developing a core implementation team and getting the entire community involved in understanding and practicing the principles taught by the curriculum. If personnel are spending time on these SEL activities, they must be spending less time on other school activities (or working longer hours!). As a result, considering the personnel time involved in implementing each program may be important for this particular decision. Note, however, that the status quo of running advisory sessions is also using teacher and administrator time, albeit in different ways. Costs of personnel time for the first year of implementing RULER are almost $10,000 more than for implementing advisory, but the school would actually save $7,400 in personnel time in the first year of implementing Character First.

When enumerating costs associated with personnel time, you should always consider how resource requirements will change relative to business as usual, as the status quo is also utilizing important resources in another way. You can see that implementing advisory already demands a lot of personnel time. RULER requires even more time, at least in the first year because it involves a dedicated implementation team and intensive initial training, while Character First uses fewer personnel resources, partly because the availability of sample lesson plans reduces the amount of time teachers need to prepare their own lessons.

+ Tools and Guidance on Cost Estimation Methods

• E$timator » A free, online tool to help estimate the costs and cost-effectiveness of educational or other social programs.
• Cost Analysis in Practice (CAP) Project » Free tools, guidance and one-on-one technical assistance (funded by Institute of Education Sciences, U.S. Department of Education) on conducting cost analysis of educational programs.

Guidance on cost estimation methods:

• The Critical Importance of Costs for Education Decisions » (2017)
• Using Cost Analysis to Inform Decisions About Professional Learning » (Webinar recording, 2018)
• Guiding the Development and Use of Cost-Effectiveness Analysis in Education » (2015)

Reports and articles illustrating cost analyses and cost-effectiveness analyses of educational programs by researchers who have rigorously estimated costs:

• A Benefit-Cost Analysis of a Long-Term Intervention on Social and Emotional Learning in Compulsory School » (2017)
• Cost-Effectiveness Analysis of Early Reading Programs: A Demonstration With Recommendations for Future Research » (2015)
• Examining Systems of Student Support » (2018)
• Leveraging Volunteers: An Experimental Evaluation of a Tutoring Program for Struggling Readers » (2016)
• Reach Cost Analysis » (2018)

These values should reflect how well each Solution Option meets your stakeholders’ criteria and, consequently, how satisfied they will be with this Solution Option. A Solution Option with an overall utility value of 10 would, in theory, meet your stakeholders’ criteria perfectly and a Solution Option scoring 0 would not meet their Evaluation Criteria at all. Clearly, choosing options with higher overall utility values should make your stakeholders happier.

If you want to understand more about how utility is calculated, click here ».

If none of the Solution Options scores very well on overall utility, you may need to think about whether there is time to start over and consider a new set of options. Alternatively, now that you have a good idea about what your stakeholders care about, it may be possible to modify one of the current options to make it more acceptable to stakeholders.

Note that these overall utility values incorporate stakeholders’ Importance Scores which indicate how much they care about each Evaluation Criterion used in the analysis. These scores are rescaled into the Importance Weights shown on the final results page. A bigger weight means stakeholders care more about the issue. It is worth checking that the Solution Option(s) you decide to adopt score(s) well on the criteria that have the highest weights.

DecisionMaker® combines the overall utility value for each Solution Option with the cost estimate you provided to show you a return on investment metric called a cost-utility ratio. This ratio is the cost per unit of stakeholder satisfaction (costs divided by utility value). A low cost-utility ratio indicates high return on investment because it means a Solution Option costs little per unit of stakeholder satisfaction. While, in theory, you should choose the Solution Option with the lowest cost-utility ratio, there may be reasons not to. One reason would be if the utility value is low, as explained above: paying very little does not help if no-one is going to be happy with the Solution Option chosen. Another reason could be that several options fit within your budget so you may decide to choose the one with the highest utility value knowing that you can afford it and make your stakeholders happier. Satisfied stakeholders are more likely to implement the chosen solution with care, which in turns means it is more likely to be successful in achieving your original goals.

We recommend you share the result of the analysis with stakeholders before making a final decision. For example, you can share some or all of the Summary Report which you can download by clicking the “Summary Report” button under the relevant decision flowchart. If stakeholders express dissatisfaction with the Solution Options that show the best results, ask them to think about whether they have other important considerations that they did not initially express. If they do, this may require identifying an additional evaluation criterion and re-evaluating the Solution Options against this new criterion to see whether the overall results change. If the issues they care about were already adequately captured in the Evaluation Criteria, it is likely that the options they favored did not perform well on some evaluation measures. Reviewing the results of the evaluation with them may help them understand the reasons some Solution Options did not fare well in the decision analysis. As with any evaluation, the numerical results should be helpful in promoting discussions based on evidence, but should not make the decision for you.

It is also helpful to see how the results of your analysis might change under different possible circumstances. What if a reading program does not produce test score results as high as you expected? What if the costs of implementing an option double next year? What if you conducted the analysis separately with teachers and administrators? Would the same Solution Option come out on top in each case or would different ones rise to the top? If the same Solution Option performs best under different scenarios, you can be quite confident it is a good choice. If not, think carefully about how likely it is that circumstances change in a way that could lead to less-than-desirable outcomes.

At some point after you have implemented one or more Solution Options, revisit the analysis to see whether, in practice, each option has performed as well as expected when it was first selected. You can update the data in the evaluation measures and cost tables to see how the options rank now with up-to-date performance data. This will help you decide whether to continue with the same Solution Options going forward, or whether you might need to change focus. It is also possible that stakeholders change over time or that new factors may need to be considered in the decision, for example, if a new regulation is imposed or if the composition of the student population changes. The analysis can be modified iteratively to ensure that you are continuously re-evaluating the options in light of changing conditions.

DecisionMaker® uses the data you entered in the Evaluation Measures table to calculate utility. The overall utility value earned by a Solution Option is the sum of the utility scores it earns on each of the Evaluation Criteria (Criterion-level unweighted utility values) multiplied by the importance weights assigned by stakeholders to the criteria.

Example:

Suppose you, as the principal of Everglades Elementary School, need to identify an educational strategy to help K-3 students improve reading comprehension. You are deliberating between two reading programs A and B. You invite your school’s Reading Specialists and all the K-3 teachers to participate in the decision as stakeholders. As a team, they develop a list of three Evaluation Criteria and assign Importance Scores to each of them. Your AP helps you identify an evaluation method for each Evaluation Criterion and collect data to assess how well Program A and Program B perform against it (see table below).

Each measure you used in your evaluation is rescaled to convert your results to a common utility scale with a minimum of 0 and a maximum of 10. The likely lowest score and the likely highest score you entered for each measure are used to set the extremes of the scale, and a straight line connects the two points. This assumes that utility changes in direct proportion to the changes in the evaluation measure.

When the rating/score on an evaluation measure is positively associated with the utility values (i.e., higher scores are better), the likely lowest score you entered is assumed to provide 0 utility and the likely highest score you entered is assumed to provide a utility value of 10. The criterion-level unweighted utility value for a Solution Option is:

When the rating/score on an evaluation measure is negatively associated with the utility values (i.e., lower scores are better), the likely lowest score you entered is assumed to provide a utility value of 10 while the likely highest score you entered is now assumed to provide 0 utility. The criterion-level unweighted utility value for a Solution Option is:

Take the criterion-level unweighted utility value of Program A for “evidence of effectiveness in improving reading comprehension” as an example. The evaluation measure is the change in % of students who perform below grade level in reading. Clearly, you want fewer students to be below grade level in reading, so the evaluation measure in this case is negatively associated with utility.

The likely lowest score is -20 (i.e., a 20 percentage point reduction in the number of students who perform below grade level).

The likely highest score is 10 (i.e., a 10 percentage point increase in the number of students who perform below grade level). Lower scores are better. The likely lowest score of -20 is assigned a utility value of 10 and the likely highest score of 10 is assigned a utility value of 0.

A straight line is used to connect the two points, under the assumption that utility changes in direct proportion to the changes in the evaluation measure (see figure below). The score/rating of -10 for Program A is transformed into a utility value of 6.7 using the following formula:

The overall utility value is the sum of the criterion-level utility values multiplied by the importance weights. The overall utility values for Program A and Program B are calculated using the following formulas:

• Overall utility value for Program A = (6.7 *0.17) + (9 * 0.33) + (8.5 * 0.50) = 8.36
• Overall utility value for Program B = (5 * 0.17) + (6 * 0.33) + (5.8 * 0.5) = 5.73

Cost-effectiveness: A Primer » by Henry M. Levin (1983) provides examples of how utility can be derived for educational programs or strategies, and the results combined with cost estimates to provide cost-utility ratios.

• Association of California School Administrators (ACSA) »
• Council of the Great City Schools (CGCS) »
• Council of Chief State School Officers (CCSSO) »

CORE Districts » (California-specific)

Visible Learning: A Synthesis of Over 800 Meta-Analyses Relating to Achievement by John Hattie (2008)

• EdSource »
• Education Week »
• X » (formerly Twitter)