Evaluating modern policy programmes
Effective government develops and implements policy in response to complex issues through instruments, such as laws, advocacy, monetary flows, programmes and initiatives. Over time developing and implementing effective policy has become increasingly difficult as a result of three factors:
- The obvious responses have been implemented - the most obvious policy responses that were readily achievable, did not require a lot of effort, and/or delivered substantial benefits, have largely been implemented; meaning that government must explore innovative policy responses to achieve further outcomes.
- Increasing socio-economic complexity – a more sophisticated economy, with increasing integration and inter-connectedness of political, structural, economic and social systems means that design of effective policy responses must conceive and respond to this complexity.
- Funding constraints – the current fiscal constraints means policy owners need to achieve outcomes in a more cost effective manner.
In response to these factors, government is beginning to adopt avant-garde, innovative and sometimes untested policy responses, with perhaps the most high profile example being the plain-packaging of tobacco products. These responses are also being enacted through new service delivery models, including engagement with citizens through digital and social channels, and partnering outside of government.
Seeking to deliver better outcomes through applying untested policy through innovative and often experimental delivery models means accepting greater exposure to policy risk. The key risks are that the response could fail, have unintended consequences or deliver adverse outcomes when subjected to real-world dynamics and human influence.
RSM’s systems approach to programme evaluation has been developed to help policy owners manage this increased exposure to policy risk.
Why the traditional approach no longer works
Programme evaluation has long been used as a tool for validating a policy programme in terms of its efficiency, effectiveness, and its relevance and appropriateness to the issue at hand. Evaluation plays a critical role in developing and implementing government programmes.
The best programme evaluations, regardless of scale, help strengthen public sector efficiency, effectiveness and accountability by:
- Acknowledging and testing for the inherent risks in the policy programme
- Strengthening resource allocation, planning and decision making across policy priorities
- Improving the selection and use of service model alternatives
While never ideal, the traditional approach to programme evaluation is ill suited to the contemporary policy environment for a number of reasons, including:
- Evaluation thinking that starts too late - The traditional approach often engages an evaluator after programme design has been completed and initial implementation has already commenced. In the contemporary context, planning for an evaluation should commence during programme design, and where possible, baseline data should be collected prior to programme implementation to allow earlier assessment of outcome alignment.
- Evaluations are often poorly focused - Traditional evaluations are often viewed as ‘compliance activities’ rather than an essential tool to manage the inherent risk in contemporary policy programmes.
- Evaluations confuse outputs with outcomes - Traditional evaluations often focus too much on inputs and outputs of a programme rather than the outcomes delivered.
- A failure to consider the complexity of third-party delivery - Increasingly, policy is being delivered through third-party service providers, adding new complexity that is often overlooked in the design of traditional programme evaluations.
- Evaluations finish too early - Real-world dynamics and human influence introduce risk and uncertainty to the delivery of programme outcomes. By concluding an evaluation prior to the completion of a contemporary programme, the policy owner risks losing sight of these factors and their impact on delivery of outcomes.
Evaluation plays a critical role in developing and implementing government programmes.
Our contemporary approach
To reduce the risk of contemporary policy responses, programme evaluation must use a systems approach implemented holistically across all phases of the policy programme.
At their core, policy programmes affect systems: collections of interconnected and interrelated elements that move or work together (directly or indirectly) to achieve a common goal.
Systems have their own dynamics, and to evaluate a programme, its impact, its success and its issues, it is necessary to consider the system as a whole.
Our approach builds a thorough understanding of a programme as well as the system that surrounds it, to ensure the outcomes are robust, pragmatic, practical and ultimately sustainable. Our systems approach asks participants to think more broadly by engaging evaluators with diverse expertise including portfolio experts, economists, actuaries, process engineers and psychologists.
RSM appreciates internal capability, our engagement is collaborative and encourages the evaluator to work closely with policy designers, third-party organisations and industry experts to help plan and execute the evaluation, and interpret the results.
Our approach is applied throughout the entire programme lifecycle, taking a holistic view, and is applied across policy programme planning and delivery:
Our approach focuses on outcomes – programmes should have clear outcomes, with the evaluation used to validate whether or not the programme is delivering to its intended outcomes. This information is used along with that developed in the earlier phases of the evaluation, to refine or redesign the policy programme.
Programme planning – allows the evaluator to assist the programme by: assessing approaches to its establishment, the appropriateness of governance and arrangements to monitor performance; forecasting programme performance; and providing suggestions on how it can deliver the intended outcomes. Incorporating ‘evaluation thinking’ into the programme from an early stage will enhance programme development and provide greater potential for evaluation findings to be used to continually improve and refine programme delivery.
Programme budgeting – allows the evaluator to investigate how an agency uses funds in delivery of the programme and its outcomes. Allocation of funding is an agency’s primary instrument for driving policy programmes and choices made with regard to this allocation: magnitude of funding, to whom and how it is allocated. This can have a significant effect on the programme outcomes and are important elements to consider in an effective evaluation.
Programme purchasing – involves assessing the process used to procure and establish the delivery function. This process is assessed against performance-based contracting approaches and provides guidance as to the roles and responsibilities of each key supplier. The process is also assessed against any specific requirements in relation to the manner in which services are to be provided, while still providing suppliers with sufficient autonomy to perform their roles as effectively and efficiently as possible.
Programme outputs – focuses on assessing how effectively and efficiently the programme has been implemented and is delivering the types of services or goods, outputs, required to achieve the policy’s objectives or outcomes. The evaluation of outputs seeks to ensure the right types of good and services are being produced, and in the right quantities; and improve programme delivery and methods.
Programme use – focuses on how citizens and clients are utilising the goods and services provided, whether there are barriers to accessing these services. Programme use also assesses the effectiveness and efficiency with which suppliers are delivering the outcomes of the programme.
RSM’s systems approach to programme evaluation
Improving evaluations after programme commencement
Although programme evaluation is most effective when embedded within the policy programme from inception we recognise that there are circumstances where external realities may inhibit a programme’s ability to do so.
Recognising this, RSM has developed a hybrid approach that can be applied to deliver the most effective programme evaluation in this circumstance. This hybrid approach allows us to quickly work with the programme to develop a clear understanding of the programme logic, objectives and outcomes. Concurrently, we identify and collect proxy data that may be used as a baseline for the programme and measure its performance over time, and identify the ways in which the programme impacts and is impacted by the systems around it.
RSM evaluation case studies
Evaluation of a national social reform programme
Members of the RSM team were appointed by the Minister to conduct an independent evaluation of the Commonwealth’s investment in the Job Network:
- a government-funded network of organisations contracted to deliver employment services to unemployed jobseekers on government income support payments.
A key component of the evaluation was establishing an evaluation framework that would allow the Commonwealth’s return (the public value created) from the investment in the Job Network to be calculated and tested. The framework was applied to analyse the policy’s performance over the preceding years. In addition, a national tour was conducted to discuss people’s experiences with the Job Network’s operational implementation. The team interviewed unemployed people, employment services providers and local business leaders in a number of regional centres across Australia.
The evaluation identified a number of significant issues that had led to a decline in the programme’s value over time, and analysed their root causes. In response, the evaluation recommended major change to the way the Job Network was structured, and how employment service providers were engaged and remunerated under the policy.
Evaluation of a customer service reform programme
Members of the RSM team led an evaluation of the outcomes achieved through its investment in the Easier, Cheaper and More Personalised (ECMP) programme at the Australian Taxation Office (ATO).
The evaluation team established an objective evaluation framework that would allow the team to determine the Commonwealth’s return (the public value created) on the investment in the reform. As the reform was focused on making the ATO far easier to work with from the perspective of taxpayers and tax agents, the evaluation framework was designed to test the real outcomes achieved by the ATO within the tax community and involved assessing and testing the change using a wide range of KPIs nationally.
Evaluation of a national health care programme
Members of the RSM team worked with the Department of Health to design the approach for evaluating the Australian Nurse Family Partnership Programme (ANFPP). This nurse-led home visiting programme has been conceived to support women pregnant with Aboriginal and/or Torres Strait Islander children; with the policy outcome being to improve their own health and the health of their babies. The ANFPP also provides valuable support and advice to mothers that helps with the baby’s development in their early years.
Given the innovative nature of the policy programme, and the fact that it was being implemented by third-parties (nurses), a contemporary evaluation approach was required. Drawing on better practice, our team members used a ‘systems based’ approach and designed a holistic evaluation framework for the ANFPP and then helped apply the framework to conduct a formative evaluation of the programme on- the-ground.
Contemporary policy owners need information about and the effect on the system.
Beyond our contemporary approach there are three key reasons why RSM is a great choice to help you succeed in evaluating complex contemporary programmes.
- RSM’s Australia-wide insights - National programme evaluations should leverage the knowledge and experience of people who have spent time living and working in the different regions across Australia. We bring regional coverage and understanding of the regional context through our network of 29 offices, across metropolitan and regional Australia.
- Global better practice - Our services are backed-up by our network of 732 offices across 112 countries, enabling clients to enjoy access to better practice, insight and expertise globally.
- Deep sector expertise - Our one-firm structure enables our clients to access industry experts across the country. Additionally, RSM has established alliances with industry experts to provide subject matter expertise when required.
If you are interested in having a conversation about programme evaluation, please contact one of our experts.