| Timestamp | Your Course ID | 1. How can developmental evaluation be more effective than summative or formative evaluations | 2. How can developmental evaluation be more effective in evaluating/monitoring community initiatives? | 2. List the principles you would use to support evaluation and monitoring of community initiatives from an ABCD perspective: | 3. List three methodologies you would use to carry out an evaluation of, or in monitoring, a community initiative that you commission or directly manage: | 4. Identify an upcoming evaluation, or an area of on going monitoring, that you can apply this learning to: | Assignments for each module will be shared with others on your course to share learning and will be visible to administrators, but will not be used for any other purpose. Do you consent to your answers being shared? |
|---|---|---|---|---|---|---|---|
| 16/08/2022 05:28 PM | newcastlejuly22 | Summative assessments can lead to a lack of retention of the information. Formative assessments are time consuming and require expertise that needs to be objective. | This can be more reflective and reactive. It enables a more flexible approach and allows for identification of issues more quickly and enables a more viable solution. | Identify the stakeholders early, what matters to them and include the community in each stage of monitoring and evaluation How will we demonstrate and communicate success Evaluation / evidence base - outputs Utilise information, particularly dynamic feedback regularly Identify impact Map the journey - what happened when | Goal or result orientated Reflective (ongoing) Outcome / impact based (qualitative and quantitative) | I would continue to utilise the reflective method to gain insight from others during process redesign projects related to introducing governance frameworks. | Yes |
| 22/08/2022 09:58 AM | newcastlejuly22 | They are restrictive and not developmental which doesn't encourage mid course correction or celebrate emerging learning. | When used as a journey, recognising that things will change and by ensuring that emerging learning is celebrated alongside people and communities. | Context, Mechanism- identifying opportunities and resources, Outcomes and understanding what types of stories we are wanting to gather, which demonstrate what the communities did for and with each other. | Developmental evaluation, Realist Evaluation, Most significant change model | 3 conversations | Yes |
| 25/08/2022 03:10 PM | newcastlejuly22 | Traditional “top-down” commissioning and funding structures require pre-determined goals, objectives and monitoring arrangements, this approach is at odds with the ABCD ethos of enabling communities / individuals to take control, identify and determine what is required for them and their community. Allowing communities to determine their own priorities and evaluating what works will require a different approach to monitoring and evaluation. A formative approach to evaluating the effectiveness of ABCD initiatives will be required with more emphasis on self-reporting, individual stories, conversations, consultation, increased community initiatives, asking questions, what’s worked, what’s improved, what would you have done differently, collecting examples that demonstrate how residents have come together, used existing strengths and collective actions to make changes, reduced dependency on services. Evidencing the impact and value of ABCD will be difficult when starting without pre-determined goals / outcomes, moving from the constraints of being governed by achieving outcomes within stringent timescales and introducing and allowing the pace of change to be determined by residents / communities will be challenging. | Developmental evaluation facilitates real-time feedback which can be used as part of the continuous development of ABCD principles. Development evaluation supports innovation development in complex system’s which have a large number of interacting and interdependent elements which has no central control e.g: communities. Development evaluation allows changes to be made to meet / support developing needs as identified | • Learning and development • Led by residents • Owned by residents • Evaluated by the residents | • Involve residents • Share / discuss findings as the project develops • Co-discovery, what works, what’s changing and what might be done next | I could apply the above learning to most of the ongoing monitoring / evaluation that I’m responsible for but it would be easier to introduce ABCD principles in the monitoring and evaluation of some new projects rather than those more established ones that already have agreed outcomes etc in place. One new project I’m involved in where I could apply this learning is the commissioning of a community-based initiative “active through football”. The aim of the programme is to improve activity levels, wellbeing, education and employability skills for 16-29 year old’s residing in Walker, one of the most deprived areas in Newcastle. | Yes |
| 26/08/2022 10:09 AM | newcastlejuly22 | Test | Test | Test | Test | Test | Yes |
| 26/08/2022 01:24 PM | newcastlejuly22 | Summative - judges the program at the end of the process, rather than helping to shape during. this may cause a sense of failure to those in the community who have dedicated a great deal of time to the project. Formative - focusing solely on the overall outcome of the work - which again if hasn't gone to plan / has faced large challenges might cause members of the community and participants to feel failure/ disengage further. | this may allow more partnership working when looking to evaluate the project or program. it helps provide more real time feedback to support rather than criticise whilst the project is being designed / gives the option to modify the direction of the project. | Learn by doing Learn from evidence and mistakes What is important to the community members ( also need to consider who the community is / acknowledge diversity within a community) provide creative methods for engagement in order to successfully evaluate Build relationships with participants | Focusing on the impact the piece of work has on the community Focusing on the participation of the evaluation of the community - not just one party ( ie the commissioning team) Observations - watching people or the program you are evaluating. | Drug and Alcohol services - annual contract monitoring - time could be spent observing the people and the staff working in particular services to gain better insight. Hold discussions with people using the service to fully understand the impact that the services within the contract have | Yes |
| 29/08/2022 10:35 PM | newcastlejuly22 | Formative evaluations: what went well, what did not, identify change and improvement. Summative evaluations: benchmarking against others, comparing with metrics, comparing with previous versions or iterations. Both have methods of quantitative data gathering, and outcomes reporting, formal and structured. For community initiatives from and ABCD perspective, no two communities are they same, you cannot compare like for like, there is no 'baseline', or tracking over time. These methods of evaluation don't lend themselves well to ABCD in terms of principles and practices. More interested in a mix of different evaluation processes which are led by citizens, owned by its people. | Developmental evaluation acknowledges the sense of a journey. It can be more effective in evaluating/monitoring community initiatives as it sees the complexity in ABCD and of communities, and celebrates it.It supports real-time learning in complex, ever changing and emerging scenarios. | Genuine participatory - harnessing the power of the 'gift exchange'. Inclusive and respectful of citizens - forming associations and shared visions, strengthening agency and less dependence on institutions. Open minded and independent perspective - strengthening the citizenship (citizens have an allegiance to one another), and building genuine trust. | People, and their stories. Case Studies. Reflection. Interviews. Community mapping - web of collaboration. | Geographical working from locality working with district nursing. | Yes |
| 30/08/2022 08:26 AM | newcastlejuly22 | Because these methods are not developmental, this means they don’t encourage mid-course corrections, they don’t encourage change and they don’t celebrate emergent learning. These evaluation types are thus difficult to use when working in a developing and complex space. | They support real time learning and working alongside citizens and partners. | Identify maximisation of gift exchange. Identify maximisation and deepening of associated life Access the changes in number of participating and coproducing residents and in their citizen power, especially with those who have been marginalised. | Case studies or case comparative methods In-depth interviews Stories of most significant change | This could be applied to the on-going monitoring of Care Homes. | Yes |
| 30/08/2022 08:54 AM | newcastlejuly22 | Tend to be an audit of what happened and don't allow for regular evaluation and opportunity to learn and change. Tend to be done to people rather than with and people might not feel genuinely part of the process. Directed by paymasters looking to seek particular answers. Often very rigid quantitative rather qualitative | Opportunity for people to comment on the level of their participation and ability to affect the process. Supports flexibility in showing how communities benefit. | 1. Maximise gift exchange. 2. Identify and maximise the deepening of associational life. 3. Assess the changes in the number of of participating and coproducing residents placing emphasis on those most marginalized. | Developmental, Realist, Stories of significant change (Case Studies) | Confirm and challenge process - Experts by experience commissioned to provide challenge to ICB commissioning intentions. | Yes |
| 30/08/2022 09:04 AM | newcastlejuly22 | It show that these methods are not developmental, they do not encourage mid course corrections. They do not encourage change or celebrate best practices and new learning. | it demonstrates that working alongside people, partners and organisations benefits the real time working. | associated life changes, gift exchange, Access the changes in number of participating and co-producing residents and in their citizen power, especially with those who have been marginalised. | Case studies or case comparative methods In-depth interviews Stories of most significant change | we could utilise this within the care homes to look at what we have on offer, where we can utilise communities and spaces to help build on better communities. | No |
| 30/08/2022 09:28 AM | newcastlejuly22 | Traditional “top-down” commissioning and funding structures require pre-determined goals, objectives and monitoring arrangements, this approach is at odds with the ABCD ethos of enabling communities / individuals to take control, identify and determine what is required for them and their community. Allowing communities to determine their own priorities and evaluating what works will require a different approach to monitoring and evaluation. A formative approach to evaluating the effectiveness of ABCD initiatives will be required with more emphasis on self-reporting, individual stories, conversations, consultation, increased community initiatives, asking questions, what’s worked, what’s improved, what would you have done differently, collecting examples that demonstrate how residents have come together, used existing strengths and collective actions to make changes, reduced dependency on services. Evidencing the impact and value of ABCD will be difficult when starting without pre-determined goals / outcomes, moving from the constraints of being governed by achieving outcomes within stringent timescales and introducing and allowing the pace of change to be determined by residents / communities will be challenging. | Developmental evaluation facilitates real-time feedback which can be used as part of the continuous development of ABCD principles. Development evaluation supports innovation development in complex system’s which have a large number of interacting and interdependent elements which has no central control e.g: communities. Development evaluation allows changes to be made to meet / support developing needs as identified. | • Learning and development • Led by residents • Owned by residents • Evaluated by the residents | • Involve residents • Share / discuss findings as the project develops • Co-discovery, what works, what’s changing and what might be done next | I could apply the above learning to most of the ongoing monitoring / evaluation that I’m responsible for but it would be easier to introduce ABCD principles in the monitoring and evaluation of some new projects rather than those more established ones that already have agreed outcomes etc in place. One new project I’m involved in where I could apply this learning is the commissioning of a community-based initiative “active through football”. The aim of the programme is to improve activity levels, wellbeing, education and employability skills for 16-29 year old’s residing in Walker, one of the most deprived areas in Newcastle. | Yes |
| 30/08/2022 01:36 PM | newcastlejuly22 | As the methods are not developmental, mid-course corrections are not encouraged as part of the monitoring cycle , they don’t encourage change and they don’t factor in learning opportunities which emerge naturally. | Real time learning is offered, which in turn allows opportunities to capture people's interest before their interest could become disengaged. | Identify maximisation of gift exchange. Identify maximisation and deepening of associated life Evaluate any changes in numbers of those participating | Case studies or case comparative methods In-depth interviews Stories of most significant change | This can be applied to the ongoing monitoring of the newly awarded support for carers service | Yes |
| 31/08/2022 11:25 AM | newcastle-septcomm | test | test | test | test | test | No |
| 21/10/2022 02:24 PM | newcastle-septcomm | Formative evaluations are useful if you want direction on how best to achieve goals, however they rely on everyone having the same needs and can prevent a holistic approach to delivery. They are also time consuming and require lots of data input & tracking. Summative evaluation collects information about outcomes and related processes, strategies, and activities that have led to them. It relies on data outcomes to prove that people have achieved rather than they're individual stories and achievements to get there. | It can support innovation and collect information in 'real time' that can be used to inform and shape a model. It can also help to make real change happen based on the impact and outcomes. | Case studies/testimonials peoples stories pictures and video evidence | Are people less lonely and isolated as a result of our Friendship model Have they overcome personal barriers to friendship by attending a friendship group Do they have a better quality of life and more meaningful connections. | Yes | |
| 23/10/2022 07:26 PM | newcastle-septcomm | Most of this information is collected at the end of a project/initiative and can only influence future activity. It's a top-down approach that tends not to allow for 'real change' to be made as part of project journey, if evaluation were to be done at various stages of the project. Summative and formative evaluations also tend to focus on the numbers and not so much on impact (what real difference is the project making) for examples, people's stories | It can help in providing real time feedback on what's working well and what's not working so well so that changes can be made at the time rather than after a period of activity and reflection. It focuses on people stories and the difference participation has made them, their lives and their families and friends. | Needs to be owned by the community, community works together to identify what's strong and what's not and may need to change. Based on people's real experiences and stories from the community should lead to action which supports the community to have the desire and skills to want to make improvements. | short video clips of people talking about their experiences (what's strong, what's not), case studies demonstrating real impact, focus groups. | People with SMI who are accessing local community activities to help with improvements in wellbeing | Yes |
| 25/10/2022 09:07 AM | newcastle-septcomm | Summative and formative evaluations come from an observational outside looking in perspective - the observations will be noted/reported then taken away to be shown to other outside people who's decisions will be based on the report. It's short term and generally gets shelved, can be largely ignored by the decision makers especially if it doesn't match their thinking | Developmental eval allows all the stakeholders to immerse themselves in the reality of an initiative. It's participatory and invites everyone involved to ask questions making it relevant and interesting to them | Participative and creative approach will enable anyone to be involved Builds on existing positive relationships within the community Enables learning and development, The process has as much value as the end product | REcording stories Case studies Network analysis | I have started developing a theory of change for my organisation and in the early stages of planning an overall evaluation for the whole organisation. We are trying to improve our monitoring systems which are traditional but would still have value being weaved into a development evaluation plan. | Yes |
| 31/10/2022 02:52 PM | hampcommissioning | These are traditional ways of evaluating which are limiting, restrictive, do not allow for much learning and are not owned by the community | This evaluation works with the community and is owned by them; they should be able to relate to the conclusions. The process should have opportunities for learning and development and should be ongoing to allow for changes in direction. | 1. it should be designed by the community, 2. it should measure what is important to the community 3. there should be opportunities for learning along with way 4. it should be useful for the community | Interviews with participants, dairies/video diaries or journals surveys | We will take this approach in developing the work in the King Arthurs area. We are linking up with community connectors and will need to discuss with them how they want to measure outcomes. This could involve "interviews" to identify stories, or video diaries - aswell as the normal counting of attendees and participants. | Yes |
| 01/11/2022 04:12 PM | hampcommissioning | because its difficult to 'quantify' particularly in terms of activity. Really evaluation emotions, cultures, meaningful people interactions, links made (and the power of them). In time there is likely to be a change in some of the NHS beans which are counted | Because it reflects the communities and individuals themselves and reinforces the positive impact they are making - positive cycle. It understands whom the evaluation is about, the context and and the mechanism - and what outcomes, which have been generated by the community themselves | changes in reciprocal gift exchange (perhaps a change of linked events), breadth and depth of association, strengthening citizenship (who are the community now seeing?) | case studies, asset mapping and analysis, participatory learning | Our first funded community builder and community health worker posts | Yes |
| 02/11/2022 10:33 AM | hampcommissioning | Because it doesn't encourage immediate course correction or celebrate emergent learning. | Developmental evaluation encourages communities to work in partnership with decision makers which in turn develops the overall ownership. | Monitoring Evaluation Learning | Developmental Evaluation Realistic Evaluation Most significant change | TBC | No |
| 02/11/2022 12:39 PM | hampcommissioning | • Summative or formative evaluation methods can be problematic when evaluating/ monitoring community initiatives as they tend to be driven by the institution outwards, rather than in a participatory way which is owned by the community. ABCD perspectives require the focus to shift more towards real time learning and monitoring as changes or situations emerge to aid learning and development. • They can be too restrictive in their approach to evaluation, and approach this to audit whether specific goals have been achieved or certain tasks completed. Community based change isn’t linear, which summative and formative methods would struggle to capture. Often, mid-course correction or change is needed to ensure we continue to work towards priorities, goals and aspirations. • Summative and formative methods don’t create the opportunity for any community ownership when driven from the top down. When evaluating from an ABCD perspective it is the communities that will be driving the change they want to see. So, it’s important they are the ones leading the learning and evaluation so that this can actually be used and built upon to take forwards. • Summative and formative evaluations are not agile enough to capture, celebrate or respond to ‘emergent learning’ which occurs in the ‘complex’ spaces created within and by asset-based community development projects. | Developmental evaluation aims to identify middle ground between summative and formative techniques which tend to be objective. Rather, developmental approaches support more adaptive learning in complex or emergent initiatives. The purpose is to aid the give real time learning and feedback to inform continual development throughout community initiatives. To be effective, the evaluator needs to be an embedded member of the community initiative and actively intervening to shape the course of development by helping to inform decision making and facilitate learning. It encourages everyone to be curious and regularly enquiring about successes, failures and unintended outcomes to enhance the overall strategy for making change. | • Identify the maximisation of gift exchange to give insight into the extent to which a community may be getting stronger. • Identifying the maximisation and deepening of associational life • Evaluation of strengthening citizenship, by identifying areas where previously marginalised groups may be now included and contributing their own gifts. • Learning through valuing what goes on between people and the relationships which are formed. As well as encouraging/ supporting participation and contribution from all in the community, to continually learn and share. | 1. Case studies and interviews with key community members 2. Stories of most significant change 3. Network mapping and analysis | Healthier communities work taking place at King Arthurs Way Expression of Interest to Simply Health to run Good Grub Clubs (sustainable food clubs run by and for local communities). Evaluation section included below: We would continue to use the Asset Based Community Development approach for the evaluation of the project. The evaluation process would be participatory – with supporting organisations working alongside to facilitate the evaluation process whilst ensuring that the community felt ‘ownership’ of the evaluation process and shared in its learning and development. The evaluation would be ongoing, starting from the initial sessions in schools, the community events, and continue to include the Good Grub Clubs. The evaluation process would use several methods to capture the agreed outcomes and inform the development of the project. In addition to recording the numbers of participants we would obtain qualitative information which would help develop the project. This would include the use of surveys and interviews with participants to capture ‘stories of most significant change’ to understand values and behaviours which mattered to them – and their views on ‘what works’ and ideas for future directions. These would be recorded and collated. We would also collate stories and case studies from other stakeholders and volunteers. We would make use of tools such as a “wishing tree” for people to records their ideas and feedback. We would work alongside participants in developing the evaluation process and in determining how information is collated, presented, and disseminated. This would include a booklet which would be developed in the first phase of the project. We would produce a summary at each phase of the project with the collated evaluation information and a project summary. Completed by Emma Wilkinson, Emma Horbury, Helen Coleman, Carl Brooks and Immy Colley | Yes |
| 02/11/2022 08:19 PM | hampcommissioning | • Summative or formative evaluation methods can be problematic when evaluating/ monitoring community initiatives as they tend to be driven by the institution outwards, rather than in a participatory way which is owned by the community. ABCD perspectives require the focus to shift more towards real time learning and monitoring as changes or situations emerge to aid learning and development. • They can be too restrictive in their approach to evaluation, and approach this to audit whether specific goals have been achieved or certain tasks completed. Community based change isn’t linear, which summative and formative methods would struggle to capture. Often, mid-course correction or change is needed to ensure we continue to work towards priorities, goals and aspirations. • Summative and formative methods don’t create the opportunity for any community ownership when driven from the top down. When evaluating from an ABCD perspective it is the communities that will be driving the change they want to see. So, it’s important they are the ones leading the learning and evaluation so that this can actually be used and built upon to take forwards. • Summative and formative evaluations are not agile enough to capture, celebrate or respond to ‘emergent learning’ which occurs in the ‘complex’ spaces created within and by asset-based community development projects. | Developmental evaluation aims to identify middle ground between summative and formative techniques which tend to be objective. Rather, developmental approaches support more adaptive learning in complex or emergent initiatives. The purpose is to aid the give real time learning and feedback to inform continual development throughout community initiatives. To be effective, the evaluator needs to be an embedded member of the community initiative and actively intervening to shape the course of development by helping to inform decision making and facilitate learning. It encourages everyone to be curious and regularly enquiring about successes, failures and unintended outcomes to enhance the overall strategy for making change. | • Identify the maximisation of gift exchange to give insight into the extent to which community may be getting stronger. • Identifying the maximisation and deepening of associational life • Evaluation of strengthening citizenship, by identifying areas where previously marginalised groups may be now included and contributing their own gifts. • Learning through valuing what goes on between people and the relationships which are formed. As well as encouraging/ supporting participation and contribution from all in the community, to continually learn and share. | 1. Case studies and interviews with key community members 2. Stories of most significant change 3. Network mapping and analysis | Healthier communities work taking place at King Arthurs Way… Expression of Interest to Simply Health to run Good Grub Clubs (sustainable food clubs run by and for local communities). | Yes |
| 03/11/2022 07:03 AM | hampcommissioning | - These are driven by impact on institutions -They do not involve the community - they can not evolve with changing contexts and emerging projects - they often overlook immediate outcomes for communities & the social value ROI | - owned by the community/participants - real time - reflects continual developments and learning process | - identify the maximize of gift/asset exchange in a community - Associations broaden & deepen - Evaluate Strengthening Citizenship - focuses on learning and sharing not on audit and counting - owned and designed by community - include social value ROI- key for ICP aims/purpose | 1) Network mapping and analysis 2) Stories of Signiant change 2) Participatory learning and reflection methods | apply to community builders who already exist as a priority, embed as mobilization of community health workers , small sparks projects when initiated | Yes |
| 02/12/2022 04:39 PM | newcastle-novcomm | Fiona, Claire, and David Summative is traditionally monitoring against targets, particularly numbers of people etc. Formative is ongoing evaluation which prepares for a final summative approach. There is less flexibility for real time changes in a project as it progresses. | The developmental approach is more constructive is it encompasses flexiblity and allows a project to pivot according to circumstances. This is less problematic from an ABCD perspective, however the feedback needs to come from service user experience linked to ABCD type outcomes. This supports timely decision making and encourages innovation. | Talking with people, looking at direction and distance of travel from individual's perspective, where a person's most significant changes have occurred, how many new things have people tried | Surveys, photos, interviews | We have just implemented a new carers contract, at the moment this focuses on summative evaluation, but we are looking to apply ABCD approach to enable better outcomes for carers. Every quarter we will be involving carers in focus groups, looking to map carers journeys particularly wrt 1:1 work, being able to link with community assets and number of links made. We will then look at the information/ statements and use this to inform the contract going forwards. | Yes |
| 08/12/2022 01:51 PM | newcastle-novcomm | Summative and formative evaluations are not developmental, and they do not allow changes or adjustments throughout the process, based on ongoing learning and feedback, rather focus on being done at the end of the process. | Developmental evaluation is happening throughout the process, allows changes and better engagement with the community as well as adjustments to support better outcomes. | Identify maximisation of gift exchange. Identify maximisation and deepening of associated life Evaluate strengthening citizenship | Case studies and complex case comparative methods In-depth interviews with key informants Network mapping and analysis. | This can be used for development of new supported living settings for people with mental health conditions. | Yes |
| 09/12/2022 04:24 PM | newcastle-novcomm | They are rigid with set outcomes for which progress and people are marked and it go against the principles of ABCD which puts the power in the hands of the community. | It allows for flexibility, adaptation and change as the project progresses | Participation Led Timely Adaptive | Listening Real Life Stories (Case Studies) Observation | A project in the west end of Newcastle we are delivering from Jan 23 to address social isolation and the impact of covid 19 on movement and mobility through the principles of ABCD encouraging and supporting used of community assets such as communal spaces and green space. | Yes |
| 09/12/2022 09:51 PM | newcastle-novcomm | Adopting traditional evaluation methodologies will not enable to capture the outcomes and impact of ABCD projects for a number of reasons. Firstly, they focus on assessing pre-determined and quantified outputs and don't allow to capture change and learning. They follow a linear "mechanical" process and ignore the essence of ABCD and how values and benefits (including those not anticipated at the outset) can be created, enhanced or transformed. Connections in particular would be difficult to capture using summative assessment. There is limited scope for learning and doing so with communities. In addition, such evaluations are top-down, i.e. conducted by professionals, "experts" with little scope to involve stakeholders and communities in the evaluation. Even though beneficiaries and users of services which are evaluated can be asked for their views, they don't take part in the evaluation process itself or its development (agreeing aims, areas/themes, tools etc). Linked to this, it is the organisation acting as the client for the evaluation or the funders which would get to "own" the outcomes, not the community. | Developmental evaluation is not an end in itself, it's a dynamic process which actively involves project actors and participants and follows their journey over a period of time. it is more likely to capture a wider range of soft outcomes including connections and relationship building. It would also enable learning and further community development, capacity building and growth. | Participatory - involve communities and residents in the evaluation process Celebratory - document learning and outcomes through events and materials Equal power - communities will be treated as equal and their knowledge and skills as valued as professional ones | developmental evaluation, realist evaluation, stories of significant change. These could be used for the high street project I currently manage. The significant change methodology will enable discussions with all project stakeholders capturing stories within the communities. This would be a powerful way to document learning and impact but also celebrate the outcomes of the project. It would give residents, shop owners and local groups ownership of the project and its achievements. | I could apply this learning to the High Street project evaluation. Together with key stakeholders, we would discuss and agree the type of evaluation and how to capture change and learning. This would go beyond an audit of quantitative and qualitative outputs and include community development outcomes and also how some other (unplanned) outcomes have been realised. | Yes |
| 12/12/2022 01:17 PM | newcastle-novcomm | Claire, David and Fiona Formative assessments are used continually/frequently for improvement – based on a single event they look at things in the ‘moment’. Process driven and serve as a gauge for improvement. Summative assessments are used at the end of a piece of work and evaluate/summarise multiple events throughout. They gauge progress and benchmarks and measure competency but may have little impact on learning and achievement as they are the ‘end product Both are traditional approaches ABCD is fluid, mistakes are expected, but more so learning. It’s not about the outcomes (these could be difficult to measure), it’s more about the innovation and the impact on individuals – maximising gift exchange e.g. time, experience, community connections, involvement of citizens | This approach is grounded in systems thinking. It supports innovation by collecting/analysing real time data in ways that lead to informed and ongoing decision making as part of the design, development and implementation process. It works well where the path(s) to success are not clear. It’s flexible and responds to changing circumstances – it can balance accountability with learning, reflection and dialogue with decision making and action. It can/does strive to balance expectations | • Identify the maximisation of gift exchange – the more ‘citizens’ contribute their gifts to their neighbours and communities (association), the healthier, safer and more prosperous they will be • Associations broaden and deepen – all ‘gifts’ are welcomed, citizens are allowed space (and time) to form associations, and for those associations to develop. Associations are an integral part of society, showing individuals’ collective interest in a topic or their connection to one another. They are a group of citizens who come together for a reason (other than to make a profit) – voluntary organisations are a good example • Evaluate strengthening citizenship – are we seeing positive and productive action by citizens, collectively, individually (those not previously involved now involved), how neighbours can work together to achieve change? • Sponsors/Funders – not always needed but can be helpful, especially to ensure that the impacts of ABCD are being evaluated – the learning and sharing - not the counting of numbers (of people turning up at events) or the auditing of finances and actions | • Case studies – to support ABCD approach by commissioned service and how this has impacted on unpaid carers • In-depth interviews or surveys with carers linked to ABCD outcomes (transformational evaluation – what do you think has been the most significant change that has occurred for you) Distance travelled tools – e.g., outcome star (based on presenting issues) | New contract with Newcastle Carers – monitoring information under discussion – some, e.g., case studies already included but there is a desire from us and them to make the figures breathe/love – this approach could do this | Yes |
| 12/12/2022 01:33 PM | newcastle-novcomm | Additional from Fiona, David and Claire Formative assessments are used continually/frequently for improvement – based on a single event they look at things in the ‘moment’. Process driven and serve as a gauge for improvement. Summative assessments are used at the end of a piece of work and evaluate/summarise multiple events throughout. They gauge progress and benchmarks and measure competency but may have little impact on learning and achievement as they are the ‘end product Both are traditional approaches ABCD is fluid, mistakes are expected, but more so learning. It’s not about the outcomes (these could be difficult to measure), it’s more about the innovation and the impact on individuals – maximising gift exchange e.g. time, experience, community connections, involvement of citizens | This approach is grounded in systems thinking. It supports innovation by collecting/analysing real time data in ways that lead to informed and ongoing decision making as part of the design, development and implementation process. It works well where the path(s) to success are not clear. It’s flexible and responds to changing circumstances – it can balance accountability with learning, reflection and dialogue with decision making and action. It can/does strive to balance expectations | • Identify the maximisation of gift exchange – the more ‘citizens’ contribute their gifts to their neighbours and communities (association), the healthier, safer and more prosperous they will be • Associations broaden and deepen – all ‘gifts’ are welcomed, citizens are allowed space (and time) to form associations, and for those associations to develop. Associations are an integral part of society, showing individuals’ collective interest in a topic or their connection to one another. They are a group of citizens who come together for a reason (other than to make a profit) – voluntary organisations are a good example • Evaluate strengthening citizenship – are we seeing positive and productive action by citizens, collectively, individually (those not previously involved now involved), how neighbours can work together to achieve change? • Sponsors/Funders – not always needed but can be helpful, especially to ensure that the impacts of ABCD are being evaluated – the learning and sharing - not the counting of numbers (of people turning up at events)) or the auditing of finances and actions | • Case studies – to support ABCD approach by commissioned service and how this has impacted on unpaid carers • In-depth interviews or surveys with carers linked to ABCD outcomes (transformational evaluation – what do you think has been the most significant change that has occurred for you) • Distance travelled tools – e.g outcome star (based on presenting issues) | New contract with Newcastle Carers – monitoring information under discussion – some, eg case studies already included but there is a desire from us and them to make the figures breathe/love – this approach could do this | Yes |
| 12/12/2022 05:36 PM | newcastle-novcomm | The y do not account for mid-course change. | They allow space to view broader benefits of an initiative and bring the people involved closer to the evaluation. | 1. Work collaboratively with stakeholders to gather evidence. 2. Respect the knowledge of people who live in communities. 3. Seek to identify strengths. 4. Commit to shared learning. | We have an upcoming evaluation of some place based work we have undertaken with the West End Foodbank. The evaluation will be undertaken in conjunction with Northumbria University using the Ripple Effect Methodology which I think is aligned to the principles of developmental evaluation. The evaluation will involve a series of conversations with people who use the foodbank as well as volunteers and professionals involved in the project to try and understand the wider impact of the initiative. | Yes | |
| 13/12/2022 01:20 PM | newcastle-novcomm | .Does not lend itself to ABCD asset based . Usually completed by an organisation rather than the people who are in the service. Becomes an audit and is too restrictive. Does not encourage mid course correction. | Is a middle ground and supports that things may change. People are part of the process. People who live in he neighbour hood are part of this and will get some thing from this | Maximisation of gift exchange Associations Broaden and Deepen Evaluate strengthening Citizenship Actively seek out sponsor an re affirm the process principles | Network mapping and analysis Case studies and complex case comparitive methods Stories of most significant change | Introduction and workings of community champion | Yes |
| 29/12/2022 12:46 PM | surreynov22 | ABCD programs are likely to be complex and emergent as different activities change the context in which they operate. Summative and formative evaluations struggle with complexity and emergence. They often try to "isolate" the activities and interventions from other issues to evaluate the impact of those interventions alone - but that's hard to do in the real world where things are interdependent. NICE guideline development groups have had to adapt their methodologies which were focused on Random Controlled Trials and systemic reviews when looking at social care programmes, and have introduced "expert" testimonies including those of people with lived experience into the panels' deliberations and synthesis of evidence and recommendations. Summative evaluations can be overly focused on accountability i.e. did you do what you said you would and achieve what you intended? This can be useful but may reduce the ability to iterate and change what you had intended to do in the light of new information and insights... | They recognise complexity and interdependence and that things will change and emerge. They are focused on learning rather than pure accountability - so it is easier to adapt if something isn't working and new opportunities present. I have used the concept of white water rafting to explain emergence in strategy ... you know where you need to get to (the other side of the rapids - and it's important to do so) however, the weather and recent rain levels will affect the water flow and how much rocks stick out, whether there are submerged logs etc. that create eddies ... so your precise path and techniques needs to adapt and adjust for these. The mechanisms of how you achieved your outcomes (stayed afloat, or rolled, maybe even stayed dry and got to the calm water are affected by the context Developmental evaluations stay the course and are more embedded so people are better aware of the purpose and how of evaluations. This promotes engagement (it could affect what they recall though so need to consider confirmation bias). They are more participative and generate insights that add value to the further actions and decisions around what the community considers to be important. | 1. Maximisation of gift exchange is to be identified - the more people share and reciprocate, the greater the strength of the community - so what exchanges and reciprocity has taken place? What relationships and meaningful (howsoever defined) interactions have occurred? 2. There will be an increase and deepening or associational life - so are there more associations that are locally resident led emerging and more links between existing associations? Has an association of associations developed a shared vision of what is important and needs to change and how people and groups can take actions to achieve this change 3. The numbers of people participating and co-producing, taking action, talking together etc. the stronger the citizenship. 4. It is important to consider the inclusion of people who have been most marginalised 5. Any evaluations and monitoring needs to consider what the people in the community want to learn through the evaluation activity? What information is useful to them? What do they need to know about what it is that has enabled positive things (outcomes) to happen? | 1. Develop a theory of change or logic model (with the community ideally) to make our program theory more explicit i.e. if we do these things, then we will bring about these results - so you can evaluate against a theory or hypothesis 2. Agree the quantitative data we will collect and the qualitative data/ information - and how we will gather this - and triangulate so it's from different sources (e.g. baseline surveys and self reports; specific activities, groups, numbers of people undertaking actions; interviews and case studies to understand how people feel they benefitted and what made things easy or difficult and what changes were made...). Analyse, draw out patterns and interim conclusions 3. Focus groups and workshops with participants to share emerging ideas and test the learning | To some extent, the work SCA does is about convening people/ organisations to identify their priorities and challenges and facilitate changes through activities and influence. For example, we are undertaking a project to co-produce new role descriptions for social care staff and get them evaluated and benchmarked, in order to create evidence for commissioners funding career structures. The aim is to improve recruitment and retention and the perceived status and wellbeing of social care staff ... it's not exactly and geography based community - but how being a member of SCA contributes to people's sense of being part of a community and taking action (rather than just complaining about a lack of parity in pay and esteem) could suit a developmental evaluation. | Yes |
| 30/12/2022 11:03 AM | surreynov22 | Such evaluation methodologies are fixed and formal, not enabling mid-evaluation change. they tend to be an audit of the expected activities (how many people attended) or of the new activity being delivered, instead of considering the more subtle issues around the local citizens being involved in and engaged fully in and with the local community developments. The ABCD approach seems to me to be one of learning as the community develops - i.e. it develops as it occurs with that local community - so we cannot determine up front what the expected outcomes might be in such a simplistic manner, so we cannot determine what metrics we want to measure to show impact. So summative and formative/traditional evaluation methodologies are not appropriate for what the ABCD approach is seeking to achieve - with its primary goal being "to enhance collective citizen visioning and production in local communities" | It gives the local community freedom to determine its own outcomes and learning in a developmental way that can then be evaluated, instead of us setting out process/audit fixed metrics to evaluate inappropriate or not the right things at the start. It responds to change during the developmental process - it is part of the ABCD community development and not a stuck on evaluation. What I personally love about this developmental evaluation approach (and have never considered before) is how you do not know what the impact/benefits might be of a local community change - you leave it open for the community to determine what are the outcomes (new vision) should be. | I could quote the 8 principles of Mr Patton's developmental evaluation talks, but shall not. But I will reflect on what of those and other principles I hope we can bring into the piece of work we plan to support in 2023 with a local community. (1) co-creation - in that the local citizens to determine what they feel are the biggest issues, and thus what is important to them, so they determine the local goal and they determine what changes and impacts they want to work together on - which in turn will determine what outcomes may be measured over time. (2) Evaluation supports and does not hinder/block the local community developments - as such the evaluation is not pre-determined and does not inadvertently direct the course of change action (3) Evaluation that informs a new way of looking at things with our system leaders - that is not the traditional mechanisms - (hence we will need to play a political game and demonstrate the ABCD development approachs in the Stanwell community, that offers different kinds of evaluation outcomes that is not pure numbers (not formative/summative only) but open our leaders minds to locally relevant impact evaluation methods in a developmental way | Developmental evaluation, Most significant change, using local stories from citizens | Stanwell local community development | Yes |
| 09/01/2023 12:23 PM | surreynov22 | Formative and summative evaluations are not encouraging change. They are normally a fixed and formal process looking at the achievement of numbers rather than the outcomes for individuals and communities. They do not foster or support the community but are normally used for reporting from a 'top down' perspective. | Developmental evaluation recognises the complexities of both communities and development. It finds a middle ground between formative and summative evaluations. It allows the community to decide what is important to them so that they can see themselves in the evaluation. A developmental evaluation brings a sense of things changing and is part of the process rather than being a start and end point. | I would want to make sure the evaluation is reflective of what is important to the community I would want to make sure that those leading the evaluation are those who take it forward- this can be facilitated but wouldn't be an external evaluation | I would combine methodologies from a developmental, realist and stories of change evaluations. I will develop a learning framework allowing for a set direction of learning and project development. Different data collection methodologies will be required but as I work across different key neighbourhoods in Surrey I will need to make sure that data can be disaggregated to reflect each community and their development within the evaluation. | I am developing a workstream to build community resilience with relation to emotional and mental wellbeing in Surrey's key neighbourhoods so will apply my learnings to this. | Yes |
| 09/01/2023 03:39 PM | surreynov22 | They don't tend to encourage learning or corrective action as you go along. They also tend to be commissioned by project funders through external evaluators and they generally therefore lack any ownership by those directly affected by the changes. They are normally focused on evaluating the impact of a specific project or change rather than an inclusive overall analysis of a situation or community. | It drives us to make changes as we go along, using evaluation to actually learn about what works and acting accordingly. It helps those overseeing a project or programme understand community-led change takes time - the concept of medium term and long term outcomes (a journey). It considers context and circumstances as a key factor. | - mapping and identifying the strengths, skills, resources available in the community to be able to track how they are shared and maximised - mapping, understanding, and establishing an evaluation plan with local associations to secure local ownership but also to help them deepen their collective work and vision - pursue indicators that showcase sustainable prosperity: citizenship, relationships, participation, civic trust, access to green space, participation in local volunteering, connections between local members of the community - focus on learning and sharing, not auditing and counting | developmental evaluation realist evaluation most significant change model | the ongoing work for community-led town partnerships programme | Yes |
| 10/01/2023 06:44 PM | surreynov22 | ABCD is primarily about enhancing collective citizen visioning and production in local contexts. Evaluation of ABCD activity should therefore reflect and enhance this goal, which speaks to an enduring shift in mindset, relationships and power dynamics (rather than pursuing one-off, time-limited initiatives), and a focus on ongoing learning. Summative evaluation (evaluating ‘results’ at the end of a process) tends to be top-down and suits projects with clearly defined outputs/outcomes, against which success or otherwise can be measured. Formative evaluation helps to inform/improve the approach to be taken for a particular project (typically undertaken during the developmental phase) but again speaks to a one-off initiative and tends to be top-down and owned/led by the entity driving the project. In essence, as well as being top-down, both summative and formative approaches to evaluation relate to time-limited activity and are therefore at odds with the ABCD approach, which advocates a focus on lasting relationships. | Developmental evaluation is designed to support ongoing learning and adaptation through an iterative and embedded approach, which aligns with the desire to enable real-time learning at the heart of the ABCD approach. In turn, this recognises the complex environment in which community development often takes place, and the predominantly emergent nature of themes. This form of evaluation enables measurement/understanding of impact by evaluating the extent to which citizens are engaged in civic life and community-driven action. | To support evaluation and monitoring of ABCD initiatives, you would: 1. Identify and maximise ‘gift exchange’ - The more citizens contribute their gifts to the wellbeing of their neighbours and their community, the healthier, safer and more prosperous all will be. - Evaluating reciprocal exchanges offers insights into the extent to which a community is getting stronger. 2. Identify and maximise associational life - Abundant communities are made up of associations that a) welcome the gifts of all; b) allow sufficient space for diverse associational life to form; and c) facilitate an association of associations to seek a shared vision and work together to produce that vision. 3. Assess and evaluate how/whether citizenship is strengthening - Monitoring changes in the number of participating/co-producing residents and their citizen power, placing particular emphasis on the inclusion of those who have been most marginalised. Working with funders/sponsors who have genuine links and insights into communities can help to ensure these principles are built into evaluation of ABCD work. In addition, evaluation must be: - genuinely participatory; - inclusive and respectful of citizens; - have access to an able local partners as evaluators; and - feature an open-minded and independent perspective. | 1. Case studies and complex case comparative methods 2. Participatory learning and reflection methods, inc. participatory statistics 3. In-depth interviews with key informants (e.g. appreciative enquiry) 4. Stories of significant change 5. Network mapping and analysis | I am part of the project team developing the council’s approach to design and change activity, which will be organisation-wide and encompass key strategic change activity. Wherever possible, I am going to embed ABCD principles and mindset in our approach, which will include elements of monitoring and evaluation. By doing so, we can reframe our understanding of how such work should be set up and carried out: re-calibrating our expectation of residents/communities, their involvement in such work and how we define success. Ultimately, this should translate into better use of public resources in improving outcomes for residents/communities. | Yes |
| 11/01/2023 10:44 AM | surreynov22 | They do not allow change or celebrate emergent learning. The evaluation should be owned by the community. | Developmental finds the middle ground between summative and formative methods. As it's name suggests it is more developmental! It is also more emancipatory and moves away from the control element of evaluation. | I would use the core principles of ABCD 1. Citizen-led 2. Relationship oriented 3. Asset-Based 4. Placed-Based 5. Inclusion focused This means that I would make sure the community owned and developed the evaluation. I would use stories from the community to really understand the impact of the initiative, particularly around how it made people feel. It would also be important to understand the local context of the initiative. | Developmental evaluation, realist evaluation, Most Significant change. | I have struggled to identify a project. Possible LTP4 which is our Local Transport Plan which we engaged with the community on but I am not sure this is a community initiative. Otherwise I could possibly do warm hubs but I don't have a lot of information about this. | Yes |
| 11/01/2023 03:44 PM | surreynov22 | The key problems are around enabling sufficient space for a developmental approach that incorporates learning, review, adaption and remodelling where necessary. | Developmental evaluation enables a adaptive approach that enables initiatives to adapt according to the monitoring and the findings that are identified as strengths and benefits for the initiative. The developmental process allows flexibility not only in the process but also in the ownership from the community. | The principles that support evaluation and monitoring from an ABCD perspective include - understanding the environment, understanding 'what works' - understanding the journey in order to understand the aspects of the initiative that can be reported, listening to what has been achieved as well as the how and why, understanding where the key points of change have occurred and why. Based on the journey to listena nd gather stories that will reflect the outcomes established form the previous points. | Identify the points of most notable change. Review how and why Democratically gather input on points of change Understand the perimeters and measure of the stories that demonstrate the change and learnings. | My interest will be the application of this process on our new enterprise. Although this is more firmly rooted in co production, I am keen to see how to apply a more flexible way of monitoring and reviewing the development of the business led y the consultants experience, learnings and solutions. Due to extreme time constraints - I have kept my response very brief. | Yes |
| 23/03/2023 01:31 PM | surreycommfeb2023 | Completed by Caroline Bedford, Rachel Taylor, Alison Keeley & Mari Roberts-Wood Outcomes can not be meaningful Not appropriate for community settings - very structured Predict outcomes - we might not know the outcomes Happen over a specific time frame, ABCD can take longer to evolve and have an impact Summative & Formative too structured/formal - more difficult to change course mid way through as required | You can change mid course and have a better impact It can be more fluid and take you down paths to a better outcome which you couldn't have predicted before you started Also enables you to bring in to the process parts of a community that you might not have originally known were there - once you are in to the ecology of a community you can go where it takes you More buy in from community = better, longer lasting outcomes | 1. Encouraging/supporting and aiming for an increase in reciprocal exchanges within a community 2.Support associations to be as broad and deep as possible - connections within a community spread and go far and wide and continue to do so 3. Are more people and neighbourhoods coming together - connecting than they were. Are they empowered and developing a 'voice' and being proactive. New projects happening that weren't originally envisaged. 4. Listening to the community voice and ensuring that funding and sponsors do the same. | Participatory Learning Stories of significant change Case Studies | Health based ophthalmology service introduced a few months ago. Opportunity to adapt monitoring approach. Initially was going to be quantitative based but will consider now client feedback, stories of users to enhance the quality of the new service to round off with the traditional stats. | Yes |
| 25/03/2023 08:15 AM | surreycommfeb2023 | They are restrictive as they are not developmental, do not encourage change or emergent learning or allow for mid course evaluation or that processes/ change are not linear | It is more objective, celebrates emergence, looks at opportunities even outside of what was planned in an intervention | Design with community who will use data, sit alongside community as they gather info | videos, case studies, stakeholder feedback | Community led walks in Sandy Hill | Yes |
| 26/03/2023 05:59 PM | surreycommfeb2023 | They focus on auditing what has been achieved against objectives rather than exploring the experience of the community and enabling learning and adjustment during the period of delivery, with a focus on the overarching goal | It can support real time working alongside the community, and is able to flex to be relevant to the range of ways working in diverse communities might evolve, rather than restrict the definition of a positive outcome to something that fits a framework designed in advance (ie without knowing the outcomes that would be achieved) | People exchange gifts (offer each other skills / support), have increased associations (more friends / supportive contacts), and the community is strengthened (more people are involved and connecting with each other, and from more diverse groups) | Participatory self assessment, stories of most significant change, network mapping and analysis | Libraries co-design - working with residents to design an approach to evaluating, including thinking about what would constitute meaningful outcomes. Designing an evaluation approach collaboratively, including factoring in ongoing review of change in relation to level, volume and type of engagement, new contacts / deepened contacts developed, any direct / indirect changes experienced as a result of engagement with the new co-designed services | Yes |
| 27/03/2023 08:58 AM | surreycommfeb2023 | Community initiatives need time to develop. Summative will not allow for learning or building on weakness as it's evaluated at a point in time to meet the project goals. | It allows you to monitor, assess and provide feedback on the development of a project when being designed. Allowing time/space for modifications. | Research, time frames, learning, communication | Goal Based, Process Based, Outcomes based | Reconnections are voluntary sector bought in to Support Discharge at the trust. We could interview patients on their experience as well as looking at thing like readmissions. | Yes |
| 27/03/2023 12:13 PM | surreycommfeb2023 | Group work - submitted by Mari Roberts-Wood | Group work - submitted by Mari Roberts-Wood | Group work - submitted by Mari Roberts-Wood | Group work - submitted by Mari Roberts-Wood | Group work - submitted by Mari Roberts-Wood | Yes |
| 27/03/2023 09:00 PM | surreycommfeb2023 | Summative and formative evaluations can be problematic when evaluating or monitoring community initiatives from an ABCD perspective because they are not developmental they don't encourage mid course correction , change or celebrate emergent learning. They do not lend themselves to working in a complex space where development may not be linear | Developmental evaluation be more effective in evaluating/monitoring community initiatives as it finds a middle ground between summative and formative traditional evaluations. It is developmental in that evaluation is ongoing, which means that feedback can be provided on a continuous basis. It is is particularly appropriate for community initiatives which can be complex and don't develop in a linear way. Because evaluation is ongoing, rather than carried out at specific points, feedback can be provided on a continuous basis. This in turn means that adjustments or mid course corrections can to community initiatives on an ongoing basis. | The principles I would use to support evaluation and monitoring of community initiatives from an ABCD perspective include: 1. Identifying the maximisation of gift exchange , namely the more citizens contribute their gifts, the healthier, safer and more prosperous all will be with the community becoming stronger 2. Associations Broaden and Deepen - Abundant communities are made up of associations that i. welcome gifts ii. allow sufficient space for diverse associational life to form iii. facilitate an association of associations to seek a shared vision 3 Strengthening Citizenship - i. Collectives drive change, ii. neighbours participating. iii. associations coming together 4. Seek out sponsors that affirm the previous three principles remembering that relationships are the primary currency and that most of the things that matter in life are 'treasured and not measured'. | Three methodologies you would use to carry out an evaluation of, or in monitoring, a community initiative that you commission or directly manage include: i. Developmental Evaluation, ii. Realist Evaluation ( Context, mechanism and Outcome) and iii. Significant change or 'the model of most significant change' as it is commonly called. | An upcoming evaluation, or an area of on going monitoring, that I can apply this learning to is the roll out of the Community Link Officer post. This is in order to determine on an ongoing basis from a resident perspective whether these paid posts are fulfilling the Community Animator role as set out in the Asset Based Community Development ( A.B.C.D) touchstones. This consists of by fulfilling a non directive role acting alongside and in support of communities. Ongoing evaluation would help to determine if this role was helping towards delivery of a shared primary goal of enhancing collective citizen visioning and production in local contexts. | Yes |
| 28/03/2023 09:11 AM | surreycommfeb2023 | These forms of evaluation are problematic from an ABCD perspective both in they way they exclude the community itself from shaping the evaluation and setting success criteria, and in the way they actually shape and limit the project itself. Local Authority evaluations of programmes/projects are often based on agendas and policy priorities of the funding organisation and may not consider all the interests, priorities or values of the people involved. Summative and Formative evaluations are often delivered within a short time frame and do not capture long-term, or generational, changes, which may be transformative for the communities involved. Therefore, this type of longitudinal work is much harder to find funding for because it doesn't fit neatly into a 1 or 2 year evaluation timeframe. Increasingly, evaluation criteria is required to be centred around a reduction in 'demand on statutory services' and short-term ROI, which fails to assess the extent to which a project has encouraged sustainable, community-generated change and does little to assess levels of social capital or active citizenship which are both needed to see these reductions on services over the long-term. There is little thought given to the value of work that creates better living experiences for people without immediate reduction on demand for services - community development work can be valuable because it creates more fulfilling lives for people who still require statutory support. They also tend to be very linear - a sort of relentless modernist march towards 'progress' - there is something quite 'colonial' about the way we approach communities and the lives of people. | We talk about co-producing projects and initiatives (though I'm not sure we really know what we're doing most of the time) but the evaluation of such co-produced programmes still tends to be fixed by funders. For example, I've been running an NHSE/DEFRA-funded Green Social Prescribing programme in Surrey for the last couple of years and there's been a lot of emphasis on 'co-design' (bit of an NHS buzz word at the moment) but the evaluation framework that was designed and fixed by DEFRA and an academic collaborative was largely quantitative and focussed on the immediate impact of interventions on individual people. This was doubly frustrating because a huge body of evidence for the value of nature connection on physical and mental health already exists so we didn't need to reinvent the wheel. The evaluation would be more valuable if instead of focusing on individuals it considered the impact of GSP on community connections - is it contributing to the sustainability of a green health eco-system within communities and is it supporting this eco-system to have better relations with the health and social care system so that people are empowered to engage across the statutory and VCSE system depending on their particular experiences at any particular time? To be more effective the evaluation framework must be co-produced by the people involved; it must allow for long-term non-linear changes that create social capital even if they don't produce an immediate ROI. | Co-production/co-design Longitudinal Mixed methods Story-telling - but actually the stories people want to tell and not just the stories that fit into a pre-determined framework Ripple effect mapping - is there evidence of social capital and active citizenship? | Participatory observation - ethnography Story-telling Ripple Effect Mapping | I have a small UKRI grant from The Young Foundation to create community knowledge about perceptions of nature, community and climate change among Muslim women and girls in Surrey so I will use this opportunity to apply this learning. | Yes |
| 28/03/2023 09:20 AM | surreycommfeb2023 | These are traditional and structured evaluation methods which will not recognise or be able to measure the impact of a community centric approach. The community initiatives are about learning and engaging citizens which formal evaluations will not meet. | It recognises that community initiatives require learning and often require midcourse corrections so that the core goal can be maintained. Developmental evaluation enables this flexibility so we can recognise the engagement in community life. | 1. We will identify the maximisation of gift exchange. 2. We will identify the maximisation and deepening of associational life. 3. We will assess the changes in the number of participating and co-producing residents and in their citizen power, placing particular emphasis on the inclusion of those who have been most marginalised. | • Case studies and complex case comparative methods • Participatory learning and reflection methods including participatory statistics. • In-depth interviews with key informants. | How to structure the ICP meetings/plan going forward - use these evaluation methods to understand if it's successful. We also need to be mindful that we have statutory responsibilities we need to to deliver. | Yes |
| 29/09/2023 03:58 PM | northumberlandcommissioning | I am new to commissioning but would feel that summative and formative evaluations can be problematic when it feels like monitoring is about power and control, especially where resources and funding are vital for the programme to continue. I have vast experience of working in a commissioned service and understand the pressure to demonstrate how you have performed against a set of key performance indicators, that may be evidenced based but not necessarily right for our communities. I understand the 3 lenses that were discussed around advocacy, therapeutic and community – I do feel as an organisation that whilst we are striving towards empowering communities, when the complexities around public funding is brought in, then comprises may need to be made, where the system isn’t set up. By starting the community conversation using the guiding principles around what they can do for themselves and what do they need help to do, it will be worth asking what success would look like?, so that communities set their own goals and measure what they value. | Working with groups around how they can measure impact would benefit from some good partnership arrangements. Residents won’t necessarily be aware of datasets that we have available that can be demonstrated using friendly methods. I have previously undertaken work around reducing teenage conception rates and would work with parent groups, who liked to see updated heat maps so that they could start and see work that they were supporting at home, was making a difference in the community. The use of warm data that includes community stories, is a powerful tool. | Work on establishing a partnership that is equal – not a commissioner / provider relationship Encourage communities to look themselves at what could be used to demonstrate success. Engage with small cycles that allow reflection and re-adjustment | Participatory research – I would be keen that we move way beyond tokenistic engagement with communities towards how we create a movement that takes responsibility for their actions and have a vested interest in making projects work. Support small stages of reflection and adjustment to ensure that the future is made and not based on a ‘you do this and that will happen’ model. Demonstrate stories of significant change in a creative way – use of case studies, video diaries, art Review any datasets that communities can be supported to demonstrate impact at a local level | We currently have a pro-active offer for families of children being measured above the 98th centile. We would like to offer a whole school community offer around the time of measurement that will include education, activity and parent engagement. I feel that we can evaluate beyond the measurement of children and look at how we measure their quality of life as a family. | Yes |
| 29/09/2023 04:51 PM | northumberlandcommissioning | They are too formal and do not focus on the concept of ABCD fully. They are more aligned to targets and KPIs and not the difference that has been made | It enables evaluation to journey alongside people and to identify the most significant change, enables gathering of stories and sharing of stories. This approach enables gathering of evaluation information that communities value and want to share | 1. Citizen-led 2. Relationship oriented 3. Asset-Based 4. Placed-Based 5. Inclusion focused | 1. Realist evaluation 2. Most significant change 3. Warm data | Young persons substance misuse service potentially | Yes |
| 29/09/2023 06:06 PM | northumberlandcommissioning | (Jane and Tara- mostly Jane) Doesn't allow for change during the project No scope for ownership of the initiative by the community Not always possible to capture real impact of initiatives for individuals – might not always be clear that the community initiative has generated some change to a person's circumstance or behaviours etc, this may require some reflection and perhaps support needed to pin-point what has been useful. Offers a closed and narrow understanding of any change Narrow scope- looks at whether the project has delivered against aims and wouldn't necessarily pick up on or present any wider impact. | Ensures that the communities are an integral part of evaluation – they own it and can see themselves in the evaluation. Focus is around what has worked and where benefit has been identified – which may not necessarily have been at first considered (so spin off opportunities and impact) A story(stories) are developed/developing which the community identifies with and reflects the initiatives in place and can also inform or naturally move the community towards next steps. | Ownership Involvement Bringing together stories Building of relationship and links across communities | Using women's health hub as an example- Quantitative/audit – need to understand increased number of women addressing health issues via the women's health hubs i.e. cervical screening uptake Qualitative- ongoing participatory learning in-depth interviews and case studies to understand the difference it has made to people's lives and understand new/strengthened connections. Also supports women to recognise the outcomes themselves, empowering them within the community to champion the pathway/process. Also to understand further opportunities. | Women's health hubs | Yes |
| 02/10/2023 11:20 AM | northumberlandcommissioning | Summative and formative evaluations will tend to focus on measuring a pre-agreed set of objectives that measure things such as activity and spend against budgets. They will often be top down processes that are quite rigid and focused on a single area, with limited opportunity to be flexible or incorporate wider learning opportunities. These type of evaluations will often be led by person or group who sit outside of the community and may not have the capacity to recognise the need for community involvement. | By ensuring the community can participate in the development and delivery of the evaluation. By ensuring the process is respectful and mindful of the every day life and work of the local community. By ensuring the evaluation puts learning at the centre of it's aims and objectives. | 1. Include 'gift exchange' ands the resulting impact in the evaluation. 2. Identify what the shared vision is between local people and organisations. 3. Evaluate where citizens and organisations are at the forefront of driving change within their own communities. 4. Keep the focus on learning and sharing not counting and auditing. | 1. Interviews and 'listening' sessions with community. 2. Personal logs and diaries of people directly involved. 3. Observational work where evaluator attends the initiative. | We have ongoing monitoring and evaluation of our commissioned services - sexual health, drugs and alcohol etc. These tend to be fairly formal and follow a pre-agreed set of criteria that is quite standardised across the Council. However, an additional layer of ABCD evaluation that follows the above principles could provide us with many new perspectives about the service and the people who access it and work to deliver it. | Yes |
| 02/10/2023 02:41 PM | northumberlandcommissioning | Depending on how they are used, they can be top-down, at end points and therefore don't allow for correction in real-time. However, in some areas around User Experience in I.T. Formative assessments can lean more towards a supportive approach, through iteration, of weaved in to a process correctly, understanding what has worked well, or not, and refining approaches to reach the end goal. | Developmental evaluation can support an initiative to generate rapid learning to support the direction of the development of a program, and/or affirm the need for a change of course. Developing relationships over time adds a potential richness to the gathering of data throughout the lifetime of an initiative, though a note of caution should be used about how these relationships can be influencers on the direction of travel. Developmental evaluation is useful in monitoring community initiatives as it is helpful in environments that are not linear, or are complex multi-agency environments. | Strengths based, local people as citizens, give people ownership of their experiences (e.g. could they also fulfil this evaluation/monitoring role?), test and learn, telling stories. | Participatory appraisal, public narrative techniques to tell stories, work with communities to understand their measures of feeling better or more engaged with their communities and build this in. | Development of a small pilot in Newbiggin around providing activities for residents, that will act as a learning opportunity for our wider Communities First approach. | Yes |
| 15/04/2024 02:48 PM | surrey2024commissioning1 | Focus on accountability for delivery does not accurately represent success of creating communities Focus on linear process does not allow for mid-course correction Do not encourage deviating from initial plan, and so do not encourage emerging learning nor accurately reflect complex/dynamic environments like communities. | Enable community to define what is important to evaluate / markers of success Enable community involvement/leadership of the evaluation | Interviews, focus groups, case studies | Current workstream supporting community projects | Yes | |
| 18/04/2024 09:47 AM | surrey2024commissioning1 | This evaluation project is in response to the Supported Living business case assignment and has been completed by Simon Montgomery, Sarah Ford and Jo Neville Rye. Development evaluation is a shift from the usual audit approach to evaluation, as it is led by the residents who are responsible to effecting change within their communities. As a result the approach addresses dilemmas of the shortcomings of audit style evaluation – which does not capture mid-course correction. Developmental evaluation is a way to monitor, assess and provide feedback on the development of a project or program while it is being designed or modified; that is, where inputs, activities and outputs may not yet be known, or may be in a state of flux. • Supports development of innovation and adaptation in dynamic environments. • Positioned as an internal team function integrated into the process of gathering and interpreting data, framing issues, surfacing and testing model developments. • Centred on the innovators' values and commitment to make a difference. Develops measures and tracking mechanisms quickly as outcomes emerge; measures can change during the evaluation as the process unfolds. | • We will identify the maximisation of gift exchange. • We will identify the maximisation and deepening of associational life. • We will assess the changes in the number of participating and co-producing residents and in their citizen power, placing particular emphasis on the inclusion of those who have been most marginalised. • Rapid, real-time feedback; diverse, user-friendly forms of feedback. Evaluation aims to nurture learning. • Learning to respond to lack of control; staying in touch with what's unfolding and responding accordingly. | • creative and critical thinking balanced; • high tolerance for ambiguity; • open and agile; teamwork and people skills; • able to facilitate rigorous evidence-based perspectives. | Supported Independent Living • Stories of significant change from local businesses, individuals and community leaders. • Participatory learning and reflection methods including participatory statistics • In depth interviews with individuals, community leaders and businesses and the workforce • Real time interactions that generate learning, evolution and development • Being flexible for measures and tracking information to quickly change as outcome emerge • User friendly forms of feedback • Interactive sessions using technology • Coproduction sessions with residents, carers and families • Accessible approaches to gathering feedback including in person and online forms in different accessible formats (i.e. easy read) | Yes | |
| 18/04/2024 03:22 PM | surrey2024commissioning1 | Developmental evaluation (DE) differs from traditional summative and formative evaluations in its focus on complex, dynamic, and emergent initiatives, especially those operating in complex environments where outcomes are uncertain and constantly evolving. Here's how developmental evaluation can be more effective in such contexts: 1. Responsive to Complexity: DE is designed to navigate complexity. It recognises that in complex environments, linear cause-and-effect relationships may not apply. Instead, it focuses on understanding the underlying dynamics, patterns, and emergent outcomes within the system. 2. Continuous Learning: Unlike summative evaluations, which typically occur at the end of a project or program, DE emphasizes continuous learning throughout the initiative's lifecycle. It provides ongoing feedback and insights to inform decision-making and adaptation in real-time, rather than waiting until the end to make adjustments. 3. Adaptive and Flexible: DE is inherently adaptive and flexible, allowing evaluators to adjust their methods, approaches, and evaluation questions in response to changing circumstances and emerging insights. This flexibility enables stakeholders to stay responsive and relevant in dynamic environments. 4. Emphasis on Innovation and Creativity: DE encourages innovation and experimentation by creating space for risk-taking and learning from failure. It provides valuable insights into what works, what doesn't, and why, fostering a culture of innovation and continuous improvement. 5. Participatory Approach: DE emphasizes collaboration and co-creation between evaluators and stakeholders. It involves stakeholders in the evaluation process, soliciting their input, perspectives, and feedback to ensure that evaluation findings are contextually relevant and actionable. 6. Systems Perspective: DE takes a systems perspective, recognising that interventions are embedded within larger social, cultural, economic, and political systems. It explores the interconnectedness of various stakeholders, factors, and influences to understand the broader context in which the initiative operates. 7. Capacity Building: DE builds evaluative capacity within organizations and communities by empowering stakeholders to engage in reflective practice, data-driven decision-making, and adaptive management. This enhances their ability to navigate complexity and drive positive change beyond the scope of the evaluation itself. 8. Focus on Emergent Outcomes: DE acknowledges that outcomes may emerge unpredictably over time, especially in complex environments. It looks beyond predefined indicators to identify and capture emergent outcomes, patterns, and unintended consequences that traditional evaluations may overlook. 9. Supports Innovation and Scaling: DE provides valuable insights for scaling successful interventions or innovations by identifying core principles, mechanisms, and contextual factors that contribute to their effectiveness. It helps stakeholders understand how interventions can be adapted and replicated in different contexts. 10. Cultivates Learning Organizations: DE contributes to the development of learning organizations by fostering a culture of curiosity, inquiry, and reflection. It encourages stakeholders to critically examine assumptions, challenge prevailing norms, and embrace new ways of thinking and working | 1. Asset Identification: Recognise and identify the existing strengths, skills, and resources within the community. This involves understanding both tangible assets (such as physical infrastructure) and intangible assets (such as skills, knowledge, and networks). 2. Asset Mobilisation: Facilitate the mobilisation of community assets to address local challenges and opportunities. Encourage community members to contribute their skills, resources, and knowledge to initiatives that align with community priorities. 3. Participatory Approach: Foster active participation and involvement of community members in all stages of the evaluation and monitoring process. This ensures that initiatives are relevant, responsive, and reflective of community needs and aspirations. 4. Capacity Building: Support the development of individual and collective capacities within the community. Provide opportunities for skill-building, leadership development, and knowledge sharing to empower community members to take ownership of initiatives and sustain them over time. 5. Collaborative Partnerships: Foster collaboration and partnership between community members, local organizations, government agencies, and other stakeholders. Recognise the complementary strengths and resources that each partner brings to the table and leverage these synergies for greater impact. 6. Strengths-Based Evaluation: Shift the focus of evaluation from deficits to assets. Assess the effectiveness and impact of community initiatives based on their ability to leverage and build upon existing strengths within the community rather than simply addressing deficits or problems. 7. Adaptability and Flexibility: Embrace adaptability and flexibility in both the design and evaluation of community initiatives. Recognise that community contexts, needs, and priorities may evolve over time, and be willing to adjust strategies and approaches accordingly. 8. Long-Term Sustainability: Promote long-term sustainability by embedding evaluation and monitoring mechanisms into the fabric of community initiatives. This includes fostering local ownership, building capacity for self-assessment, and creating systems for ongoing learning and improvement. 9. Celebration and Recognition: Celebrate successes and achievements within the community to reinforce positive behaviours and outcomes. Recognise and acknowledge the contributions of individuals and groups who have played a role in driving positive change. 10. Learning Culture: Cultivate a culture of learning and reflection within the community. Encourage openness to feedback, continuous learning, and adaptation based on lessons learned from both successes and failures | 1. Stakeholder Interviews and Surveys: Conduct interviews and surveys with various stakeholders involved in the management of extra care housing, including residents, staff, management, local authorities, and healthcare professionals. This qualitative approach allows you to gather insights, feedback, and perspectives on the effectiveness of the management processes, quality of services provided, and overall satisfaction levels. 2. Performance Metrics and Data Analysis: Establish key performance indicators (KPIs) relevant to the management of extra care housing, such as occupancy rates, resident satisfaction scores, staff turnover rates, budget adherence, and health outcomes. Collect and analyze quantitative data regularly to track trends, identify areas for improvement, and assess the impact of management decisions on the quality of care and operational efficiency. | Yes | Yes | |
| 19/04/2024 07:31 PM | surrey2024commissioning1 | It recognises that a community's development journey isn't always linear and allows for deviation, flexibility and mid-way correction in response to the conditions encountered. | Involving and/or enabling the community to define what success looks like and conduct the evaluation of outcomes. | Stories/case studies (including videos) from members of the community; Realistic where the Terms of Reference for evaluation are co-created with the community; Use contextualising warm data that demonstrates the social/community context that surrounds the data. | The impact that Community Link Officers are having in the community. | No | |
| 07/05/2024 03:34 PM | surrey2024commissioning1 | It acknowledges learning and is a more participatory approach. | Celebrates change, self evaluation, recognise learning, focus on most significant change | Developmental Evaluation, Realistic Evaluation, Most Significant change model | Stanwell place based work | Yes | |
| Timestamp | Your Course ID | 1. How can developmental evaluation be more effective than summative or formative evaluations | 2. List the principles you would use to support evaluation and monitoring of community initiatives from an ABCD perspective: | 3. List three methodologies you would use to carry out an evaluation of, or in monitoring, a community initiative that you commission or directly manage: | 4. Identify an upcoming evaluation, or an area of on going monitoring, that you can apply this learning to: | Assignments for each module will be shared with others on your course to share learning and will be visible to administrators, but will not be used for any other purpose. Do you consent to your answers being shared? |
Please wait while this table loads