Terms of Reference (ToR) for the Endline Evaluation of the Project: Addressing protection and multiple socio-economic needs through a triple nexus approach in South Sudan and Ethiopia

Introduction and Context

DanChurchAid (DCA) is a Danish Humanitarian and Development Organization, which primarily works with both local and international NGO partners, and it’s a strong member of international networks/ alliances including churches. DCA is headquartered in Copenhagen and has offices in 19 countries worldwide. It seeks to assist the most marginalized populations through its three global goals – Save Lives, Build Resilient Communities, and Fight Extreme Inequalities with a cross-cutting focus on Gender Equality, Youth Engagement, and Environmental responsibility. DCA works with local communities and is increasingly involved in building the capacity of partners, communities, and local government institutions as well as its staff. DCA’s South Sudan’s office is based in Juba and implements programmes through its partners in Jonglei, Upper Nile, Eastern Equatoria, and Central Equatoria states.

This Terms of Reference (ToR) for Consultancy describes DCA South Sudan’s plan to evaluate one of its projects titled: “Addressing protection and multiple socio-economic needs through a triple nexus approach in South Sudan and Ethiopia.” The ToR outlines the project background and specific outcomes, objectives of the evaluation, methodology, and the expected deliverables. Guidance has also been provided for the competitive bidding and sourcing of the External Consultant.

Project Background

The humanitarian crisis in South Sudan and Ethiopia reflects dire challenges, characterized by extreme food insecurity, violence, and massive displacement affecting millions. In South Sudan, the food insecurity rate has reached unprecedented heights since the country’s independence, compounded by catastrophic malnutrition rates, conflict, hyperinflation, and recurrent flooding, with over 60% of the population requiring humanitarian aid. Conversely, Ethiopia faces exacerbated challenges from the Tigray conflict, climatic disturbances, and heightened Gender-Based Violence (GBV) stemming from ethnic tensions.

This project aims to address not just the immediate needs but also the underlying vulnerabilities of affected populations in South Sudan's Jonglei and Greater Pibor areas, alongside Ethiopia's Afar region. Specifically, DCA is implementing the project through our partner, Nile Hope (NH), in Akobo County of Jonglei state and Pibor County of the Greater Pibor Administrative Area (GPPA). It intends to provide life-saving assistance while focusing on resilience-building and capacity-strengthening for refugees, IDPs and local communities.

The overarching goal is to alleviate the socio-economic impacts of ongoing crises and enhance resilience against future shocks. This will be achieved through a comprehensive approach known as the triple nexus, integrating humanitarian actions with development and peace-building efforts. Priority will be given to vulnerable groups, especially youth, women, and girls, through the establishment of community-led protection committees and initiatives aimed at improving income-generating opportunities.

The project underscores the importance of a rights-based approach, ensuring the dignity of all individuals is respected. Collaboration with other humanitarian actors through established frameworks is undertaken to enhance the efficacy of interventions. Ultimately, the project seeks to instill hope, and foster sustainable development amidst ongoing conflict and instability, paving the way for peace and resilience through collective community action.

Project Summary

Project name

Addressing protection and multiple socio-economic needs through a triple nexus approach in South Sudan and Ethiopia

Overall objective

The overall objective of the project is to support vulnerable groups among refugees, internally displaced people and affected local communities to mitigate the social and economic impacts of COVID-19 and strengthen resilience to natural and man-made disasters in South Sudan and Ethiopia

Project outcomes

1. Outcome 1: Social cohesion among refugees, IDPs and host communities is strengthened, and vulnerable people protected.
2. Outcome 2: Communities affected by multi-disasters (conflict, floods, drought, desert locust and COVID-19) have improved food security and access to basic needs.
3. Outcome 3: Self-reliance and livelihoods capacities of targeted communities are enhanced to sustainably meet food and other household needs

Key indicators

• Proportion of the population who have experienced a dispute in the past two years and who accessed a formal or informal dispute resolution mechanism, by type of mechanism. (SDG indicator 16.3.3)
• Percentage of beneficiaries who report that social cohesion is enhanced and who feel safer in their communities
• Average Reduced Coping Strategies Index (rCSI) for the target population
• Percentage of targeted beneficiaries who report increased self-reliance as a result of the intervention.

Project target: 15,740 (5990 male, 9750 female)

Project locations: Akobo (Jonglei State), and Pibor (GPPA) – South Sudan

Project duration: 1st January 2022 – 31st December 2024

Partner: Nile Hope

Evaluation Purpose

The main purpose of this endline evaluation is to assess the relevance, coherence, effectiveness, efficiency, sustainability, and potential impact of the project interventions. The evaluation will help to provide practical recommendations and document the lessons learned for adoption in other projects designed by DCA and partners.

Specifically, four key evaluation themes will be covered during the evaluation process: project logic, participation, partnership, and sustainability. The Consultant will also assess cross-cutting themes, including disability inclusion, gender and social inclusion, accountability to the affected population, social cohesion (IDP vs host community) and environment during the evaluation process. The pre-and-post-project performance analysis with the baseline values compared against the end-of-project targets, significance of the change in the indicator performance will also be determined during the endline evaluation.

The specific objectives of the evaluation will be:

1. To establish whether and to what extent the project design and interventions have been relevant to the needs of the target participants.
2. To assess the extent to which the project has achieved the planned outcomes and outputs.
3. To assess the extent to which the components of social-cohesion and livelihoods interventions have been efficient.
4. To establish the indications of the impact of the project interventions on the target participants.
5. To assess if the benefits of the project are likely to continue beyond the project life span.
6. To assess the extent of corrective measures identified and addressed in a timely manner during the implementation process.
7. To document lessons learnt and good practices that can be replicated in future projects.

Scope of the Evaluation

Geographically, the endline evaluation will cover the project locations of Akobo County of Jonglei state and in Pibor county of the Greater Pibor Administrative Area where the Nile Hope implemented the project interventions. The evaluation will broadly assess the key achievements of all the expected outcomes as outlined in the project logframe. The evaluation will also assess the different project activities as in the logframe, with a strong emphasis on the following:

• Determining the extent to which the project outcomes and outputs have been achieved and whether there were unexpected outcomes
• Determining the relevance, effectiveness, coherence, efficiency, sustainability, and impact of the project
• Documenting the challenges, lessons learnt and key recommendations for improvement.

The endline evaluation will take place between November and December, with the Final Report expected by 15 December 2024.

Users of the evaluation

DCA and NH will be the primary users of the evaluation findings for learning and decision-making. The evaluation report will also be useful to the donor (DANIDA). Moreover, the evaluation findings will provide appropriate measures to be taken and recommendations to improve future programming.

Evaluation Questions and Criteria

The endline evaluation will be guided by the Organization for Economic Co-operation and Development's/Development Assistance Committee (OECD/DAC) and Active Learning Network for Accountability and Performance (ALNAP) criteria for evaluating humanitarian actions of relevance, effectiveness, efficiency, coherence, impact, and sustainability criteria)[2].

Table 1. Evaluation Matrix

Criteria

Evaluation Questions

Relevance:

The relevance of the project will be assessed by looking into whether the project is tailored to local needs, increasing ownership, accountability, and cost-effectiveness accordingly.

• Are interventions tailored to the needs of the target community (men, women, PwDs, IDPs) and other vulnerable groups?
• To what extent did the project build local capacities and worked towards improving the resilience of communities and people affected by the humanitarian situation?
• Were the project participants involved in the process of developing, implementing and evaluating the project?
• How well was the project aligned with the Humanitarian Response Plan (HRP) of South Sudan and DCA goals of saving lives, building resilience and fighting inequality?

Coherence:

The extent to which the interventions of different actors are harmonized with each other promotes synergy, avoiding gaps, duplication, and resource conflicts.

• To what extent has the project engaged in stakeholder coordination, including local authorities and clusters?
• Which factors have restricted coordination, and which factors have supported it? Are there any coordination success factors that can be transferred to other situations?
• How did the various components of the project work together to achieve the overall goals and objectives?
• Are the project activities and outputs logically connected and aligned with the project's purpose and intended outcomes?
• To what extent did the project adapt and adjust its strategies in response to changing circumstances and new information while maintaining its overall coherence?
• Do the staff understand the approach to the project and how their individual roles contribute to its success?

Partnership:

The assessment and analysis of collaborative partnerships between organizations to achieve common goals.

• How would the communication and collaboration between DCA and NH be rated?
• In what ways did DCA and NH effectively share resources and support each other to achieve the project objectives?
• Has the partnership between DCA and NH resulted in measurable benefits or outcomes for the parties involved? If so, what are the key examples?
• How effectively did DCA and NH address any challenges or conflicts that might have arisen during the implementation? How satisfied are DCA and NH with the level of transparency and trust exhibited during the project implementation?
• Have there been any missed opportunities or areas where DCA and NH could have worked more effectively together?
• What recommendations are proposed for strengthening and enhancing the partnership between DCA and NH in the future?

Effectiveness:

The effectiveness will involve determining who is supported by the project, and why.

What were the main reasons that the intervention provided or failed to provide major population groups with assistance and protection, proportionate to their needs?

• To what extent have the intended outcomes (and use of outputs) been achieved by the end of the project?
• Were the project participants identified and assisted in a timely manner?
• What factors contributed to the achievement or non-achievement of planned project results? What are the main operational bottlenecks, what is causing them and how can they be addressed?
• To what extent are the most vulnerable participants being reached by the project interventions? Do men, women and PwDs have equal access to the project interventions? How successful have DCA and NH been reaching the most vulnerable groups in the most affected geographic areas?
• To what extent has the affected population been properly targeted and reached by the project? If not reached, what were the impediments? What strategies can be used to improve targeting?
• Is the project covering an appropriate number of people in need?
• Is the project adequately responding to the changing humanitarian context?

Efficiency:

This will assess the project output, both qualitative and quantitative, in relation to the inputs i.e., were inputs, staff, time, funding, equipment used in the best possible way to achieve outputs.

• To what extent were the components of social-cohesion, livelihoods and cash assistance efficient?
• Did the project leverage on other projects in the same area?
• How did the costs compare to other projects targeting similar outputs?
• Did the project adhere to established timelines and milestones?
• Were there any unexpected challenges or obstacles that affected the project efficiency?

Sustainability:

The project sustainability will be assessed by establishing to what extent the net benefits of the intervention continue or are likely to continue?

• To what extent was the project able to connect short-term interventions to medium-term strategies and longer-term perspectives?
• Environment: To what extent did the project adopt different approaches towards environment-sensitive protection programming? What are the concrete adjustments applied both operationally and programs side?
• Exit Strategy: To what extent did the project build the capacity of the local recipients and structures in contributing to the exit strategy? How feasible is the exit strategy considering the context and capacities?
• What is the likelihood of the continuation of positive project outcomes beyond the end of the project (both by primary stakeholders and duty bearers)?
• How does the project seek to safeguard sustainability?
• Are there any factors threatening the sustainability of project outcomes? How does the project seek to mitigate these risks?
• To what extent was the target group becoming more aware and resilient and their aid-dependence is considered less than before?

Impact:

The project impact will be assessed by establishing to which extent the intervention has generated or is expected to generate significant positive or negative, intended, or unintended, higher-level effects?

• Is the project contributing to systemic changes, e.g to improved food security, self-reliance, and social cohesion among the IDP and host communities etc (contribution analysis)?
• What evidence is there that the expected outcomes have been realized by the project?
• What have the achievements of the project been in relation to these outcomes and, to what extent have other contextual and operational factors played an influential role.
• What are the unintended positive and negative impacts of the implementation of the project? If it has, what measures have been and can be taken to eliminate or reduce the negative impacts?

Lessons Learnt:

The evaluator will document challenges, programmatic lessons learnt and key recommendation

• What are the key lessons learned?
• What are the recommendations for improvement of the project or similar projects elsewhere in South Sudan?
• What best practices are to be adopted for such projects in future?
• What mistakes should be avoided if the project were to be replicated?
• Is there any identifiable harm caused by the project either for participants or non-beneficiaries?

Evaluation Methodology

The evaluation will employ a participatory approach. The results of the evaluation and lessons learned will be used to improve future projects by DCA and the implementing partners. The evaluation will adopt both quantitative and qualitative methods, using both primary and secondary data sources. Interviews will be conducted with the project participants, and the key stakeholders. Document reviews will also be conducted.

A range of existing information will be made available to the Consultant upon notification of the award. These will include Project proposal and logframe, MEAL plan, IPTT, baseline, mid-term evaluation and post-distribution monitoring reports, monthly and quarterly reports, financial reports and other relevant documents. As part of this assignment, the Consultant will apply different methodologies, including primary data collection and a review of existing resources. The primary data collection will include Key Informant Interviews (KII) with implementing partners (IP)- duty-bearers and key stakeholders; Most Significant Change (MSC) guide; beneficiary surveys; and substantiating photos and videos.

The Consultant will present and discuss the findings, conclusions, and recommendations with DCA and NH, reformulate them if necessary and identify key actors and methods to respond to these recommendations in future programming.

The Consultant will provide a detailed plan of the proposed methodologies in the inception report. The proposed methodology should include an evaluation matrix to reflect evaluation questions, judgement criteria, indicators linked to the judgement criteria, data sources, data collection and analysis plan. DCA will review the methodology proposed by the Consultant and provide feedback before the evaluation process begins.

The methodology used and the final report must adhere to the minimum standards of quality of evidence outlined in BOND Evidence Principles Checklist, including voice and inclusion, appropriateness, triangulation, contribution, and transparency. To demonstrate the impact of the project, the consultant will make use of three approaches as follows.

• Comparing the performance of outcome and impact level indicators at Baseline and the Endline Evaluation. The difference in the two (positive or negative) provides an indication of the change that has transpired because of the interventions of the project
• Determining if the change proposed is significant with 95% confidence that indeed there has been an impact of the interventions of the project when the baseline is compared with evaluation findings
• Documentation of impact or most significant change stories to qualitatively complement the quantitative data collected.

While maintaining independence, the Consultant will seek the views of all parties, including the affected population. The emphasis of analysis and learning will be in the targeted areas of the project, result achieved, and process adopted, as well as coordination and collaboration among partners.

Data privacy and protection

The Consultant is expected to sign the DCA data protection policy and demonstrate an understanding to ensure protection of personal data collected during this assignment. The Consultant in the (inception report) needs to elaborate on how participant data will be collected and protected; what equipments will be used to store the data, and how long this data will be stored. It is the Consultant’s responsibility to ensure that all staff involved are clear on the evaluation’s aim and purpose, as well as all project details.

Translators, if needed, must be organized by the Consultant and should strictly comply with the above measures. It is the role of the Consultant to ensure that interviewers are trained in confidentiality procedures. Interviewers need to be trained in obtaining verbal consent for interview participation. Staff should have experience in program monitoring, surveying, and data collection and effectively use technology-based analysis software to collect and analyze data. The Consultant will prepare all manuals, guides, and training material used to train data collectors.

All tools will be designed in English and approved by DCA prior to field data collection. The Consultant will be expected to translate all the tools into relevant languages in case needed. The main language of reports and tools will be English.

Management of the Evaluation Process

The DCA MEAL Manager and Programme Manager will be the main contact persons for the evaluation process. DCA will manage the contract and engagement with the Consultant in accordance with the terms of the contract. The Consultant should make the necessary arrangements and coordinate with DCA before field work to ensure no issues arise during the data collection. Relevant contacts will be shared with the evaluator. The summary of the roles and responsibilities are outlined below.

Role: Evaluation Commissioner
Responsibility: Commissions/authorizes the evaluation study, the main user of the evaluation results. Title: Programme Manager

Role: Evaluation Manager
Responsibility: Overall management of evaluation and technical support if needed. The MEAL Manager will provide technical support during the end line evaluation process to ensure that the evaluation is of the required quality and standard.
Title: MEAL Manager

Role: Evaluator
Responsibility: Responsible for carrying out the evaluation as agreed upon in the ToR (and the Inception Report).
Title: External Evaluator

Role: Logistical support
Responsibility: Make sure that the evaluation administration regarding the finances and procurement is compliant with the existing donor/organization’s regulations.
Title: Head of ProLog

Role: Field coordination
Responsibility: Day-to-day coordination and communication with the evaluator during field data collection
Title: NH MEAL Manager

Expected Deliverables

The following deliverables are expected to be produced by the Consultant. All the deliverables must be submitted to DCA in soft copy:

1. Inception Report (IR)

• The IR should set out the planned design and methodology to meet the above-mentioned objectives and to answer the evaluation questions. Furthermore, the overarching evaluation questions should be specified further in the IR
• The IR should also reflect the limits of the suggested design and methodology and explore the feasibility for answering the evaluation questions and reflect on the ToR, describe the overall approach of the evaluation and how data will be collected by providing an evaluation matrix, data collection tools such as questionnaires and interview guidelines as well as the evaluation schedule.
• The IR should follow a standard outline which will be provided to the evaluator(s) after contracting and needs the approval of the contracting party.

2. Data collection tools

The Consultant shall develop the data collection tools and have them approved by DCA before field data collection. The data collection tools shall be shared and approved together with the IR.

3. Draft Evaluation Report

The Consultant shall prepare the draft evaluation report with details of findings, recommendations and lessons learned for review by the DCA and NH.

4. Validation of Results

A virtual meeting or face-to-face to discuss the main evaluation findings, draft report with DCA and NH.

5. Final Evaluation Report

The Consultant will share the final evaluation report after incorporating the comments from DCA.

6. Evaluation Management Response Matrix

The Consultant, as part of the assignment shall elaborate a Management Response Matrix, listing the recommendations and the person responsible for each recommended action. The priority level for each recommendation shall be provided in the Evaluation Management Response Matrix provided by DCA.

Time frame/schedule

The duration of evaluation shall be 45 days, starting from 1st November 2024. The table below presents the tentative schedule prepared to guide the evaluation process.

1. Review documents, develop data collection tools and IR - 7 days

2. Review of the draft IR by DCA and NH - 5 days

3: Finalisation and approval of the IR - 3 days

4. Digitalisation of data collection Tools and mobilisation of logistics - 2 days

5. Field data collection - 8 days

6. Data analysis and preparation of the draft evaluation report - 10 days

7. Review of the draft evaluation report - 5 days

8. Preparation of the Final evaluation report - 5 days

Total: 45

Confidentiality

All documents and data acquired from documents during interviews and meetings are confidential and used solely for the evaluation. The deliverables and all material linked to the evaluation (produced by the evaluators or the organization itself) are confidential and remain the property of the contracting party.

Expertise of the Evaluators

This assignment is open to Evaluators with sound experience in the services outlined above.

The Consultant/team must meet the following requirements:

• Post-graduate qualifications in Sociology, Development, International Humanitarian studies and a qualified team member with experience in assessing food security, livelihoods and social-cohesion projects
• Proven experience of carrying out endline evaluations focusing on project implementation of food security, livelihoods and social cohesion
• Extensive experience in applying various evaluation methodologies. Relevant training and demonstrated expertise in participatory methods, quantitative and qualitative studies
• Minimum of 10 years and demonstrated experience in evaluating development and/or humanitarian programs implemented by International Aid Organizations; experience of evaluating DANIDA funded projects.
• Knowledge and experience of South Sudan political and cultural contexts

Technical and Financial Offer

The Consultant shall be required to submit the following:

1. Technical and Financial offer
2. The technical proposal should among others detail the following:
   1. Consultants understanding of the assignment with specific attention to the objectives of the assignment and how the objectives will be measured
   2. Methodology: A clear description of the overall design and methodology of the endline evaluation, and clear work plan
   3. Proposed team with CVs of the Individual Consultant or Consulting team that will be involved in the assignment.
   4. At least 2 sample evaluation reports conducted in the last 12 months for assessing the quality of work of the Consultant.
3. The financial proposal shall include the following:
   1. Proposed budget for the evaluation in United States Dollars (USD)
   2. Professional consultancy fees and logistics costs (include all the professional fees, travel and data collection expenses, and clearly state the level of effort for each member of the proposed team)
   3. Proof of professional registration, taxation and other relevant documents may be required.

Consultant Evaluation Criteria

The following ranking criteria will be considered for the contract granting:

Technical and Financial Criteria

Technical Expertise of the Consultant

1. The number of the proposed team is adequate, with expertise in the key sectors to meet the requirements of the assignment (score: 5)

2. Relevant academic and professional qualifications of the Consultant in conducting evaluations (include CVs) (score: 5)

3. Consultant’s knowledge and experience in the context, or similar contexts with the ability to work in the geographical location of the assignment (score: 5)

4. Demonstrated previous experience in similar evaluations by submitting 2 sample reports. A review of these reports demonstrates quality work (score: 5)

Sub-total: 20

Understanding of the assignment and Methodology

1. The technical proposal clearly describes the understanding of the ToR, responding to the objectives of this evaluation (score: 10)

2. Demonstrated understanding of the project and the logic model (score: 5)

3. Clear description of how the evaluation questions will be answered, including the evaluation matrix (score: 10)

4. The proposed evaluation design is appropriate, with clear description to meet the requirements of the evaluation (score: 10)

5. Sampling criteria is clearly elaborated and appropriate for the evaluation (score: 10)

6. Clear description of the data collection, and analysis tools and procedures (score: 5)

7. Mechanisms for quality and ethical procedures are well articulated (score: 5)

8. Detailed work plan with realistic time estimates of each major segment of the work plan/milestones (score: 10)

Sub-total: 65

Financial proposal

1. The budget is clear and realistic, aligned with the work plan (Include all the professional fees, travel and data collection expenses) (score: 15)

Sub-total: 15

Overall maximum score: 100

8. Payment Terms.

30% Payment will be made upon submission and approval of the Inception Report, and the remaining 70% upon submission and approval of the Final Evaluation Report with other key deliverables. The payment will be subject to local or national tax laws.

[2] For more details: https://www.alnap.org/help-library/evaluating-humaintarian-action-using-the-oecd-dac-criteria

If interested in applying for the aforementioned consultancy, please send your CV along with a written technical and financial proposal, outlining a plan and approach for executing the consultancy in line with the points mentioned in section 2 to viona@dca.dk with agym@dca.dk and kamm@dca.dk in cc.

The deadline for application is the 31-10-2024 @ 15:00 PM CAT. The subject of the application letter or e-mail should read Application for Consultancy to Evaluate one of its projects titled: “Addressing protection and multiple socio-economic needs through a triple nexus approach in South Sudan and Ethiopia