Consultancy: Evaluability Assessment of the National School Safety Framework (NSSF), Pretoria, South Africa, 3 months (Remote)

The appointed service provider will conduct an evaluability assessment of the NSSF under the general supervision of the Adolescent Development Specialist (Education and Adolescent Development) with close technical inputs and insights from PMRE with overall guidance from Chief of Education. The assignment will also be delivered in close collaboration with the Department of Basic Education (Director: School Safety).

UNICEF works in some of the world’s toughest places, to reach the world’s most disadvantaged children. To save their lives. To defend their rights. To help them fulfill their potential. 

Across 190 countries and territories, we work for every child, everywhere, every day, to build a better world for everyone. 

And we never give up. 

For every child, education. 

Education | UNICEF South Africa

Due to the forever increase of different kinds of violence in schools DBE in 2014 working in close collaboration with UNICEF and in partnership with the Centre for Justice and Crime Prevention (CJCP) developed the School Safety Framework (NSSF) which was approved by the Minister of Basic Education in April 2015. The NSSF was developed in order to provide an all-inclusive strategy to guide schools, districts, and provinces to establish early warning signals to violence in schools, develop school safety plans, monitor the implementation of the plans, and report violent incidences using standardized school safety indicators. It also provides a platform for engagement of key partners in the community to ensure overall safety in schools.

The overall aim of the NSSF is to create a safe, violence and threat-free, supportive learning environment for learners, educators, principals, school governing bodies and administration. The broad objectives of this NSSF are to:

• Assist the school in understanding and identifying all security issues and threats;
• Guide schools to effectively respond to identified security issues and threats;
• Create reporting systems and manage reported incidents appropriately; and
• Help the school to monitor its progress over time.

It is important to also note that the NSSF is not a duplication of already existing policies and strategies already mentioned above but rather it consolidates all of these and provides the thread that links them all together to enable schools to respond to all forms of violence (which include Corporal punishment, Sexual abuse, Gender-Based Violence, Assault, different forms of bullying including cyberbullying, xenophobia, homophobia and gang violence, just to mention a few) holistically.

Additionally, the NSSF adopts the Ecological Risk Model which provides a structure within which violence and its complex relationships can be understood and addressed. Its value lies in the fact that the theory recognises that there is an interplay between various individual, relational, community and societal factors. Violence in schools is therefore viewed as the result of an accumulation of risk factors that operate on these various levels. The combination of risk and protective factors is influenced by the external environment and impacts on how any individual adapts to their environment.

The NSSF has the training manual and implementation tools. The manual provides a detailed account of how each of these tools are administered which includes:

• How to determine school’s readiness to implement the framework
• How to engage educators and non-educator staff in identifying, responding, and assessing interventions being implemented in a school
• How to gather information on learners’ experiences of crime and violence, and their suggestions to improve school safety

As mentioned above, the NSSF was approved by the Minister of Basic Education in April 2015. Following the approval of the NSSF in 2015 a series of capacity development workshops covering 44 education districts were conducted in all nine provinces reaching over 1,500 master trainers. Since 2015 to date at least 47,000 copies of the NSSF were printed and distributed nationwide by UNICEF.

In 2018 UNICEF further supported 472 provincial, districts officials including educators, learners and SGBs members reaching 250 schools in Northern Cape, North West, Eastern Cape and Kwa-Zulu Nata (KZN) provinces. 

Between 2022 and 2023, the following number of schools (School Management Teams (SMT), members of School Governing Bodies (SGB), educators and learner) were physically trained on the NSSF: 1441 schools were trained in the North West, 3220 schools were trained in KZN, 229 schools were trained in Gauteng, 535 schools were trained in Northern Cape, 508 schools were trained in Mpumalanga, 1483 schools were trained in the Eastern Cape, 151 schools were trained in the Western Cape, 18 schools were trained in the Free State.

The DBE together with its partner the Wits Reproductive Health and HIV Institute further developed an NSSF digital training course for school communities, to enable access to the training anywhere in the country. The course has been accredited by the South African Council of Educators (SACE). As a result, educators will receive 15 Professional Development Points (PDP) for successfully completing the course. The DBE released a circular in September 2022, compelling all school-based personnel and school safety committees to complete the digital training by 31 March 2023. To date, over 130 000 people have completed the course. 

How can you make a difference? 

The appointed service provider will conduct an evaluability assessment of the NSSF under the general supervision of the Adolescent Development Specialist (Education and Adolescent Development) with close technical inputs and insights from PMRE with overall guidance from Chief of Education. The assignment will also be delivered in close collaboration with the Department of Basic Education (Director: School Safety).

The main purpose of the Evaluability Assessment (EA) of the NSSF

Despite the government’s several announcements, including its advocacy efforts regarding the existence and implementation of the Framework since 2015, no major evidence-generation effort has been conducted to evaluate its effectiveness or impact. As a result, the extent to which the framework has achieved its intended objectives remains to be seen. Since its adoption and approval, the Framework has never been evaluated and it is difficult to tell if the intervention is successful without a formal evaluation.

UNICEF policies expect interventions to be evaluated. UNICEF’s Performance Review calls for periodic independent evaluations of interventions intended to support government efforts in realising the protection of children's rights, to help meet their basic needs and to expand their opportunities to reach their full potential.

The evaluation of the NSSF will assist in identifying the effectiveness of the intervention regarding performance levels, achievement of results and lessons learned. However, before the actual evaluation is conducted it is important to perform an Evaluability Assessment (EA) which is intended to assess if its practical to do the actual evaluation, how much the evaluation will cost, and what elements of the NSSF needs to be evaluated. Therefore, the main purpose of the EA for the NSSF is to assess the strategic and technical soundness of the proposed evaluation.

The objectives of the evaluability assessment of the NSSF are:

• Assess Program Design and Clarity: Evaluate the clarity of NSSF’s objectives, the logic of its Theory of Change, and the alignment of activities with intended outcomes.
• Assess Data Availability: Assess the availability and quality of data to gather insights on program implementation and the feasibility of conducting a future evaluation.
• Determine Evaluation Feasibility and Design: Develop key evaluation questions, identify suitable designs, and assess the practical feasibility of conducting a comprehensive evaluation.

Period to be covered:

The assessment will cover the period since the NSSF was approved by the Minister (April 2015) to date. However, the assessment timeframe should be informed by the available information in the form of reports on the programme implementation and other supporting documents which include work plans, mission and workshop reports, programmatic documents, provincial and districts reports, baseline surveys, monitoring data (which could be obtained from Annual School Readiness Surveys, South Africa School Administration and Management System [SASAMS]) as well as information from Learner Unit Record Information and Tracking System [LURITS]), country data and any other information relevant to the programme.

Geographic coverage:

The assessment is intended to cover the whole country, starting with national consultation with officials from relevant government departments, to provincial and district officials as well as representatives from School Management Teams (SMT), School Principals, School Governing Bodies (SGB), Representative Council of Learners including ordinary learners and educators.

Programme  Focus:

Evaluability assessments will intentionally focus narrowly on the technical elements surrounding programme logic and measurement (e.g., the existence of a theory of change, monitoring, and evaluation plans, the SMART indicators, and data availability).

Summary of the key activities for the EA:

1. Clarity of Program Objectives: Assess if the NSSF’s objectives are clearly defined and understood by relevant stakeholders.
2. Programme Design and Logic: Develop, review and revised the NSSF’s Theory of Change (ToC) and assess or evaluate if the ToC or logic model logically links activities, outputs, outcomes, and impacts. This involves reviewing the programme’s implementation plan, activities, and expected results.
3. Stakeholder Engagement: Identify and consult with key stakeholders, including government officials at various levels (national, provincial, district and circuit level), educators, learners, other government departments, funders, and other relevant parties. Solicit stakeholders’ perspectives and expectations in their understanding of the framework’s context and the feasibility of an evaluation. Ascertain that a shared understanding exists among implementers and key stakeholders of the objectives, results, implementation strategy. This could be done using a hybrid approach which could be face to face or online, depending on the situation.
4. Data Availability and Quality: Assess the availability and quality of data needed for the evaluation. This includes reviewing existing monitoring (e.g. school readiness surveys, schools’ incident reports, provincial reports, etc.) and evaluation systems, data collection methods, and data sources. Confirm information requirements and data sources, and to articulate and/or refine indicators of success.
5. Evaluation Questions: Develop and refine key evaluation questions that align with the framework’s objectives and stakeholders’ information needs. Ensure these questions are clear and answerable given the available data and resources.
6. Evaluation Design: Identify appropriate evaluation designs and methods that can answer the evaluation questions. This could include experimental designs (e.g., randomized controlled trials), quasi-experimental designs (if appropriate) (e.g., matched comparison groups), or non-experimental designs (e.g., pre-post comparisons), as well as contribution analysis, etc.
7. Feasibility and Resources: Assess the feasibility of conducting the evaluation, considering the available resources, budget, time, and expertise. Ensure that the evaluation is realistic and manageable within the program’s constraints.
8. Sampling strategy/criteria: Develop a sampling strategy for programme documents and primary data sources (both government and UNICEF and other key partners).
9. Instrument development and validation: A conceptual/measurement matrix (or a checklist) for the EA will be developed, as well as draft data collection and data analysis instruments. These will be reviewed and approved, as part of the inception phase. The EA consultant will propose an approach to pilot and validate instruments in order to mitigate any conceptual and/or measurement error and submit an updated version of the data collection instruments in the early days of the execution phase.
10.  Four distinct data analysis components are proposed for the EA: (i) stakeholder analysis; (ii) desk-based review of planning documents, processes, and activities, and analysis of programming tools; and (iii) analysis of key interventions/activities at the country-level, (iv) analysis of monitoring indicators and data:
    • Stakeholder analysis. Key stakeholders will be identified including UNICEF and targeted non-UNICEF partners. Their roles and inputs will be assessed through a stakeholder analysis.
    • Desk-based review and analysis of national documents (policies, law, legislations, etc.): The EA consultant will conduct broad background reading of government, UNICEF and other key stakeholder documents.
    • Analysis of key national, including UNICEF programme activities/interventions and from other relevant partners: The EA consultant will review both the technical and management aspects of the NSSF initiatives at the country level.
    • Analysis of government monitoring and evaluation systems and data: A particular focus will be on capacities for monitoring and evaluation, the relevance and evaluability of the results framework at the national level, and outcome and output indicators. The EA consultant will carefully review the sources and reliability of information, determine what gaps there may be in the information required, and suggest activities needed to fill the gaps.

Evaluability Assessment Approach and Methodology

The EA for the purposes of this assignment will follow the approach suggested by the DFID Working Paper on Evaluability Assessment that identified these dimensions of evaluability: 

• Evaluability “in principle”, given the nature of the project theory of change.
• Evaluability “in practice”, given the availability of relevant data and the capacity of management systems able to provide it.
• The utility and practicality of an evaluation, given the views and availability of relevant stakeholders. 

The assessment will be conducted by utilizing mixed method approach

One of the first methods to be utilized will include desk-review of the documents, which may be expanded in discussion with stakeholders. The documents will include the programmatic documents, provincial and districts reports, annual School Readiness Surveys, South Africa School Administration and Management System (SASAMS), including reports from the Learner Unit Record Information and Tracking System (LURITS).

Individual and group interviews will also be conducted. Particularly, key informant interviews with the representatives of national institutions responsible for implementation and reporting of different results and/or interventions of the NSSF will need to be conducted. Round-table discussions and individual consultations with development partners, implementing partners and CSOs active in the field of violence prevention and response in schools will also be needed to understand their perspective, issues they raise, validate the TOC and develop recommendations.

Deliverables

1. Month 1: 12% of the budget
   • Inception Report (IR) developed and submitted – IR should describe preliminary desk review, the EA methodology, implementation plan, outline of the EA Report, dissemination plan.
2. Month 1: 8.40% of the budget
   • Desk-top review of both international and local literature of school violence prevention and response as well as various surveys and reporting documents from national, provincial and district as well as school level is conducted.
3. Month 1: 6% of the budget
   • ToC/ result framework developed, reviewed and analyzed
4. Month 1: 6% of the budget
   • Relevant programme activities/projects are mapped
5. Month 2: 3.60% of the budget
   • Sampling strategies developed and submitted
6. Month 2: 18% of the budget
   • Partners/stakeholders, documents, programme activities/interventions and M&E systems including relevant data is identified and analyzed. All documents or materials gathered to be shared with UNICEF.
7. Month 2: 6% of the budget
   • Round table/workshops are conducted
8. Month 2: 6% of the budget
   • Draft EA report is prepared and submitted.
9. Month 3: 12% of the budget
   • Feedback received from members of the Steering Committee (SC) is reviewed, revised and incorporated into the final document. The SC comprises of key personnel from DBE and UNICEF.
10. Month 3: 6% of the budget
    • PowerPoint Presentation as well as the final report of the EA is submitted. Based on the findings refine and update the current draft TOR for the planned NSSF evaluation
11. Travel costs: 16% of the budget

Limitations include:

• Lack of systematic documentation of the design and implementation of some interventions may limit the assessment.
• Not all key informants might be available or reachable at the time of the assessment.

The assessment should be conducted via virtual consultations when possible or by adhering to the restrictions and requirements of the country and institutions when a physical meeting is organized.

The final report should include the executive summary, background, progress on the deliverables, monitoring and evaluation, lessons learned, challenges and mitigating factors, recommendations, next plans and timelines, annexes. All the documents including the final report should be written in English and submitted electronically to the supervisor (Adolescent and Youth Development Specialist)

To qualify as an advocate for every child you will have… 

• An advanced university degree (Master’s or higher) in  Social Services, Research or Development
• A minimum of five years of relevant professional experience in monitoring and evaluation with a proven record of conducting evaluations
  • Knowledge of how UNICEF or other UN agencies or developmental agencies and the South African government system works at national, provincial and local level
  • Excellent conceptual and analytical skills
  • Good skills in Microsoft Word, MS Excel, PowerPoint, and other necessary software e.g. Adobe, graphics, etc.
  • Excellent and concise English writing skills
  • Proactive and energetic approach to the work
  • Ability and commitment to work to a tight time frame
  • Ability to manage and supervise evaluation teams and ensure timely submission of quality evaluation reports
  • Ability to deal with multi-stakeholder groups
  • Ability to write focused evaluation reports
  • Willingness and ability to travel to the different project's sites in the country and must be in possession of a valid driver’s license

For every Child, you demonstrate… 

UNICEF's values of Care, Respect, Integrity, Trust, Accountability, and Sustainability (CRITAS). 

To view our competency framework, please visit  here. 

UNICEF is committed to diversity and inclusion within its workforce, and encourages all candidates, irrespective of gender, nationality, religious or ethnic background, and persons with disabilities, to apply to become a part of the organization. To create a more inclusive workplace, UNICEF offers paid parental leave, breastfeeding breaks, and reasonable accommodation for persons with disabilities. UNICEF strongly encourages the use of flexible working arrangements. Click here to learn more about flexible work arrangements, well-being, and benefits.

According to the UN Convention on the Rights of Persons with Disabilities (UNCRPD), persons with disabilities include those who have long-term physical, mental, intellectual, or sensory impairments which, in interaction with various barriers, may hinder their full and effective participation in society on an equal basis with others. In its Disability Inclusion Policy and Strategy 2022-2030, UNICEF has committed to increase the number of employees with disabilities by 2030. At UNICEF, we provide reasonable accommodation for work-related support requirements of candidates and employees with disabilities. Also, UNICEF has launched a Global Accessibility Helpdesk to strengthen physical and digital accessibility. If you are an applicant with a disability who needs digital accessibility support in completing the online application, please submit your request through the accessibility email button on the UNICEF Careers webpage Accessibility | UNICEF.

UNICEF does not hire candidates who are married to children (persons under 18). UNICEF has a zero-tolerance policy on conduct that is incompatible with the aims and objectives of the United Nations and UNICEF, including sexual exploitation and abuse, sexual harassment, abuse of authority and discrimination based on gender, nationality, age, race, sexual orientation, religious or ethnic background or disabilities. UNICEF is committed to promote the protection and safeguarding of all children. All selected candidates will, therefore, undergo rigorous reference and background checks, and will be expected to adhere to these standards and principles. Background checks will include the verification of academic credential(s) and employment history. Selected candidates may be required to provide additional information to conduct a background check, and selected candidates with disabilities may be requested to submit supporting documentation in relation to their disability confidentially.

Remarks:  

Only shortlisted candidates will be contacted and advance to the next stage of the selection process. 

Individuals engaged under a consultancy or individual contract will not be considered “staff members” under the Staff Regulations and Rules of the United Nations and UNICEF’s policies and procedures, and will not be entitled to benefits provided therein (such as leave entitlements and medical insurance coverage). Their conditions of service will be governed by their contract and the General Conditions of Contracts for the Services of Consultants and Individual Contractors. Consultants and individual contractors are responsible for determining their tax liabilities and for the payment of any taxes and/or duties, in accordance with local or other applicable laws.