Apprenticeship SME Brokerage Pathfinder: Early Evaluation — Department for Education

Small and Medium Enterprises (SMEs) may face greater barriers to hiring apprentices than larger businesses. These include not understanding where to begin in the hiring process, struggling with the perceived complexities of the apprenticeship landscape, and a lack of capacity to hire and train apprentices.

The Department for Education’s (DfE’s) Apprenticeship SME Brokerage Pathfinder aims to overcome these barriers by commissioning independent brokers to support SMEs in hiring apprentices. The brokers provide SMEs with information on apprenticeships and their benefits, help with setting up Apprenticeship Service accounts, assist in choosing an appropriate Apprenticeship Standard and training provider, and recruit candidates for the role.

The Pathfinder operates in four areas: Greater Manchester and East Lancashire, Merseyside and Northwest Lancashire, West Yorkshire and South Yorkshire, and the North East. Within the Pathfinder, four sectors are of interest, deemed a priority due to recent declines in apprenticeship starts: construction, manufacturing, adult social care, and digital.

Alma Economics was commissioned to conduct an early evaluation of the Apprenticeship SME Brokerage Pathfinder, providing insights into the delivery of the Pathfinder and enabling an understanding of why participation by SMEs has been lower than anticipated.

As part of the early evaluation, we conducted:

• A desk-based review of policy documents and Pathfinder monitoring data,

• Interviews with key stakeholders (brokers, training providers, and SMEs), covering regional and sectoral challenges and barriers, as well as their experience with engagement, onboarding, and apprenticeship starts, and

• A survey of SMEs in relevant geographic areas to explore their interest in the type of support offered within the Pathfinder.

➥ You can find all our findings and recommendations in the research report, here.

Teaching relationships education to prevent sexual abuse — Department for Education

Alma Economics was commissioned by the Department for Education to review the evidence on teaching relationships education to prevent sexual abuse.

Our team conducted a flexible Rapid Evidence Assessment (REA) to identify and review the most relevant literature within a short timeframe. Our evidence review covered a wide range of research questions on issues including boundaries, consent, privacy, sexual violence, sexual harassment, and domestic and non-peer violence. We also formed an 8-member Expert Group comprised of academics and professionals with expertise in topics including relationships education, sexual and online abuse, and gender-based violence. With their advice and support, a final list of 18 recommendations was created.

The report aims to inform practitioners, ensure relationships education is based on the best possible evidence, and ultimately contribute to preventing sexual abuse.

➥ You can find all our findings and recommendations in the research report here.

Numeracy skills interventions for adults — Department for Education

A significant portion of the adult population in the UK has numeracy skills below grade C at GCSE level. This presents high costs to individuals and society in the form of lower individual wages, losses in productivity, and forgone tax revenue.

Our team was commissioned by the Department for Education (DfE) to explore evidence around the numeracy skill level of adults in the UK, and the policy landscape around adult skills over the last 20 years. We analysed Wave 11 of the UK Understanding Society database to determine what factors are correlated with low numeracy attainment and created a search strategy to systematically review the academic and grey literature on adult numeracy policy from the UK and other OECD countries. Our systematic review and data analysis was supplemented by the testimony of experts in four semi-structured interviews and a nine-person focus group.

Our resulting report and evidence map provide a comprehensive overview of the literature and identify gaps and areas to target for future research.

➥ Read our report here.

➥ Explore our Evidence Map here.

International Evidence Review for Disability Employment — Department for Work and Pensions

Alma Economics was commissioned by the Department for Work and Pensions (DWP) to conduct an evidence review of programmes supporting disabled people and people with health conditions to enter the labour market and remain in employment.

Our team followed a systematic and transparent methodology to retrieve evidence from the academic and grey literature. The review explored (i) international and UK interventions aiming to help people move into and progress in employment, (ii) the differences in outcomes across different demographic groups, and (iii) the factors that proved successful for disabled people and people with health conditions in moving closer towards employment.

We also developed an interactive online tool which enables users to explore the evidence of interest swiftly and identify gaps in the literature by using the available filters.

Access the Evidence Map here.

Mapping Early Learning and Childcare (ELC) provision for 1- and 2-year-olds in Scotland — Scottish Government

From August 2021, the funded entitlement to Early Learning and Childcare (ELC) in Scotland increased from 600 to 1,140 hours per year for all 3- and 4-year-olds as well as eligible 2-year-olds. In the 2021 Programme for Government, the Scottish Government further committed to expanding funded ELC to all 1- and 2-year-olds. Alma Economics was commissioned by the Scottish Government to map existing ELC provision for 1- and 2-year-olds in Scotland to understand what provision currently exists for this age group, identify good practice, and understand challenges that need to be addressed within the design of the new policy.

Through analysis of secondary data, a survey of local authorities, interviews and focus groups with relevant stakeholder groups, and interviews and a survey of ELC providers, we mapped national ELC provision for 1- and 2-year-olds in Scotland and created detailed case studies focusing on 4 local authorities. We found that childminders and nurseries constitute the most common provision type for younger children. However, across all provider types, there are gaps in provision for 1- and 2-year-olds. Those gaps are more pronounced in rural, island and remote communities. The recruitment and retention of staff were emphasised as primary challenges in providing ELC to younger children, together with capacity issues and the need for facility changes and additions to appropriately address the needs of 1- and 2-year-olds. Holistic ELC programmes that support both children and their families and programmes tailored to the varying needs of children and families were identified as examples of good practice.

➥ Read the report here.

Teachers’ pay and conditions: international comparisons — Independent Welsh Pay Review Body

To inform a strategic review carried out by the Independent Welsh Pay Review Body (IWPRB), Alma Economics was commissioned to research teachers’ pay terms and conditions in Wales. The aim was to inform best practice or changes that could be introduced in Wales to help deliver the Welsh Government’s national mission.

Through benchmarking analysis, correlation analysis, and case studies, our team researched the pay and conditions for teachers and school leaders in primary and secondary education across OECD countries. We examined the correlation of pay and conditions with pupils’ education outcomes and with teachers’ recruitment, retention, and professional development. We also held a workshop to present and discuss findings with key stakeholders.

Our final reports were cited in the IWPRB’s strategic review and included in full as annexes.

➥You read can the strategic review and our reports here.

Economic Implications and Benefits Assessment of an Updated UKRI Open Access Policy for Peer-Reviewed Research Articles

UK Research and Innovation (UKRI) asked Alma Economics to carry out an independent review of the economic implications of Open Access (OA). Under UKRI’s new OA policy all UKRI-funded research output will be published Open Access. The report will be used alongside the results of the OA Review’s consultation and other Review activities to inform UKRI’s new OA policy.

To address the research objectives, we adopted a mixed-methods approach that comprised a desk-based review, engagement with key groups of stakeholders, as well as quantitative data analysis and modelling. We developed a comprehensive model to provide detailed estimates of the impact of the new OA policy on publishers, universities, and funding bodies under different policy scenarios. Combining different data sources with qualitative information gathered during our interviews with universities, publishers, funders, and researchers, we estimated the costs of publishing OA. To this end, we estimated additional OA expenditures under the new OA policy using an econometric model to predict publication fees based on journals’ CiteScore and academic subject area.

We then compared the costs with the benefits of OA. Our model focuses specifically on quantifying two key benefits: the efficiency gains throughout the research process due to the lower costs of OA publishing models, and the additional social returns to investment in R&D as knowledge in OA-published research forms the basis of further research or real-world applications across the public and private sector as well as universities.

➥ Our report is available on UKRI’s website.

Economic Evaluation of Fair Start Scotland — Scottish Government

The Scottish Government commissioned Alma Economics to carry out an economic evaluation of Fair Start Scotland. This is the Scottish Government’s first fully devolved employment programme and it is aimed at people who may find it difficult to find employment due to disabilities or long-term health conditions.

To deliver the evaluation, our economists worked closely with our developer team to create a bespoke algorithm to streamline the analysis. This led to more accurate results and helped us generate additional insights into the wider impact of the service.

In order to ensure that the Scottish Government met its goal of inclusive growth and improving wellbeing, we went beyond financial analysis to ensure that the service was reaching everyone and helping them achieve good quality jobs. Our report makes specific recommendations to improve the service. Crucially, these include adjusting incentive payments to providers so that the service can reach more people with disabilities and participants can achieve better work conditions and fairer pay.

➥ The full report can be found here.

Review of benchmarking methodologies — Office for Students

Alma Economics conducted a review of benchmarking methods for the Higher Education (HE) sector. The goal of this project was to list and assess prospective techniques for the Office for Students (OfS), the regulator of HE providers in England, to make meaningful comparisons across the sector.

Our research provided the OfS with up-to-date evidence on benchmarking methods and a thorough assessment of the strengths and weaknesses of each approach.

This review covered complex benchmarking approaches in a clear, concise manner, presenting both an accessible version and technical extensions to tailor findings to different audiences and maximise the impact of the analysis.

➥ Our review of benchmarking methodologies is available on the OfS website.

New approaches to regulating student outcomes and the Teaching Excellence Framework — Office for Students

Alma Economics was commissioned by the Office for Students (OfS) to analyse the responses to two consultations. The consultations, which were open from January to March 2022, set out a proposed new approach to the Teaching Excellence Framework (TEF) and to regulating student outcomes. They sought views from anyone interested in the regulation of the higher education sector and received around 500 responses in total.

Our team carried out quantitative and qualitative analysis of the responses. We produced summary statistics to understand the degree of agreement with the consultation proposals and conducted a thematic analysis of open-text responses. Firstly, we developed a preliminary set of themes based on the consultation document and our understanding of OfS regulations. Then, additional themes were added during the review of the open-text responses. This method allowed a consistent approach to reading while allowing the flexibility to capture additional insights from each response.

We developed a codebook recording all qualitative themes identified by our team to summarise the views of respondents. Our reports summarise the key themes raised by the respondents, aiming to inform the OfS approach to regulating higher education in England.

➥ The Teaching Excellence Framework (TEF): Analysis of consultations responses

➥ A new approach to regulating student outcomes: Analysis of consultation responses

NHS apprenticeship modelling — Health Education England

Alma Economics was commissioned by Health Education England (HEE) to assess the cost implications of clinical apprenticeships in London Trusts. Our team constructed a microsimulation model to compare the financial implications of expanding apprenticeship delivery with a “business-as-usual” scenario.

By examining the difference between these two scenarios, we estimated the expected costs and savings associated with apprenticeships in London Trusts over a 10-year period. The model was informed by administrative data provided by HEE, qualitative information gathered from an online survey of 21 London Trusts, and interviews with education providers and apprenticeship leads.

Interviewees also provided access to internal workforce modelling and ongoing research. Some of the main costs we assumed in our analysis were backfill, hiring, administrative, and supervision costs, as well as earnings. In terms of benefits, we considered retention rates, the value of career development opportunities, the value of internally trained staff to Trusts and the reduced use of bank and agency staff to fill vacancies.

Consultation on contingency arrangements for the award of GCSE, AS, A level, Project and AEA qualifications in 2022 — Department for Education/Ofqual

Ofqual asked us to analyse responses to a joint Department for Education and Ofqual consultation on contingency arrangements in case summer 2022 exams could not take place safely or fairly.

Our team read all responses to the consultation, mapping each individual or organisation to a respondent type (such as school/college staff, students or education/training providers). For each of the consultation questions, we either carried out a segmentation analysis across respondent types (for closed-ended questions) or conducted a thematic analysis to identify key ideas and perspectives (for open-ended questions). As part of this process, we applied a machine-assisted approach, with codes generated by machine learning models in Python used as a quality assurance check of manually applied codes. In addition, we developed a codebook that summarised the distribution of key themes across different respondent types.

This work helped inform the arrangements for the Teacher Assessed Grades (TAGs) process in the event that summer 2022 exams would have been cancelled, ensuring that students would be in a position to successfully progress to the next stage of their education or to employment even if public health restrictions meant that exams could not go ahead.

➥ Find more on the consultation here.

➥ The report is available here.

Consultation analysis on arrangements for an autumn 2021 exam series — Ofqual

Ofqual commissioned our team to analyse the responses to the consultation on arrangements for an autumn 2021 exam series. We conducted segmentation analysis of the responses to the closed questions, presenting frequencies of answers broken down by respondent groups. For the open-ended questions, we carefully read all responses in full, and we summarised the key themes which emerged from respondents’ answers. Additionally, using a Python programme we developed, we created a codebook of answers which recorded all qualitative themes identified by responses.

This work helped identify the best way the exams should run in autumn to create equal opportunities in education for all students, despite the difficulties caused by the Covid-19 pandemic.

➥ Find more on the consultation here.

➥ The report is available here.

Consultation on proposed changes to the assessment of GCSEs, AS and A levels in 2022: Analysis of responses — Ofqual

Ofqual commissioned our team to analyse responses to a joint DfE and Ofqual consultation on arrangements for the summer 2022 exam series. Our team carried out segmentation analysis of quantitative questions, measuring frequencies of answers across respondent types. In addition, we conducted thematic analysis for responses to open-ended qualitative questions, identifying key ideas and issues raised by respondents. To support this, we developed a Python program to create a codebook of answers which recorded all qualitative themes identified by responses.

This project was delivered within tight timescales (under 2 months) by a large team of experienced researchers. Our work helped identify the best way exams and assessments should run in summer 2022 to reduce the impact of Covid-19 disruptions on students’ education, helping students be in the best possible position to successfully progress to the next stage of their education or to employment.

➥ Find more on the consultation here.

➥ The report is available here.

Statistical analysis of Alternative Provision in the UK — Department for Education

Alma Economics conducted research for the Department for Education (DfE) to gather information about how alternative provision (AP) in local areas is organised. AP refers to specialised educational arrangements designed for students unable to attend mainstream school.

Our analysis explored the factors that influence demand for AP, and what makes for an effective “local AP system”. Using survey and publicly available data, our team carried out statistical analysis to understand which variables appear to affect the operation of the AP market, e.g. places, pupils placed, costs, outcomes, and duration of placements.

Cost-benefit analysis of education reforms: A rapid evidence assessment — Welsh Government

The Welsh Government commissioned Alma Economics to evaluate the potential benefits of proposed reforms in the regulation of post-compulsory education and training in the country.

Our team carried out a Rapid Evidence Assessment (REA), reviewing evidence from a range of sources, including academic literature and government publications. We gathered evidence on the benefits of similar reforms in the rest of the UK and overseas, as well as the impact of similar reforms in the broader field of education. We also interviewed key organisations in the sector, gathering their views on the potential costs, benefits, and challenges of the proposed bill.

The review provided a thorough assessment of the evidence on the expected benefits of the reforms, informing Welsh Government consultations on the Tertiary Education and Research Bill.

➥ Our REA was published as part of the Regulatory Impact Assessment of the Bill and can be found here.

Headmasters’ survey project — Children’s Commissioner’s Office

The Children’s Commissioner’s Office (CCO) asked Alma Economics to carry out a comprehensive analysis of a large survey administered to all Head Teachers in England. The survey examined the current provision of education programmes related to the prevention of child sexual abuse in English schools.

We examined the data collected and provided recommendations on ways to support and strengthen the work. The analysis was integrated into a final report published by the CCO.

➥ The report can be found here.