Randomised Controlled Trials (RCTs) are considered by many to be the gold standard in demonstrating the real impact of an intervention such as the introduction of a new medicine; but translating the approach to the social sciences is not without challenges. Within the field of public policy, conducting an RCT can be viewed as a challenge to traditional institutional practices, raise ethical and political concerns, and be too complex for non-expert staff to design and implement.
New Economy and NatCen Social Research recently concluded research to identify a business support service in Greater Manchester that could become the subject of an RCT. Our study offers a number of policy insights for both researchers and programme leads interested in testing RCTs on innovation, entrepreneurship, business growth and social policy.
A simple way of assessing the suitability of RCT candidates
Our study employed a matrix of qualitative judgements on anticipated programme sample sizes, the potential for generalisability of research findings, the availability of relevant monitoring data, the number and types of potential comparison conditions, and the ease of implementing a trial. More detail on the matrix is available here. Using the matrix, we were able to appraise seven different business support services in a relatively short time and identify which service had the greatest potential to be the subject of an RCT.
Here are a few important lessons we take away from the project:
Early engagement between researchers and programme managers
Many of the business support services we reviewed are delivered by a combination of in-house and external resources, with sub-consultants delivering some but not all aspects of the service under contract with a prime provider who oversees overall delivery and monitoring. This model of delivery is also seen in policy areas such as worklessness and skills. The Service Heads we spoke to saw this complexity as a potential barrier to implementing an RCT. They mentioned contractual obligations towards funders to support all businesses in need and contractual agreements with sub-contractors over the numbers of businesses they would get to work with. Service Heads were concerned that randomising the companies who receive support, and hence which sub-contractors receive fees, would contravene the signed delivery contracts. Early engagement between researchers or evaluators and programme managers can help avoid this type of situation later on.
A clear strategy on communicating with businesses
Staff involved in delivering business support services told us that without clear communication around why randomisation is being applied, rejected applicants or those applicants who perceive they are receiving less support despite meeting selection criteria may be dissuaded from engaging with business support programmes in future and/or may make their grievances known within the local business community. At a time when the funding available for delivering business support is limited and highly competitive and marketing budgets are stretched, negative feedback from businesses is obviously unwelcome. Delivery staff aim to provide clear and consistent communications to businesses but with lots of other information to communicate there is concern around whether a brief mention of a trial being conducted would be fully understood.
If these concerns can be addressed we are optimistic that RCTs of business support services could proceed. All the Service Heads we spoke to were interested in and, in many cases, aware of how RCTs operate and the added value they can offer when it comes to redesigning a service.
The research above was funded by the Innovation Growth Lab (IGL) Grants Programme. We offer research funding to randomised controlled trials that help build the evidence base on the most effective approaches to increase innovation, support entrepreneurship and accelerate business growth. For more information please click here.
Image copyright David Dixon
Researchers
David Morris, Head of Evaluation, New Economy
Martina Vojtkova, Head of Evaluation, NatCen Social Research