Evidence-based HR: Make better decisions and step up your influence

The basic idea of evidence-based practice is that high-quality decisions and effective practices are based on critically appraised evidence from multiple sources. When we say ‘evidence’, we mean information, facts or data supporting (or contradicting) a claim, assumption or hypothesis. This evidence may come from scientific research, the local organisation, experienced professionals or relevant stakeholders. We use the following definition from CEBMa:

“Evidence-based practice is about making decisions through the conscientious, explicit and judicious use of the best available evidence from multiple sources… to increase the likelihood of a favourable outcome.”

This technical definition is worth unpacking.

Conscientious means that you make a real effort to gather and use evidence from multiple sources – not just professional opinion. Good decisions will draw on evidence from other sources as well: the scientific literature, the organisation itself, and the judgement of experienced professionals.

Explicit means you take a systematic, step-by-step approach that is transparent and reproducible – describing in detail how you acquired the evidence and how you evaluated its quality. In addition, in order to prevent cherry-picking, you make explicit the criteria you used to select the evidence.

Judicious means critically appraised. Evidence-based practice is not about using all the evidence you can find, but focuses only on the most reliable and trustworthy evidence.

Increased likelihood means that taking an evidence-based approach does not guarantee a certain outcome. Evidence-based people professionals typically make decisions not based on conclusive, solid evidence, but on probabilities, indications and tentative conclusions. As such, an evidence-based approach does not tell you what to decide, but it does help you to make a better-informed decision.

The importance of evidence-based practice and the problems it sets out to solve is explained in more detail in our factsheet and thought leadership article but, in essence, it has three main benefits:

1. It ensures that decision-making is based on fact, rather than outdated insights, short-term fads and natural bias.
2. It creates a stronger body of knowledge and as a result, a more trusted profession.
3. It gives more gravitas to professionals, leads to increased influence on other business leaders and has a more positive impact in work.

Before making an important decision or introducing a new practice, an evidence-based people professional should start by asking: "What is the available evidence?" As a minimum, people professionals should consider four sources of evidence.

Evidence from people professionals

The expertise and professional judgement of practitioners, such as colleagues, managers, staff members, employees and leaders, is vital for determining whether a people management issue does require attention, if the data from the organisation are reliable, whether research findings are applicable, or whether a proposed solution or practice is likely to work given the organisational context.

Evidence from scientific literature

In past decades, a large number of scientific studies have been published on topics relevant to people professionals, for example on topics such as the characteristics of effective teams, the drivers of knowledge worker performance, the recruitment and selection of personnel, the effect of feedback on employee performance, the antecedents of absenteeism, and the predictors of staff turnover. Empirical studies published in peer-reviewed journals are especially relevant, as they provide the strongest evidence on cause-and-effect relationships, and thus what works in practice.

Evidence from the organisation

This can be financial data or performance indicators (for example, number of sales, costs, return on investment, market share), but it can also come from customers (for example, customer satisfaction, brand recognition), or employees (for example, task performance, job satisfaction). It can be ‘hard’ numbers such as staff turnover rates, medical errors or productivity levels, but it can also include ‘soft’ elements such as perceptions of the organisation’s risk climate or attitudes towards senior management. This source of evidence typically helps leaders identify the existence or scale of a need or problem, possible causes and potential solutions.

Evidence from stakeholders

Stakeholders are people (individuals or groups inside or outside the organisation) whose interests affect or are affected by a decision and its outcomes. For example, internal stakeholders such as employees can be affected by the decision to reduce the number of staff, or external stakeholders such as suppliers may be affected by the decision to apply higher-quality standards. However, stakeholders can also influence the outcome of a decision; for example, employees can go on strike, regulators can block a merger, and the general public can stop buying a company’s products. For this reason, this evidence is often an important guide to what needs or issues an organisation investigates, what improvements it considers and whether any trade-offs or unintended consequences of proposed interventions are acceptable.

The importance of combining all sources

Finally, none of the four sources are enough on their own – every source of evidence has its limitations and its weaknesses. It may not always be practical to draw on all four sources of evidence or to cover each of them thoroughly, but the more we can do, the better decisions will be.

Since the 1990s, evidence-based practice has become an established standard in many professions. The principles and practices were first developed in the field of medicine and following this, have been applied in a range of professions – including architecture, agriculture, crime and justice, education, international development, nutrition and social welfare, as well as management. Despite differing contexts, the approach broadly remains the same.

Below, each step is discussed and illustrated with examples. It is important to note that following all six steps will not always be feasible. However, the more that professionals can do – especially when making major or strategic decisions – the better.

Asking questions to clarify the problem and potential solution and to check whether there is evidence in support of that problem and solution is an essential first step. Without these questions the search for evidence will be haphazard, the appraisal of the evidence arbitrary, and its applicability uncertain.

Asking critical questions should be constructive and informative. It is not about tearing apart or dismissing other people's ideas and suggestions. By the same token, evidence-based practice is not an exercise in myth busting, but rather seeking to establish whether claims are likely to be true and potential solutions are likely to be effective.

Example 1 shows the type of questions you can ask.

Example 1: Autonomous teams – an example of asking critical questions

Consider a typical starting point: a senior manager asks you to develop and implement autonomous teams in the organisation. Rather than jumping into action and implementing the proposed solution, an evidence-based approach first asks questions to clarify the (assumed) problem:

• What is the problem we are trying to solve with autonomous teams?
• How do we know we have this problem? What is the evidence?
• What are the organisational consequences of this problem?
• How serious and how urgent is this problem? What happens if we do nothing?

The next step would be to ask more specific questions on whether there is sufficient evidence confirming the existence and seriousness of the problem. The senior manager explains that the organisation has a serious problem with absenteeism, and that a lack of autonomy – employees’ discretion and independence to schedule their work and determine how it is to be done – is assumed to be its major cause. Important questions to ask are:

• Do experienced practitioners (for example, supervisors, managers) agree we have a serious problem with absenteeism? Do they agree lack of autonomy is a major cause?
• Do the organisational data confirm we have a problem with absenteeism? How does our rate of absenteeism compare to the average in the sector? Is there a trend? Do the data suggest the problem will increase when nothing is done?
• Does the scientific literature confirm that lack of autonomy is an important driver of absenteeism? What are other common causes?
• How do stakeholders (for example, employees, supervisors) feel about the problem? Do they agree lack of autonomy is a major cause?

Based on the answers, you should be able to conclude whether there is sufficient evidence to support the senior manager’s claim that the organisation has a problem with absenteeism, and that this problem is most likely caused by a lack of autonomy. If one or more questions can’t be answered, this may be an indication that more evidence is needed. The next step would be to ask the executive manager critical questions about the proposed solution:

• Do we have a clear idea of what autonomous teams are? How are they different from ‘traditional’ teams?
• How exactly are autonomous teams supposed to have a positive effect on absenteeism? How does this work? What is the causal mechanism/logic model?

The final step would be to ask questions to check whether there is sufficient evidence from multiple sources indicating the proposed solution will indeed solve the problem:

• Do experienced practitioners (for example, supervisors, managers) agree that the introduction of autonomous teams is the ’best’ solution to lower the organisation’s absenteeism rate? Do they see downsides or unintended negative consequences? Do they see alternative solutions that may work better?
• Can organisational data be used to monitor the impact of autonomous teams on absenteeism?
• Does the scientific literature confirm that autonomous teams have a positive effect on absenteeism? Does the literature suggest other solutions that may work better?
• How do stakeholders (for example, employees, supervisors) feel about the introduction of autonomous teams? Do they think it will have a positive impact on absenteeism?

.

Based on the answers to the questions in Step 1, we should have a good understanding of whether there is sufficient evidence from multiple sources to support the assumed problem and preferred solution. In most cases, however, the available evidence is too limited or important sources are missing. In that case, we proceed with the second step of evidence-based practice: acquiring evidence.

Acquiring evidence from practitioners

This could be through:

• face-to-face conversations: this is the easiest way. While it can be prone to bias, sometimes simply asking people about their experience can give good insight.
• more structured interactive group meetings, workshops or other ways of collecting views, such as surveys.

Practitioner expertise is a useful starting point in evidence-based practice to understand the assumed problem and preferred solutions. It is also helpful in interpreting other sources of evidence – for example, in assessing whether insights from scientific literature are relevant to the current context.

Acquiring evidence from scientific literature

This could be through the following types of publication:

• Peer-reviewed academic journals, which can be found in research databases. These are usually behind a paywall or are accessible only through a university, but CIPD members have access to one such database in EBSCO's Discovery Service. It is also worth noting that different databases focus on different specialisms – for example, like the Discovery Service, EBSCO's more expansive Business Source Elite and ProQuest's ABI/INFORM cover business and management in general, whereas the APA's PsychINFO focuses on psychology (these are all available via CEBMa). However, even once you access them, peer-reviewed articles often contain theoretical and technical information that's hard to understand for non-researchers.
• ‘Evidence reviews’ such as systematic reviews (see below) and shorter rapid evidence assessments (REAs). These are easier to use as they aim to identify and summarise the most relevant studies on a specific topic. They also do the work of identifying the best research, and selecting and critically appraising  studies on the basis of explicit criteria. The CIPD produces evidence reviews on a range of HR and L&D topics – you can access these via our Evidence review hub.

Acquiring evidence from the organisation

This could be through the following sources:

• Internal management information: Often the finance department and the HR/personnel department are the key custodians of people data and analytics.
• Internal research and evaluation: This could be conducted via trials of interventions, bespoke surveys or focus groups.
• External sources such as census bureaus, industry bodies, professional associations and regulators. However, sometimes relevant organisational data is not available, either because collecting it is too time-consuming and costly, because of data sensitivities and a lack of disclosure (for example on employee diversity), or simply due to a lack of analytical capability in processing and interpreting data.

Acquiring evidence from stakeholders

Organisational decisions often have lots of stakeholders both inside and outside the organisation. A stakeholder map is therefore a useful tool to identify which stakeholders are the most relevant. A stakeholder’s relevance is determined by two variables:

1. The extent to which the stakeholder’s interests are affected by the decision (harms and benefits).
2. The extent to which the stakeholder can affect the decision (power to influence).

When the most important stakeholders are identified, often qualitative methods such as focus groups and in-depth interviews are used to discuss their concerns.

Unfortunately, evidence is never perfect and can be misleading in many different ways. Sometimes the evidence is so weak that it is hardly convincing at all, while at other times the evidence is so strong that no one doubts its correctness. After we have acquired the evidence we therefore need to critically appraise what evidence is ‘best’ – that is, the most trustworthy.

Appraising evidence from practitioners

When appraising the evidence from practitioners, the first step is to determine whether their insights and opinions are based on relevant experience or personal opinions. Next, we need to determine how valid and reliable that experience is. We can assess this by considering:

• whether the experience concerns repeated experience
• whether the situation allowed for direct, objective feedback
• whether the experience was gained within a regular, predictable work environment. 

For example, based on these three criteria, it can be determined that the expertise of a sales agent is more likely to be trustworthy than the expertise of a business consultant specialised in mergers. In general, sales agents work within a relatively steady and predictable work environment, they give their sales pitch several times a week, and they receive frequent, direct and objective feedback. Consultants, however, are involved in a merger only a few times a year (often less), so there are not many opportunities to learn from experience. In addition, the outcome of a merger is often hard to determine – what is regarded as a success by one person may be seen as a failure by another. Finally, consultants accompanying a merger do not typically operate in a regular and predictable environment: contextual factors such as organisational differences, power struggles and economic developments often affect the outcome.

Professional expertise that is not based on valid and reliable experience is especially prone to bias, but any evidence from practitioners is likely to reflect personal views and be open to bias (see Building an evidence-based people profession). For this reason, we should always ask ourselves how a seemingly experienced professional’s judgement could be biased and always look at it alongside other evidence.

Appraising evidence from scientific literature

Appraising scientific evidence requires a certain amount of research understanding. To critically appraise findings from scientific research, we need to understand a study’s ‘design’ (the methods and procedures used to collect and analyse data). Examples of common study designs are cross-sectional studies (surveys), experiments (such as randomised controlled trials, otherwise known as ‘RCTs’), qualitative case studies, and meta-analyses. The first step is to determine whether a study’s design is the best way to answer the research question. This is referred to as ‘methodological appropriateness’.

Different types of research questions occur in the domain of people management. Very often we are concerned with ‘cause-and-effect’ or ‘impact’ questions, for example:

• What works in managing effective virtual teams?
• Does digital work affect mental wellbeing?
• How can managers help employees be more resilient?
• What makes goal setting and feedback more effective in improving performance?

The most appropriate study designs to answer cause-and-effect questions are RCTs and controlled before-after studies, along with systematic reviews and meta-analyses that gather together these types of study.

Other research questions that are relevant to people professionals are questions about prevalence (for example, “How common is burnout among nurses in hospitals?”), attitudes (for example, “How do employees feel about working in autonomous teams?”), prediction (for example, “What are drivers/predictors of absenteeism?”) or differences (for example, “Is there a difference in task performance between virtual teams and traditional teams?”).

Each of these questions would ideally have a specific study design to guarantee a valid and reliable (non-biased) answer.

An overview of common study designs and what they involve can be found in the Appendix. Following this (also in the Appendix) is an overview of types of questions and the appropriateness of each design. This gives a useful guide for both designing and appraising studies. For example, if you want information on how prevalent a problem is, a survey will work best; a qualitative study will give the greatest insight into people’s experiences of, or feelings about, the problem; and an RCT or before-after study will give the best information on whether a solution to the problem has the desired impact.

Of course, which design a study uses to answer a research question is not the only important aspect to consider. The quality of the study design – how well it was conducted – is equally important. For example, key considerations in quantitative studies include how participants are selected and whether measures are reliable and valid.² For systematic reviews and meta-analyses, a key question is how included studies are selected.

Finally, if an impact study seems to be trustworthy, we want to understand what the impact is of the intervention or factor being studied. Statistical measures of ‘effect sizes’ give us this information, both for an intervention or factors of influence itself, and how it compares to others. Being able to compare effect sizes is very important for practice. For example, the critical question is not simply, “Does a new management practice have a small or large effect on performance?” but rather, “Is this the best approach or are other practices more impactful?” For more information on effect sizes, see Effects sizes and interpreting research findings in the Appendix.

Appraising evidence from the organisation

When critically appraising evidence from the organisation, the first thing to determine is whether the data are accurate. Nowadays, many organisations have advanced management information systems that present metrics and KPIs in the form of graphs, charts and appealing visualisations, giving the data a sense of objectivity. However, the data in such systems are often collected by people, which is in fact a social and political endeavour. An important appraisal question therefore is: “Were the data collected, processed and reported in a reliable way?”

In addition to the accuracy of data, several other factors can affect its trustworthiness, such as measurement error, missing contextual information and the absence of a logic model. Some organisations use advanced data-analytic techniques that involve big data, artificial intelligence or machine learning. Big data and AI technology often raise serious social, ethical and political concerns as these techniques are based on complex mathematical algorithms that can have hidden biases and, as a result, may introduce gender or racial biases into the decision-making process.

Appraising evidence from stakeholders

Unlike the scientific literature and organisational data, which serve to give objectifiable and trustworthy insights, stakeholder evidence concerns subjective feelings and perceptions that can’t be considered as facts. Nonetheless, we can make sure that stakeholder evidence comes from a representative sample, so that it is an accurate reflection of all relevant stakeholders.

Evidence-based practitioners should present stakeholders with a clear view of the other sources of evidence. That is, they should summarise what the body of published research and organisational data tell us, as viewed with the benefit of professional knowledge. This can serve as the basis for a well-informed and meaningful two-way exchange. For example, the scientific literature may point to a certain solution being most effective, but stakeholders may advise on other important aspects that should be weighed up against this evidence – for example, whether the intervention is difficult to implement, ethically questionable or too expensive.

Use the best available evidence

The purpose of critical appraisal is to determine which evidence is the best available – that is, the most trustworthy. Sometimes, the quality of the evidence available is less than ideal; for example, there may not be any randomised controlled trials on your intervention of interest. But this does not leave us empty-handed. We can look at other studies that are less trustworthy but still go some way to showing cause-and-effect. Indeed, it’s possible that the best available evidence on an important question is the professional experience of a single colleague. However, even this limited evidence can still lead to a better decision than not using it, as long as we are aware of and open about its limitations. A useful maxim is that “the perfect is the enemy of the good”: if you don’t have the ideal evidence, you can still be evidence-based in how you make decisions.

After we have acquired and critically appraised the different types of evidence, how should we bring them all together? The broad process of knitting together evidence from multiple sources is more craft than science. It should be based on the question you wish to answer and the resources available.

A potential approach is illustrated in Figure 3 below. The steps illustrated here are as follows:

1. The starting point is evidence from the organisation in the form of people data; for example, let’s say employee survey results and key performance indicators have identified a likely problem.
2. This evidence informs the next phase: workshop or roundtable discussions with practitioner experts and stakeholders on issues the organisation faces.
3. Once there is agreement on the priority issues, the project managers scope researchable questions, which are examined in an evidence review of the published scientific literature.
4. The review finds little research on a practice of particular interest, so to fill the evidence gap, researchers run an in-house trial.
5. The findings of this pilot are presented to practitioner experts and stakeholders, discussing with them the implications for practice.
6. All the sources of evidence, including expert and stakeholder views, are then brought together into a final report with recommendations.
7. These are presented and discussed with stakeholders in a final workshop.

In most cases, the answer as to whether to implement a new practice is not a simple yes or no. Questions to consider include the following:

• Does the evidence apply to our organisational context?
• Is the intervention in question the most effective or are others more effective?
• Are the anticipated benefits likely to outweigh any risks?
• Are there ethical issues to consider; for example, if the benefits aren't evenly distributed among shareholders?
• Do the costs, necessary resources and timescale fit the organisation’s needs?

The final part of this step concerns how, and in what form, the evidence should be applied. This includes the following possible approaches:

• The 'push' approach: Actively distributing the evidence to the organisation’s relevant stakeholders, often in the form of a protocol, guideline, checklist or standard operating procedure. The push approach is typically used for operational, routine practices (for example, hiring and selection procedures, or how to deal with customer complaints).
• The 'pull' approach: Evidence from multiple sources is actively obtained and succinctly summarised. When it concerns summarising findings from the scientific literature, rapid evidence assessments (REAs) are often used – see Step 2 and the CIPD Evidence review hub. The pull approach is typically used for non-routine decisions that involve making changes to the way an organisation operates (for example, the implementation of an autonomous team, or the introduction of performance feedback).
• The 'learn by doing' approach: Pilot testing and systematically assessing outcomes of the decisions we take in order to identify what works (see also Step 6). This is especially appropriate if there is no other option, for novel or hyper-complex decisions when evidence is often not (yet) available (for example, starting a new business in an emerging market).

The final step of evidence-based practice is assessing the outcome of the decision taken: did the decision (or the implementation of the new practice) deliver the desired results?

Unfortunately, organisations seldom evaluate the outcome of decisions, projects or new practices. Nevertheless, assessing the outcome of our decisions is something we can and always should do. Before we assess the outcome, however, we first need to determine:

• whether the decision or new practice was executed/implemented
• whether it was executed/implemented as planned.

After all, if we don’t know with certainty if a practice was implemented as planned, we don’t know whether a lack of impact is due to poor implementation or the practice itself.

When we assess the outcome of a decision, we are asking whether the decision had an effect on a particular outcome. As discussed in Step 3, for a reliable answer to a cause-and-effect question, we need a control group (preferably randomised), a baseline measurement, and a post-measurement – also referred to as a randomised controlled trial. However, often there is no control group available, which leaves us no other option than to assess the impact of the decision or practice by comparing the baseline with the outcome. This type of assessment is referred to as a before-after measurement.

When we do not (or cannot) obtain a baseline, it is harder to reliably assess the outcome of a new practice or intervention. This is often the case in large-scale interventions or change projects that have multiple objectives. But even in those cases, assessing the outcome retrospectively is still beneficial. For example, a meta-analysis found that retrospective evaluations can increase performance by 25%.

People professionals may be thinking: “How do I follow the six steps if I’m not a qualified researcher?” But evidence-based practice is not about trying to turn practitioners into researchers. Rather, it’s about bringing together complementary evidence from different sources, including research and practice. Some aspects of evidence-based practice are technical, so people professionals may find it useful to work with academics or other research specialists.

For example, it’s unlikely all people professionals in an organisation will need to be highly trained in statistics, but an HR team may benefit from bringing in one or two data specialists, or hiring them for ad hoc projects. Similarly, conducting evidence reviews and running trials requires well-developed research skills – practitioners could either develop these capabilities in-house or bring them in from external researchers. Academics are also usually keen to publish research, so it may not be necessary for practitioners to do much additional work to support this. Events like the CIPD’s annual Applied Research Conference can be a good way for people professionals to develop networks with academic researchers.

A good aim for practitioners themselves is to become a ‘savvy consumer’ of research, understanding enough that one can ask probing questions; for example, about the strength of evidence and the size of impacts. This is underpinned by skills in critical thinking, in particular being clear about what questions are really of interest and what evidence will do the best job of answering those questions.

To develop your knowledge and capability, visit the CIPD’s online course on evidence-based practice or, for more advanced skills, look at CEBMa’s online course. 

While it is fair to say that evidence-based practice and HR is still in its infancy compared to some other professions, people professionals can begin with these practical steps to pump prime their decision-making:

• Read research.
• Collect and analyse organisational data.
• Review published evidence.
• Trial new practices.
• Share knowledge.
• Above all, think critically.

Our thought leadership article gives further insight into how the people profession can become more evidence-based.

Evidence-based practice is about using the best available evidence from multiple sources to optimise decisions. Being evidence-based is not a question of looking for ‘proof’, as this is far too elusive. However, we can – and should – prioritise the most trustworthy evidence available. The gains in making better decisions on the ground, strengthening the body of knowledge and becoming a more influential profession are surely worthwhile.

To realise the vision of a people profession that’s genuinely evidence-based, we need to move forward on two fronts:

1. We need to make sure that the body of professional knowledge is evidence-based – the CIPD’s Evidence review hub is one way in which we are doing this.
2. People professionals need to develop knowledge and capability in evidence-based practice. Resources such as the CIPD Profession map and courses from the CIPD and CEBMa can help. Our case studies demonstrate how people professionals are already using an evidence-based approach to successfully address issues in their organisations.

In applying evidence-based thinking in practice, there are certain tenets to hold onto. For substantial decisions, people professionals should always consider drawing on four sources of evidence: professional expertise, scientific literature, organisational data, and stakeholder views and concerns. It can be tempting to rely on professional judgement, received wisdom and ‘best practice’ examples, and bow to senior stakeholder views. But injecting evidence from the other sources will greatly reduce the chance of bias and maximise your chances of effective solutions.

Published management research is a valuable source of evidence for practitioners that seems to be the most neglected. When drawing on the scientific literature, the two principles of critical appraisal (‘not all evidence is equal’) and looking at the broad body of research on a topic (‘one study is not enough’) stand us in excellent stead. This has clear implications for how we look for, prioritise and assess evidence. A systematic approach to reviewing published evidence goes a long way to reducing bias and giving confidence that we’ve captured the most important research insight.

Becoming a profession worthy of the label ‘evidence-based’ is a long road. We need to chip away over time to see real progress. HR, learning and development, and organisational development are newer to evidence-based practice than other professions, but we can take inspiration from them, for whom it has also been a long road, and be ambitious.

This appendix explains some important technical aspects of appraising scientific research, which is inevitably the trickiest aspect of evidence-based practice for non-researchers. As we note in this guide, most people professionals won’t need to become researchers themselves, but a sensible aim is to become ‘savvy consumers’ of research.

To support this, below we explain four aspects of appraising scientific research:

1. The three conditions that show causal relationships.
2. Common study designs.
3. Assessing methodological appropriateness.
4. Interpreting research findings (in particular effect sizes).

We hope that this assists you in developing enough understanding to be able to ask probing questions and apply research insights.

Three conditions to show causal relationships

In HR, people management and related fields, we are often concerned with questions about ‘what works’ or what’s effective in practice. To answer these questions, we need to get as close as possible to establishing cause-and-effect relationships.

Many will have heard the phrase ‘correlation is not causality’ or ‘correlation does not imply causation’. It means that a statistical association between two measures or observed events is not enough to show that one characteristic or action leads to (or affects, or increases the chances of) a particular outcome. One reason is that statistical relationships can be spurious, meaning two things appear to be directly related, but are not.

For example, there is a statistically solid correlation between the amount of ice-cream consumed and the number of people who drown on a given day. But it does not follow that eating ice-cream makes you more likely to drown. The better explanation is that you’re more likely to both eat ice-cream and go swimming (raising your chances of drowning) on sunny days.

So what evidence is enough to show causality? Three key criteria are needed:³

• Association: A statistical relationship (such as a correlation) between reliable measures of an intervention or characteristic and an important outcome.
• Temporality or prediction: That one of these comes before the other, rather than the other way round. We obtain this from before-and-after measures to show changes over time.
• Other factors (apart from the intervention or influencer of interest) don’t explain the relationship: We obtain this from various things: studying a control group alongside the treatment group to see what would have happened without the intervention (the counterfactual); randomizing the allocation of people to intervention and control to avoid selection bias, and controlling for other relevant factors in the statistical analysis (for example, age, gender or occupation).

Common study designs

Different study designs do better or worse jobs at explaining causal relationships.

Single studies

• Randomised controlled trials (RCTs): Conducted well, these are the ideal method that meets all three criteria for causality. They are often referred to as the ‘gold standard’ of impact studies.
• Quasi-experimental designs: These are a broad group of studies that go some way towards meeting the criteria. While weaker than RCTs, they are often much more practical or ethical to conduct, and can provide good evidence for cause and effect. One example is single-group before-and-after studies. Because these don’t include control groups, we don’t know whether any improvement observed would have happened anyway, but by virtue of being longitudinal they at least show that one thing happens following another.
• Parallel cohort studies: These compare changes in outcomes over time for two groups who are similar in many ways but treated differently in a way that is of interest. Because people are not randomly allocated to the two groups, there is a risk of ‘confounders’ – that is, factors that explain both the treatment and outcomes, and interfere with the analysis. But these studies are still useful as they show change over time for intervention and control groups.

These research designs go much further to show cause-and-effect or prediction than cross-sectional surveys, which only observe variables at one point in time. In survey analysis, statistical relationships could be spurious or the direction of causality could even be the opposite to what you might suppose. For example, a simple correlation between ‘employee engagement’ and performance could exist because engagement contributes to performance, or because being rated as high-performing makes people feel better.  

Other single study designs include controlled before-after studies (also called 'non-randomized controlled trials' or 'controlled longitudinal studies'), controlled studies with post-test only, and case studies. Case studies often use qualitative methods, such as interviews, focus groups, documentary analysis, narrative analysis, and ethnography or participant observation. Qualitative research is often exploratory, in that it is used to gain an understanding of underlying reasons or opinions and generate new theories. These can then be tested as hypotheses in appropriate quantitative studies.

Systematic reviews and meta-analyses

Systematic reviews and meta-analyses are central to evidence-based practice. Their strength is that they look across the body of research, allowing us to understand the best available evidence on a topic overall. In contrast, even well-conducted single studies can give different results on the same topic, due to differences in context or the research approaches used.

Characteristics of these are as follows:

• Systematic reviews: These are studies that summarise the body of studies on the same topic. They use consistent search terms in different scientific databases, ideally appraise the quality of studies and are explicit about the methods used. The CIPD conducts evidence reviews based on rapid evidence assessments (REAs), a shortened form of the systematic review that follows the same principles.
• Meta-analysis: This is often based on a systematic review. It is a study that uses statistical analysis to combine the results of individual studies to get a more accurate estimate of an effect. It can also be used to analyse what conditions make an effect larger or smaller.

More information on research designs can be found in CEBMa resources.

Assessing methodological appropriateness

When conducting an evidence review, we need to determine which research evidence is ‘best’ (that is, most trustworthy) for the question in hand, so we can prioritise it in our recommendations. At the same time, we assess the quality of research evidence to establish how certain we can be of our recommendations: well-established topics often have a strong body of research, but the evidence on new or emerging topics is often far less than ideal.

This involves appraising the study designs or research methods used. For questions about intervention effectiveness or cause-and-effect, we use tables such as that below to inform a rating of evidence quality. Based on established scientific standards, we can also estimate the trustworthiness of the study. Hypothetically, if you were deciding whether to use a particular intervention based on evidence that was only 50% trustworthy, you would have the same 50/50 chance of success as tossing a coin, so the evidence would be useless. On the other hand, using evidence that was 100% trustworthy would give you certain success. Of course, in reality nothing is 100% certain, but highly trustworthy research can conclusively demonstrate that, in a given context, an intervention has a positive or negative impact on the outcomes that were measured.

Table 1: Methodological appropriateness of effect studies and impact evaluations

Notes: Trustworthiness takes into consideration not only which study design was used but also how well it was applied. Table reproduced from CEBMa (2017), based on the classification system of Shadish, Cook and Campbell (2002)⁴ and Petticrew and Roberts (2006)⁵.  

There are two important points to note about using such hierarchies of evidence. First, as we discuss in this guide, evidence-based practice involves prioritising the best available evidence. A good mantra here is ‘the perfect is the enemy of the good’: if studies with very robust (highly methodologically appropriate) designs are not available on your topic of interest, look at others. For example, if systematic reviews or randomized controlled studies are not available on your question, you will do well to look at other types of studies, such as those with quasi-experimental designs.

Second, although many questions for managers and people and HR relate to effectiveness or causality, this is by no means always the case. Broadly, types of research questions include the following:

Table 2: Types of research question

Different methods are suited to different types of questions. For example, a cross-sectional survey is a highly appropriate or trustworthy design for questions about association, difference, prevalence, frequency and attitudes. And qualitative research is highly appropriate for questions about experience, perceptions, feelings, needs and exploration and theory building. For more discussion of this, see Petticrew and Roberts (2003).

Effect sizes and interpreting research findings

Even if practitioners wanting to be evidence-based can search for and find relevant research, they are left with another challenge: how to interpret it. Unfortunately, academic research in human resource management is often highly technical, written in inaccessible language and not closely linked to practice. A recent analysis found that in a sample of 324 peer-reviewed articles, half of them dedicated less than 2% of the text to practical implications, and where implications were discussed, this was often obscure and implicit.  

Even if published research does include good discussion of practical implications, it’s helpful and perhaps necessary for practitioners wishing to draw on them to understand the findings. This can be tricky, as they contain fairly technical statistical information.

Statistical significance

There’s an obvious need to simplify the technical findings of quantitative studies. The typical way to try to simplify research findings is to focus on statistical significance, or p-values. Reading through a research paper, this may seem intuitive, as the level of significance is identified with asterisks: typically, * means sufficiently significant and ** or *** means highly significant. However, there is a lot of confusion about what the p-value is – even quantitative scientists struggle to translate it into something meaningful and easy to understand – and a growing number of scientists are arguing that it should be abandoned. What’s more, statistical significance does nothing to help a practitioner who wants to know if a technique or approach is likely to have a meaningful impact – that is, it does not answer the most important practical question of how much difference an intervention makes.

Effect sizes

The good news is that effect sizes do give this information. The information is still technical and can still be hard to understand, as studies often use different statistics for effect sizes. Fortunately, however, we can translate effect sizes into every-day language. A useful tool is 'Cohen’s Rule of Thumb', which matches different statistical measures to small/medium/large categories.⁶  

According to Cohen:

• a ‘small’ effect is one that is visible only through careful examination – so may not be practically relevant
• a ‘medium’ effect is one that is ‘visible to the naked eye of the careful observer’
• a ‘large’ effect is one that anybody can easily see because it is substantial. An example of a large effect size is the relationship between sex and height: if you walked into a large room full of people in which all the men were on one side and all the women on the other side, you would instantly see a general difference in height.

The rule of thumb has since been extended to account for very small, very large and huge results.⁷

Effect sizes need to be contextualised. For example, a small effect is of huge importance if the outcome is the number of fatalities, or indeed, sales revenue. Compared to this, if the outcome is work motivation (which is likely to affect sales revenue but is certainly not the same thing) even a large effect will be less important. This shows the limits of scientific studies and brings us back to evidence from practitioners and stakeholders, who are well placed to say what outcomes are most important.

¹ Gifford, J. (2016) In search of the best available evidence: Positioning paper. London: Chartered Institute of Personnel and Development.

² For a discussion of reliability and validity in performance measures, see People performance: an evidence review.

^3,4 Shadish, W. R. Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Belmont, CA, Wadsworth Cengage Learning. 

⁵ Petticrew, M., & Roberts, H. (2006). How to appraise the studies: an introduction to assessing study quality. In: Systematic reviews in the social sciences: A practical guide, pp125-63. Oxford: Blackwell.

⁶ For a table showing different measures of effect sizes according to Cohen’s Rule of Thumb, see CEBMa Guideline for Rapid Evidence Assessments in Management and Organizations, p 20.

⁷ Sawilowsky, S. S. (2009) New Effect Size Rules of Thumb. Journal of Modern Applied Statistical Methods. Vol 8(2), pp 597–599.

CIPD evidence reviews are available on a range of HR and L&D topics.

Barends, E. and Rousseau, D. M. (2018) Evidence-based management: how to use evidence to make better organizational decisions. London: Kogan Page.

Barends, E., Rousseau, D. and Briner, R. B. (2014) Evidence-Based Management: the basic principles. Amsterdam, Center for Evidence-Based Management.

Guadalupe, M. (2020) Turn the Office Into a Lab. INSEAD Economics & Finance – Blog.

Petticrew, M. and Roberts, H. (2003) Evidence, hierarchies, and typologies: horses for courses. Journal Of Epidemiology And Community Health. Vol 57(7): 527.

Pfeffer, J. and Sutton, R. I. (2006) Hard facts, dangerous half-truths, and total nonsense:profiting from evidence-based management. Boston, Mass., Harvard Business School Press.

Pindek, S., Kessler, S. R. and Spector, P. E. (2017) A quantitative and qualitative review of what meta-analyses have contributed to our understanding of human resource management. Human Resource Management Review. Vol 27(1), pp26–38.

Rousseau, D. M. (2006) Is there such a thing as "evidence-based management"? Academy of Management Review. Vol 31(2), pp256–269.

Rousseau, D. M. (2020) Making Evidence-Based Organizational Decisions in an Uncertain World. Organizational Dynamics. Vol 49(1): 100756.

This report was written by Jonny Gifford and Jake Young of the CIPD and Eric Barends of the Center for Evidence-Based Management (CEBMa).

Please cite this guide as: Gifford, J., Barends, E. and Young, J. (2023) Evidence-based HR: Make better decisions and step up your influence. Guide. London: Chartered Institute of Personnel and Development.