Avoid this trap
We all use tests to help us make decisions. We take a test, and based on the results, we make a decision. A common mistake occurs when we lose sight of the fact that there are two types of tests in terms of purpose:
- Tests meant to choose the best option for our needs;
- Tests meant to simplify the choice and eliminate options.
One could argue that mathematically, these are the same, but pragmatically, they are very different. Even more problematic is when we conduct tests to eliminate options to simplify decision-making and later use them as if they were meant to choose the best option for our purpose.
Tests for selecting candidates
An example would be the tests given to choose candidates for certain positions in the job market, for which candidates have to pay to practice or where there is no transparency regarding the type of questions or the grading criteria used for evaluation (for those surprised by this idea, I remind you that hiring tests rarely have grading systems communicated in advance in the private sector). There is a vast market of tests that, by design, aim to narrow down the number of candidates for the decision-maker and generate profits for the test authors. No problem here, as everyone has the right to pursue their own goals. Problems arise when these types of tests are used to argue that the selected individuals are the best suited for the position. If you have to pay to prepare for a test to enroll in a university, course, or job competition for a position at a company, this is a test designed to make the selection easier for the decision-maker, under the guise of a skills test. If everyone can prepare for the test based on their own intellectual/physical/skill and resources, then it is a test meant to select those with the most suitable capabilities. If barriers, like payments or lack of access in information prevents some people from preparing for the test, then the purpose of the test is not to select the best, but to ease the choice for the decision-maker. Unfortunately, personal financial resources are not a good indicator of merit for anything other than the ability to make money in the current societal context. It is certainly a skill that not everyone has, but it cannot be an indicator for anything beyond itself.
Tests for evaluating researchers
Another example would be the development of scientific journal excellence standards based on quartiles and the evaluation of researchers based on the quartile in which the journals they publish in are. This is a problem because scientific journals which aim to be top ranked use tests designed to narrow down the number of accepted articles (because they are positively evaluated if they have a very low acceptance rate) rather than select articles that contribute to science. As long as articles are rejected because they are not interesting to the journal’s audience (which is different from a problem that does not fit the journal’s topic), that journal strays from the objective of selecting articles with scientific results within its field and leans towards profit and audience goals, which are more suited to commercial journals without scientific excellence goals. Furthermore, as long as a researcher’s development depends on the funds of the institution in which they develop and the opportunities available during their development, and as long as certain countries, fields of science, research institutions, and universities have access to very large research funds while others have very little, researcher evaluation must take these inequalities into account, using systems of publishing scientific results based on contributions to science rather than the need to reduce the acceptance rate of articles in highly ranked journals.
Procedural solutions and their limits
For decision-makers
As a decision-maker, when we outsource to a decision consultancy and accept certain tests that it proposes, we have the responsibility to check a few things if we want to select the most suitable options/people and accept that these options/people can have any background and come from any environment:
- What kind of options/candidates have access to those tests;
- What kinds of barriers they might face in order to perform well.
This is where values become important for our decision: Is it important for us to accept that some very suitable candidates or options can come from any environment/context, or not?
Some decision-makers argue that many resources are needed to select the most suitable candidates, but this is not correct. This is at least a false problem and, at most, one meant to avoid the effort of self-awareness regarding organizational needs. A deep understanding of one’s own needs and the ability to request exactly what is necessary for that position is required. In fact, many organizations do not know very well the criteria needed to perform certain job tasks and believe that doing what everyone else does will bring them the success they see in others. In reality, if we look closely, how many managers and recruiters know exactly what is needed to perform the tasks of a particular position? Generally, very few, especially those who have never performed those tasks themselves. Additionally, we must accept that sometimes we will simply use the first viable option from many that are suitable. It is not always necessary to choose the best version but the first that meets our criteria (the “satisficing” criterion). However, in these cases, transparency is a decision-making value that may be essential to the context in which we make the choice, especially if it involves decisions with ethical implications, such as hiring personnel or evaluating researchers, as discussed earlier.
For organization and institution designers
For situations where decision-makers do not know the necessary details for evaluation, various institutional mechanisms have been developed. For example, in the scientific evaluation of articles for publication, the institution of referees has been developed. The editor, not being specialized in all areas of a subject, even one as narrow as those around which scientific journals are built, seeks the advice of a specialist in the narrow field of an article. To avoid conflicts of interest, neither editors, nor referees are paid to give a verdict. This system worked as long as referees were financially independent and had control over how they allocated their research time and, moreover, shared the ethical values of the scientific community. However, today referees do not always meet these first two conditions, making it increasingly difficult for editors to find referees specialized in the field of an article who also have financial independence and control over their daily tasks to dedicate the time needed to evaluate an article.
As in the case of decisions about allocating the most suitable candidates to vacant positions, in this case, the necessary information for making decisions that lead us to select the best researchers for their positions is closely tied to knowing the research work in detail when making decisions about evaluating and choosing excellent researchers: a deep understanding of the skills needed to perform academic research in each specific field and a clear definition adapted to the context of each research field. These are things that would guide decision-makers toward identifying excellence in research. For now, the level of understanding of these issues remains unexplored as many studies on performance in research focus on evaluating results from tests meant to make choices easier, not to select those who contribute to scientific development.
Limits and implications of solutions
You will notice that in the two specific cases discussed, I did not mention “the best” because this is always a theoretical measure relative to the chosen goals. In concrete situations, we always need to discuss the particular goal. The goal of research is the advancement of knowledge, while the goal of personnel selection is to identify individuals who can perform job tasks while optimizing the goals of the organization that hires them.
It should also be noted that in this case, we are not discussing a meritocratic system but one oriented toward achieving a pragmatic goal. When the goal is clearly chosen, from an organizational design perspective, we can talk about designing meritocratic institutions or not. Additionally, a meritocratic institution is not related to the idea of an individual who “deserves” to become or develop in a particular direction. Meritocratic systems do not necessarily reward potential, but results, even in the early stages of an individual’s development, when, pragmatically, it would be unlikely to have results. For those wondering what a non-meritocratic system oriented toward achieving a goal looks like, an example is systems that select candidates based on potential and then provide them with resources to achieve the goal, sometimes rewarding the ability to reach the goals with minimal resources, other times rewarding reaching the goal with the given resources. Of course, there are other such examples of institutional design.
In conclusion, the use of the two types of tests (those oriented toward simplifying the decision through elimination and those oriented toward identifying the optimal option) is always justified in the right context. A major error is to use a simplification test and claim that it yields the optimal solution. Decision-makers must always be aware of what kind of test they are using, even when these services are outsourced, or especially in this case. And individuals who, for various reasons, are subject to an evaluation system, must distinguish what type of test they are entering to avoid using it for self-assessment when it is not appropriate.
Decision counseling as a solution
Experience and knowledge in decision-making are also essential to understanding the implications of the type of decision you are using to make choices. If you are facing difficult decisions as an individual, entrepreneur, manager, or employee, a specialized decision counselor can help you choose what serves all your goals, not just one. A specialized decision counselor will never tell you that your goals are incompatible but will support you in refining and finding the most appropriate way to achieve all the goals you genuinely pursue. Sign up for a decision counseling session (https://decisionresearch.eu/) to understand what decision method is suitable for your context. Sometimes, new errors or difficulties you thought you had overcome may hide in the complex details of the particular decision-making situation you are in. Your decision counselor can help you save time in identifying these and adopting the most suitable decision-making approach.
