Testing Chronicles. Chapter I
This is the first part of a detailed guide about user testing and checking if your ideas work. It includes a basic introduction, how to test your ideas, and getting ready for the tests. For information on different types of tests, tips on how to do them, and how to understand the results, check out Chapter II.
Plenty has already been said about the obsolescence of old development methods. We used to pour tens of millions into chiseling a product for 4 years, only to discover it was unwanted and shrug. So we take a deep breath and move on. That doesn’t fly anymore. Everyone wants to know if an idea will work before spending a single penny. Experimentation has become a modern necessity, and no endeavor seems possible without the principle of “Fail fast”.
Fortunately, with increasing demand comes supply. The rapidly evolving market offers many tools to uncover issues before a single line of code is written or a penny spent. These things, let’s be frank, don’t come cheap. Smart folks realized that testing saves tonns of money, time, and effort. So, why not take a bigger slice of that pie? For a company, spending a few days on an experiment that tanks an idea equals saving many months of work (read: tens of thousands of dollars). And if the test confirms suspicions, it’s an extra brick in the foundation of confidence, risk minimization, and decision rationale. Why not shell out a couple of hundred for such a test?
For us, as designers, testing helps gauge how good our solutions are, whether our vision is blurred, and if the interaction is deeply thought through. Yeah, I know, you already understand all this because what designer wouldn’t want to constantly test their work? But beyond these obvious advantages of hedging and covering one’s behind (risk minimization to not miss the product-market fit, especially among the younger crowd), there’s another less apparent benefit.
In this era paranoid with data, where no one wants to budge until the numbers say so, everyone wants to hire people skilled in this area. Almost every interview will grill you about working with data. The paradox, however, is that not every project allows you to gain such experience. It’s costly, and embellishing your portfolio with beautiful figures at the client’s expense is a rare occurrence.
That’s where testing comes to our rescue. The simplest and cheapest form of testing, unmoderated quantitative testing, surprisingly provides quantitative data that you’d only get after months of working with a digital analyst and a front-end developer. I’m not discouraging you from replacing product analytics with testing, believe me, they should work together. Yet, if a solution is needed right now, and you lack the time for deploying measuring equipment, data collection, and cleansing, testing becomes a perfectly viable substitute. You’ll also get numbers, the stakeholders’ beloved, though they’ll only be relevant for a specific scenario, but for now, isn’t that enough? And such cases, like the cherry on top, fit beautifully in your portfolio, attracting clients like magnets.
In conclusion, more and more factors weigh in favor of making testing part of our daily practice. But I hope I don’t need to convince you. So the next reasonable questions is “How to do it right? “That’s precisely why I crafted this epic long read, addressing the most common questions about testing. Besides the introduction, a whole adventure awaits us, which I’ve dubbed the Testing Chronicles.
Where does testing begin?
You can’t just start testing something without a clear direction. First, we need to have assumptions or hypotheses about the expected user or market reactions. Hypotheses should exist even before testing starts and should even initiate it. But where do they come from?
That’s a great question, thanks for asking. There are several ways to generate hypotheses:
- Domain-driven: New technologies emerge, enabling more complex functions like personalization or LLM implementation. You want to test the applicability of a new technology in your product and form a hypothesis.
- Field-driven: Esteemed figures in the domain, such as cool competitor companies, introduce novelties. You need to understand if it’ll work in your context.
- Common sense and expert opinion: Leveraging your experience, you can logically formulate assumptions about improving interaction or creating something new.
- Results from exploratory research: Identifying user goals, problems, and mental models leads to assumptions that aligning interactions with this data would benefit users — these are hypotheses.
- And my personal favorite. Thought fountain from your CEO or manager, albeit they come not as hypotheses needing proof but as directive instructions.
Today, the prevalent notion is about product hypotheses, constructed using the following template:
“If we do this (introduce a new feature, change current behavior, create a new product, etc.), we will get this reaction (attract a new audience, make users happier, increase conversion, etc.), which will reflect on these metrics (number of active users, user revenue, company valuation, etc.).”
This tool works well, and yet you need not confine yourself to it. Consider the user’s mental model, preferences, and perception. For instance, when talking about the end product, you can hypothesize which path the user will prefer. During the test, don’t restrict their navigation and see if it truly works or if it’s easier for people to “cut corners.” Sure, you’ll need to create a realistic prototype, but it opens additional avenues to improve the product. Add measurable metrics to the mix, and it becomes a fairytale.
What can you test?
Perhaps you’ve got the mistaken impression that testing is only for a finished product or prototype. That’s not entirely true — testing can occur at any stage of development, from idea inception to product sunset. To avoid being tied down to a specific stage of the lifecycle, let’s talk about testing hypotheses since they can exist at any time.
When the product is ready, finding assumptions to check for issues usually isn’t a problem. We can use our astute eyes to examine the user flow and identify potential “pain points.” Thus, hypotheses often imply some effect from our actions. For instance, “If we change the layout of our catalog from cards to a listing, users will find the catalog more convenient.” It’s a hypothesis in its purest form, not yet suitable for testing — a problem we’ll discuss later. But that’s if we have something tangible to test. But what if we don’t?
We’ve already discussed how hypotheses can arise. However, in practice, we’ve become so accustomed to testing a finished product that the question of how to test an idea can stump us. We have nothing to show users, so how do we proceed? In this case, problems and insights obtained from exploratory research can fuel hypothesis generation: retrospective interviews with users, shadowing, domain exploration, etc. In this scenario, we’d end up with more of a business hypothesis, focused on financially viable solutions for newly identified needs.
Of course, the testing methods will differ. In one scenario, we have access to a finished product with existing users, allowing us to conduct basic measurements for comparison. In the other case, there’s nothing but fractals born of consciousness, and at first glance, it’s unclear what to do with them. Perhaps that’s where we’ll start… or continue.
In general, to ensure an idea will work, we need to check its alignment with three criteria: user value, business viability, and feasibility. It’s like collecting Pokémon — all three are essential; otherwise, it won’t work. You’ve got it right, three criteria, no rocket science here. But despite its apparent simplicity, it’s not always easy to find the right experiment that confirms at least one of these conditions. If the experiment is chosen incorrectly, our conclusions could be misleading.
Usability testing is straightforward: create a scenario, give it to users, if they can complete the task — good; if not — bad. However, at the early stages, when there’s nothing to show or describe, many designers struggle when asked to test an idea for a new product. They tend to revert to previously solved tasks and think: “Alright, we have an idea, but describing it in words won’t cut it. We need to make it possible. Therefore, we must implement the idea in mockups and conduct testing as I know how.” Instead of benefiting from early testing, the designer implements the idea and tests it using a familiar method, shows it to the development team, and starts counting potential profits.
Undoubtedly, this approach has its place since we still obtain results before the development stage, but it consumes our time and thereby increases the risks of a complete loss of investments. Is it possible to do without the full visual representation? Absolutely. Let’s take another look at the three criteria an idea must meet to survive in our harsh world:
- It must bring value to the user, or else no one will use the product.
- It must be financially effective and viable, generating profit or at least covering its costs.
- It must be feasible; otherwise, it won’t be possible to create.
Now, let’s take a closer look at each criterion and explore how they can be tested.
Value for the User
The closest to our spirit — value for the user. It means that a new feature or product addresses a pressing user need in a way that they’re willing to reach into their pockets and pay for it. This is the foundation for designing experiments.
First and foremost, let’s look at the problem our idea will solve. How do people currently solve this issue? Humans are adaptable creatures, some might say resistant to change, so it’s likely that people have already found a potential solution on their own. Perhaps it won’t be sophisticated or elegant but somehow functional. Most likely… on Excel, but that’s not important.
The essence lies in conducting an exploratory investigation. It won’t likely be the type of research we’re used to; instead, it might be a series of field interviews without a specific agenda. We’ll ask loads of questions on the fly, make tons of notes, and birth a multitude of ideas. During these sessions, you might pick up on potential areas for improvement by their intonation: characterized by sighs, eye rolls, clenched fists, and a barrage of complaints from users. If you observe any of these signs, you’re on the right track, ready to proceed with the experiment. Yes, just like that, without explanations, without writing a scenario.
If people talk about an inefficient part of the process or their lives but don’t know how to solve the problem, it presents a unique opportunity for you to solve their problems at your expense. Ha-ha. It’s called…
Concierge Test
The simplest approach is to first ask how difficult they find this part of their work. This often yields inflated estimates. Then you suggest that someone else will solve their problem instead of them and ask how happy they’d be in that scenario. This usually results in deflated estimates. You could, of course, stop there — take the average of these two numbers (if they’re not both zero). It’s the first way to give you a very distant discrete view of the value in your idea. But it’s definitely not enough to calculate the economics based on this data. Avoid mentioning money at this stage. Otherwise, I can already see dollar signs in your pupils, only to find out later that even $5 is too much.
But if you’re keen to touch on the topic of crisp green bills, an improved version of this test involves confirming intentions. Say you’ll do their work for them and offer to pay for it right there and then, seriously. Ask them to open their wallet and hand you the bill or make a transfer. This will significantly dampen their enthusiasm. But you could go further — actually take on and complete this part of their work using good old-fashioned manual labor.
Someone says, “I always prepare for trips and have to search for sights, google restaurants, plan logistics, create schedules,” — you respond, “Oh, I have access to a prototype of a service that does something similar, albeit slow right now, but let’s give it a try?” After confirmation, you go ahead, do the work yourself, and return with the result. After constructive feedback and blows to your work and shattered self-esteem, ask if they’re willing to subscribe to the waiting list.
Even if you don’t see wild enthusiasm from the person, you’ll understand how complex a problem you’re trying to solve.
Cost of Loss
It’s different when people are already using makeshift means, competitors’ offerings, or a whole chain of different services to solve a problem. For instance, YouTubers use a competitor’s app to record a podcast, although they used three different apps before: one for recording, another for post-processing, and a third for publishing. Ask how much money someone is willing to pay out of their pocket to avoid losing the ability to continue using one product. This type of question works best when it involves deprivation, as humans tend to assess fear of loss more easily than potential gain (just think about how you evaluate new features).
With these simple actions, we can eliminate the factor of “nothing needs to change” from the equation and be a bit more confident that we won’t waste money. Speaking of money…
Business Viability
Until recently, when not every other designer added “product” to their title, I often heard about the importance of considering economics. For some reason, it seemed to me that everyone did it. In one interview, some turbo-lead explained how brilliantly he solved problems with a psychologist and wanted everyone to be involved in it. But creating a separate service for this wasn’t profitable because he crunched the numbers, and it didn’t add up. Yet, a year later, I see precisely such a service advertised all over YouTube. Maybe the folks rethought their business model, but I prefer to think that a product’s economics, like quantum physics, sometimes aligns and sometimes doesn’t.
Jokes aside, it’s essential to worry about money in advance. But how do you do that when you only have an idea in hand? Very simply: by deceit and betrayal.
Selling the Idea
Even without a ready prototype, nothing stops you from creating a sales-driven landing page. Yes, yes, why are you surprised? Is this the first time you’re selling air? Have you never done pre-sales or consulting? Alright, I’m amused.
The method works, really. If you’ve identified a problem and a potential solution, you don’t need to plan everything meticulously, spending months of work. Spend a day or two creating a landing page with a value proposition where you describe the problem and mention that you have an idea on how to solve it. Naturally, there probably won’t be screenshots of the app or descriptions of its advantages because… you don’t have it. But if the problem resonates with the target audience, they’ll undoubtedly sign up for the waiting list for the upcoming novelty. Furthermore, if you’ve dared to calculate in advance how much money per user you need to break even (although I seriously doubt the accuracy of such calculations), add another 50% and offer the person a choice of a pricing plan.
After a small investment in targeted advertising, you can reasonably assess the level of interest in your idea, and then in terms of money.
Prototypes
Another way to check the feasibility of an idea without significant investment would be a prototype of the final solution. And here, I’m not talking about the gray mock-ups we’re used to, but a solution made of spit and sticks. Remember when we talked about exploratory research? I mentioned that people probably already solve their problem somehow, most likely using Excel.
Essentially, almost any new app or service processes and provides access to information, whether it’s a list of rental apartments, sea tours, available ice cream flavors, or a list of sworn enemies. Everything can be boiled down to tables. Or a multitude of different tables. Therefore, you can create a reasonably satisfactory solution with just Notion or Coda, without any design.
Let me share an example from my own practice. Before our company created a specialized service for professional growth, one that displayed a person’s career path, indicated what they lacked, and how to “boost” it, I created something similar in Coda. Yes, the solution lacked sophistication, visual originality, or beauty. Yes, each person had to manually create their file. Yes, it wasn’t possible to view general statistics. Yes, it only worked for designers. But it did its job.
By creating such a template now and sharing it with everyone wandering in the career abyss, I could see how much demand there was for such a service before investing a million dollars. Oh, wait, that’s what I did, except the money wasn’t mine, and users had no choice, but still. I checked that my approach to solving the problem was valid, effective, and yielded results before even opening a notebook, let alone Figma.
Feasibility
In the example above, by the way, I simultaneously tested the feasibility of my idea. Even if I, with my wonky attempts, managed to create a working solution without much programming, for experienced programmers, it would be like transmitting two bits.
However, validation for feasibility doesn’t always yield success. For instance, Netflix conceived the idea of a streaming platform long before becoming a media giant. Everything aligned for them, people wanted to instantly watch their favorite movies, and their business model even worked with DVDs by mail. However, the hitch was the available technology — the internet was too slow for streaming video. Therefore, such a brilliant idea wasn’t implemented until the point when data transmission at that speed became feasible.
So, if you have a brilliant idea born out of user pain that can bring in money — show it to your development team, or at least a savvy expert who specializes in making things work. Most likely, they will immediately spot the system’s weak points capable of derailing the whole plan. These are the areas you should test using software prototypes — rudimentary implementations of system fragments, without aiming for quality, just to ensure that it’s physically possible.
Only after receiving a firm or not-so-firm “yes” from the team can you move forward.
The Ideational “Sum-Up”
Testing ideas, well, it’s an ungrateful business, really: you can do everything right, ask everyone everything, confirm everything, but still flush money down the toilet because of those black swans that landed on your grand plan. There are far more of these swans than you can imagine, so there’s no 100% guarantee that you’ll shoot for the stars. However, that doesn’t mean testing ideas isn’t necessary: some form of protection against risks is still better than none at all. That’s precisely why checking your idea for user value, business viability, and feasibility is simply essential.
Read more:
It seems we’ve sorted out the ideas. Congratulations if your idea has survived the meat grinder of validation and is still alive. You’ve probably even managed to create a prototype by now. However, before your product sees the light of day, it still has many trials to undergo.
We will be considering the situation where our idea has already taken shape, either in the form of a prototype or as a functioning product. So, we are moving from the abstract (ideas) to something more concrete, something we can confidently show to users. The methods and tools may vary slightly, but the main narrative remains unchanged.
That’s why, since users will play a major role in the tests, this process is called user testing. However, depending on the goals and context, the testing methods can vary. We’ll talk about specific types of tests a bit later, but for now, let’s delve into the general principles.
How to Prepare Tests?
As strange as it may sound, just three years ago, most designers practically ignored testing. At best, they engaged in guerrilla testing, a topic we’ll discuss later. While this was better than nothing, the effectiveness of solutions for end users remained a mystery. However, today, the testing market has matured: numerous solutions have emerged capable of supporting this crucial initiative at a reasonable cost, and testing has become an integral part of the process.
But, as you’ve probably guessed, you can’t simply show users bright pictures and ask, “What do you think?” Although, technically, you can do that, the results will be unreliable. Users are emotional beings, and their responses can depend on numerous factors, from the weather outside to the scent of your cologne. Therefore, instead of words, it’s essential to assess facts, and not all facts, only those that matter.
If you already have a test candidate in mind, let’s together, step by step, plan your testing. For this purpose, I’ve prepared a simple template. Copy it, open it, and let’s get started.
Defining the Goal and Success Criteria
Let me remind you that we’re discussing user testing, which means we already have a solution we want to evaluate. Before starting to recruit participants, it’s crucial to define the goal of the testing and its success criteria.
Ask yourself: “What do I want to achieve through this testing?” and “How will I know the testing has yielded results?” An answer like “I want a report on testing” is an empty sound, as it means testing just for the sake of it. It’s important to ensure the goal aligns with stakeholders’ expectations to avoid rendering the testing results useless.
Let’s consider a live example. Suppose you have an idea to streamline the checkout process on your online store by eliminating the order confirmation step — the seemingly redundant page displaying all information the user entered three seconds ago. Why make them look at it again? You’ve created a prototype and want to conduct testing on this idea. In this case, the goal could be formulated as follows:
Testing Goal: To determine if the new approach to checkout significantly improves over the current process and make a decision about its implementation.
Success Criteria. The new process will be considered successful if:
- The average checkout completion time decreases by more than 10% compared to the current process.
- The number of user errors decreases by more than 15% compared to the current process.
- The User Satisfaction Level (CSat) increases by 1 point.
In this example, we first defined the questions we can answer through testing and then developed criteria to measure success. It’s not necessary to define all the details 100%, as after obtaining additional data, we can refine the goals and success criteria. These are broad strokes, so to speak.
Defining Hypotheses
To be frank, this stage should have been done by you already, as you created solutions based on some assumptions, right? You didn’t just randomly generate an image you want to show users, did you? I hope not.
But hold on a second, didn’t we talk about hypotheses at the very beginning? Yes, I’m that Tarantino guy. But if you recall, the basis of our testing was more about product hypotheses — assumptions about how the product state will change in response to our actions. And some of these product hypotheses undoubtedly resemble the goal of our testing. However, in our preparation, we need to delve deeper.
You need to break down your assumption into many more specific elements, focusing on the user’s mental model. Personally, I try to look at the solution from the user’s perspective and anticipate what might be unclear to me, where gaps might arise in interaction, which terms could cause an intellectual deadlock, and so forth. All these assumptions are neatly included in the list of hypotheses for testing, complementing the “main” ones.
Once the assumptions are formulated, it wouldn’t hurt to also outline KPIs (Key Performance Indicators) for each of them — metrics we’ll look at during testing to confirm or refute them.
Target Audience Identification
Just like with delineating hypotheses, defining your target audience might precede any testing, and I can’t stress enough the importance of this critical preparatory step. Alongside the significance of understanding what and how you’re testing, it’s equally crucial to pay attention to whom you’ll be targeting during your experiments. The definition of your target audience largely influences the final research outcome, as an incorrect choice of audience can significantly distort the facts. For instance, if you’re developing an app for financial brokers and test it among your designer friends, you might receive low satisfaction scores with comments like “not enough white space” or “poor font choice,” which aren’t relevant to the target audience — brokers.
Apologies for the strictness, but I’ve encountered situations where people disregarded the importance of the phrase “You are not your users” and failed to see the difference in choosing the audience for testing.
Personally, I prefer engaging with the audience by creating personas and trying to understand their motives, goals, and tasks (and if possible, their mental model), and then correlating them with our hypotheses. For example, if I have a hypothesis that removing the confirmation page will speed up the ordering process, I need to gauge how much the user values speed and whether they are willing to take the risk of a possible error for that speed. If I have doubts even at this stage, the testing may not be meaningful, and it’s better to return to the lab for additional research.
By considering your target audience, you ensure more precise testing and safeguard yourself from mistakes right at the test creation stage, as well as from misinterpreting results. Ultimately, this helps you avoid a situation where you sped up one process but broke everything else. Although, of course, you can fix what’s broken and provide the customer with another bill, right?
Creating a Testing Roadmap
Once you’ve fully grasped the testing goal and the necessary criteria for determining its success, it’s time to focus on hypotheses and KPIs. Reflecting on how to obtain data that validates or, ideally, refutes the hypothesis is crucial. The thing is, data can be collected in various ways, but the results aren’t always reliable or credible.
At times, achieving high credibility requires more than one testing method. Qualitative testing methods can enrich the hypothesis list and track the emergence of issues, but they don’t always suit extrapolating results to the entire target group. Quantitative testing methods are often employed to confirm the relevance of the problem, backing hypotheses with data.
For instance, if a user complains that the new design doesn’t fit on their screen and suggests reducing its size, it’s not advisable to immediately put this task into the backlog based solely on one opinion. It could be a localized problem. We can turn this into a hypothesis for the next stage and ask ourselves, “How can we verify this?” Initially, consulting analytics to view the audience’s device distribution might be helpful. However, this data won’t tell us if people genuinely find the design too large. To verify this, another testing phase with session recordings could be launched, asking users to perform a simple task and then inquire about what they would like to improve or found inconvenient. If no one mentions scaling, it might not be considered a significant issue. Instead of directly asking about size, studying testing session recordings could help understand how users perceive the interface. Your 27-inch monitor probably differs from a user’s 10-year-old laptop set at 150% scaling.
Thus, for each hypothesis, you need an associated method or series of methods to test it. From my experience, one can start untangling this knot from either end by asking oneself, “What data do I need to get an answer?” and “How can I best collect it?”
Describing User Scenarios
When the preliminary plan gains coherence, it’s still too early to start looking for users. We’ve only defined the data flows that will help test the hypotheses, so now it’s time to think about the users, how they will navigate the tests, what questions to ask them, and how they will move from one question to another. It’s crucial to avoid mistakes that could skew the research results because even a single incorrect word can significantly impact perception and spoil the data, which, in turn, affects the conclusions.
We’ll discuss specific formulations and approaches a bit later when we move on to specific types of testing. Right now, I want to draw your attention to the story that the test will tell. This isn’t a joke.
Testing shouldn’t resemble an interrogation or the process of registering for government services. In my view, it should be akin to a conversation that naturally unfolds. It’s important to remember that during testing, users will learn about your product and form their perception of it. Hence, each question and stage of the test should have its place in this conversation. For example, you can’t ask a user how much they’re willing to pay for your product before they’ve seen it. But you also can’t simply show the product to the user because you don’t know if the problem you’re solving is relevant to them.
Therefore, you need to clearly define the sequence of questions and allow for the possibility of skipping certain parts of the test if the user lacks necessary knowledge. Otherwise, we might end up with distorted data and misinterpretations.
Once you’ve formulated your scenario, mentally go through it from the user’s perspective. Does everything seem logical? If not, you know what to do (fix it).
Now that we have our roadmap and scenario ready, let’s delve deeper into each type of test to polish our plan to perfection.
Types of Tests
If we don’t delve into specifics, all tests can be broadly categorized, but most often, they fall into the following classifications:
- Face-to-Face / Remote,
- Moderated / Unmoderated,
- Qualitative / Quantitative,
- Exploratory / Comparative.
It’s essential to understand that any testing method can fit into a specific type. For example, card sorting can be conducted both face-to-face, using paper cutouts, and remotely, employing advanced 23rd-century technology. Interviews can provide qualitative data, which, with perseverance and sufficient funding, can become quantitative (as seen with researchers who secure university grants).
This typology exists independently of the primary testing methods and serves more as an overlay, adapting to specific usage contexts. Let’s explore each of these pairs.
Face-to-Face and Remote Tests:
I’ve almost forgotten the times when we conducted face-to-face meetings with users, trying to dig into their true motives and emotions. Ah, those were the days. Now, you can wake up at 8:59, wipe the drool, and within a minute, be engaging with a respondent on the other side of the globe. Undoubtedly, remote communication has made testing more efficient — no need to spend ample time organizing meetings. However, the remote format has its drawbacks. Not everything can be tested remotely; if it involves physical devices, you’re unlikely to mail a banking terminal to a respondent (though such instances have occurred). Additionally, remote testing limits your perception of human facial expressions, gestures, and intonations.
Moderated and Unmoderated Tests:
As the names suggest, moderated tests involve a researcher’s participation, who can either remain silent or actively engage with the respondent by asking clarifying questions or encouraging them. Unmoderated tests don’t require your presence, and users go through them independently. This doesn’t mean that moderated tests are always conducted face-to-face, and unmoderated tests are remote. For instance, a Zoom interview is an example of a remote moderated test. Conducting a scenario in a laboratory without your involvement by the respondent is an example of a face-to-face unmoderated test.
Qualitative and Quantitative Tests:
It’s worth noting that this typology applies not to the tests themselves but to the results obtained. Sometimes the boundaries between them blur, as qualitative data can be transformed into quantitative and vice versa. Let’s start with qualitative methods. They aim to study individual behavior patterns, uncover mental models, and understand how people think and feel. Quantitative methods, on the other hand, seek to confirm or refute hypotheses and identify patterns based on large volumes of data. Qualitative methods include exploratory user tests mentioned earlier, surveys with open-ended questions, and similar methods. Quantitative methods encompass remote unmoderated tests, closed surveys, and so forth.
Exploratory and Comparative Tests:
The differences between them relate to the goals of the test. If the test aims to deeply explore the current interaction implementation, identify all issues that users face without any prior solution, it’s an exploratory test. If there already exists a solution, and the testing is geared toward comparing its effectiveness with the old solution, it’s a comparative test. Usually, comparative tests are conducted to validate the correctness of the chosen path or identify significant metric improvements. Exploratory tests include various research sessions where users are given a task, and their approach is analyzed in current conditions — testing the product’s current implementation to find potential enhancements. Comparative tests involve A/B/n testing, comparing the effectiveness of the new solution to the previous one.
This presented classification, like any other that could be devised, is conditional and mainly serves for a better understanding of context. It acts as a framework for testing methods, helping determine where and which method to use. I intentionally omitted some other classifications, such as based on the functionality being tested (testing on live products or isolated prototypes) and the level of access (open and closed), as they are applicable in limited cases. We’ll return to this topic, exploring each test type in more detail. Literally in a few seconds.
