Topic Definitions #
Methods for Evaluating Ideas in Project
Introduction to Idea Evaluation
Expert Evaluation
In project management, most idea evaluations are often based on expert opinions. This method involves professionals with deep knowledge and experience in a specific field providing subjective assessments of ideas, tasks, or alternatives based on their expertise and intuition. Expert evaluations are an essential tool, especially in situations of uncertainty or when quantitative data is unavailable or difficult to obtain.
- What is Expert Evaluation?
Expert evaluation is the process in which specialists in a given area offer their opinions on how promising, realistic, or useful an idea, project, or proposal is for achieving specific goals. Experts may assess a wide range of factors, such as technical feasibility, economic viability, timelines, risks, and expected outcomes. These evaluations are typically based on the experts’ previous experience, their knowledge of the industry, and their ability to foresee potential challenges or successful outcomes.
- Why are Expert Evaluations Important?
- Expert evaluations are crucial because many ideas and proposals, especially at the early stages of planning, cannot be accurately measured or evaluated using quantitative methods. Experts help fill in data gaps and provide qualitative judgments that take into account complex, often hard-to-formalize factors. While these assessments may be subjective, they are often highly accurate due to the experts’ extensive experience and understanding of the project context.
- Relying on expert evaluations is entirely appropriate, especially when quantitative data is limited or the process is too complex for precise calculations. Experts’ subjective opinions help speed up decision-making and guide the team towards the most likely successes or potential risks. However, it’s essential to remember that expert evaluations, like any other method, should be part of a comprehensive approach to idea assessment, incorporating both qualitative and quantitative data.
Project Idea Evaluation Methods #
Goals: #
- Deciding which ideas to include in the project;
- Determining the most valuable ideas and their potential impact on the project;
- Allocating resources between tasks;
- Mitigating risks by thoroughly assessing alternatives.
Let’s take a closer look at some popular evaluation methods that can be applied for these purposes.
1. Impact and Effort Evaluation #
The impact and effort evaluation method allows teams to analyze ideas based on two factors:
Ideas are classified into four main categories, as shown in the image:
-
Easy Wins #
Ideas with high impact and low effort. These are the most prioritized tasks because they deliver significant results with minimal resource investment.
-
Big Bets #
Ideas with high impact but requiring considerable effort. These tasks can yield high rewards but require thorough planning and a significant amount of resources.
-
Incremental #
Ideas with low impact and low effort. While they don’t have a substantial effect, they can be executed without significant time or resource commitment.
-
Money Pit #
Ideas with low impact and high effort. These tasks are not worth the investment and typically should not be pursued.
The image also shows intermediate positions, such as tasks with moderate effort and impact, which may be tackled if resources allow. This method helps project teams quickly identify which ideas will bring the greatest benefit for the least cost.
2. Team Voting #
The impact and effort evaluation method allows teams to analyze ideas based on two factors:
The process typically involves: #
1. Presenting ideas for the team’s consideration.
2. Each team member votes for a set number of ideas they think are the most important.
3. The ideas with the most votes become prioritized for further development.
The image illustrates various types of voting that can be used within this method:
-
Ranked Choice Voting #
Each participant votes by ranking the ideas in order of preference. As a result, each idea is placed in a priority list based on the cumulative rankings.
-
Points Allocation #
Participants are given a limited number of points, which they can allocate to different ideas. This helps gauge the relative importance of each idea to the team members.
-
Pairwise Comparison #
Ideas are compared in pairs. Participants choose between two alternatives, and priorities are determined based on comparisons across all proposed ideas.
-
Dot Voting #
Participants are given a limited number of votes (in the form of dots) which they can distribute among the proposed ideas. Ideas with the most votes become prioritized.
-
MaxDiff Analysis #
Participants select the most and least preferred ideas from the proposed set. This creates a preference gradient and helps identify which ideas have the highest and lowest value.
Using these methods helps teams objectively evaluate and prioritize ideas, incorporating the input of each team member, making the decision-making process more transparent and democratic.
3. Impact Mapping #
Impact Mapping is a visualization method that helps link business goals with specific actions and outcomes, providing clarity on how various factors affect project success. It is a tool that allows teams to focus on actual business needs, identify key actors and their actions, and predict how their behavior may influence project outcomes.
The image above shows the following steps to apply the Impact Mapping method:
“Why?” — Define the Goal: #
- The first step in the process is to clearly define the business goal the project is aiming to achieve. It’s crucial to answer the question “Why?” — why is this goal important to the business, what problem does it solve, and how will it contribute to success?
“Who?” — Identify Key Actors: #
- Next, it’s essential to identify “Who?” — the key actors who can influence the achievement of the goal. These could be both external and internal stakeholders, such as users, customers, or employees. Each actor affects the project in different ways, and this impact must be considered.
“How?” — Analyze the Impact: #
- The following step is to answer the question “How?” , — how can the behavior or actions of the actors influence the business goals? It’s important to identify both the positive and negative impacts of each action on goal achievement. The diagram shows how actors’ actions are linked to different levels of impact.
“What?” — Determine the Deliverables: #
- Finally, you need to answer the question “What?” — what results or key project elements (such as epics, features, or user stories) will be delivered if the goals are achieved. This stage involves defining the specific product elements needed to realize the goal.
4. Expected Monetary Value (EMV) #
The Expected Monetary Value (EMV) method is based on using probabilities and monetary estimates to evaluate different scenarios for executing ideas. It helps assess the expected cost or benefit of implementing an idea, taking into account possible risks and the likelihood of their occurrence. The formula for calculating Expected Monetary Value (EMV) :
Where:
- P {\scriptscriptstyle 1} is the probability of each outcome occuring (from 0 to 1);
- V {\scriptscriptstyle 1} is the monetary value or benefit of each outcome.
EMV is calculated as the sum of the products of the probabilities of various outcomes and their corresponding monetary values. This formula helps determine the expected financial impact of an idea, taking into account both its potential benefits and associated risks.
EMV method is often represented as a decision tree, which involves the following steps:
-
1. Define the Decision (Decision Node): #
The first step is to define the decision that needs to be made. In the example shown, it’s “Should we build a new plant or renovate the existing one?”.
-
2. Define Scenarios and Probabilities (Probability Node): #
Next, for each decision option, possible scenarios are outlined. For example, a “strong demand” scenario (with a 60% probability) or a “weak demand” scenario (with a 40% probability). Each scenario has a likelihood associated with it.
-
3. Calculate Costs for Each Scenario: #
For each scenario, the potential cost or benefit is calculated. In the example, building a new plant would result in an $80M profit under strong demand or a $30M loss under weak demand.
-
4. Calculate EMV (Expected Monetary Value): #
Ideas with low impact and high effort. These tasks are not worth the investment and typically should not be pursued.
Where:
- P {\scriptscriptstyle 1} is the probability of each outcome occuring (from 0 to 1);
- V {\scriptscriptstyle 1} is the monetary value or benefit of each outcome.
In the example, the EMV for building a new plant is $36M, while for renovating the existing plant, it’s $46M.
-
5. Choose the Best Option (End of Branch): #
The final step is to choose the option with the highest EMV. In this case, renovating the plant is more favorable, as its EMV ($46M) is higher than that of building a new plant ($36M).
The EMV method, combined with decision trees, is a powerful tool for risk analysis and making optimal decisions under uncertainty.
5. Weighted Shortest Job First (WSJF) #
The Weighted Shortest Job First (WSJF) method is used for prioritizing tasks in a project. The core idea of WSJF is to evaluate tasks based on their business value, time criticality, and ability to reduce risks or enable opportunities. These values are then compared against the size of the task (the cost of completing it), allowing teams to create a priority queue for execution.
WSJF formula:
Where:
- Business Value: How important the task is to the business or users.
- Time Criticality: How urgent it is to complete the task soon.
- Risk Reduction or Opportunity Enablement: How much the task helps reduce risks or opens new opportunities.
- Job Size: The estimated complexity or effort required to complete the task.
Steps for Applying WSJF:
1. Define Task Metrics:
- Evaluate each feature or task based on metrics such as business value, time criticality, and risk reduction or opportunity enablement. In the table provided, these values are scored from 1 (low) to 10 (high).
2. Assess the Task Size:
- Evaluate the complexity or size of the task (Job Size) on a similar scale. A higher score means more effort or time is needed to complete the task.
3. Calculate the Priority:
- For each task, calculate the priority by dividing the sum of business value, time criticality, and risk reduction by the task size.
For example, for “Feature A”:
WSJF = {10 + 1 + 1 \above{1pt} 1} = 10And for “Feature D”:
WSJF = {1 + 8 + 10 \above{1pt} 8} = 1,25
4. Rank Tasks by Priority:
- Rank the tasks by their WSJF score: the higher the value, the higher the priority. In the example, Feature A gets the highest priority with a score of 10.
This method helps focus on tasks that deliver the highest value in the shortest possible time, with minimal effort.
6. RICE (Reach, Impact, Confidence, Effort) #
RICE is a framework used to prioritize features or tasks based on four key factors: Reach, Impact, Confidence, and Effort. The goal of the RICE method is to ensure that the most important tasks are prioritized based on their potential value and cost of implementation.
RICE Formula:
Where:
- Reach:
How many people or segments will be impacted by this feature or task? This factor is usually represented as a number of users or affected entities over a defined period. - Impact:
How strongly will this feature or task affect the user or the business? This is usually rated on a scale from 1 (minimal impact) to 5 (massive impact). - Confidence:
How confident are we in the accuracy of the reach and impact estimates? This is expressed as a percentage. - Effort:
How many resources (measured in person-months, for example) are required to complete the task? A higher score indicates a larger effort.
- Reach:
The RICE score helps project managers focus on tasks that provide the most value with the least effort.
Steps to Apply the RICE Method Using the Example in the Image:
1. Define the Feature or Task:
- List the features or projects that need to be prioritized. In the image, the list includes:
- Biometric Authentication
- Stripe Pay Integration
- Shopify Ecommerce Cart
- API Dashboard
2. Evaluate Each Feature Across the Four RICE Factors:
Reach: #
- Estimate how many users or stakeholders the feature will affect. In the image, Biometric Authentication has a reach of 81 users, while API Dashboard affects 43 users.
Impact: #
- Rate the impact on a scale from 1 to 5. In the example, Biometric Authentication has a maximum impact score of 5, while Stripe Pay Integration has a slightly lower impact score of 3.
Confidence: #
- Provide a confidence percentage to show how sure you are about the accuracy of your reach and impact estimates. Biometric Authentication has 75% confidence, whereas Shopify Ecommerce Cart has 50%.
Effort: #
- Rate the effort required to implement the feature on a scale of 1 to 5. A lower score indicates less effort. In the image, Biometric Authentication has a relatively low effort score of 2, while API Dashboard has a higher effort score of 3.
3. Calculate the RICE Score:
- Apply the formula for each feature:
- For example, for Biometric Authentication, the calculation would be:
- For Stripe Pay Integration:
4. Rank the Features by RICE Score:
- After calculating the RICE scores, rank the features from highest to lowest. In the image, Biometric Authentication ranks the highest with a RICE score of 152, indicating that it should be the top priority, followed by Stripe Pay Integration (35), Shopify Ecommerce Cart (29), and API Dashboard (14).
The RICE method allows you to focus on the most important functions, ensuring the most efficient use of resources. This method allows you to balance the impact of an idea and its implementation, taking into account resource costs.
7. MoSCoW (Must, Should, Could, Won’t) #
The MoSCoW method is a task prioritization approach that categorizes tasks into four groups based on their importance to the project:
Step 1: Simple Task Distribution Using MoSCoW
The image below illustrates a simple task distribution using the MoSCoW methodology, without the use of buffers. Each task is categorized as follows:
-
Must Have: #
These tasks occupy no more than 60% of the total effort, and they are essential for achieving the project goal.
-
Should Have: #
Tasks that should be completed but are not critical, typically occupying around 20% of the effort.
-
Could Have: #
Additional tasks that can be completed if there are resources available, taking up around 20% of the effort.
-
Won’t Have: #
Tasks that will not be included in this cycle.
This approach clearly allocates time and resources based on task importance, with no mention of buffers.
Step 2: Detailing the Approach with Buffers
The image below provides a more complex workflow, showing how buffers are used during the Planning Phase and Execution Phase:
In this phase, 60% of the resources are allocated to “Must Have” tasks.
- External organizations or individuals who provide materials, equipment, or services for the project.
In the execution phase, “Must Have” tasks remain the top priority, and the project cannot be completed without them.
The remaining tasks can be completed if time and resources allow, utilizing the buffer.
This flowchart demonstrates the importance of having a buffer to manage unexpected delays or additional tasks.
Step 3: Example of Feature Allocation Using MoSCoW
The image below presents a matrix where project features are distributed according to the MoSCoW categories
-
Must Have: #
Mobile app integration and API for third-party integrations are critical tasks that must be completed.
-
Should Have: #
User profile customization is important but not critical to project success.
-
Could Have: #
Multi-language support can be added if resources are available.
-
Won’t Have: #
The dark mode feature will not be implemented in this cycle.
This approach helps the team focus on the most important tasks, effectively allocating effort and resources based on priority.
Using different methods of evaluating and prioritizing ideas helps project teams make informed decisions based on the impact, effort, risks, and long-term value of ideas. It is important to choose the methods that best fit the specifics of the project, its goals, and resources.
