CDW
Global E-Commerce Experimentation Program
- Industry
- Retail / B2B
- Timeframe
- 1 year
- My role
- Experimentation Program Manager
- Year
- 2023
CDW Global E-Commerce Experimentation Program
Industry
Retail / B2B
Timeframe
1 year
My role
Experimentation Program Manager
Year
2023
About the company
CDW Corporation (Nasdaq: CDW) is a leading multi-brand provider of information technology solutions to business, government, education and healthcare customers.
Challenge
CDW’s split-testing ran as a black box — an external consultancy owned the method, and no one inside could see how it worked behind the curtain. The task was to rebuild it as a glass box: reconstruct the process without disrupting live operation, then improve the process and the materials — while handling confidential data and the reality that the internal know-how and commitment didn’t exist yet.
Objective
I designed and installed the whole system of work — standards and tools for scoring, statistical analysis, results presentation, and communication — and ran the program as its manager: a three-person team executing tests across the entire platform, with me involved at every stage from intake to recommendations.
PHASE 1: Blueprinting the Future-State Strategy
Phase one was an audit: I mapped how the existing process aligned with strategic and tactical goals and how it actually ran across workflows, tools, stakeholders, communication, and metrics. That assessment became the blueprint for replicating the end-to-end process with a new, transparent approach.
Highlights
Mapped new process with roles and milestones
Improved specification and workflow for submitting experimentation needs Project Description
Experiment description and results template for better learnability and process efficiency
Experimentation specification and knowledge source - know how transfer was one of the main program goals
PHASE 2: Working with the experimentation cycle in the agile model
The strategy and materials I built let us launch fast and run experiments across many product areas. The honest frictions were a dev bottleneck — test builds needed product-team engineers who weren’t always free — demand that came in waves, and the biases of the people commissioning tests.
Featured hypotheses & experiments
Showing top filters in the product search typeahead will make it easier for customer's to refine their search. “If I'm right, well see an increase in typeahead engagement.”
Ranking search results using New Al-driven Search Engine will push more relevant products to the top of the results. This might increase Search Engagement and Search to Cart rate
Modifying the appearance of inventory messages for items with a lead on the SERP will increase search engagement for those items
Change the "Reorder Now" button to "Reorder" and add an "Edit and Reorder" option to streamline the reordering experience for customers
Main metrics
Revenue
Measured by how experiments contribute too boosting revenue (revenue lift, revenue safed)
Impact
Assessed by how experiemnts influence product metrics and cognitive goals (lift in primary metrics, amount of Winners, Learners and Loosers)
Execution
Evaluated by the efficiency of experimentation delivery proces (amount of implemented tests)
Outcomes
Experimentation at enterprise scale — run inside CDW’s $21B B2B technology business.
Product teams began relying on experiment insights to prioritize initiatives, de-risking new features and strategies.
The in-house program replaced an external experimentation consultancy — hypothesis-driven testing became a fully internal capability.
Detailed lift metrics are client-owned and confidential — the program scored every initiative against a Revenue / Impact / Execution framework.
CDW Global E-Commerce Experimentation Program
Industry
Retail / B2B
Timeframe
1 year
My role
Experimentation Program Manager
Year
2023
About the company
CDW Corporation (Nasdaq: CDW) is a leading multi-brand provider of information technology solutions to business, government, education and healthcare customers.
Challenge
CDW’s split-testing ran as a black box — an external consultancy owned the method, and no one inside could see how it worked behind the curtain. The task was to rebuild it as a glass box: reconstruct the process without disrupting live operation, then improve the process and the materials — while handling confidential data and the reality that the internal know-how and commitment didn’t exist yet.
Objective
I designed and installed the whole system of work — standards and tools for scoring, statistical analysis, results presentation, and communication — and ran the program as its manager: a three-person team executing tests across the entire platform, with me involved at every stage from intake to recommendations.
PHASE 1: Blueprinting the Future-State Strategy
Phase one was an audit: I mapped how the existing process aligned with strategic and tactical goals and how it actually ran across workflows, tools, stakeholders, communication, and metrics. That assessment became the blueprint for replicating the end-to-end process with a new, transparent approach.
Highlights
Mapped new process with roles and milestones
Improved specification and workflow for submitting experimentation needs Project Description
Experiment description and results template for better learnability and process efficiency
Experimentation specification and knowledge source - know how transfer was one of the main program goals
PHASE 2: Working with the experimentation cycle in the agile model
The strategy and materials I built let us launch fast and run experiments across many product areas. The honest frictions were a dev bottleneck — test builds needed product-team engineers who weren’t always free — demand that came in waves, and the biases of the people commissioning tests.
Featured hypotheses & experiments
Showing top filters in the product search typeahead will make it easier for customer's to refine their search. “If I'm right, well see an increase in typeahead engagement.”
Ranking search results using New Al-driven Search Engine will push more relevant products to the top of the results. This might increase Search Engagement and Search to Cart rate
Modifying the appearance of inventory messages for items with a lead on the SERP will increase search engagement for those items
Change the "Reorder Now" button to "Reorder" and add an "Edit and Reorder" option to streamline the reordering experience for customers
Main metrics
Revenue
Measured by how experiments contribute too boosting revenue (revenue lift, revenue safed)
Impact
Assessed by how experiemnts influence product metrics and cognitive goals (lift in primary metrics, amount of Winners, Learners and Loosers)
Execution
Evaluated by the efficiency of experimentation delivery proces (amount of implemented tests)
Outcomes
Experimentation at enterprise scale — run inside CDW’s $21B B2B technology business.
Product teams began relying on experiment insights to prioritize initiatives, de-risking new features and strategies.
The in-house program replaced an external experimentation consultancy — hypothesis-driven testing became a fully internal capability.
Detailed lift metrics are client-owned and confidential — the program scored every initiative against a Revenue / Impact / Execution framework.
CDW Global E-Commerce Experimentation Program
Industry
Retail / B2B
Timeframe
1 year
My role
Experimentation Program Manager
Year
2023
About the company
CDW Corporation (Nasdaq: CDW) is a leading multi-brand provider of information technology solutions to business, government, education and healthcare customers.
Challenge
CDW’s split-testing ran as a black box — an external consultancy owned the method, and no one inside could see how it worked behind the curtain. The task was to rebuild it as a glass box: reconstruct the process without disrupting live operation, then improve the process and the materials — while handling confidential data and the reality that the internal know-how and commitment didn’t exist yet.
Objective
I designed and installed the whole system of work — standards and tools for scoring, statistical analysis, results presentation, and communication — and ran the program as its manager: a three-person team executing tests across the entire platform, with me involved at every stage from intake to recommendations.
PHASE 1: Blueprinting the Future-State Strategy
Phase one was an audit: I mapped how the existing process aligned with strategic and tactical goals and how it actually ran across workflows, tools, stakeholders, communication, and metrics. That assessment became the blueprint for replicating the end-to-end process with a new, transparent approach.
Highlights
Mapped new process with roles and milestones
Improved specification and workflow for submitting experimentation needs Project Description
Experiment description and results template for better learnability and process efficiency
Experimentation specification and knowledge source - know how transfer was one of the main program goals
PHASE 2: Working with the experimentation cycle in the agile model
The strategy and materials I built let us launch fast and run experiments across many product areas. The honest frictions were a dev bottleneck — test builds needed product-team engineers who weren’t always free — demand that came in waves, and the biases of the people commissioning tests.
Featured hypotheses & experiments
Showing top filters in the product search typeahead will make it easier for customer's to refine their search. “If I'm right, well see an increase in typeahead engagement.”
Ranking search results using New Al-driven Search Engine will push more relevant products to the top of the results. This might increase Search Engagement and Search to Cart rate
Modifying the appearance of inventory messages for items with a lead on the SERP will increase search engagement for those items
Change the "Reorder Now" button to "Reorder" and add an "Edit and Reorder" option to streamline the reordering experience for customers
Main metrics
Revenue
Measured by how experiments contribute too boosting revenue (revenue lift, revenue safed)
Impact
Assessed by how experiemnts influence product metrics and cognitive goals (lift in primary metrics, amount of Winners, Learners and Loosers)
Execution
Evaluated by the efficiency of experimentation delivery proces (amount of implemented tests)
Outcomes
Experimentation at enterprise scale — run inside CDW’s $21B B2B technology business.
Product teams began relying on experiment insights to prioritize initiatives, de-risking new features and strategies.
The in-house program replaced an external experimentation consultancy — hypothesis-driven testing became a fully internal capability.
Detailed lift metrics are client-owned and confidential — the program scored every initiative against a Revenue / Impact / Execution framework.