Background & Highlights
- Internal Users provided feedback that the manual deployment of the Strategic Planner via excel and SQL was slow and confusing
- But there were some internal users who were close to self-sufficient
- Through data analysis and user interviews, we discovered that the first time users were struggling with the deployment
- We launched a self-service data management tool to address the feedback and automated a lot of the manual process for the deployment
- Deployment time and internal user satisfaction increased
Situation
At Analytic Partners, one of our main customer facing products was the Strategic Planner, which allowed enterprise customers (marketing teams of Fortune 500 companies) to simulate their quarterly marketing budget spending. To be more specific, it was an interactive UI dashboard that allowed users to re-allocate their marketing budget % by different channels (tv, online, print, etc.) in order to simulate and observe the changes to their predicted sales and ROI (return of investment) per channel. One of the feedback was that although the Strategic Planner was popular and fun to use by the enterprise customers, the internal users felt that the process of manually preparing the input data required to set up the Strategic planner was too long and confusing. This was a surprising feedback for the development team as we had dedicated technical resources to help out the internal users via JIRA Service Desk and documentations with demos readily available for the internal users.
Behavior
The Strategic Planner was a very popular product to convert the main analytic service of Analytic Partners, called the MMM (marketing mix modeling) into actionable insights and was updated on a quarterly basis. Preparation of the input data for the Strategic Planner was a two step process where:
- Internal users manually format the output of MMM in excel
- Technical support creates a table and insert the input data via SQL
The development team had a mixed response to the feedback since we assumed that the process was straightforward and there were some internal users who were very self-sufficient when it came to preparing the input data. To better understand the feedback, we decided to dig deeper with both qualitative and quantitative data points. In order to generate qualitative data points, we conducted a survey among the internal users (~ 20 teams) with 5 point likert scale to measure the difficulty, feasibility, satisfaction and a separate question to estimate the # of hours required to prepare the input data for the Strategic Planner.
For the quantitative data points, we decided to look at our JIRA service desk tickets and measure:
- Average duration of support request per ticket (generated every quarter): start time - end time
- # of excel attachments per ticket (indicative of errors and mistakes)
- Average response time of our dedicated technical resources based on SLA (service level agreement) timers
After going through the analysis, we learned that both metrics were within our reasonable threshold as:
- We averaged ~ 4 (out of 5) for all responses in the qualitative measurement
- Around 6 work hours spent by the internal users
- Average duration of support request ticket: ~ 1 week
- # of excel attachments per ticket: ~ 3 spreadsheets
- Average response time: ~ 2 hours
Although there’s room for improvement, the development team was quite satisfied with the metrics and felt that there wasn’t an urgent need to prioritize the pain point reported by our internal users. However, something felt off and I decided to filter the metrics by the duration of contracts for the Strategic Planner by grouping contracts that have been live for less than 2 quarters as “new clients” and the rest as “on-going clients”. The numbers told a different story when this grouping was applied. In general, the metrics from the internal users who were supporting the “on-going clients” were satisfactory, but the problem was with the internal users supporting the “new clients”. Below were the metrics from the internal users who were supporting “new clients”:
- 5 out of 20 projects were considered as “new clients”: 25%
- ~2 (out of 5) for all responses in the qualitative measurement
- Around 15 work hours spent by the internal users
- Average duration of support request ticket: ~ 2.5 week
- # of excel attachments per ticket: ~ 9 spreadsheets
- Average response time: ~ 3 hours
It seemed that the learning curve of manually preparing the input data was high and despite having dedicated technical resources and documentations, the process required multiple back and forth between the support team and the internal users, especially during the initial phase of setting up the Strategic Planner. With this insight, there were multiple options to consider in order to address the pain point of our internal users:
- Provide better documentation
- Encourage the internal users to better communicate and share resources when it comes to setting up or updating the Strategic Planner
- Build a self-service data management tool to prepare the input data for the Strategic Planner
Among the 3 options, we determined that option 1 wasn’t an ideal solution to solve the internal user’s pain points and option 2 wasn’t too feasible since we had 20 projects spread across the global office and proposing to share our internal users across APAC regions and North American regions was not ideal. Therefore, we decided to run with option 3 as it was the most scalable solution via automation. Within 2 sprints, the development team was able to launch a self-service data management tool within the Strategic Planner that automatically transformed and uploaded the input data CSV file.
Impact
The story of how our development team launched a self-service data management tool for the Strategic Planner was a good learning experience as it required us to dive deeper on what seemed like a “business as usual” situation and better empathize with the internal users. If we were to just rely on high-level averages, we would not have been able to identify the root-cause of the internal user’s feedback. Personally, I felt that there were more data points to measure like the average experience of the internal users, day of the week when support tickets were registered, # of columns manually transformed by the internal users, better classification of on-going clients, and more. However, the insights gained from our initial analysis were significant and the development team was able to act on it. By launching the self-service data management tool for the Strategic Planner, we were able to improve the overall metrics for the internal users:
- ~4.5 (out of 5) for all responses in the qualitative measurement
- Around 2 work hours spent by the internal users
- Average duration of support request ticket: ~ 3 days
- # of excel attachments per ticket: ~ 0.5 spreadsheets
- Most internal users didn’t have to attach spreadsheets to support request tickets anymore
- Average response time: ~ 0.5 hours
- Dedicated technical resources were mostly responsible for bugs instead of being involved in the entire process
The biggest improvements came from the internal users who were involved with “new clients”:
- ~4 (out of 5) for all responses in the qualitative measurement
- Around 2 work hours spent by the internal users
- Average duration of support request ticket: less than 1 week
- # of excel attachments per ticket: ~ 1 spreadsheets
- Average response time: ~ 1 hours
These improvements may not have had a direct impact on the performance of Strategic Planner, but we would like to assume that the overall quality of our internal user’s service to the actual clients have improved with our new enhancement. The average duration of a support request ticket implies our deployment time of the Strategic Planner and watching it reduce from ~ 2.5 weeks to less than a week for the new clients was very meaningful. Above all, I was glad that we got to rely less on manual transformations via excel spreadsheets and make our internal users' lives easier :)