Coding exercises
Company: Udemy, 2022-2023
Role: Principal product designer
#productdesign #uxresearch #strategy #designsprints #workshops #CoDesign #ParticipatoryDesign
As the design lead, I worked closely with the product manager to identify instructor domain strategy for the following years. We created decks to showcase the importance of practice for our instructors, learners, and Udemy's upcoming strategy. We onboarded the team and gained buy-in from the senior leadership to focus on improving our practice tools, starting with the Coding Exercise Tool.
I was an integral part of end-to-end product development, from defining the product strategy & roadmap to design and delivery. In addition, I planned the user-centered design process and led collaboration in a remote workplace, with a multidisciplinary team distributed in different time zones.
Designing a learning tool for an unfamiliar technical domain provided a perfect opportunity to instrumentalize various human-centered design tools and demonstrate the benefit of a solid design process.
You can scroll down to read the whole process, or click here for the final designs.
Learning science proves that practice is vital for learning. Udemy offers its learners a unique course experience by combining video lectures and practice activities. Technology and development domains are a big focus for Udemy, and Udemy hosts the best instructors' content. Despite this, instructor adoption of the coding exercise tool, the primary practice tool for development courses, was extremely low.
We started this project to increase instruction adoption rates, bring instructors who use third parties for practice back to the Udemy platform, and consequently provide learners with a better learning experience.
Problem domain
Discovery
Understanding the legacy tool
As the core team of product manager, designer, and engineers working on instructor tools, we first had a heuristic evaluation of the legacy coding exercise in addition to analyzing the existing documentation. Then, we quickly realized we needed more insights from domain experts.
Since we had very talented engineers in the company, we launched an internal test as the fastest way to gather insights. Within a few days, 35 people experienced the tools and shared their findings with the project team. This study helped us to map the high-level pleasure and pain points.
In the meantime, we completed a benchmark study to straighten our understanding of the current landscape in the industry.
Internal user test
Why don't instructors use the Udemy coding exercise tool?
We launched an instructor survey within the product dashboard to understand the reasons for low adaptation numbers.
Top reasons
Creation of coding exercise is hard
Learner experience isn’t good enough
Kickstart to solution design
with a design sprint
After collecting a lot of insight, I organized a three-day design sprint. We prototyped an innovative solution that includes creating a coding exercise pool for instructors.
Hypothesis
IF we provide ready-to-use, quality coding exercises to instructors, THEN they will include them in their courses (and increase their learner engagement)
Change in strategy
On the last day of the sprint, we invited five instructors to test our prototype, and they rejected the concept synonymously.
Key learnings
Each coding exercise is unique and closely-tight to the learning design of each course. So, ready-to-use coding exercises are not the answer.
Instructors were aware of the fundamental need for practice, and they would prefer to provide it with a Udemy tool to create a holistic learning experience in their courses. However, they don't choose it now mainly because it doesn't have a good learner experience.
New focus
Designing the learner experience in the first phase, then moving on to the instructor tool in the second phase.
Collaborative
design process
Participatory design sessions
Structure
Coding exercise creation journey
While instructors showed and told us how they created their exercises, we took notes on Miro board and collected screenshots.Feature ideas
Instructors reviewed the journey in Miro and added more feature ideas.Sketch of the ideal learner experience
We sketch the ideal learner experience with them using the toolkit I prepped.Feature prioritization
Finally, they prioritized the features in their sketches.
The UX researcher and I designed 1:1 participatory design sessions with Udemy instructors to create the learner tool to benefit from their expertise in learning offerings.
Applying learning design principles
Udemy learning science team prepped a learning requirement documentation to guide the design of the ideal coding exercise learning experience.
Udemy learning science team prepped a learning requirement documentation to guide the design of the ideal coding exercise learning experience.
Coding exercises should allow for quick understanding checks. In addition, they should guide learners to help in between attempts while having an authentic, IDE-like experience.
Frequent design reviews
with the core stakeholder team
At the end of each two-week sprint, I facilitated an interactive design review session with the core stakeholder team. I first presented the outcomes, such as; user flows, layout alternatives, detailed wireframes, prototypes, and UIs. Then participants added their feedback with sticky notes. Lastly, we reviewed the comments and discussed them as a team.
Participants
Pod (the core team of PM, product designer, and engineers), Product marketing manager, domain leads of product, design, and engineering, A designer from another team experienced in the domain, Learning designer
After the reviews, I met with the PM, UX researcher, and Engineering lead in identifying the following actions based on feedback.
Co-design sessions
with the engineering team
Design reviews sometimes surfaced use cases I can't grasp fast enough to design for. In such circumstances, I organized small co-working sessions with a couple of volunteer engineers since they were the domain experts, and we designed for the use case together.
Final design
Authentic coding experience
The new IDE-like design has more editor capabilities. It opens expanded, and allows users to adjust all panels and remember their customization through the session.
Learner guidance in every step
Test cases and error details
Hints
Solution explanation by instructors
User-centered rollout approach
The new learner experience was first launched in beta gradually to selected courses. After the beta release, we tested the usability of the live tool with 12 learners.
We monitored user behavior through Fullstory recordings.
We collected both learner and instructor feedback during beta and evaluated them frequently. In this process, on top of fixing bugs, we improved usability and fine-tuned delight elements in the experience.
Outcome
%81
Learner CSAT
“I have only done 3 coding exercises, but I’m already in love with them! Really love the course and how it comes with actual exercises and tests to see how you have been progressing.”
Learner quotes
“It is amazing, I would do more courses if this sort of activity was heavily used!”
“It works well and allows me to practice in a learning environment. It is very useful to receive feedback when I submit my code to make sure I am doing what was taught.”
%30 increase
in # practice activities created by top instructors
3x increase
in # new courses with coding exercises
“This is fantastic! I love the layout and options to show or hide, and the test report is great. Being able to access hints and a solution explanation from the exercise will be very helpful to students.”
Instructor quotes
“The new tabs (“Hints” and “Solution explanation”) are very helpful. Now I can explain the solution in more detail. Thank you for adding this to the coding exercises”