Coding exercises
Coding exercises are instructor-created challenges that help learners in development and data science practice their coding skills right in the Udemy course-taking experience.
1.Strategy
Storytelling with actionable insights
As the design lead, I built a strong partnership with the product manager in my domain. In our first year in the domain, we gained a good amount of knowledge about our users and the Udemy instructor platform. We wanted to reflect that in upcoming years' product strategy.
We started collecting data on previous instructor research, content consumption, and competitor analyses. When the possible direction became more evident, we created decks to gain buy-in from senior leadership. We gradually grew our audience and content until we agreed on a high-level domain goal for the following year. Together with the PM, we lead our team to create detailed OKRs for the next quarter to get closer to achieving our goal.
New strategical theme:
Improve the quality and adoption of
in-course practice tools
Learning science proves that practice is vital for learning. Udemy offers its learners a unique course experience by combining video lectures and practice activities.
First, reimagining the coding exercise tool
Low
consumption rates
Importance of technology domains
Company strategy on skill development
Udemy prioritizes technology and development education and provides top-notch instructor content. However, the coding exercise tool, primary for development courses, has low usage among instructors.
Top online course platforms to learn to code
Project Goals
increase adoption rates
bring instructors who use third parties for coding practice back to Udemy
as a result, provide learners with a better learning experience.
2.Designs
Learner tools
Legacy view
New, IDE-like coding experience
With its new features like test cases, hints, and solution explanations, the coding exercise tool guides learners in every step. Adjustable panels of the design help learners to customize their coding experience experience.
Instructor tools
Legacy view
New instructor experience
New creation flow guide instructors with learning design best practices to create quality coding exercise while also giving flexibility to seasoned instructors to still use their flow.
Example codes, generated with AI feature and result panel with error details, help instructors to excel at creating test cases.
They can create learning-facing content in one final step.
3.Outcome
%81
Learner CSAT
“I have only done 3 coding exercises, but I’m already in love with them! Really love the course and how it comes with actual exercises and tests to see how you have been progressing.”
Learner quotes
“It is amazing, I would do more courses if this sort of activity was heavily used!”
“It works well and allows me to practice in a learning environment. It is very useful to receive feedback when I submit my code to make sure I am doing what was taught.”
%80.4
Instructor CSAT
“This is fantastic! I love the layout and options to show or hide, and the test report is great. Being able to access hints and a solution explanation from the exercise will be very helpful to students.”
Instructor quotes
“The flow of creating coding exercises is very simple to understand to newbies also!”
3x increase
in new courses with coding exercises
2x increase
in coding exercises created in a year
Instructor engagement
4. Process
Problem domain
As the core team of product manager, designer, and engineers working on instructor tools, we first had a heuristic evaluation of the legacy coding exercise in addition to analyzing the existing documentation. Then, we quickly realized we needed more insights from domain experts.
Heuristic evaluation
Since we had very talented engineers in the company, we launched an internal test as the fastest way to gather insights. Within a few days, 35 people experienced the tools and shared their findings with the project team. This study helped us to map the high-level pleasure and pain points.
Internal user test with all Udemy teams
In parallel, we completed a benchmark study to straighten our understanding of the current landscape in the industry.
Competitor benchmark
Instructor survey
We launched an instructor survey within the product dashboard to understand the reasons for low adaptation numbers.
Top reasons for not using Udemy's coding exercise tool
Creation of coding exercise is hard
Learner experience isn't good enough
Changing the course
After collecting a lot of insight, I organized a three-day design sprint. We prototyped an innovative solution that includes creating a coding exercise pool for instructors.
Kick start with design sprint
Hypothesis
IF we provide ready-to-use, quality coding exercises to instructors, THEN they will include them in their courses (and increase their learner engagement)
On the last day of the sprint, we invited five instructors to test our prototype, and they rejected the concept synonymously.
Key learnings
Each coding exercise is unique and closely-tight to the learning design of each course. So, ready-to-use coding exercises are not the answer.
Instructors were aware of the fundamental need for practice, and they would prefer to provide it with a Udemy tool to create a holistic learning experience in their courses. However, they don't choose it now mainly because it doesn't have a good learner experience.
New focus
Designing the learner experience in the first phase, then moving on to the instructor tool in the second phase.
Applying learning design principles
I created close collaboration with the Udemy learning science team so we can reflect most recent learning and instructional best practices. They helped the team to create foundational rules for both learner and instructor tools. They were also a big part of the design reviews, always bringing a learning science perspective to the table.
Backwards design process is recommended while creating coding exercises, which suggests an instructor first identify their learning objective and create the solution. Creating the problem description is the last step. This ensures instructors frames the problem with the solution in mind.
Collaborative design process
Participatory design sessions with instructors
The UX researcher and I designed 1:1 participatory design sessions with Udemy instructors to create the learner tool to benefit from their expertise in learning offerings.
Structure
Coding exercise creation journey
While instructors showed and told us how they created their exercises, we took notes on Miro board and collected screenshots.Feature ideas
Instructors reviewed the journey in Miro and added more feature ideas.Sketch of the ideal learner experience
We sketch the ideal learner experience with them using the toolkit I prepped.Feature prioritization
Finally, they prioritized the features in their sketches.
When we started the solution phase of the instructor tool, we already launched the learner solution and made many iterations on it. So everyone on the team internalized the domain and instructor needs.
I started the instructor tool design process with a small design sprint to ideate together and create solution sketches that could be the starting point for our instructor-phasing solution.
Ideation workshop with the team
Frequent design reviews
with the core stakeholder team
At the end of each two-week sprint, I facilitated an interactive design review session with the core stakeholder team. I first presented the outcomes, such as; user flows, layout alternatives, detailed wireframes, prototypes, and UIs. Then participants added their feedback with sticky notes. Lastly, we reviewed the comments and discussed them as a team.
Participants
Pod (the core team of PM, product designer, and engineers), Product marketing manager, domain leads of product, design, and engineering, designers from another team experienced in the domain, Learning designer
After the reviews, I met with the PM, UX researcher, and Engineering lead in identifying the following actions based on feedback.
Co-design sessions
with the engineering team
Design reviews sometimes surfaced use cases I can't grasp fast enough to design for. In such circumstances, I organized small co-working sessions with a couple of volunteer engineers since they were the domain experts, and we designed for the use case together.
User-centered rollout approach
We first launched the learner experience and then the instructor experience in beta gradually to selected courses. We facilitated additional usability tests with learners and instructors during beta.
I also monitored user behavior through Fullstory recordings and organized watch parties with the team to share the excitement and the insights.
We collected feedback within the tool during beta and evaluated it frequently. In this process, on top of fixing bugs, we improved usability and fine-tuned delight elements in the experience.
