Why self assessments improve learning
O’Reilly’s assessment tool puts the focus on the learner, not arbitrary scores.
Studying (source: O'Reilly)
We just don’t get it. Looking across the industry, most services promoted as “assessments” are basically memory tests or trivial click-to-own certifications. In one case, we discovered a site where 10 clicks through a video path was all that it took to get awarded a certificate, obtained in less than 10 seconds. Moreover, too much emphasis gets placed on scores provided to some third party, not on feedback to professionals so they can improve their learning experiences. We decided learners deserve better and we can deliver better.
The Safari learning platform just made learning more effective by bringing self assessments to our popular Learning Paths. We’re adding more structure and support for those wanting to master a given subject, accelerating your “speed to understanding” by combining our expert curated Learning Path with new self assessments available exclusively in Safari.
Learn faster. Dig deeper. See farther.
Different assessments for different outcomes
While a variety of approaches exist, let’s consider the contrast between summative assessment and the formative assessment we use as a self assessment.
Summative assessment measures how much students have learned up to a particular point in time, generally to meet some standard. Examples include final exams in university courses, or professional certifications. While those serve important needs in testing, their results are intended for someone other than the person taking the exam. Think: grades.
In contrast, formative assessment gives feedback during testing. It’s considered part of the learning experience, and need not be graded. Questions are constructed such that if you understand the material, the answers are quick. On the other hand, if you’re struggling with a subject, you’ll need to spend much more time working through the questions.
After carefully evaluating the options, O’Reilly chose to use formative assessment for the new self assessments in our Learning Paths. In other words, you get feedback for each answer. This approach is found in some state-of-the-art K-12 online learning, though rare in adult education and almost nonexistent for online learning in industry.
When a professional is working through a self-paced Learning Path, having several waypoints of self assessment provides the feedback needed. It’s what is most appropriate for the medium. More to the point, it’s what’s most relevant for the learner.
How assessments enhance learning
Part of our job at O’Reilly is to help learners gauge how well they understand important topics. Individuals, teams, and organizations need that feedback. In one view, we’d simply create comprehensive exams for each of the most important subjects, require people to take those exams, then provide grades. Even if that kind of linear, scholastic world ever existed, it’s long gone.
There’s a problem with how some traditional approaches in education tend to view the world. Aggregate raw scores from employees taking exams won’t help your organization succeed. Instead, you need to know how much your employees are struggling with information overload, huge learning curves, typical misconceptions, industry “fear, uncertainty, and doubt” (FUD), etc.
Also, keep in mind that the learning materials needed by people in industry span a range of important subject areas: design thinking, software architecture, data science, systems engineering, programming languages, security, machine learning, product management, leadership, and more. Most of these areas require expertise with increasingly complex technology stacks, often organized in several layers.
Here’s an example: Popular open source projects tend to push new releases frequently—every few months. So the challenge of maintaining “hands on” familiarity becomes more complex. Meanwhile, vendors have incentives to spread good claims about their products, and not-so-good claims about their competition. Developers, product managers, and execs alike get bombarded with FUD. The presumption that any individual can keep pace with all the changing layers and components in a given tech stack becomes nearly absurd. That’s why these mythical people are called “unicorns.” Pulling those factors together, how do you develop a process of creating a comprehensive exam, reduced to a numeric score, that can measure someone’s actual understanding of a complex technology stack? Frankly, it’s an imaginary thing.
Instead, O’Reilly approaches learning and feedback at a human scale. We’ve been moving away from large books and lengthy videos, breaking out of those arbitrary containers to provide much more readily accessible learning experiences.
Likewise, O’Reilly has a new approach to “what” we measure through assessment, and “how” we measure that, addressing the challenges mentioned above. Self assessments go through a rigorous process with the goal of distinguishing between those people who have invested time to learn a technology and those who are “phoning it in” or perhaps have been confused by industry FUD. The result is a feedback mechanism that helps learners learn.
O’Reilly has a team of experts in pedagogy and user experience creating assessments, with a goal of rolling out the first large-scale formative assessment program in the industry. We’re now introducing self assessments—feedback while learning—at waypoints along Learning Paths to improve learning outcomes. Self assessment is a natural next step in our evolution to meet the needs of learners. We look forward to building on these to provide truly personalized learning through our learning platform.