A practical guide for turning learning objectives into useful AI drafted assessments

AI can write quiz questions in seconds. That does not mean the questions measure the right thing!

Type “Write a 10-question quiz from this training” and an AI tool will do it quickly, before you finish your coffee. The efficiency is real, but so is the risk.  A professional-looking quiz can still test minor details, flatten a nuanced idea or reward good guessing instead of proving that someone can do the job.

The solution is not to avoid AI completely, but to give it better direction and keep a human responsible for the final decision. Begin by defining what employees must be able to do (performance), how well they must do it (standard), and what will show they can do it successfully (evidence). Then use AI to draft and improve the assessment.

This guide shows you how to make the decisions AI can’t make for you. A free interactive prompt builder will then turn those decisions into a customized assessment prompt, challenger prompt and human-review checklist.

What the Cassano study found

Cassano and colleagues tested a large language model framework that created multiple-choice questions from 45 short training video transcripts. Seven instructors reviewed 257 generated questions. Most questions were clear, answerable and usable, and instructors estimated that the approach could save substantial drafting time. But the review also revealed a problem that matters for workplace training: approximately 19 percent, were rated low in relevance, not closely connected to the learning objectives.

The weak questions often treated a passing example as though it were the main concept, focused on peripheral details, removed important context, or turned a nuanced explanation into an absolute statement. In other words, the AI could produce a technically correct question that missed what mattered.

Four rules for AI-generated multiple-choice questions. Graphic supplied by the author.

Start with the job instead of the quiz

Before you ask AI to write anything, decide what successful performance looks like. A learning objective such as “understand the scheduling process” is too vague to guide an assessment. Replace it with an observable action, a clear standard and credible evidence.

Decision	Question to answer	Example
Performance	What must the employee do?	Enter a client request correctly in the scheduling system.
Standard	What makes the performance acceptable?	Complete every required field, choose the correct service code and make no privacy errors.
Evidence	What would prove the employee can do it?	Complete a realistic scheduling scenario in a practice environment and explain one exception.

This step often reveals that a quiz is not the best assessment. If employees must perform a procedure, handle a conversation, or create work, stronger evidence may come from a demonstration, role-play, simulation or scored work sample.

Use the STEER cycle

STEER is an Elevated Video Solutions framework for keeping human judgment at the center of AI-assisted assessment design. It is informed by Cassano et al., but it is not a tested study protocol.

S  Specify  Define the performance, standard and evidence before requesting questions. Also identify the audience, job context, source material and assessment purpose.

T  Task  Give the AI the full assignment. Tell it what to use, what to produce, what to avoid and how every item must connect to the objective.

E  Examine  Review the draft for relevance, accuracy, context and nuance. Check whether it assesses a core concept or job decision rather than an incidental detail.

E  Edit  Revise wording, distractors, scenarios, scoring criteria and feedback. A subject-matter expert should confirm the content; an instructional designer should confirm the evidence.

R  Refine  Use the rejected or weak items to improve the prompt, then generate a new version. Treat the prompt as a design document that gets better with each review cycle.

Build the prompt in two stages

The best way to prevent AI from generating poor questions is to separate planning from generation. First, ask for an assessment blueprint. Review it. Only then ask for the actual items.

Stage one asks AI to plan

The blueprint should map each objective to the evidence required, recommend an appropriate assessment format and flag missing information. This is where you catch a mismatch before AI creates 20 versions of the wrong question.

Stage two asks AI to draft

After you approve the blueprint, ask AI to generate the questions, scenarios, performance tasks or scoring criteria. Require a rationale for every item so you can see why the model thinks the item proves learning.

What a strong assessment prompt must include

A useful prompt gives AI more than a topic and a request for ten questions. It supplies the instructional decisions that determine whether the assessment will collect meaningful evidence. Before asking AI to draft, include:

The employees’ roles, experience and work environment

The observable performance the training should support

The standard for acceptable performance, including important conditions and unacceptable errors

The evidence that would demonstrate competence

The approved source material the AI may use

The purpose of the assessment, such as practice, a readiness check or final verification

The most authentic assessment format for the required evidence

Explicit constraints that prevent common AI question-writing problems

Tell AI to focus on core concepts, use examples only as context, preserve qualifications, provide enough information for a fair decision and avoid unsupported details. For multiple-choice questions, require one clearly best answer and plausible distractors based on realistic mistakes.

Build your customized prompt. The free AI Assessment Prompt Builder will use these inputs to create a detailed drafting prompt, a separate challenger prompt, and a human-review checklist.

What to check before you approve the assessment

A well-structured prompt improves the draft, but it does not approve the assessment. A subject-matter expert must verify the content, and an instructional designer must decide whether the assessment actually produces the evidence the objective requires.

☐  Does every item map to an approved learning objective?

☐  Does it test the core concept or job decision rather than a minor detail?

☐  Would success provide the evidence you actually need?

☐  Is the assessment method authentic for the performance?

☐  Is every answer grounded in the approved source material?

☐  Does the item provide enough context and preserve important nuance?

☐  Is there one clearly best answer or a scoring rule clear enough for consistent judgment?

☐  Could someone succeed through guessing or test-taking tricks rather than understanding?

Use a second AI model as a challenger

A second AI model can provide an independent critique because it has not followed the same reasoning path. Give it the objective, performance standard, required evidence, source material and draft assessment. Ask it to identify misalignment, missing context, unsupported claims and ways a learner could succeed without mastering the content.

For a consequential assessment, use the challenger twice: review the critique, revise the assessment and challenge the new version again. The second model is still only a reviewer. A human must decide which recommendations improve the assessment.

The interactive prompt builder will generate both parts of this cycle: the prompt for creating the assessment and a separate prompt for challenging it.

When multiple choice is not enough

Multiple-choice questions can efficiently check knowledge, recognition and judgment. They rarely prove that someone can complete a procedure, use a tool, lead a conversation or produce acceptable work under real conditions. If the required evidence is performance, ask AI to help create the practice scenario, observation checklist or scoring rubric instead of forcing everything into a multiple choice quiz.

For example, a customer-service course might include a short scenario-based knowledge check, followed by a role-play scored for listening, accuracy, tone and appropriate escalation. The quiz supports learning; the observed performance provides stronger evidence of readiness.

The prompt is only the beginning

The most valuable part of an AI assessment prompt is the human thinking inside it: what employees must do, what acceptable performance looks like and what evidence would justify a decision about readiness.

AI can reduce the time spent drafting and revising. Humans must still decide whether the assessment is relevant, authentic and fair. Specify the evidence, STEER the output, challenge the draft and approve only what truly proves learning.

Ready to try the process?

Use the free AI Assessment Prompt Builder to create a drafting prompt, challenger prompt and review checklist. At Elevated Video Solutions, I help small businesses turn expert knowledge into training and assessments that support workplace performance.

 

Source

Cassano, G., Ebbert, D., Mirriahi, N., et al. (2026). “Instructors’ perceptions of a large language model framework for generating automated multiple-choice questions from video transcripts.” Educational Technology Research and Development.https://doi.org/10.1007/s11423-026-10646-7