Educational Harness Engineering

Presentation context

NOTE

Media School department workshop, July 6, 2026 Educational Harness Engineering: Designing Teaching and Assessment That Resist Student Shortcuts in the AI Era Presentation materials

The collapse of an outcome-centered educational assessment model

Higher Education Policy Institute (HEPI) findings (2026)

• Higher Education Policy Institute (HEPI) survey (2026):
  • A survey of 1,054 undergraduates found an increase in students using AI in some way for their studies.
  • From 2024 to 2026, the proportion rose from **66% → 92% → 95%**.
  • The five most common uses were text generation (56%), summarization (38%), sentence revision (37%), translation (31%), and image generation (22%).

AI cheating and shortcuts are students' default

• Large-scale cheating in a Yonsei University remote exam (2025.10.15)
  • Students answered multiple-choice questions on an online site.
  • Cameras were used, but according to a poll posted on Everytime, 211 respondents said they had cheated.
• Cheating in an in-person Statistics Laboratory exam at Seoul National University (October 2025)
  • In a Statistics Laboratory course with around 30 students, some students using classroom computers were found apparently using AI without the teaching assistant's knowledge.
• Answer-sharing through KakaoTalk at Korea University (2025.10.25)
  • The midterm for Interdisciplinary Understanding of an Aging Society, a remote general-education course with around 1,400 students
  • involved students sharing questions and answers through a KakaoTalk open chat room. The entire midterm result was invalidated.

Outsourcing thinking among Korean university students

• AI is convenient, but habitual reliance erodes the need to think (Korean University Newspaper) 2026 신년기획/AI&미래고등교육③ , 2026. 1. 31)”
  • The article's own title was Studying became easier, and the need to think diminished as reported.
  • Students identified convenience as a major benefit but also said they had lost the need to struggle and think through answers.
• Lee (20, philosophy, Korea University): Why practice writing? Expressing thoughts in writing is an essential skill, but AI produces a reasonably similar result, so much of the reason to practice writing myself has disappeared.
• Jeong (26, economics, Kyungpook National University): Problems are solved easily, but I remain the same AI makes difficult points in studying very easy to resolve, yet it feels less like personal growth and more like easily getting past an obstacle that I still cannot overcome myself.
• Bae (25, political science and diplomacy, Dankook University): My thinking feels simplified Whenever I get stuck even slightly, I immediately search with AI. It feels as though my thinking has stopped and become simpler.

Detection and bans are not the answer

OpenAI's Text Classifier: discontinued in 2023 (no further updates)

QUOTE

OpenAI: discontinued for low accuracy (2023.07.20). As of July 20, 2023, the AI classifier is no longer available due to its low rate of accuracy. We are working to incorporate feedback and are currently researching more effective provenance techniques for text, and have made a commitment to develop and deploy mechanisms that enable users to understand if audio or visual content is AI-generated.

AI detectors falsely accuse non-native English-speaking students

• Liang et al. (2023) - doi: 10.1016/j.patter.2023.100779
  • Key finding 1: AI detectors falsely label non-native English speakers' writing as AI-generated.
  • Key finding 2: a single prompt can also make AI-generated text appear not to be AI-generated.
• Liang et al. (2023)
  • When GPT detectors assessed non-native English TOEFL essays, the results showed more than half were incorrectly labeled AI-generated (mean false-positive rate in the paper: 61.3%)
    • Use richer vocabulary, like a native speaker was enough to reduce AI detection to **11.3% (a sharp increase in false negatives)**.
    • Restricting the vocabulary of real eighth-grade native-English student writing raised false positives from 5.1% to 56.9%.
  • GPT detectors assessed non-native English US college-admission essays with a false-positive rate of **70.0%**.
    • Use a more literary variety of expression was enough to reduce AI detection to **3.3% (a sharp increase in false negatives)**.
  • When GPT detectors assessed Research abstracts written by non-native English speakers had a false-positive rate of **43.9%**.
    • Use a more literary variety of expression The same single prompt reduced AI detection to **19.1% (a sharp increase in false negatives)**.
• Hadra, Cambridge & Mesbah (2026) - doi: 10.1007/s40979-026-00213-1
  • Key finding 1: a study of essay detection with Turnitin and Originality reported
    • Turnitin that 900–1,100-word essays had a 32% false-positive rate
    • Originality that and 450–550-word essays had a 37% false-positive rate in that study.
  • Key finding 2: both Turnitin and Originality showed very high false-positive rates for STEM academic writing: 49% and 42%, respectively.
    • Accuracy for humanities writing was 86% and 96%, respectively, but STEM papers are often brief and limited to numbers and facts, making AI detectors unsuitable for this disciplinary writing.

Detection itself has limitations

QUOTE

Ardito (2023) - Contra generative AI detection in higher education assessments Reliance on detection mechanisms is at odds with an educational reality in which AI is becoming ubiquitous. A strategic shift is needed toward robust assessment methods and educational policies that embrace generative AI while preserving academic integrity.

QUOTE

Ardito (2023), Contra generative AI detection in higher education assessments: traditional plagiarism detectors and AI detectors operate differently. Plagiarism detectors such as Turnitin compare text against an existing original, whereas AI detectors examine the final output for sentences thought likely to have been generated by a typical LLM. Unlike comparison against original text, there is no independent means of verifying the assessment result. (AI detectors operate on a fundamentally different principle. They assess the degree to which textaligns with the most common outputs of supported large language models and do not provide anymeans to independently verify their assessment.)

QUOTE

Ardito (2023), Contra generative AI detection in higher education assessments: consider two students, Alice and Bob. Alice personally drafts the essay and works through the entire thinking and writing process, but because she uses AI for the final polish, her AI score is high. Bob, in contrast, generates the entire essay with GPT, then changes words and sentences immediately before submission to evade the detector and receives a low AI score. Is it valid to conclude that Alice's essay does not contain her original ideas solely because its AI score is high?

If neither banning nor detecting AI works, the answer is design

• Ban, leave alone, or design: three paths, one answer
  • Ban it: do not use AI Impossible to verify, and students do not develop AI proficiency.
  • Detect it: catch it with a detector ):
    • OpenAI officially discontinued its Text Classifier
    • Bias against non-native English speakers and STEM students
    • Fundamentally different detection mechanisms and their limitations
  • Leave it alone: use AI and submit the result : the worst direction, with no control for the instructor.

IMPORTANT

Shift toward design, beyond detection and bans One answer remains: design. Design assessment so that students cannot pass without thinking for themselves, even when using AI. This is the starting point for education in the AI era.

Educational Harness Engineering starts by asking what shortcuts students might take

From prompts to context, then to harnesses

• AI use often begins with a series of requests to do more, such as do this or write it more specifically.
• For example, if you prompt it to fact-check, AI does not know what you mean by fact-checking.
  • Instead of the phrase fact-check, specify the Journalists Association of Korea ( www.journalist.or.kr ) website's Journalist of the Month awards pages, and ask it to compare the text only against the bodies of the winning articles linked there. Only after spelling everything out like this does it begin to work properly. This is **Context Engineering**.
• Thinking carefully, asking it to compare only against that website also means not consulting other sites at its own discretion: you are stating constraints more strongly.
  • Being more specific means making prohibitions and constraints more explicit in practice. That is Harness Engineering the point.

A shift in mindset: think like an attacker

QUOTE

Bruce Schneier — The Security Mindset (2008) "Good engineering involves thinking about how things can be made to work; the security mindset involves thinking about how things can be made to fail. It involves thinking like an attacker, an adversary or a criminal. ... if you don't see the world that way, you'll never notice most security problems." Good engineering considers what will make something work . A security mindset considers what could make it fail . Unless you think like an attacker, adversary, or criminal, you will miss most security problems.

Educational Harness Engineering

• An ineffective approach (harness failure):
  • I worked so hard to write this exam or assignment. How dare you take shortcuts?
• An effective approach (harness success):
  • Use GPT or another AI to help plan, design, and review the harness.
  • I plan to run a remote exam or assignment with these instructions. The attachment contains the instructions and the planned procedure. Students will try every possible way to avoid thinking for themselves, asking AI to write the essay and submitting it with minimal effort.
    • a) What shortcuts could they use?
    • b) How can I design it so that reliance on AI alone necessarily fails, regardless of cheating or shortcuts?
    • c) How can I assess the thinking process so that logical errors are exposed when students do not think, plan, review, or criticize for themselves?

• The mindset needs to change:

• if students take shortcuts,

• the assessment design has failed to anticipate and block those shortcuts.

• Students will certainly seek shortcuts that avoid thinking:

• start the design of every exam and assignment from that premise.

Educational Harness Engineering case 1: AI-Assisted Research Methods

AI-Assisted Research Methods: course structure

• Class schedule: In-person teaching
  • Every Monday (three hours)
• Teaching format: Lectures + discussion + practice
• Course content: Social-science research methods + prompt engineering and AI use
  • Traditional research theory and methodology from a social-science communication perspective
  • AI use from a social-science perspective (top-down versus bottom-up thinking)
• Discussion and practice: NotebookLM + reading papers in English
  • Read papers on prompt engineering and related topics using AI within a 15–20-minute limit
  • Instructor: use Socratic questions to ask whether students can explain the material aloud themselves

QUOTE

Achmage's Socratic questioning example. Instructor: What did you read in Attention Is All You Need? Student: looking at and reading from the screen, I asked about the equations. According to AI's explanation... Instructor: deliberately interrupting and looking directly at the student, Leave the equations aside. Can you explain how an LLM works according to this paper? Student: after thinking briefly, Sorry, I do not know.

If you do not understand, ask: I still do not understand. Explain it more simply, so that an elementary-school student could understand and keep asking until you do. If you still do not understand, use the slide-generation button and ask: Draw an educational comic for children that helps explain this paper then read it and continue until you understand. That is the first step in prompt engineering. "

AI-Assisted Research Methods: in-class writing

• Each week during class, for around thirty minutes to one hour, students write to discover how they themselves view social phenomena.
• Choose writing topics for which having AI write instead of the student has no value to the learning purpose.

Week	Category	Question 1 · Type A	Question 2 · Type B
Week 2 (3/16)	Self-awareness and self-narrative	What kind of person do I think I am, and when was that belief recently shaken?	Have I ever felt that how others see me differs from how I see myself? If so, what differed?
Week 3 (3/23)	The gap between words and actions	What do I often say is important but fail to practice consistently?	Is there something I repeatedly do despite saying I dislike it? Why?
Week 4 (3/30)	Examining criteria for judgment	When I first meet someone, what makes me judge whether they are trustworthy?	When I read or hear something, what makes it seem plausible?
Week 5 (4/6)	How I change across relationships	How do I differ when I am alone and when I am with others?	Around whom do I become talkative, and around whom do I become quiet? Why the difference?
Week 6 (4/13)	Rules and conformity	Choose a rule I normally take for granted: whom does it benefit?	Is there a rule I follow despite finding it strange? Why do I keep following it?
Week 7 (4/20)	Comparison and fairness	What makes me feel that something is fair, and what makes me feel wronged?	Why can I accept the same outcome in one situation but feel angry about it in another?

AI-Assisted Research Methods: final project

• Final project: June 1–21 A three-week individual daily-writing challenge
• Core objective: Help learners discover through writing how they view social phenomena
• Phase 1: Everyday observation and records (June 1–11)
  • How do I view social phenomena each day?
    • Method: write at least one Obsidian note every day about myself and social phenomena.
    • Content: responses to news, thoughts while walking, perspectives discovered in conversations—anything is welcome.
    • Goal: develop a habit of retaining fragments of thought in text instead of letting them disappear
• Phase 2: Deeper writing informed by an AI metacognition report (June 12–21)
  • How do my records change after I become aware of my thinking patterns?
    • Starting point: move to phase two after the **AI metacognition report** and in-class practice provided on June 12, the make-up week.
    • Method: continue daily records while consciously attending to blind spots and unconscious patterns identified in the analysis.
    • Goal: use feedback to move beyond listing phenomena toward deeper, more refined writing.

AI-Assisted Research Methods: personalized AI metacognition reports

• Students submit their weekly in-class Markdown (.md) writing as an individual bundle
• AI meta-analysis: Design a Skills prompt that analyzes all writing submitted by each student together
• Grounded in the submitted text:
  • Design AI prompts so every piece of feedback is grounded in the student's writing. No flattery or gaslighting: feedback and challenges must refer to the submitted text.
  • Example AI feedback: In week two, you wrote that you do not seem to judge others hastily. Yet in week seven, sentences such as I do not understand why people act that way and they seem to be like this frequently make unsupported judgments based on your own criteria. This suggests a cognitive gap or bias between your awareness of your standards for judging others and the standards you actually apply.
• 2-track feedback
  • Thinking and metacognition profile (Track A) : a comprehensive assessment across at least eight dimensions, including metacognitive monitoring, self-regulation and execution, framing of social phenomena, information credibility, self-narrative and change, self-discrepancy and role tension, attribution and allocation of responsibility, and interpretation of emotion
  • Writing proficiency profile (Track B): Assess both weaknesses in writing and thinking patterns across at least five dimensions: organization, argument development, vocabulary, style and voice, clarity, and metacognitive writing
• Theory-based
  • Feedback for every student is grounded in social-science communication theory to support metacognitive analysis
  • Every analysis report must include links to the original theoretical papers and references by design.

AI-Assisted Research Methods: sample AI metacognition report

AI-Assisted Research Methods (spring 2026): final exam (in person, essay questions disclosed on the day)

• How AI read me, and how I reread myself:
  • Throughout the semester, you wrote every week about yourself, relationships, rules, fairness, information credibility, digital habits, failure and responsibility, and interpretations of emotion. Your collected writing then informed an individual metacognition analysis report . You also wrote daily from June 1 to 21, and from June 12 were asked to attend to weaknesses and recurring patterns in your thinking. This final exam does not ask you simply to accept AI's analysis. Using your own records, it assesses your ability to verify, challenge, correct, and extend that analysis for yourself.
• Exam question:
  • Write one complete essay based on the following three sources.
    • Your own metacognition analysis report
    • written from June 1 to 11: Phase 1 daily-writing records
    • written from June 12 to 21: Phase 2 daily-writing records
• a) How did the AI report read me?
  • Choose one or two analyses in your metacognition report that you consider most important. You may choose, for example, from the following.
    • Frames for viewing social phenomena
    • Self-narrative
    • Criteria for trusting information or people
    • Interpretations of failure and responsibility
    • How emotions arise and are interpreted
    • Weaknesses in thinking
    • Writing habits
  • Do not simply summarize the report: explain why that analysis felt important to you in particular.
• b) Do I agree or disagree with the analysis?
  • Choose one of the following positions on the AI report's analysis.
    • I agree
    • I partially agree
    • I disagree
    • I initially agreed, but changed my mind after rereading my records
    • I initially disagreed, but saw some truth in it after rereading my records
• Your answer must provide at least two pieces of direct evidence from your own writing. Direct evidence includes:
  • Sentences I actually wrote
  • Expressions I used repeatedly
  • Recurring judgments in similar situations
  • Changes in how I recorded things before and after June 12
• Note: do more than say that the AI report is right or wrong. Show the process of reassessing its analysis against your records.
• c) How did my writing change before and after June 12?
  • Compare your Phase 1 records before June 12 with your Phase 2 records afterward.
  • Answer the following questions.
    • Before June 12, how did I usually record social phenomena or myself?
    • After receiving the AI metacognition report, what did I pay more attention to while writing?
    • After becoming aware of weaknesses and recurring patterns, what changed in the depth, evidence, questions, or expression in my writing?
    • What did not change?
    • If something did not change, why do I think it remains the same?
  • Note: the point is not to claim improvement. Distinguish what actually changed from what remains unchanged.
• d) Did AI read me well, or plausibly misunderstand me? :
  • Identify at least one part where AI read you well and one where it misread you or missed context. Include all of the following.
    • What AI read well
    • Why I judge it that way
    • Context AI missed
    • What AI could have interpreted differently
    • What additional material or questions AI would need to read me more accurately

AI-Assisted Research Methods (spring 2026): sample student exam work

QUOTE

Student A: I had never really written before. Brief notes were only reminders of tasks; this was probably my first time expressing my thoughts and stories... [omitted] ...Before reading the AI metacognition report, I tended to make a strong claim and then become vaguer and less certain as the text grew longer. So my endings always shifted responsibility onto the reader ... [omitted] ...After receiving the report, however, the major change was that I began trying to state my thinking clearly and understand other people's positions. Feeling exposed by the report, I chose an approach other than making excuses.

QUOTE

Student B My weaknesses were habitually using very and ending sentences with it seems, blurring my thoughts without confidence. I had not realized that expressions I used unthinkingly could be weaknesses. Once I recognized my habits, I noticed myself writing very and immediately deleted it. After seeing the problem myself, I resolved to choose precise words rather than dress sentences in vague adverbs. Learning weaknesses I had not noticed made writing habits feel more important than anything else. Writing was a little difficult after reading the AI metacognition report. Trying to correct weaknesses I had not known about made me hesitate repeatedly... [omitted] ...After June 12, I wrote while consciously trying to correct them. As I wrote every day, I felt my habits gradually changing. Previously I would have written that writing which attacks and provokes others seems wrong. Now I write that it is wrong. The revised writing now feels more natural, and I wonder why I used to write that way.

QUOTE

Student C: [omitted] ...I did not want to deal with my inner feelings. I just wanted to rest and did not want to think further about relieving stress. Putting these feelings into writing instead cleared my head. The strain I had been living with eased, and time seemed to slow down. It was an unusual experience, and I compared it with earlier writing to see why it read more clearly. Earlier pieces tried to make claims or feelings sound impressive, with statements such as think positively and work–life balance is a mistaken phrase. I liked some essays, but others did not feel like me. After that, I began writing thoughts as they came, even disjointedly, including expressions such as this is annoying or disgusting. I accepted that the person writing these things was also part of me, and thought it might be enjoyable to live a little more unapologetically. Over those days, some writing sounded like a child and some like a young adult. Seeing these varied sides of myself, I think I understand a little better what it means to feel that knowing myself is enough.

QUOTE

Student D: I chose two main points. First, I can reframe problematic situations positively, and was surprised to see my usual thinking so clearly in my writing. Second, the AI report described me as accepting information unconditionally, easily believing what I heard on YouTube or from people in relevant fields without verification. How did my writing change after the report? Before June 12, I viewed social phenomena and myself very positively, sometimes with excessive optimism. After receiving the report, I realized that excessive positivity can paradoxically lead to laziness and complacency, so I tried to examine and record my own problems and social issues more sharply. ... [omitted] I was astonished that it analyzed not only my writing but my thinking patterns and habits in ways I had not considered. The excessive positive reframing and unconditional acceptance identified by AI are things I need to keep reflecting on and improving. The course helped me realize the power of records and the meaning of encountering a self that has developed through them.

AI-Assisted Research Methods (spring 2026): semester outcomes

• Total writing practice about social phenomena and myself:
  • In-class writing: Eight sessions
  • Three-week daily-writing challenge: June 1–21 (21 days, three weeks)
  • Individual AI metacognition analysis: Once per student (June 12)
• Marked improvement in students' overall metacognition and awareness of problems:
  • More than 90% of students submitted writing explaining their own sentence-level habits and cognitive weaknesses ( based on final exam submissions, handwritten without any digital devices )
  • Improvement in communication:
    • The instructor asked students to share personal reflections on what they gained during the semester.
    • All students could speak without looking at a laptop screen, maintaining eye contact and calmly, naturally, and sincerely expressing their feelings without a trembling voice, indicating substantial improvement in nonverbal communication as reported by the instructor.

Educational Harness Engineering case 2: AI Digital Graphic Design (spring 2026)

AI DGD: classroom examples

• Class schedule: In-person teaching
  • Every Tuesday/Thursday (3-2-2)
• Teaching format: Lectures + practice
• Course content:
  • First half, through the midterm: traditional 2D graphic design with Affinity v3
  • Second half, through the final: HTML + vibe design + vibe coding with AI

AI DGD: final exam instructions

Item	Content
Exam format	Create an HTML presentation using HAI-GPT within the time limit
Topics	Imagine competing for youth entrepreneurship support or investment funding. Create an HTML pitch for a startup or investment proposal based on interests, major knowledge, hobbies, activities, part-time work, clubs, or projects you know well. Treat actual judges, rather than the instructor, as your audience
Assessment criteria	Why does this problem matter now? What do I know about this problem firsthand? What real material supports my claims? Do the sources I used actually exist on the web? Can the viewer navigate directly to those sources from the HTML presentation? Did I verify what AI produced? What thinking and working process led to this presentation?
Deliverables	An eight-slide HTML presentation + the complete AI chat workflow log (an immediate 50% deduction if omitted)

Category	Example	Reason
Weak topic	Starting an AI-based shopping mall	Too broad; difficult to tell whether the student knows the subject
Weak topic	Making an app for university students	Unclear whose problem it addresses and what that problem is
Stronger topic	An HTML newsletter with local living information for international students	The intended users and information problem are relatively clear
Stronger topic	A comparison service for nearby gyms and exercise routines for beginners	Allows firsthand experience, research, comparison tables, and an explanation of market potential

Item	Criterion
Clickable web sources	At least five recommended
Authoritative sources	At least one government, public institution, university, company official page, paper, or report recommended
Where to cite	At the bottom of the relevant slide or on a references slide
Link format	Clicking must open the actual page

• Examples of good sources:
  • Government or public institution websites
  • University or research institute material
  • Official company websites
  • Actual service introduction pages
  • Newspaper articles
  • Papers or reports
  • Statistics Korea, local governments, and public data
  • Official startup-support program pages
• Examples of weak sources:
  • Sources that AI merely made sound plausible
  • Links that do not exist when clicked
  • A real source whose content does not support the slide's claim
  • A citation with an unclickable link

IMPORTANT

The central assessment criterion, regardless of visual quality, is whether the planning intent is clear as explained to students.

AI DGD: actual final exam example

AI DGD: sample instructor assessment

• Missing app UX/UI: It is a tourism app—why is there no app mockup?
• Budgeting problem: In an era when some people report earning KRW 10 million through vibe coding with no initial capital, **why should funding go to someone who says development will only begin after receiving it?**
• Missing evidence: You say overseas-targeted marketing. How, specifically, will you recruit influencers?
• Design mismatch: Does this color palette fit the Korean Wave concept? **Inconsistent brand colors**

IMPORTANT

Average grading time: under five minutes. If the proposal mentions an app or platform but has no app or platform mockup, the score is automatically low A saturated concept or inadequate evidence: the score is automatically low Why should I give money to you in particular? the score is automatically low

AI DGD (spring 2026): implications from the instructor's perspective

A. Assignment design principles (Input Design)

• Design the planning logic and logical loop: Create assignments where unthinking use of GPT inevitably encounters logical errors it cannot resolve alone
• Design so that shortcuts are ineffective: Raise the difficulty to professional standards, either beyond what simply running AI can solve or, as in AI DGD, with AI use explicitly assumed
• Choose topics AI cannot substitute for: The writing example from AI-Assisted Research Methods: If you ask AI to write about yourself, what could you possibly use that for?
• Require submission of the process: For example, immediately deduct 50% of the exam or assignment score when workflow logs are missing

B. Assessment and grading (Output Evaluation)

• Assess the thinking process itself: Assess both the final result and the process. Requiring working records while shortening the time limit exposes weaknesses in thinking even when AI is used extensively
• Socratic oral verification: If a presentation sounds memorized, **interrupt immediately and ask a sequence of questions**.
• Do not set questions with a single correct answer: Students commonly struggle with questions that have no single right answer. Watching how they solve these for themselves allows an instructor to **assess them in just a few minutes**.

C. Educational Harness essentials (Core)

• Think like an attacker:   Design on the assumption that shortcuts will be attempted
  • Draft the exam or assignment first,
  • then ask GPT or a similar tool to review what shortcuts students could take—like reviewing and improving Skills
  • Ask for a revised design that anticipates those shortcuts and incorporates sections A and B above
• Design instead of detection or prohibition:  
  • Treat failure as a design failure, a failed prompt, rather than simply blaming students
  • Even an AI prompt only begins to work well after the reasons for its failure are incorporated: a core principle of harness engineering
  • Instead of only becoming angry at an unexpected shortcut, record the new case in instructor notes immediately and incorporate it into next semester's class: update the harness

• Do not block AI:

• Design so that students cannot pass without thinking, even when using AI.

That is Educational Harness Engineering.