majeedkazemi

yeah, friction should be designed to be meaningful and not just feel frustrating or like a barrier. it should empower the user.

in our research, we found that an AI agent which involves the user in each step of the process (e.g. asking them to check the AI’s assumptions, or edit them directly if they’re not good) ends up being a bit slower -- i.e. more friction -- but gives the user more control. And when compared with an AI agent that provides less control but faster, users preferred the slower agent which provided more agency.

https://dl.acm.org/doi/abs/10.1145/3654777.3676345

"leverage"

Thanks!

> did you measure the correlation between CodeAid use and test scores?

Unfortunately, we couldn't measure such correlations due to many external factors that can impact students' performance on test scores.

However, our previous research specifically compared students learning python coding for the first time with and without LLM code generators. Austin and I wrote a summary on it here: https://austinhenley.com/blog/learningwithai.html

We found that students who performed higher on our Scratch programming pre-tests (before starting the Python lessons), performed significantly better if they had access to the code generator. However, the system that we used in that study was very different, it showed the exact code solution to the students' query. Here with CodeAid we were trying to avoid producing direct code solutions!

I think my stance is pretty clear about "utilizing" AI in educational settings. We absolutely don't need AI the same way we need air to breathe. But AI could potentially provide some solutions (and create new problems or have adverse effects as well), so why not explore it properly to find out where it works and where it doesn't?

No. Students' usage was anonymized, so the course instructors did not know who used the system in what way. This was to make sure that students could use the tool freely without feeling like the instructors are watching their usage.

LLMs are tools. They're not everything. Yes, they can't sympathize or empathize. But if they can help a student to be more productive and learn at the same time, then I'm all in for designing them properly to be used in such educational contexts... "as an additional tool."

We need both humans and AI. But there are problems with both, so that's why they can hopefully complement each other. Humans might have limited patience, availability, etc. and AI lacks empathy, and can be over-confident.

> Why are we doing this to academia when the better approach would be giving TAs better training in actual teaching?

Sure, that is a fantastic idea and some researchers have explored it.

But, what's wrong with doing exploratory research, in a real-world deployment? In the paper we describe both where CodeAid failed and where students and educators found it useful, in a very honest way.

Doesn't this make AI agents better? Given that human TAs do make mistakes a LOT, or in many cases are just unprepared (e.g. haven't done the programming assignment themselves)

Human TA's have ego, AI doesn't. With proper tools, you should be able to steer an AI agent.

I think both humans and AI agents both have their drawbacks and benefits. That's why the last section of the paper discusses that we even need to teach students (or provide tools) to help them decide where to use AI vs non-AI tools.

The same goes with human TAs that are extensively used in undergrad introductory programming classes. They can also be unreliable in many cases.

1. Provide students with the tools and knowledge to critically verify responses, either coming from an educator or a an AI agent. 2. Build more transparent AI agents that show how reliable they are on different types of queries. Our deployment showed that the Help Fix Code was less reliable, while other features were significantly better.

But totally agree that we should be discussing the ethical implications much more.

I'm the lead author of this paper. Feel free to ask me anything! - MK

CodeStruct is a structured editor that includes:

→ a context-aware toolbox of explorable tutorials

→ allows point-and-click + char-by-char code insertions

→ automatically handles syntax

→ provides immediate feedback + suggestions while coding

→ provides visual aids: holes + scope

We also evaluated CodeStruct in an 11-session study with 26 high school students. Some of the results:

→ Transitioning to CodeStruct reduced syntax errors (4.35x) and help requests (4.63x) when compared to transitioning directly to Python.

→ Interestingly, learners that used CodeStruct performed equally well after the scaffolds were removed.

Learn more here: https://dl.acm.org/doi/10.1145/3501712.3529733

This editor is amazing!! You should take a look at CodeStruct as well: https://code-struct.vercel.app/

We've had a similar goal and were trying to create a tool to help students transition from Scratch to Python using an intermediary structured editor that provides scaffolds such as handling syntax.

Here's a research paper that we published on CodeStruct: https://dl.acm.org/doi/10.1145/3501712.3529733