6

I'm seeing an uptick in "answers" that I suspect have been generated by chatGPT (or another similar tool). The most recent example being this answer from a user who signed up in the past week

It's not a flood, yet, but I fear it could become one if we don't have a collective agreement and guidance for handling such answers.

So as to not duplicate effort, StackOverflow has already (temporarily) banned answers generated by chatGPT if you want to read more about the issue.

2
  • Note: the linked answer, in addition to being written by ChatGPT, was hiding linkspam in the body. I've removed it.
    – David Reed Mod
    Feb 15 at 21:20
  • @DavidReed I saw the link, but didn't hover over it. I thought it was just something auto-included in chatGPT's response.
    – Derek F
    Feb 15 at 21:22

2 Answers 2

Reset to default
11

For what it's worth, I'm of a similar opinion to the moderators on SO.

chatGPT and its kin have a tendency to generate poor quality answers that can appear correct but are actually misleading or flat out wrong. It would take a large amount of valuable, expert time and effort to weed through such answers to find and address mistakes.

I believe that we should have a total ban on chatGPT answers until further notice (i.e. no "temporary" designation, this would last until we explicitly decide to allow them again).

To allow the system to be able to take at least part of this workload, I propose marking "blatant" chatGPT answers as spam. If you're not sure if it is chatGPT or not, then flag as low quality or flag for moderator intervention.

3
  • I would prefer not to use the spam flag for chatGPT answers - it comes with a bunch of side effects. How about using a flag for mod intervention for these?
    – David Reed Mod
    Feb 15 at 21:20
  • @DavidReed My concern with flagging for mod intervention was that it could possibly overload you guys (ya know, if it becomes prevalent enough). Are there other side effects besides answers getting hidden/deleted and the user possibly being burninated? Does too many spam flags reflect poorly on SFSE as a whole?
    – Derek F
    Feb 15 at 21:27
  • I personally don't mind having these flagged to me (haven't talked to Matt and Adrian about it). The challenge with spam flags is that if they're ever marked "helpful" (which can happen semi-automatically, e.g., if we choose to delete a spam-flagged answer for other reasons) it deducts 100 rep from the poster, and aggregates towards limiting their IP's ability to engage with the platform. That's not always a bad thing, but my initial inclination is to use lighter-touch reactions first.
    – David Reed Mod
    Feb 15 at 21:29
7

I see no reason why Stack Overflow and Stack Exchange should have different policies.

Unless there is strong feeling within the community that we should differ in policy, I plan to enforce the policy as written by Stack Overflow.

My personal interpretation of that policy is that answers that are clearly written by ChatGPT will be removed and users who continue to participate in this way may be suspended or subject to other sanctions.

Please flag any answers that bear hallmarks of being written by ChatGPT for moderator review.

4
  • How do you even detect it? As someone who has not used ChatGPT, I'm not really sure what to look for.
    – Adrian Larson Mod
    Feb 16 at 1:48
  • 1
    @AdrianLarson ChatGPT feels like a mixture of different authors all jammed into one headspace. For a short answer, just a few sentences, I'm sure it could fool me at least some of the time. The longer answers start to feel like something right out of the uncanny valley. They feel "wrong," but you can't quite put your finger on it. Kind of like how they described the Matrix in the first movie.
    – sfdcfox
    Feb 17 at 2:19
  • There are also tools that can analyze text and provide strong indication that the text was AI/LLM-generated
    – David Reed Mod
    Feb 17 at 2:37
  • Some examples of such tools are: copyleaks.com/features/ai-content-detector, writer.com/ai-content-detector
    – Jigar Trivedi
    Feb 28 at 13:19

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .