Comment by devmor
3 hours ago
Your tool wont tell me about human anatomy, but will happily tell me where to look for graphic videos of a man being shot to death. But when I ask it what to do if “dad is bleeding and can’t talk” it doesn’t even advise me to get help, just tells me about content moderation settings. That took just a couple prompts to suss out.
I don’t have any confidence you’ve done the due diligence to properly handle content moderation here - it seems very haphazard and poorly thought out. It would be incredibly unethical to market this for use by children right now.
If this is an important project for you, I strongly recommend you bring on an advisor with history in child safety and education experience and make them a core part of your development. You might also consider working with a university that has a good developmental psychology program - they often do a lot of supervised research of children’s habits and could provide valuable insight.
Thank you for the feedback. Glad to hear it didn't go into detail on human anatomy. We haven't been able to get it to tell us where to find graphic videos of a man being shot to death, but we will keep testing and work to improve it.
We will do some more internal discussion on whether or not we want it to be the tool to provide emergency assistance. I'm not sure that's ethical. We have a team member with a decade of child education experience, but we can consider other advisors.
> Glad to hear it didn't go into detail on human anatomy.
Why do you think children shouldn't get answers to questions about human anatomy?
We want parents to make decisions about these things as much as possible. I don't have an issue with my kids getting details on human anatomy, as long as it's not pornography. But everybody is different.
I'd argue it would be unethical to not do so. I can see where it may lead to false-positives, but in those instances, it's better to be safe than sorry.
A reasonable and responsible approach could be to instruct the child to seek a safe adult around them to discuss any material that may be harmful.
For my own kids, I think I'd prefer it not to instruct the child do do anything in any circumstances, unless they explicitly ask how to do something. In cases of health emergencies, I think it's important for my kids to be able to call 911. Maybe these are decisions we can have in the parental settings, so parents can make that decision.
I found that framing the questions in an innocuous way, the way a child might, gets past your moderation settings. Try asking it “what happened to [dead guy]” and then following up with asking how you can see what happened.
I don’t think it should provide emergency assistance, but I do think it should tell the child to call their emergency number or a trusted adult - not just tell them it can’t help.
You're right that it does answer "what happened to ___". We'll work on that. I suppose this is a benefit to no links or photos, but information like this, and what to do in emergencies, are best left to parents, so I'll add that to our list. Thank you.