Comment by hntester123

13 years ago

>You already have the data I'd use: the text of the comments.

Wouldn't that require real AI though? I thought for a minute that NLP (Natural Language Processing, not the other meaning(s) of the acronym) might help, but then thought that it may not work for cases where the comment is quoting another comment. Note: I'm not at all an expert in any of those fields, just interested.

7 comments

hntester123

Reply
adimitrov  13 years ago

Sounds like a job for Sentiment Analysis [1]. Modern systems are pretty good at discerning negative from positive comments.

You could probably find a way to mark negative and positive comments. Whether the resulting algorithm would be fine-grained enough to semi-reliably mark 'middlebrow dismissal,' I really don't know. Actually, as somebody who has worked on that stuff in the past, I don't think it would be very easy.

[1] http://en.wikipedia.org/wiki/Sentiment_analysis

  • pmelendez  13 years ago

    Thanks for the link.

    Agreed I don't think it would be an easy task, but I wonder how would perform a "bag of word" approach.

    Harder part as I see it would be to categorize the comments on middlebrow dismissal / Not dismissal. It seems like we would be spending more time preparing the data than in the algorithm itself.

    • disgruntledphd2  13 years ago

      Welcome to statistics/data science/machine learning.

      The hardest part is always getting the data into usable form. Its not as much fun as fitting models, but its definitely the majority of any role where people pay you to do this kind of stuff.

      There's a lot of good research on forums (pm me if you want a bibliography i collected for a previous role), and short texts have become a bigger deal post Twitter. I completely agree with pg on the somewhat annoying nature of comments such as the GP.

      2 replies →