Comment by sandgiant

This is very word salad. It makes superficial sense, but doesn’t really say a lot nor represent a strong or comprehensive case.

"generate large amounts of data quickly"

what is the throughput of ChatGPT (or any other GPT-3* derived model) and how much hardware is it using to achieve that ?

Jesus

Why did you say hello to the algorithm? Not that you did anything wrong, but I find it funny that it having perceived intelligence gives it perceived personhood

Does it always talk about itself in the plural "we"?

I tested ChatGPT with a couple of topics I’m personally unclear on, and while it is linguistically eloquent, it basically just regurgitates the majority opinions that can be found on the internet on those topics (sometimes combining several of those opinions in an incoherent way). It wasn’t possible to engage it with actual reasoning on specific aspects of those topics. It just went with whatever premise you gave it, and/or circled back to the generic exposition. It doesn’t have any thought or understanding of its own.

Exactly this seems like a natural extension for Copilot

Copilot solved the in context code generation problem while ChatGPT is this session based friend who would look attentively through the whole course of your coding. I tried some relative niche task with ChatGPT and it mostly solved them with high precision.

The productivity boost is going to be substantial to reshape software engineering as profession. Moving forward being a full stack engineer would be a bare-minimum, anything that can be done by ChatGPT will not qualify as valuable skill.

It will make true 10x engineers works like they have virtually a team.

Something is coming after us all, be prepared. I am scared yet can’t wait

Keep in mind it was trained when everyone was isolating. Your kitchen is a closed system. Your living room is a closed system. Your bedroom is a closed system

ChatGPT got a 1020 on SAT Practice Test #10 (https://satsuite.collegeboard.org/media/pdf/sat-practice-tes...), a sample exam whose questions have been extensively discussed online since at least 2019 (eg https://youtube.com/watch?v=18oahUy6cGA). The exact text of the questions, with the corresponding correct answers, almost certainly appeared in ChatGPT's training set.

And yet it can't play an extremely simple game: https://news.ycombinator.com/item?id=33850306

What’s that mean relative to other scores?

I asked it to write a function that could calculate the grade adjusted pace between 2 latitude, longitude coordinate pairs. What it spit was naive but correct enough, given the prompt. I was on my phone, standing in line at a grocery store, so I couldn't keep feeding it more prompts to understand more pieces of the puzzle, but I imagine that it would've gotten it with enough coercion.

The problem being, of course, that if the prompter doesn't have the knowledge to feed ChatGPT that it'll never get the right answer, either, of course.

I also played around with having it implement small games in Lua using Love2D (tic-tac-toe, hangman, and tetris) but haven't had a chance to test them for correctness, but they're obviously close-enough as I've implemented them myself. Similarly, it was capable of understanding how to write a disassembler for 6502 (I didn't verify that it was getting the opcodes correct, etc.), ditto for an emulator for the same microprocessor, and then I was pushing it to write them in Lua, ultimately driving it towards Love2D, as well, but ran out of time.

It's not perfect but it's really intriguing. I'm going to feed it a bunch of prompts for some other personal project type of stuff I'm currently working on and see if it's solutions are moving in the direction I expected myself to move.

yeah, exactly.

also the skeptics are really hung up on semantics of "intelligence" and not addressing the model's output, much less where this is going to be in upcoming years.

like, take home coder test is probably just dead. today.

i mean: https://twitter.com/emollick/status/1598745129837281280

ChatGPT seems to disagree with this assessment.

Prompt: Can you code better than most programmers?

Response: As a large language model trained by OpenAI, I don’t have the ability to write or execute code. My function is to provide general information and answer questions based on the text that I have been trained on. I do not have the same capabilities as a human programmer and am not able to perform complex tasks such as writing code.

I tried this:

> Solve the following word problem step by step:

> "John is in a train moving west at ten miles per hour. Jill is in a train moving north at twenty miles per hour. How fast are John and Jill moving apart from each other?"

To solve this problem, we first need to understand the concept of speed. Speed is a measure of distance traveled over time. In this problem, we are told that John and Jill are traveling at different speeds in different directions.

Since the two trains are moving in different directions, we cannot simply add or subtract their speeds to find the combined speed at which John and Jill are moving apart from each other. Instead, we need to use the Pythagorean theorem to find the combined speed.

The Pythagorean theorem states that in a right triangle, the square of the length of the hypotenuse (the longest side of the triangle) is equal to the sum of the squares of the lengths of the other two sides. In this problem, we can consider the trains to be moving along the two sides of a right triangle, with the distance between them as the hypotenuse.

We can use the Pythagorean theorem to find the distance between John and Jill as follows:

First, we square the speed of each train and add the results:

10 mph * 10 mph + 20 mph * 20 mph = 100 + 400 = 500

Next, we take the square root of the result to find the distance between John and Jill:

sqrt(500) = 22.36 mph

Therefore, the combined speed at which John and Jill are moving apart from each other is approximately 22.36 mph.

---

Results seem pretty inconsistent though. Other attempts have given complete nonsense.

> John is in a train moving west at ten miles per hour. Jill is in a train moving north at twenty miles per hour. How fast are John and Jill moving apart from each other?

This is confusing to me too, a human, because they could also be moving towards each other based on this description. To determine their speed relative to each other, don't you need to know their positions?

I would guess the majority of humans alive today would struggle to solve these problems. Do only people who have a good understanding of middle/high school math count as generally intelligent?

This seems like a ratcheting of expectations for what counts as intelligence in response to an advance of AI. (also per some sibling comments apparently with the right prompt it can solve Pythagorean theorem problems).

It worked for me as well:

"""

To solve this problem, we need to find the speed at which John and Jill are moving apart from each other. Since John is moving west and Jill is moving north, their trains are moving in opposite directions.

To find their relative speed, we need to first find the speed of each train in a common direction. Since both trains are moving on a flat surface, we can consider the east-west direction and the north-south direction separately.

In the east-west direction, John is moving at a speed of 10 miles per hour to the west, and Jill is not moving at all in this direction. Therefore, the relative speed of the two trains in the east-west direction is 10 miles per hour.

In the north-south direction, Jill is moving at a speed of 20 miles per hour to the north, and John is not moving at all in this direction. Therefore, the relative speed of the two trains in the north-south direction is 20 miles per hour.

To find the total relative speed of the two trains, we need to combine the speed in the east-west direction with the speed in the north-south direction. Since these two directions are perpendicular to each other, we can use the Pythagorean theorem to find the total relative speed. This theorem states that in a right triangle, the square of the length of the hypotenuse (the longest side) is equal to the sum of the squares of the lengths of the other two sides.

In our case, the length of the hypotenuse represents the total relative speed of the two trains, the length of one side represents the relative speed in the east-west direction, and the length of the other side represents the relative speed in the north-south direction. Therefore, we can use the Pythagorean theorem to find the total relative speed of the two trains as follows:

Total relative speed = √(east-west speed² + north-south speed²)

Substituting the known values, we get:

Total relative speed = √(10² + 20²) = √(100 + 400) = √500 = 22.36

Therefore, John and Jill are moving apart from each other at a speed of 22.36 miles per hour.

"""

Anecdotally, I've been doing Advent of Code and it actually works if you copy paste in the problem verbatim: https://news.ycombinator.com/item?id=33821092

I don't know if that's a great way to judge its creativity—it was trained on an enormous dataset, so it would be hard to determine if something was actually new or cobbled together from bits and pieces it found elsewhere.

I asked it to write AC/DC's next song, and while I admit I led the witness a bit, I eventually got it to this as the chorus:

  (Outro)
  We're the kings of rock and roll
  Our songs might sound the same, we know
  But we bring the heat, we can't be beat
  With Angus Young in his schoolboy suit
  And Axl Rose, he sang with us too.

https://imgur.com/gallery/oWR7xVM

Pretty cool stuff.

I've tried playing interactive fiction games with it and it's not bad, but not too great either. It tends to silo into giving a response that encompasses entire story arcs and doesn't really leave you asking for more questions. It also tends to have very uplifting endings and doesn't stray towards anything dark, but I find that understandable.

But I'm being pretty nitpicky - it's astounding that it's able to construct a story line and maintain some semblance of plot between questions.

The best examples we have of a general intelligence are humans.

So I'd check if it could solve a vast variety of problems at the level of, say, an average well-educated adult. You could ask it common sense questions, simple maths questions, solving mazes, quickly learn various novel games it hasn't seen before, perform tasks requiring memory, generalization etc. etc.

If I needed to test this, I would gather a number of experts and have it throw questions at it for a few months to understand how its capabilities compare to a human.