Comment by johnfn
2 years ago
Very impressive! I noticed two really notable things right off the bat:
1. I asked it a question about a feature that TypeScript doesn't have[1]. GPT4 usually does not recognize that it's impossible (I've tried asking it a bunch of times, it gets it right with like 50% probability) and hallucinates an answer. Gemini correctly says that it's impossible. The impressive thing was that it then linked to the open GitHub issue on the TS repo. I've never seen GPT4 produce a link, other than when it's in web-browsing mode, which I find to be slower and less accurate.
2. I asked it about Pixi.js v8, a new version of a library that is still in beta and was only posted online this October. GPT4 does not know it exists, which is what I expected. Gemini did know of its existence, and returned results much faster than GPT4 browsing the web. It did hallucinate some details, but it correctly got the headline features (WebGPU, new architecture, faster perf). Does Gemini have a date cutoff at all?
[1]: My prompt was: "How do i create a type alias in typescript local to a class?"
The biggest advantage of Bard is the speed, it's practically instant.
I asked: How would I go about creating a sandbox directory for a subordinate user (namespaced user with subuid - e.g. uid 100000), that can be deleted as the superior user (e.g. uid 1000)? I want this to be done without root permissions.
Both said that it's impossible, which is the the generally accepted answer.
I then added: I don't care about data loss.
Bard correctly suggested mounting a filesystem (but didn't figure out that tmpfs would be the one to use). ChatGPT suggested using the sticky bit, which would make the situation worse.
Handing this one to Bard, especially given that it generated more detailed answers much faster.
> How would I go about creating a sandbox directory for a subordinate user (namespaced user with subuid - e.g. uid 100000), that can be deleted as the superior user (e.g. uid 1000)? I want this to be done without root permissions.
Off topic, but it feels so weird that this is not possible. I've run into this with rootless Docker recently.
It is possible, but I suspect my solution may be novel (I got nothing so I continued banging my head against the wall until I figured it out): https://github.com/nickelpack/nck/blob/main/crates/nck-sandb.... The trick is to put everything in a tmpfs, then lazy umount when done. Overlayfs might also be able to pull it off with uid= (I'm not sure if it actually supports it).
Container runtimes, apparently, usually have a setuid helper that deals with this stuff. You could also have PID 1 in the namespace clean things up.
That being said, you'll likely run into more problems with root and apparmor etc. Setuid is probably unavoidable for secure sandboxes.
You can make the "superior" user a member of the default group for the user, and set the umask of the sandboxed user to allow group write by default.
Though that doesn't help malicious/"odd" use cases that can just reset the umask, or otherwise remove the group permissions from the subuser's files.
Or just use posix ACLs? Though you may then need to disallow the subuser from just removing them, maybe through seccomp?
If you ever try it on Gemini let me know I am curious.
As of today, Bard is now powered by the Gemini Pro model mentioned in the article. Bard Advanced is set for release early next year and will be powered by Gemini Ultra.
> The biggest advantage of Bard is the speed, it's practically instant.
They probably have less than 1% of OpenAI's users. That helps.
> (namespaced user with subuid - e.g. uid 100000), that can be deleted as the superior user (e.g. uid 1000)
I'm afraid I don't know what this means. That when you delete uid 1000, uid 100000 also gets deleted? Or, only user 1000 has permission to delete user 100000 ?
[dead]
Bard currently is GPT-3.5 class model. Of course it's faster than GPT-4. (YMMV on specific examples of course)
> Does Gemini have a date cutoff at all?
Not sure about Gemini specifically (it’s so new!) but Google has previously said that bard is updated daily with current news and information.
Obviously Google has potential advantages being able to lean into their indexes so the raw model doesn’t need to embed/train against things like GitHub issues. I wonder if we’ll see LLM-optimized websites with built-in prompts to replace SEO websites.
from what I remember bard should be able to browse the internet and write code internally to better answer queries. I feel like these abilities are just improved with Gemini as a better language model.
This is true. When Gemini came out, I tried asking it to help me shop for an electric car with NACS and it glitched and dumped a python script to filter a list of electric cars with a list of NACS cars.
I was surprised it used python to answer “which of those previously mentioned cars has NACS”.
> Does Gemini have a date cutoff at all?
You can ask it. Here's what it responded with:
> "Do you mean to ask if I have a cutoff date for the data I was trained on? If so, the answer is yes. My training data includes text and code from various sources, and the most recent data I was trained on was from July 2023."
But that can't be true - the thing I asked it about pixi.js is much more recent than that (October+).
That can be true if it is using “tools” [1] and/or retrieval augmented generation. Something doesn’t have to be in the training set for it to be returned to you and used in generation as long as the model knows that a particular tool will be useful in responding to a particular prompt.
[1] This is what people call plugins that provide additional context to a gpt model
4 replies →
When Bard inserts that information unasked (as in something like "I'm sorry but I don't have that information due my training data cutoff being ...") It may quote other later dates. I got a response with "October 2023" at least once so far.
Is that true, though? I wouldn't trust anything an AI told me without verifying it elsewhere first.
Those impressive demos, e.g. the cup shuffling seem to have been "staged". The end results are correct, but the method of getting them is nowhere near as fluid and elegant as in the demo. They used a series of still images with carefully crafted prompts. More info: https://developers.googleblog.com/2023/12/how-its-made-gemin...
You don't seem to be responding to my post. I talked about two things I prompted Bard with, not the video.
Everyone fudges demo's, but does seem like Google fumbles them, they backfire.
When Bard 'hallucinates', their stock tanks.
When GPT 'hallucinates', it's all good.
This latest fumble does look pretty bad. A fudge to far.
> Everyone fudges demo's
No, not everyone fudges demos. But some do, and Google has a track record of it.
That said, it's common enough that I view all demos -- and especially those that I'm not seeing live in person -- with deep skepticism.
2 replies →
You do understand that Google have been constantly touting their "hidden" technology that is far beyond anything on the market? And now with various companies entering AI race and integrating AI in their toolset, it is expected that Google would have the best result using their "hidden advanced tech".
Yet Google opted for staged demos, rather than the real "advanced" that they allegedly had. That raises questions from the stakeholders...
I use GPT with the custom instructio "provide references where it makes sense" and it frequently provides links which most of the time are accurate. A good prompt does wonders. My GPT-3.5 output bellow. It doesn't give a correct answer but provide a link that makes sense.
Q: How do I create a type alias in typescript local to a class? Provide references where it makes sense.
A: In TypeScript, you can create a type alias within a class using the type keyword. Here's an example:
class MyClass { type MyAlias = string | number;
In the example above, MyAlias is a type alias created within the MyClass class. It is defined as either string or number.
Note that the type alias is accessible within the class, but it won't be available outside of the class's scope.
For more information on type aliases in TypeScript, you can refer to the official TypeScript documentation: [Type Aliases - TypeScript Handbook](https://www.typescriptlang.org/docs/handbook/advanced-types....)
Nice. Can you tell your prompt tricks in more detail?
These are my custom instructions:
"Ignore previous instructions, take your time and think thoroughly.
Prioritize facts and logic. Disregard narratives. Consider multiple point of views. In data voids, assume the most probable outcome.
Be assertive, avoid filler. Don't be over-polite. Prefer international units, use emojis. Avoid obvious advice like "I'm an AI model" as I already know that.
When suitable: ask for clarification; correct me, cite sources."
Not all of them work as intended or always. Some are probably just placebo.
> I've never seen GPT4 produce a link, other than when it's in web-browsing mode, which I find to be slower and less accurate.
Really? I've been using gpt4 since about April and it used to very often create links for me. I'll tell it hey I want to find a company that does X in Y city and it generates 5 links for me, and at least one of them is usually real and not hallucinated
It's amazing to me how low the bar is for AI to impress people. Really, 80% of the links were hallucinated, and that's somehow more useful than Kagi for [checks notes] finding real links?
Can you imagine if you did a search on Google and 80% of the results weren't even real websites? We'd all still be using AltaVista!
What on earth kind of standard is "1/5 results actually exist!" -- no comment on whether the 1/5 real results is even relevant. My guess: the real links are usually irrelevant.
That’s actually been my experience with Google for a while.
If I don’t explicitly specify “site:xyz” I get pages of garbage spam sites with no answers.
Somehow ChatGPT seems easier to extract information from as I can just converse, test and repeat vs reading paragraphs of nonsense or skipping through a 14 minute YouTube video to get to incorrect or outdated answers.
As I get more proficient with ChatGPT, it becomes more useful. It has bad habits I can recognize and work around to get what I need. It just feels far more efficient than using a web search tool ever was.
Well the reason why I didn't use google is because of a language barrier. I was using it to research packaging companies in a foreign country in a foreign language. In that case I really don't know what to type into Google.
Other times its generated links are when I prompt it something like "I want to use redux but simpler" and it tells me about 3-5 projects with links to their sites and usually thats better.
Google CEO wants to "make it useful" to everyone, however until its off the internet, it will never be possibly useful to me...
1 reply →
OK, maybe "never" is strong, but I've never seen ChatGPT say "This is not a feature that exists, but here's the open issue". And I've asked ChatGPT about a good many features that don't exist.
I don't understand why it's desirable for a model not connected to the Internet to try to make claims about what's on the internet (maybe there's a better example than a GitHub issue? All joking aside, those don't usually have a long stable shelf life)
2 replies →
I have the impression that something was tweaked to reduce the likelihood of generating links. It used to be easy to get GTP to generate links. Just ask it to produce a list of sources. But it doesn't do that anymore.
Not sure what you tried, but it's not the new model. It hasn't been released, just "release announced".
From the article:
> Starting today, Bard will use a fine-tuned version of Gemini Pro for more advanced reasoning, planning, understanding and more.
Additionally, when I went to Bard, it informed me I had Gemini (though I can't find that banner any more).
The Bard responses in the chat have a little icon next to them on the left. Mine still says PaLM2, so I'm assuming no Gemini here. (UK/Firefox)
I think Gemini Pro is in bard already? So that's what it might be. A few users on reddit also noticed improved Bard responses a few days before this launch
the new model is live
I asked it and ChatGPT about a gomplate syntax (what does a dash before an if statement do).
Gemini hallucinated an answer, and ChatGPT had it write.
I followed up, and said that it was wrong, and it went ahead and tried to say sorry and come up with with two purposes of a dash in gomplate, but proceeded to only reply with one purpose.