Comment by ksaun
10 days ago
Overall, I've been very pleased, though I have seen inconsistency like you mention.
First, some background: At the start, I used Sonnet, but after a couple weeks, I'd switched to mostly Opus. I've used Fable when available. For any significant new feature or work, I use /brainstorming (superpowers plug-in). For Vestiges, I haven't written any design documentation per se. I'll start off a /brainstorming with a hefty prompt, maybe 5-15 lines. Throughout the brainstorming steps, I'll think of additional ideas/details about the feature, etc. I'll just add the change/addition to my answer to whatever question it had asked and it seems to handle that fine.
Sometimes I catch things in the subsequent design review process and it handles new guidance well there, too. Typically, I've left some details underspecified and Claude fills in the gaps -- often quite well, but I have to pay attention. The details that it decides later in the process (the implementation steps) are often solid, but when they're not, it's easy to iteratively fix later. Most of the time when Claude asks for clarity on a detail, its recommended option is what I wanted.
It's been rare for Claude to do something that just didn't work (but it has happened occasionally). I have found that it's not good at imagining the player's perspective; it doesn't consider the player's cognitive load for UX considerations, for example. That seems to be consistent with your observation about what they can and cannot verify. A recent instance was when I added a tutorial (which was much more scripted than regular gameplay, providing a more specific experience); I ultimately had to get very directly involved to get to the quality I wanted.
Vestiges doesn't have any gameplay involving timing like Pong does. I could imagine that sort of gameplay being harder for it to get right.
(I hope that covers some of what you were seeking!)
>Vestiges doesn't have any gameplay involving timing like Pong does. I could imagine that sort of gameplay being harder for it to get right.
Thanks. Yeah I think that's the main thing. They can look at pictures (although I've had some amusing results with that, too...), so theoretically they could play a game frame by frame if they run it very slowly. That would just very slow (like 10 minutes per 1 second of gameplay) and very expensive.
But basically, they can't "see video" yet, so they're blind in that domain. They can only see static images, not movement.
I expect this to be solved eventually, either with some new architecture (Google is doing a lot of interesting work in this space), or -- and this is kinda funny -- just running them really fast. The labs are experimenting with specialized hardware (e.g. Cerebras) lately, which can do inference at thousands of tokens per second, and several of them have plans to offer it in the next few weeks.
In other words, I think we're going to get around the "time-blindness" issue, in the short term, by just... running them really fast. (Then it would only take 1 minute per second of gameplay, instead of 10...)
Unfortunately, specialized inference hardware massively increases the per-token costs, as well as the tokens per second, so... it's not going to be a practical option for a little while!
--
Re: Superpowers: Thanks for the tip, that sounds very cool.
Re: superpowers/brainstorming: in an attempt to be more helpful, below are a couple examples of prompts I used. (I've been surprised at how infrequently people talking about genAI share their prompt text.)
I've found it's fine to start /brainstorming in a new session; Claude has been usually good at gathering up enough relevant context to directly dive in. I use high or xhigh for brainstorming; sometimes I ask Claudes opinion regarding level of effort (or whether using Fable would matter). In both cases below, I had more details in mind than I included in the prompt text; I try to provide enough context and direction to drive the discussion. I think I started the first one in a new session, with Opus, in parallel to another Opus whose work sparked my interest in this feature. The second prompt was during a Fable session that I felt had been going very well. It was working on a tutorial and I thought continuing the session with it having that additional context might be beneficial.
**
/brainstorming We're currently fixing up the Disrupted Memory panel, which leads into a Signal Lock match. Through that process, we decided we want to let opponents effectively have "items" like the player can. These would be authored specifically by opponent (i.e., Null has <item1> <item2> etc.). For passive items, the opponent would simply get the effect during the match. For Active items, the opponent would need to have their logic/strategy modified so that they can decide when to use it. I'm imagining the Signal Lock screen/UI shows/represents the opponent's abilities in a way that's symmetrical to what we're doing for the player's. It seems possible that we'll need other/new visual effects for when opponent active abilities are activated, to more clearly indicate what's happening. The design objective here is to help the opponents differ more significantly from each other, adding to the sense of them having personality, increasing difficulty, and influencing the player's own strategy. Please help me delve into the specific items for each opponent and then implement all aspects of this feature.
**
/brainstorming A tangent that's related to the tutorial: We want to make Mara more of a "character" in the game. "Mara" being the tool, or the AI agent within the tool. We use Mara more actively than we have been. For example, instead of just static system log messages, Mara "talks" to the player: when first starting the game, Mara says something like "Hi! It's been 2423 days since I was last activated. You could not be Erik, so I assume you are a new analyst interested in accessing digitized memories. Since you are presumably new here, would you like me to give you an overview of how all of this works?" etc. Going this route would involve a few things: 1) In elevating Mara's presence, we'd probably want to define her voice differently, probably give it a little more personality. 2) We'd want to adjust the tutorial as if it is being delivered by Mara. Both using this new personality and maybe adjusting some other details to make it feel more like what the tool itself would want to explain. 3) We'd want to audit the system log and see what messages it contains would be better presented by a Mara message (either instead of or both). 4) We'd want to revist the Ask Mara content of the Console, maybe adding more questions or modifying some of the answers. 5) We'd reposition Signal Lock's place in the fiction. (E.g., "Restoring disrupted memories is very complex and poorly suited for human consumption. I understand that many humans excel at card games, so I've translated the process into a card game.") 6) A writing pass on all of the text that's supposed to be in Mara's voice to better fit the new voice. 7) Probably we'd want other places/opportunities for Mara to say something to the player. 8) We should re-envision the UI a bit -- maybe we stop the map from reaching all the way to the bottom, and repurpose the screen to the right of the system log for Mara (like the Tutorial does). Maybe we want to allow the player to "talk" with Mara, at least in some situations. Like... Mara might ask the player a question and the player can answer and there's reactivity to these choices later. I see two routes we could take: a) fix up the things in the list above (1-7) and let that help us to see how to better leverage Mara and then revisit 8 and iterate; b) Design out 8 and implement this, then figure out how to cascade the implications of 8 through the rest of the game (i.e., 1-7 + more?).