← Back to context

Comment by ciberado

11 hours ago

DS being unable to precisely view Playwright screenshots is the only thing I really miss from Sonnet. This is promising.

> Images are converted into tokens based on their dimensions, and these tokens are billed together with your text tokens.

> Before inference, every image is automatically resized:

> - Images with a total pixel count below roughly 384×384 are scaled up while preserving their aspect ratio.

> - Larger images are scaled down while preserving their aspect ratio so that the total pixel count after resizing is roughly that of an 800×800 image.

> As a result, there is an upper bound of 384 tokens per image: for example, a 2000×2000 image and a 5000×5000 image consume the same number of tokens after resizing. When a request contains multiple images, each image is counted independently under the same rule—there is no separate calculation for multi-image requests.

400 tokens per image results in 2,500 images per dollar, if I’m not mistaken.

edit: format.

29 comments

ciberado

Reply
knollimar  11 hours ago

Oof 800 by 800 kills a lot of use cases

  • johndough  11 hours ago

    Might still be fine. The most recent crop of vLLMs proactively use whichever programs are available on the system (e.g. ImageMagick or PIL) to "zoom in" by cropping subimages if they can't quite make out the details.

  • wongarsu  11 hours ago

    For most use cases you can fix that in the harness. Just give the model a tool to request a crop of specific coordinates of any image it has in its context. Call the tool "zoom" and it should be intuitive for the model

    Maybe there are some use cases where you need high detail everywhere at once, but for OCR of small text and the like a zoom ability should be sufficient

    • embedding-shape  10 hours ago

      For really dumb models I've also had success automatically cropping it into a grid of N images with the max size, then processing each cell individually, then once all been processed, do one final call with resized image + all other context previously generated per cell. Basically a workaround to the image dimension restrictions without loosing fidelity. Works well with even dumb 7B models.

      Can't remember if I stole this idea from some existing public harness though, can't remember. If someone knows of public harnesses that do this already, please share them :)

      3 replies →

  • stronglikedan  6 hours ago

    I don't know about a lot. Probably more like a few. I take a lot of screenshots for various reasons, and over 800 seems like I could have done a better job framing and cropping.

  • shadyr  11 hours ago

    It might also be due to its experimental status. Wouldn't surprise me if the GA version allows for larger input. Either that or the eventual pro version.