Comment by anthonypasq

mostly cost of hardware going down. as models scale, nvidia produces a new hardware generation that outputs more tokens per watt, but those speed gains get eaten by the fact that the model is bigger ie. more expensive to serve.

Also we have no clue whether Anthropics inference margin is compressing or not and they just want to maintain the price.