← Back to context

Comment by pimeys

14 hours ago

Speed and price. You should try the 4.1 Flash on one of the fast providers. It's pretty crazy how fast this model works, and it's great for iterative software development because you can steer it fast to where you want to go.

https://www.lithosai.com/

https://www.coralbricks.ai/

8 comments

pimeys

Reply
swiftcoder  14 hours ago

How are these high-speed providers close to matching the API rates from deepseek?

  • pimeys  14 hours ago

    Seems that V4.1 Flash is quite cheap to serve. Especially how it implements the cache, meaning the cache pricing can be very low depending on how you implement it in your inference engine.

    Depends on your use case. But if you do software development with a harness such as oh-my-pi, your cache rates are probably somewhere above 98%. Coralbricks has free cache reads, meaning it's much cheaper than the Deepseek API rates.

    Lithosai is more expensive especially in the ultra tier. It's if you want extremely snappy and fast responses, which is important e.g. for a personal assistant where every second counts.

    If you generate a lot of output, you're better off with the Deepseek API rates.

andrepd  14 hours ago

Where does one even get decent ZDR provider? I'm wary of sending sensitive data but not at a scale where I would build my own machine.

  • therealpygon  11 hours ago

    You just search around, or heck, just ask Google for inference providers that are ZDR and whatever else you want. Just make sure to read their terms/privacy and review the security center. You will have to request access if you are wanting to get into the deeper details.

    Outside of major players like Amazon/Google which have compatible offerings available, I believe together.ai, fireworks.ai, and deepinfra.com are, as best as I can tell, US-based (if that matters in your situation) SOC2/HIPPA/ZDR. Each of them offer serverless model hosting with caching, which is going to be the cheapest overall. Most of them also offer private AI deployment which is more expensive but your entirely own instance. Some of them more heavily quantize their models (deepinfra) so take that into your workload considerations. I included deepinfra because they tend to have some the new models sooner than others or have models the others don’t (like mimo). IMO, together > fireworks > deepinfra.

    However, you’ll have to keep in mind that for true ZDR you give up caching which increases your inference costs, so it depends on how “Zero” you need the zero to be and on what timeframe. If you just don’t want your data being trained on, normal use is fine imo. If you are working on something actually sensitive (like, because of NDA agreements and such), then they might not be the right choice and you may be looking at more expensive offerings or building a machine.

    One note: you have to pay attention to the source of the inference. Many (similar to open router) offer models that are pass-throughs to a providers API, but they are pretty clearly marked. Just make sure the model is marked as serverless when evaluating new ones. This is where a proxy comes in handy so you can prevent your LLM from accessing a different models from the specific ones you’ve vetted.