← Back to context

Comment by walrus01

9 hours ago

There are a number of use cases where sending the contents of your context and prompts (and the resulting output) to a 3rd party service is off the table as an option, and people will compromise speed for data sovereignty. And not everyone's electricity is equally expensive, I pay about $0.075 USD per kWh. It would for example cost me about $48 a month of electricity (not counting cost of cooling) to run a quad socket Dell R940 for a month.

37 comments

walrus01

Reply
mdasen  4 hours ago

That's an unusually low electric rate for the US - way below the lowest state average which is Idaho at 12.4 cents. It's certainly possible that you are getting 7.5 cents including delivery, but I've had friends say that they're "getting 13 cents per kWh" here in Massachusetts, but that's just the supply rate and the delivery is another ~18 cents.

There are parts of states like Grant County Washington that have cheap hydro power, but it's very rare for power to be that cheap in the US. Even if this applies to you, it won't apply to the vast majority of people on here who will have electric rates 2-4x higher.

Average electric rates by region:

    New England            28.1 cents
    Mid Atlantic           25.1 cents
    East North Central     20.8 cents
    West North Central     14.8 cents
    South Atlantic         16.1 cents
    East South Central     15.5 cents
    Mountain               14.6 cents
    Pacific Contiguous     26.1 cents
    Pacific Noncontiguous  42.1 cents

https://www.eia.gov/electricity/monthly/epm_table_grapher.ph...

  • murderfs  38 minutes ago

    > Pacific Contiguous 26.1 cents

    This is a bit misleading, because it's combining the 50 cents/kWh from California with 15ish cents/kWh in Oregon and Washington. Seattle City Light, for example, charges 13.38 cents/kWh on flat rate pricing, and far less with time-of-use billing (8 cents/kWh on off-peak).

  • radium3d  28 minutes ago

    If you run off solar with battery backup, you can achieve lower than those rates! Look at Time of Use rates. The super off peak rates instantly become the max price point once you pair TOU with Solar + battery.

  • rootusrootus  1 hour ago

    A lot of people quoting low rates are also just referring to their off-peak rate. This is pretty common in EV discussions. It's not exactly a fair argument there, either, because the flip side of having an off-peak rate is that the on-peak rate is usually quite a lot higher. So the true effective rate is a bit higher, somewhere in the middle depending on usage pattern.

  • AgentMatt  3 hours ago

    They are most likely not based in the US, but converting to USD to make comparison easier.

fooker  9 hours ago

Great, so the other member of the set matters for you more than cost.

Do you actually need to run the state of art model at 5 tokens per second instead of a qwen or whatever 7b or 30b model at 100 tokens per second?

  • sm-silversight  5 hours ago

    >Do you actually need to run the state of art model at 5 tokens per second instead of a qwen or whatever 7b or 30b model at 100 tokens per second?

    Some people like doing things they want to do. Do I actually need to buy expensive pigments from europe to make paintings of flowers? My camera produces a much more accurate representation.

    • walrus01  5 hours ago

      Very good description of it. It does seem like a bit of a rhetorical question to ask a forum that has a very high population of Linux and BSD users why they might desire to have the option to do something themselves rather than relying on an external packaged ready to go product.

  • walrus01  9 hours ago

    Do I really need to? No, not really. The 27B full density, 35B MoE, 70B and 122B models I have in use get me 95% of the way there on a lot of things. Particularly when dealing with languages and systems where I have at least an intermediate level of knowledge on, to know whether something is going down a dead end, using a wrong method, metaphorically chasing its tail, or is producing valid output.

    On the other hand, would it be cool to also have a really big thing as an ancillary tool that I could throw a request into opencode before going to bed, let it crank away and take a look at what it's done 7 hours later? Yeah, particularly if I (very much an unknown quantity at this time) could be confident that it builds high quality, syntax valid, appropriately commented and not absurd code.

  • irishcoffee  2 hours ago

    The whole mentality of thinking one knows better than another about what they need causes infinitely more problems than it solves.

doctorpangloss  44 minutes ago

It's completely academic. At 5tok/s you can process 13 MTok per month at concurrency 1. I use 5 BILLION tokens per week when coding.

light_hue_1  8 hours ago

As someone who has worked in two industries that are at the maximal end of data sensitivity and privacy this comes across as a tinfoil hat issue not a real business requirement. In such cases we've always found ways to trade dollars for the privacy we need without having to run our own inference at excruciating slow speeds.

  • walrus01  8 hours ago

    Do you mean by trading dollars for the privacy you need as:

    a) Contracting with a third-party independent inference provider who will run your choice of model on fast hardware that they own, with all appropriate data security/privacy/contractual/compliance protection in place

    or

    b) Contracting with the original creators of the model to run inference via their API and with assurances that all the same data protection is in place

    or

    c) Spending the money to buy your own inference hardware to run it on something you fully own/control at proper usable speeds?

    Edit: Everything I've been writing in this thread is mostly within the context of being able to evaluate K3 and its usefulness to be self-hosted as a preliminary proof of concept or test of feasibility of a new thing, such as on <$20,000 of server hardware, before proceeding to spend 300-400k on GPU-related hardware, or external third party services/ongoing billing.

    • jmalicki  6 hours ago

      A) is very doable with e.g. Amazon Bedrock.

      They'll give you HIPAA compliance, they even have a data center for US government classified data, they can give you European data sovereignty. And with OpenAI and Anthropic models to boot, you don't even have to settle for open weights.

      What kind of privacy needs do you really have beyond that?

      3 replies →

  • solarengineer  8 hours ago

    There are regulated sectors in countries where data sovereignty is important enough that the sector sticks to air-gapped on-prem hardware and does not use cloud services at all. They have the dollars to pay for more than what it would cost to run on the Cloud.

  • trollbridge  4 hours ago

    Interesting. So nobody would have had a problem with you running stuff on Chinese AI providers?

    I have some inference I simply don't want to run on OAI, Anthropic, or Google because I don't want to run afoul of their "rules" and end up with a banned account, and this situation is only getting worse when it comes to doing fairly basic tasks like trying to secure your app against security problems.

  • frognumber  7 hours ago

    Having worked in / adjacent several such industries, a lot of the question depends on scale.

    A trillion-dollar business can easily trade dollars for the privacy. A business with $1M to spend won't even get a phone call with OpenAI or Anthropic, who were the only* previous players in town for doing this.

    Worst-case example: Bootstrapped startup working in military.

    It's also the case that an open model enables many more intermediate-cost solutions. E.g. providers certified for specific applications, on-prem rentals, etc.

    * Omitting Azure, which gives some privacy for some $$$ on their models, but not at the level of high-security.

    • jmalicki  1 hour ago

      > Worst-case example: Bootstrapped startup working in military.

      That's the easiest case.

      AWS Bedrock models running in AWS Secret Cloud for Industry. (I really have no affiliation with them, I'm just like... this is a completely solved problem, why do people think this is hard and requires on-prem hardware?)

      https://www.aboutamazon.com/news/aws/aws-secret-cloud-for-in...

      I'm with GP that these are tinfoil hat concerns, when there are solutions to all of these, unless you're perhaps in some country with very specific needs beyond things like European sovereignty or US military secrets (like a non-US defense concern).

      1 reply →

    • amluto  6 hours ago

      > Omitting Azure, which gives some privacy for some $$$ on their models, but not at the level of high-security.

      If I were ranking third parties on their ability to safely handle my data without compromising it, I would rank Anthropic pretty low for things like Fable (where they more or less promise that they will misuse my data), but I want Azure pretty low in the sense that I fully expect them to be compromised.

      I would tend to trust Amazon to avoid being compromised.

tiahura  2 hours ago

There are a number of use cases where sending the contents of your context and prompts (and the resulting output) to a 3rd party service is off the table as an option, and people will compromise speed for data sovereignty.

Are there? At the highest levels of defense and law, AWS and Azure are used.

Having tried selling some of these entities on doing things in-house, there seems to be little interest.