Zed Editor automatically downloads binaries and NPM packages without consent

2 years ago (old.reddit.com)

This is what I hate about vscode, and they at least ask for consent. Some of the stuff vscode needs for golang are (to me) developed by Random Joe on github. It's just a matter of time before it is abused for supply chain attacks.

  • > Some of the stuff vscode needs for golang are (to me) developed by Random Joe on github.

    Pretty much all of it seems to be on golang.org, github.com/google, or are by people working on gopls. The suggestions that there are tons of dependencies from random people is just not true.

    • Comment states some of the tools, not tons of dependencies. Your comment that "pretty much all" is not a contradiction of what the parent comment said.

  • yea I was recently dismayed to find vscode extensions have full network access and there's no way to prevent it. It's been an open issue since 2018 and not on the roadmap.

    https://github.com/microsoft/vscode/issues/52116

    • Is that something people really expect of their IDEs these days?

      In my mind, an IDE needs to be able to read all the files I can read myself, it needs to be able run arbitary tools like compilers and linkers then run the resulting compiled code, it needs a debugger that can attach to running processes and mess with them, it needs to be able to pull from the language's package repository when you ask it to, it needs to talk to your git server when needed, it needs access to your SSH keys to do that. About the only thing the IDE doesn't need is to run as root - and if you're working with Docker it basically needs that too.

      The idea of an IDE with pluggable language support safely running plugins from untrusted sources? I can't imagine how such a thing could even be possible?

      4 replies →

    • The idea of everything on a system having open net access is going to have to go, not just for this reason but also privacy. It’s kind of amazing it’s still the default everywhere.

      Also supply chain attacks are a freight train barreling down the tracks. The gate is down and the crossing bells are dinging but our car is stuck on the tracks.

      Ever since the xz thing almost worked every black hat group on the entire planet is trying their hand at this. The days of software dev as a high trust environment are going to be over fast.

      1 reply →

    • At our org we run code-server in docker with no network access except pre-allowed DNS, and a white listed squid proxy, due to this.

      It works really well for us and prevents potentially bad extensions, python libraries, etc exfilling our code, data, etc.

      1 reply →

  • Go suppport on VSCode was originally done by an intern at Microsoft, and then later there was an agreement with Google, for Google Go team to take over it.

    • No wonder. Of all the languages I code in in VSCode, Golang is the most frustrating to use as I can’t jump to definitions with Cmd + Click the way I can in JS, PHP or Java.

      It there a decent alternative for Golang on the Visual Studio Code marketplace?

      19 replies →

  • Not sure if it was always like that, the C/C++ extension used to download the language server and some tools during startup as well. They now bundle it with the extension, which is a lot better especially in an offline environment.

  • I just want a dumb text editor that doesn't bundle shitton of crap I don't want (like LLMs), doesn't phone home, isn't bloated and slow. I feel that still the only good editor that does this apart from vim+emacs is sublime text.

    • I've been using Sublime more and more lately when I'm not in a JetBrains IDE. I can't feel a difference between it's speed and Zed's on my machines. If I could get Supermaven in it I'd SWITCH completely.

While I share similar concerns, I also want to point out that:

  - Zed is (currently) free
  - still pre 1.0 release
  - being developed quickly by a small group of developers

For those of us who enjoy Zed, we should give appreciation for what they have created.

As someone who's maintains OSS myself, the onslaught of people who can swarm in fast to piss on your hard/long efforts can demoralize you.

So let's be kind in our words (and tone) to these folks.

  • I just don't buy this argument. None of what you've listed gives the right to install binaries without permission. A simple opt-in notification could resolve this but they decided against this for ease of use. Free or not, pre 1.0 or not, small team or not this puts users at risk for a pretty bad attack vector.

    • If DockYard.com had a security incident for a free/beta service, I'm sure you'd want users to show some compassion.

      (It doesn't make it right, all I'm saying is - showing compassion goes a long way with developers while they re-evaluate)

      7 replies →

I don't really see the big deal here. Who wants to approve and configure all of their language servers?

If you open a file for that language, is there ever a time you would deny the download?

I just don't want a huge amount of popups like VSCode.

Also, the binaries are downloaded from their release on github. As long as that is secure I don't see a problem.

  • > I don't really see the big deal here

    Here's an idea: someone sends a dev at some company, or even a freelancer, some code. Code references a module with a malicious npm package (say, with a postinstall script). Dev opens it in zed

    Now, my untrusted code is running on your machine, probably without your knowledge

    • >someone sends a dev at some company, or even a freelancer, some code

      The code itself could be malicious and have backdoors. Really you shouldn't run anything untrusted outside a sandbox or some other form of isolation

    • Why the hell does npm support a postinstall script? There really shouldn't be a need to run arbitrary code provided by the package for something like this.

      6 replies →

    • How is that any different from the VS Code extensions that have one star and are just copies of other extensions… waiting to get high stars and then switch-a-roo? Same goes for browser extensions.

      Unless you’re auditing everything while taking Trusting Trust into account, you’re drawing the line somewhere saying “ok I can’t be bothered past this point verifying”.

      … everyone has a line somewhere on the trust-but-verify spectrum

      1 reply →

  • > I don't really see the big deal here. Who wants to approve and configure all of their language servers?

    everyone

    > If you open a file for that language, is there ever a time you would deny the download?

    every time

    default should be offline imo. want to download something? do it manually and place in folder etc

    • > everyone

      Are you really so self-centered that you can't imagine anyone doesn't care?

      I'll raise my hand and say I personally am happy to trust Zed and am fine with auto-installing whatever binaries they propose.

      I do think it's reasonable to have an option for people with different security postures, but don't assume you speak for everyone.

    • Just migrated from Vim to NeoVim. Downloading and configuring all the various LSPs has genuinely been one of the most frustrating aspects.

      6 replies →

    • a classic HN comment! "everyone is just like me"

      absolutely incorrect, i loved that it does it for me and do not care even in the slightest and in fact this is 1000x preferable to vscode asking me.

      what on earth made you think this was a reasonable reply.

  • Yes, whenever I’m reading untrusted code, I don’t want to be using a language server - most of them execute arbitrary code, and I do not want that.

    • That’s a completely separate concern, it’s not like a new language server is downloaded for each file you open. I don’t know if Zed has a “safe mode” like some other editors, if it doesn’t you should ask for that instead. Unless of course you never open untrusted files in a language you’re familiar with, which would make you extremely peculiar.

      1 reply →

    • Open it in something else then.

      If you use Zed you must have known the language server was running when you tried it, how did you think that was happening?

      2 replies →

  • What if one language server adds a function to use your code for AI training? Are you okay with that as long as it came as a gitthub binary?

    And these modern editors introduce another issue with their modularazied design. For each supported language Vscode installs tons of other crap beside the language server itself. And the language server alone has a quite long list of dependencies

    https://github.com/golang/vscode-go/blob/master/extension/go...

  • GitHub release builds provide no whatsoever guarantee of having been built by GitHub from the corresponding source, if I remember correctly

  • I usually don't use language servers at all. AI auto complete does the same for me faster... So yes, I would like to not download any of them.

    • IntelliJ now comes by default with a local-only AI auto-completer. I noticed that almost always, it "knows" the autocompletion better than the older intellisense.

      However, sometimes (very often) you need to explore the API and just check every available method and check their docs to find which one is appropriate to use.

      So, even though I can see AI replacing a lot of auto-completions, it just can't replace it completely.

      1 reply →

  • > Who wants to approve and configure all of their language servers?

    I think you're asking the wrong question. The correct one would be: "who wants to be asked if they want to approve and configure all of their language servers?"

    It's not what zed does, it's doing it behind your back!

  • It's okay for a browser to download and use anything from any site, maybe, with mature cross origin policies and billions in security work, but the fact it's done without saying anything is just a bug that can be fixed. Fixing clarity is the real win.

    What's really funny is it was found because it was crashing and the user was running another libc. If they're really concerned about 14MB of download, they should add a firewall or something, but they saw it crashing. Finally, all these versions of everything sitting around, nodeJS, glibc, etc, very UNIX, a recipe for small breakages. Though I guess that's just the problem we deal with.

This broke Zed for me and had to go back to Neovim at my workspace. The corporate AV software was going crazy with all these automated downloads and installations. It wasn’t blocking them but just vetting them was taking so long, I just didn’t find it worth using

Zed is my favorite editor, but I'm not going to minimize concerns that people raise simply because I think the editor is stupefyingly awesome overall.

Questions: What control does a user have right now over what gets installed automatically? What are the levers we can pull to get more control? (These levers include configuration options, pushing back on the project, and so on.)

P.S. Not that this is an excuse, but VS Code's security posture (sandboxing, prompting users, etc.) probably didn't happen overnight without user pressure. Who knows the history?

  • Some highlights from the Github issues thread [1]:

    > Ideally you would be able to turn off auto-download but still be able to use a [language server] if it’s already on the users system.

    > There is not a binary choice between "friendly to end users" and "secure". You can have your cake and eat it too. The main thing that should be considered is less of a user facing popup or preference (where opt-in vs. opt-out is an issue, as is noise & friction), but developer facing options. If Zed offered compile time configuration this could be fixed to everybody's satisfaction. Distros could ship a pre-configured package with all the dependencies already provided so that the user experience is just "install and run" while also not having an app that downloads (or attempts to download) binaries behind teh users back. Meanwhile if Zed wants to ship a binary package upstream that defaults to downloading things they can.

    [1]: https://github.com/zed-industries/zed/issues/12589

Zed is supposed to be a lightweigh and fast text editor. That was my hope when trying it. This is not the case. When I was editing some JS or HTML file I noticed that my laptop is quite warm. I checked all processes and there was some node process taking up 100% of one of CPUs. It was some language server running in the background in some non-efficient way. The problem with Zed is that its mission is to be "engineered for performance", while in the background they cut corners and run some heavy unoptimized stuff. I think this is not a right strategy, even cosindering it is still in beta.

  • The language server isn't part of Zed though is it? You should report the issues on their github page.

    • it is not, but as the title says: it all happens in the background, without you being aware what happens.

      The good thing is that this can be turned off with this option: "enable_language_server": false

They could ask during install whether silent installation of LSPs should be done or whether Zed should ask explicitly for every LSP.

With Zed, I have another issue. I don't understand which niche it is trying to fill. The advertising story doesn't convince me. The performance bottlenecks are typically the LSPs after all, not if text is rendered in 10 or 20 ms. Startup time is secondary. Yes, memory usage is a concern. I get that and that's where Zed is miles ahead of VS Code and Jetbrains IDEs. But overall I think:

- If you want easy and free, go VS Code.

- If you want ultimate IDE features and mouse and GUI, go Jetbrains.

- If you want ultimate productivity, follow this beautiful guide: https://lazyvim-ambitious-devs.phillips.codes/

I don't get why having a modal for each tool asking for consent is too hard.

  • Because people want a fast out of the box editing experience. Not clicking yes for every language server.

    But what they should have is a CDN with their own extensions and verified binaries. This way they can ship new versions of extensions without bumping their editor version.

    • > Not clicking yes for every language server.

      How many language servers are we talking about here for the average dev? Three?

      3 replies →

    • >Because people want a fast out of the box editing experience. Not clicking yes for every language server.

      That strikes me as more of a UX problem. Doing a bunch of sketchy things behind the user's back is absolutely not a solution though.

    • The versions should generally match what's specified in the user's package.json. It doesn't make much sense then to have a separate registry.

  • I don’t want that. Popups are one of the reasons I stopped using VSCode. They drive me nuts. Just let me write code.

    • It annoys me a lot as well, though it took me a couple of minutes to turn off the popups.

      Once you've done that, it's similar to emacs for me, everything has to be evoked via a shortcut (or Action Palette which works very well in VS Code). The shortcut to show "help" or "docs" is Cmd+K Cmd+I, by the way - easy to type and remember...

    • The popups in VScode seem explicitly intended to annoy.

      Why is showing the release log the default for so many things? Is the average user really going to read them?

      It makes you wonder what they are doing with all that telemetry

  • [flagged]

    • It's important to note that this isn't a dev response that is meant to directly address the issue in the OP. Someone else just saw that it was a dev comment in a related issue and linked it.

      If you look at the linked thread the quote came from [1], the quote is actually answering a similar issue titled "Why are there nodejs files in my zed install". Judging by the response, the dev had interpreted the issue title as a "what does zed uses nodejs for" and not "why does zed downloads nodejs without informing the user" and answered accordingly.

      There are more relevant links to PRs and comments further down the GitHub thread (the one in the OP) where the zed devs acknowledge that they are still thinking about how to best implement the UX for extensions downloading LSPs and whatnot.

      [1]: https://github.com/zed-industries/zed/issues/7054#issuecomme...

    • That is about the idea of rewriting existing tooling in Rust to get rid of node_modules folders, not about prompting users whether to download a language server or not.

    • "Action would be too difficult / we don't like it" =/= "there is no action available".

      This is just refusing to take responsibility for their decision. "We don't feel like doing it" is the truth, and it would be best to state it plainly. Of course there is no obligation to do otherwise, which makes it strange to play with words.

    • That's nothing to do with having a dialog to ask for each too. That's talking about the amount of work it would be to rewrite all these tools themselves so it was first party Rust code.

    • That's not a Rust issue (not having a prompt to update LSPs). Lapce[1] is also a Rust editor, and it didn't keep downloading JS or stuff without a prompt. You can do what VSCode does, have extensions that ask for update, then update on change (even if using binary is the only solution, which I also doubt). Or if the issue is running LSP, ask if you trust a project folder on project start.

      [1]https://github.com/lapce/lapce

Being it binary or not, it doesn't make any difference.

It's the "modern times" craze about plugins pulled from different unauditable, unknown sources. The fact that it is on GitHub or any other "publicly available" source it is irrelevant.

I keep using vim and Kate and manually install anything I need form my distro (Arch Linux) repos. If it is not there, then, sorry I cannot use it.

  • This is a fallacy though, unless you’re also code reviewing all of the Archb libraries you pull down

    • I think the main add of distro repositories is the repo maintainers sit as a review step between you and the project updates on e.g. GitHub, not that it enables you to better audit the code yourself. I'm not sure it's really all that effective in practical terms though.

      1 reply →

Zed is version 0.1-something, you can't expect them realistically have their own maintained packages at this stage. And these things do happen when you use software at the early stages, just wait for 1.0 and see what happens then.

The security side of free editors and IDEs is not great anywhere today for JS development. Once you start wanting more features and integrations, you start facing an apparent choose-any-2 of security, convenience, and productivity.

I don't think it has to be this way. I think we can have both better compartmentalization and tighter workflow integration without having it becoming a part-time job.

Here is my ongoing attempt at addressing the issue, currently scoped for neovim[0]:

https://github.com/legobeat/l7-devenv

(I did share this to crickets as a Show HN the other day, hope it's on-topic enough to OK to reshare here)

[0]: The same framework should, at least in theory, be extensible to do something similar with Code/VSCodium. While working on this I realized there is some overlap with their Dev Containers and am yet to look into if and how one would run those in a similar fashion and if they could be leveraged to the same end

This was also documented here, nearly 1-year ago:

https://github.com/zed-industries/zed/discussions/6659

Where there is a VSCode theme importer for Zed.

And what it does is silently install a Home Brew package and attempt to execute it on your machine.

Not asking the user for consent for software updates is quite common.

My corpo rejects a lot software, because they do exactly that.

There is a balance between asking too many confirmations and not asking at all. VS code had this feature called "Workspace Trust" or something like that. It was so incredibly annoying. Always asking me for my own repos or repos which are in my org, if I trust the authors. I ended up disabling it completely and it will remain that way. I hope Zed finds a way to strike the balance in a better way than bombarding the user with confirmations, otherwise I'll be completely disabling that too, probably to the detriment of the security of my computer.

I lost my investment capital and profits trading online, they kept on requesting for extra funds before a withdrawal request can be accepted and processed, in the end, I lost all my money. All efforts to reach out to their customer support desk had declined, I found it very hard to move on. God so kind I followed a broadcast that teaches on how scammed victims can recover their fund through the help of Gavin ray a recovery specialist, I contacted his email provided for consultation, I got feedback after some hours and I was asked to provide all legal details concerning my investment, I did exactly what they instructed me to do without delay, to my greatest surprise I was able to recover my money back including my profit which my capital generated. I said I will not hold this to myself but share it to the public so that all scammed victims can get their funds back. Contact his email:gavinray78@gmail.com or whatsapp +1 352 322 2096

This might be a very very silly question so bear with me, why would it need to download these binaries? I'm on sublime text atm, and I can't think of a reason why it would download anything other than the app itself or an update to the app when I'm asked. I know that might sound very stupid and I'm sorry.

Hey, Antonio here. Co-founder at Zed.

Sorry that we haven't replied to that GitHub issue yet. We try our best to listen to the community (here, on GitHub, on Discord, ...), but we're a small team and, admittedly, it's tricky to keep up with everything.

I agree that we should ask users for consent before downloading language servers (and other executables).

For everybody who's come across the ticket here or on Reddit and hasn't worked with the Zed codebase yet, let me provide some context on how language support is implemented.

In Zed, we have three ways of supporting a language (and its language servers):

1. Extensions that users can install from the `zed-extensions` repository [0]

2. Pre-bundled extensions that ship with the Zed binary, but still need to be installed [1]

3. Built-in language support [2].

For (2) and (3), the code is owned by the Zed team and we make a conscious effort to review contributions from the community in that area.

That code can automatically download language servers, but we try to vet which exact scripts/binaries are downloaded from where. For example: we heavily use rust-analyzer ourselves and keep up to date with its releases, the Go language server `gopls` is downloaded from the Go team using the official `go` tooling, the ESLint language server comes from Microsoft, etc.

For the longest time, we only had built-in language support (3). A couple of months ago, we shipped extensions for Zed (point 1 and 2 above, parts of it described in [3]). The goal was for built-in language support (3) to gradually move to pre-bundled extensions (2) so that users had the ability to choose which ones to install. We did make some progress, but we haven't ported all languages yet.

We're a small team and can only do so many things at once. So after investing quite a bit of time into extensions, we chose to pause that work and invest into other areas for a while (porting Zed to Linux, for example). Once those areas are in a better state, we plan to come back to extensions, build them out some more, and port the remaining languages.

So, TL;DR: we hear you loud and clear. We try to vet things that are currently installed automatically. But we agree that we should ask users whether they want to install arbitrary binaries on their computer. We also plan to transition all language support to manually-installed extensions once we finish other projects.

[0]: https://github.com/zed-industries/extensions

[1]: https://github.com/zed-industries/zed/tree/main/extensions

[2]: https://github.com/zed-industries/zed/tree/main/crates/langu...

[3]: https://zed.dev/blog/language-extensions-part-1

  • Thanks for this response. Zed is awesome, haven't been able to switch to it full-time due to [1] https://github.com/zed-industries/zed/issues/5065, but do appreciate what's being built.

    It's a new product, which is clearly seeing quick changes every week, so hopeful you'll get to this one soon. The internet will always be extreme around any issue, and make it seem like the end of the world, to those folks, maybe try zed again later? It's still a good editor to keep in mind.

    Although one minor thing about this, getting users accustomed to this flow and then later asking for consent might also raise issues, like this one, just with a more "Zed now prompting for every little thing" in future. So might want to keep that in mind.

    Loving zed for go development (especially with a decent suite of Vim bindings), just haven't been able to use things like Flutter , due to lack of debug support, which VSCode does quite well (albeit crashes a bunch).

  • I think some middle ground might be including an extension lock file that gets committed to the repo. For internal projects, users will get auto configured by trusting the other internal users that setup the repo.

    For external projects, users need to trust the project they're pulling down anyway since it's arbitrary code.

    That would also help mitigate the risk of supply chain attacks (since versions are pinned and ideally verify package integrity)

  • Hi Antonio, thanks for Zed, love it.

    If I may offer a small piece of feedback based on your comment here:

    Apart from its speed, the single thing I loved about Zed and made it stand out from the likes of vscode, was the built-in language support.

    This made for a great first time experience where I didn't have to search and download anything and everything was working out of the box.

    In fact it was so good an experience, that when I had to download another thing for terraform support, it struck me as much more annoying than usually

    Reading that your plan is to move in the exact opposite direction is disappointing and, I'm sorry to say, I hope it doesn't come to pass.

  • I've seen this guy on the Zed YouTube channel, and yeah, dude loves to talk for hours without saying anything useful.

  • this answer makes things even worse. "we are a small team so its ok for us to expose your pc to risks".

    If you dont have the bandwidth to do things securely dont do them at all.you are asking the devs to wait until you have time to return to it to plug the holes you have opened because you needed a bulletpoint for your release.

    and this is not just a lack of time, its your attitude in regard to the devs, see also the optout telemetry for another example of total lack of respect for privacy.

[flagged]

  • The language used to report the issue is very reasonable. Maybe it's cultural, but the fake version you're suggesting is something I'd find insulting, in addition to upsetting me because it's wasting my time. I prefer it if they get to the point.

    > And people wonder why foss devs burn out.

    These are full-time devs, working for an investor-backed company that plans to make money off the editor. The FOSS part allows them to profit off the work of volunteers.

  • Zed is OSS but not free. There's a company behind it, not volunteers. They are doing quite some marketing lately. I don't see anything wrong with calling the current issue "completely unacceptable". Forced opt-in is what happens if the language is not offensive.

    • Interesting. Do paying customers get a different bug tracker from the open source repos or do they all get directed to the same place? If people are paying for this editor then the tone may make a lot more sense.

      I can't find anything about a commercial offering so I don't know what the non-free version entails.

      1 reply →

  • I agree with you, it's a standalone package. It can be assumed to have some setup permissions. Also it's far better than the packaging the remote code with the installer or binary.

    The same people who will complain about this do t really understand how package managers work also. Take npm as an example, you manually install one package. You do not consent explicitly to have all of its dependencies added also.

    This sounds like typical Reddit behaviour.

    As you said, a better approach would have been to ask the maintainers to mention it in the readme. No drama required.

    • I don't use NPM, but that means NPM's behavior isn't that great and maybe shouldn't be an example for others to follow.

      Linux package managers with which I'm familiar will absolutely prompt you with the list of dependencies they'll install when you ask for some package and give you the possibility of bailing out.

      2 replies →

  • Maybe make a cve out of it since it is an obvious exploit path running unchecked automatically downloaded binaries without user interference.

  • > Instead they go on calling it "completely unacceptable " repeatedly, using language that implies that the devs have caused grave offense.

    Downloading and executing untrusted code is a security vulnerability. If a library does so accidentally, avoiding such an accident should be the primary focus of the report. If a library does so intentionally due to an accidental error in design of a feature, then the report can focus on how to provide the same functionality without introducing a security vulnerability.

    This is neither of those cases. This is a feature whose core functionality, automatic download and execution of arbitrary code, cannot be introduced without causing a security vulnerability. This trade-off, in which marginal functionality is introduced by sacrificing any and all security, was a decision made at some point.

    > but would it have killed the person reporting it to have formulated it something like "I appreciate the convenience of automatic downloads but I'd prefer to be able to opt-out because of [...]".

    This phrasing is not equivalent. Stating "I appreciate the convenience of automatic downloads" does not seem accurate at all. Nowhere does the convenience show up as something that the reporter appreciates. Stating "I'd prefer to be able to opt-out" implies that an opt-out is sufficient. Avoiding a security vulnerability based on a per-user opt-out is something that should only be done for a hotfix until a better solution can be implemented.

    I could see the report being updated with a minimum list of design changes that would be necessary for the feature to be implemented in a safe manner: "While locating and recommending a package to be downloaded is convenient, the download must only be performed when the user explicitly approves it, with the user informed of the recommended package, its version and checksum, and the download URL prior to any download. Anything less than that is a security vulnerability." However, I don't fault the report for not doing so, as a reporter may not be familiar with a project's design roadmap. Describing an existing feature's design as "completely unacceptable" is sufficient.

  • And everything is now a "cyber security flaw" or exploit. I don't know why but it seems like there's been a recent crop of less technical people that know just enough to throw around cybersec buzzwords that are completely meaningless in context. Like I've seen people call this an exploit or a code execution vuln (on other platforms). Like what the hell.

  • I really wish these "torch and pitchfork" posts were declared off-topic. A discussion on what/when to auto-download and how would be useful, but comments on these kind of submissions are almost always just ranting/complaining about how bad $x is, what idiots people are, and things like this.

    > And people wonder why foss devs burn out.

    I have slowly become convinced that the open source community has been infiltrated by trolls from, eh, I don't know – something or someone that doesn't like open source. I have no direct evidence for this, but it does seem to align with observed facts.

    A few days ago someone posted some hobby project they worked on, and of course one of the replies was some unhinged rant about how the chosen $language wasn't any good and how they would "rather kill myself" than use that language... Okay... I don't think any normal personal can get get that triggered by someone's hobby project, hence my conclusion: infiltration by trolls.

At this point I really believe we need a consent popup after every letter typed, got forbid you typed a wrong letter.