This comes up a lot when people discuss anything related to npm modules. It's easy to simply dismiss these trivial one-line modules as "insanity" and move on, but there's actually plenty of good reasons as to why many prefer to work with multiple small modules in this manner. This GitHub comment by Sindre Sorhus (author of over 600 modules on npm) is my favorite writeup on the topic:
TL;DR: Small modules are easy to reason about, and encourage code reuse and sharing across the entire community. This allows these small modules to get a tremendous amount of real world testing under all sorts of use cases, which can uncover many corner cases that an alternative naive inlined solution would never have covered (until it shows up as a bug in production). The entire community benefits from the collective testing and improvements made to these modules.
I also wanted to add that widespread use of these small modules over inlining everything makes the new module-level tree-shaking algorithms (that have been gaining traction since the advent of ES6 modules) much more effective in reducing overall code size, which is an important consideration in production web applications.
Yes they are, in the same way that a book in which every page consists of a single word is easier to understand than one with more content per page.
By focusing on the small-scale complexity to such an extreme, you've managed to make the whole system much harder to understand, and understanding the big picture is vital to things like debugging and making systems which are efficient overall.
IMHO this hyperabstraction and hypermodularisation (I just made these terms up, but I think they should be used more) is a symptom of a community that has mainly abandoned real thought and replaced it with dogmatic cargo-cult adherence to "best practices" which they think will somehow magically make their software awesome if taken to extremes. It's easy to see how advice like "keep functions short" and "don't implement anything yourself" could lead to such absurdity when taken to their logical conclusions. The same mentality with "more OOP is better" is what lead to Enterprise Java.
They really aren't, when I'm reading is-positive-integer(x) and wonder if 0 is positive I need to hunt down the definition of positive through two packages and as many files. And it gets wrose if both your code and one of your dependencies required 'is-positive-integer' and I have to also figure out which version each part of the code base is using.
If you had written (x > 0) I would have known immediately, it also wouldn't be doing the same thing as is-positive-integer(x) but how many calls to is-positive-integer are actually correct in all the corner cases that is-positive-integer covers?
And then there's the other problem with dependencies: you are trusting some unknown internet person to not push a minor version that breaks your build because you and Dr. is-positive-integer had different definitions of 'backwards compatibility'.
You'd have a good point, if all of those tiny but probably useful modules were given the use you're describing.
Discoverability though is so poor that most of those modules are most likely just used by the author and the author's co-workers.
If a typical npm user writes hundreds of packages, how the hell am I supposed to make use of them, when I can't even find them? Npm's search is horrendous, and is far from useful when trying to get to "the best/most supported module that does X" (assuming that random programmers rely on popularity to make their choice, which in itself is another problem...).
If anything looking at sindresorhus's activity feed: (https://github.com/sindresorhus) perfectly supports the author's point. Maybe some people have so little to do that they can author or find a relevant package for every single ~10 line function they need to use in their code and then spend countless commits bumping project-versions and updating package.json files. I have no idea how they get work done though..
How is that superior to copypasting from a stackoverflow answer?
If it's a popular issue, lots of people had the same issue, many will be nice enough to add their edge cases and make the answer better, most will not. Same goes for contributing to a package.
With a package you would be able to update when someone adds an edge case, but it might break your existent code and that edge case may be something that is not particular to your system.
If you don't want to get too deep in the issue, you can copy paste from SO, just the same you can just add a package.
If you want to understand the problem, you can read the answers, comments, etc. With the package you rely on reading code, I don't know how well those small packages are documented but I wouldn't count on it.
The only arguments that stands are code reuse and testability. But code reuse at the cost of the complexity the dependencies add, which IMO is not worth the time it'll take you to copy and paste some code from SO. Testability is cool but with an endless spiral of dependencies that quite often use one or more of the different (task|package|build) (managers|tools) that the node ecosystem has, I find it hard to justify adding a dependency for something trivial.
The simple rebuttal to that is that modules that are collections of small functions are easy to reason about as well, and don't the downside of increasing the metadata management to useful function ratio nearly as much. Why have just an average (mean) function, when it makes sense to provide a median and mode as well? Even then, you might find that there's a bunch of extra math operations that are useful, and you might want to include some of those.
Just sugar-coated kool aid I'm hearing. Community benefits? First of all, I'm coding to get paid and this recent madness proved that JS ecosystem is semi-garbage. Back to the original question - do people really can't program that they need packages like left-pad or is-integer which had their own dependencies? Before writing those cool toolchains (which would likely work in a specific machine with a specific setting with all the real world testing the community has) can we at least pretend that we know the computer science basics?
They do NOT encourage code re-use because the effort required to understand your need, refrain from writing code, and hunt down a module in npm, far outweighs the effort to just write the code and stick it in your in-house utils library.
I think that there's a certain wishfulness bordering on naïveté to this pursuit. We tell ourselves that we are managing complexity by having small, well-tested units that are reused by an entire community. But software complexity continues to exist. We think we are mitigating the complexity of our own software. But the complexity really just shifts to the integration of components.
Anyone that has been around long enough has war stories about getting two relatively simple pieces of software working with each other. In my experience, integration problems are often the most difficult to deal with.
I'm not at all clear why this blog post is touted as evidence that the tiny modules approach is correct. I think it might be all the people after it congratulating him.
"It's all about containing complexity." - this completely ignores the complexity of maintaining dependencies. The more dependencies I have the more complicated my project is to maintain.
Dependencies are pieces of software managed by separate entities. They have bugs and need updates. It's hard to keep up to date.
When I update a piece of software I read the CHANGELOG, how am I expected to read the CHANGELOG for 1,000 packages?
Depending on a bigger package (handled by the same entities, who write one changelog, in the same form) is more straight forward.
I'm not saying this is wrong - but there's a balance here, and you must not ignore the complexity of increasing your number of dependencies. It does make things harder.
My problem with this, as an occasional JavaScript developer, is "discoverability" (as many others have mentioned). If I decide I need a left-pad function, and search on NPM, how do I choose which one is best? The one with the most downloads? Not always the best indicator of quality; perhaps it's just the oldest.
Not to mention the cognitive overhead of stopping programming, going to NPM, searching/finding/installing the module, then reading the documentation to understand its API. Isn't it simpler to `while (str.length < endLength) str = padChar + str;`? How can there be a bug in that "alternative naive inlined solution"? Either it works or it doesn't!
I don't see how your linked comment brings more to the table than the basic arguments for code reuse.
But naturally, with any code reuse there's a benefit and a cost to instance of internal or external reuse.
The Benefits for external reuse include ideal reliability as your describe as well as not having to create the code. The costs for external reuse include having your code tied to not just an external object but also the individuals and organizations creating that object.
I think that means that unless someone takes their hundreds of modules from the same person or organization and is capable of monitoring that person, that someone is incorporating a layer of risk to their code that they don't anticipate at all.
Percentage of module owners who you can't trust to not screw up their module: H
Risk of indirectly hosing a project with N module owners providing dependencies: 1-((1-H)^N)
Let's say H is very small, like 0.05% of module owners being the type who'd hose their own packages.
3 module owners: 0.15% chance your own project gets hosed
30 module owners: 1.49% chance your own project gets hosed
300 module owners: 13.93% chance your own project gets hosed
Keep in mind it's not just your dependencies, but your entire dependency chain. And if you think a module owner might hose some modules but not others, maybe H is actually the number of modules in which case 300 starts getting pretty attainable.
Upshot:
Not everyone is trustworthy enough to hang your project on. The more packages you include, the more risk you incur. And the more module owners you include, definitely more risk.
The micromodule ecosystem is wonderful for all the reasons described, but it's terrible for optimizing against dependency risk.
Takeaways:
Host your own packages. That makes you the module owner for the purposes of your dependency chain.
If you're not going to do that, don't use modules dynamically from module owners you don't trust directly with the success of your project.
I love collaborative ecosystems, but some people suck and some perfectly non-sucky people make sucky decisions, at least from your perspective. The ecosystem has to accommodate that. Trust is great...in moderation.
I agree with you, besides the tree-shaking (nice word btw).
It's like NPM shaking the Christmas tree and then say "Have fun cleaning up the floor". Remember that NPM is not like apt-get, where the packages are Managed for you by an expert. In NPM you have to manage the packages! And where you can't have NPM and build dependencies, like in production, maintenance now becomes much harder!
My problem is one of productivity. There's already a standard library, and if it's a language I've been using for a while, I probably remember most of it. I can pretty much go straight from my thought of what I want done to typing it out, much like typing English. If you force a 'cache miss' and force me out of my head and into a documentation search, well, that's going to have a significant effect on my performance. If the function is well-named, it has less of a cost in reading the code, but there's still a cost, because what if it's slightly different? I have a pretty good feel for the gotchas in much of the standard library of the languages I use. I have to stop and check what the gotchas of your function are.
Yes, at some point the complexity cost of gluing together the standard library functions to do something becomes greater than the lookup cost of finding a function that does what I want; but I am saying that adding more functions is not costless.
Small modules are also often the result of dealing with different javascript implementations over the years. I've recently seen a simpler version of the left pad that would not have worked on multiple Safari versions or <IE6
The derision is unwarranted, due to a failure in critical thinking from otherwise smart people.
It's interesting to me that people find this convincing. I find it to be complete insanity. People need their libraries, but putting everything in tiny buckets is just not working. Why aren't people working on good utility libraries instead?
There's even some guy calling for a "micro-lodash". To me, as a Python engineer, lodash [1] is already a tiny utility library.
I guess it's also about the fact that JS is a pretty bad language. That you need a one-line `isArray` dependency to `toString.call(arr) == '[object Array]'` is crazy.
it reminds an apt proverb: missing forest for the trees.
Too many modules do not necessarily become a good thing. They may appear to get rid of complexity but in reality, you will have to face the complexity some level above and in fact, the sheer number of small modules will most probably add more complexity of themselves.
The reasoning makes sense for small modules that might change in the future, but as he says himself, most of his modules are finished and will never change. That makes many arguments in his post moot and the modules should probably be snippets instead that are implemented directly.
Author of "is-positive-integer" here. I will admit the implementation is pretty funny, but I move-out all single-purpose utils out of projects to modules for a bunch of reasons. DRY is the most obvious one, but one that may be less obvious is for better testing.
I move out modules so I can write really nice tests for the independent of the projects I am using them in. Also, I tend to write projects w/ 100% test coverage, breaking out utils allows me to test projects easier and faster.
Also note, the implementation of this module changed a few times today. With it being open source and having the collaboration of other engineers, we ended up with a very performant version, and discovered interesting quirks about "safe integers" in JS.
- Breaking out is-positive-integer hasn't reduced the number of paths to test. You have not gained anything, you've added overhead.
- 100% test coverage is rarely a good thing. It is required for safety critical areas like avionics. I can guarantee that your JS code is not making into any safety critical environment!
Just to pick at a nit with you, it's a little meaningless to say "100% test coverage", without specifying whether you're talking about line coverage, branch coverage, path coverage...
This is especially true for one-liner modules in js, where any test at all might let you claim 100% statement coverage, without accounting for branches or loops within method calls.
Actually, thats a good reason to use trivial functions like the one described. Hopefully the author has discovered all of the quirks in Javascript that might affect this. It will likely be a lot more tested than any version I would write.
As someone who spends 80% on the back end, I often get bit on the ass from JavaScripts quirks when I need to add some front end stuff.
It would be really, really great if this function was not in its own module, but was part of a larger library in which all such functions of related modules were captured, without the (cognitive and other) overhead of the separate packaging.
var average = require('average');
var result = average([2, 5, 0, 1, 25, 7, 3, 0, 0, 10]);
console.log('The average for all the values is:', result);
It's hard to not stare at that in complete disbelief; someone thought that it was worthwhile to create a package for determining the mean of an array of numbers.
You know what's worse? Javascript numbers are all floating point numbers, which means integers are 53 bits long. So, you might think this library would try to address issues this can cause, but nope, this is the average statement you'd write if you didn't know was a mantissa was and had never heard of big.js, bignumber.js, decimal.js, crunch.js or even strint (which represents integers as strings because wtf not).
Well a bunch of standard libraries have sum defined, right?
JavaScript has suffered from a lack of a standard library for a while. Having a small package like this means that (in theory) everyone is using the same bug free version of summing instead of writing their own.
Honestly having JS start building a standard library at this point would be wonderful.
Sounds like the author was aiming for something to put on his resume: e.g., "Author of 25 libraries on NPM, some with more than 500K downloads." etc...
This sounds like a symptom of an inadequate standard library. I do expect to be able to call "average" on a list of numbers without writing it myself, but I expect that to be part of the language not a 3rd party package.
I'm thinking that someone wanted to learn about building and publishing a package and the ecosystem so they made this computationally trivial thing as a practical exercise.
Pretty much every package management system gets cruft in it like this. Example: for a long time someone had uploaded a random Wordpress core install into Bower.
While it demonstrates the problem of npm lacking namespaces (such that the word "average" is wasted on such a trivial implementation)...it doesn't seem anyone was using that library
I wanted to use a javascript tool that would make my life easier, and when I looked at the npm dependency tree it had 200+ dependencies in total.
If I used that javascript tool, I'd be trusting hundreds of strangers, lots of which had absolutely no clout in github (low number of stars, single contributor projects) with my stuff.
And not just them, I'd be trusting that no one steals their github credentials and commits something harmful (again, these projects are not very popular).
It doesn't help that npm doesn't (AFAIK) implement code signing for packages which at least would let me manage who I choose to trust
In all the debate about this, why is the trust-dependency-fuck-show not getting more attention?
Every dependency you take is another degree of trust in someone else not getting compromised then suddenly finding all sorts of horribleness making it into your production environment.
This is more a reflection of how bad the JS language is than anything. Real programming languages have means of standardizing the most common UX and simple patterns into a standard library. Javascript is a consortium hell that never gets updated sufficiently, and has no good standard library, so NPM basically replaces the standard library with a thousand micropackages.
Also it is a lot easier to get it wrong in JS. Is it null? Is it undefined? Is it bird? Is it a plane? No it's a string!(but sometimes a number). Good programming languages make it easy to write is negative e.g. isNegative = (<0) where implicitly by the 0 and < it will take a Num and return a bool and this is type checked at compile time.
Yeah, there are a few shitty examples on npm. It's an open system and anyone can upload anything. The market speaks on how valuable those are. Cherry picking poor modules says nothing about the rest.
Plus, if you think that's too small, write your own broader module that does a bunch of stuff. If people find it valuable, they'll use it. If they find it more valuable than a bunch of smaller modules, you'll get 10,000 downloads and they'll get 10.
The module you roundly ridicule has had 86 downloads in the last month, 53 of which were today (at the time of this writing). I imagine most of those 53 were after you posted. So that's 40 downloads in a month, as compared to the express framework which has had 5,653,990 downloads in the last month.
The wailing and gnashing of teeth over this module is ridiculous.
DRY taken to the dogmatic extreme where everything is made up and the github stars are the only thing that matters.
This article touches on things that are wrong in the javascript culture. I always had this nagging feeling when working with NPM, this article brings it to light. For what it's worth I never felt this while writing Ruby, C# or Go.
It's the -culture- that needs to change here, not the tools.
The recursive folder structure in npm-modules was the first indication. At least Java had a single tree with com.bigco.division.application.framework.library.submodule.NIHObject.java
That recursive node_modules or whatever it is called was what made me hate this whole npm thing, specially because it is not centralized somewhere in my computer.
And that means the same files a few times repeated on my drive just eating space.
Being a Java developer I don't understand why the approach was not more like maven.
It’s written in such a way that every time you call...
passAll(f1, f2, ..., fn)(args..)
... there are something like 5 + 2n attribute accesses, 5 + 3n function calls, 3 + n new functions created, as well as some packing and unpacking of arguments, not including the actual application of the functions to the arguments that we care about. That’s in addition to the several functions defined in the dependent submodules, which you only have to pay for constructing once.
[From my quick eyeball count. These numbers could be a bit off.]
I'm more disappointed that it doesn't short-circuit the operation at all. It applies all the functions, THEN it determines whether all of them passed. Even worse, it uses `every` (which does short-circuit) to determine that all the functions are, indeed, functions, but apparently the ability to use that function to determine whether every predicate passes was missed.
Lol, nice, I wrote "pass-any" first. I then copied the code and replaced "or" w/ "and" to create "pass-all".
I will probably have go back and change this now that I know about it. In general though, not gonna a lie, I am not very concerned about micro performance optimizations.
I think it actually is not. That's from some years ago and my memory of it is fuzzy, but at that time it was surprisingly hard to check whether a variable is a positive integer – maybe it was a negative one though and that was harder? You'd think it is just checking whether it is an integer and bigger than 0, or just checking whether it is bigger than 0. And it is. But to get that code to work reliably, regardless of whether it gets a string or float or an undefined, with the JS type system of that time and in multiple browsers, even the crappy ones, that took some time. There was one specific edge case involved.
Not that it was impossible, but I still remember having to search for it and being astonished that that was necessary.
Yea. I don't think many argue against abstracting complexity into more easily understood orthogonal modules. But some of these modules aren't abstracting complexity. They are ludicrous. They are nonsense. They are a symptom of a very real problem with how JS is programmed, how people expect to program in JS, and the difficulty people have with decomposing problems.
So many people on this page have written about how these are well tested, performant, and correct modules. But, the modules aren't even correct in many cases, let alone providing incomplete coverage over edge cases or the slow performance and horrendous dependencies of many of the modules.
Formerly 9 dependent modules... but who cares maybe it was a homework assignment or a joke.
I don't use NPM so maybe it doesn't really matter aside from the level of abstraction being implemented being relatively ridiculous.
However, if my build system had to go out and grab build files for every X number of basic functions I need to use, grab Y number of dependencies for those functions, run X * Y number of tests for all those dependent packages, AND then also fell apart if someone threw a tantrum and removed any one of those packages basically shutting me down for a day... then I'd question every single thing about my decisions to use that technology.
[Quick Edit] Basically I'm saying "Get off my lawn ya kids!"
This implementation reads almost as parody, although I don't suspect that the author meant it as such. If you didn't have a sense of what lurked behind the abstraction, it would be kinda beautiful.
I can't decide what's crazier to me: that such a package exists, or that JavaScript is such a ridiculously bad language that an "is positive integer" function is non-trivial to write.
I end up spending most of my working life working on other peoples code, rather than new features I end up debugging and fixing bad code. (I actually rather like it)
The majority of code I have ever seen is awful (20 years across large + small companies) but that is why I am hired to fix awful code so I am skewed. The amount of times I have seen people implement something simple, in a convoluted error prone was is unbelievable.
I know this seems ridiculous but when you see time and time again how people fail to do the simplest things it seems like a good idea.
I had several arguments about JS and I was shocked how many developers consider this platform great. I am not sure why these extremely bad practices are defended by devs, what are they getting out of it? I am only hoping we are moving towards more sensible development environment, there are many of them with better best practices and more sane libraries.
> If npm was invoked with root privileges, then it will change the uid to the user account or uid specified by the user config, which defaults to nobody. Set the unsafe-perm flag to run scripts with root privileges.
I can't stop laughing. I think you have to admire the elegance of the concept as performance art though: this is cheap insanity. In fact, I've got to hand it to them, this is the most fun I've had looking at something programming related in a while. I recall the opening lines of SICP,
> I think that it's extraordinarily important that we in computer science keep fun in computing. When it started out, it was an awful lot of fun. Of course, the paying customers got shafted every now and then, and after a while we began to take their complaints seriously. We began to feel as if we really were responsible for the successful, error-free perfect use of these machines. I don't think we are. I think we're responsible for stretching them, setting them off in new directions, and keeping fun in the house. I hope the field of computer science never loses its sense of fun. Above all, I hope we don't become missionaries. Don't feel as if you're Bible salesmen. The world has too many of those already. What you know about computing other people will learn. Don't feel as if the key to successful computing is only in your hands. What's in your hands, I think and hope, is intelligence: the ability to see the machine as more than when you were first led up to it, that you can make it more.
Quoted in The Structure and Interpretation of Computer Programs by Hal Abelson, Gerald Jay Sussman and Julie Sussman (McGraw-Hill, 2nd edition, 1996).
The same could be said for your post. When people ask why commenting on message boards is terrible, show them this.
If you don't want to write modules this way, don't. Nothing about javascript requires that you even read articles about modules that you don't want to use. Or read articles and then follow up by posting on message boards about articles about modules you aren't going to use.
Bang some code out instead. Your opinion of javascript is about as valuable as the opinion of the person who wrote the modules.
A good micro-module removes complexity. It has one simple purpose, is tested, and you can read the code yourself in less than 30 seconds to know what's happening.
Take left-pad, for example. Super simple function, 1 minute to write, right? Yes.
The fact of the matter is: every line of code I write myself is a commitment: more to keep in mind, more to test, more to worry about.
If I can read left-pad's code in 30 seconds, know it's more likely to handle edge cases, and not have to write it myself, I'm happy.
The fault in this left-pad drama is not "people using micro-modules". The fault is in npm itself: all of this drama happened only because npm is mutable. We should focus on fixing that.
> every line of code I write myself is a commitment
That's true. However:
Every dependency you add to your project is also a commitment.
When you add a dependency, you're committing to deal with the fallout if the library you're pulling in gets stale, or gets taken over by an incompetent dev, or conflicts with something else you're using, or just plain disappears. If you add a dependency for just a few lines of code, you're making a way bigger commitment than if you'd just copy/pasted the code and maintained it yourself. That's why so many people are shaking our heads at a 17-line dependency. It's way more risk than it's worth. If you need a better stdlib for your language (some of us write PHP and feel your pain) then find one library that fills in the gaps and use that.
> If you add a dependency for just a few lines of code, you're making a way bigger commitment than if you'd just copy/pasted the code and maintained it yourself.
This is a problem with NPM, not with dependencies. With different package management systems with stable builds and lockfiles, then you pin to a specific version and there is no way upstream can cause problems. A lockfile is a pure win over vendoring.
Maintaining a dependency on a library should be much less effort than maintaining 17 lines of code. If it isn't that's a deficiency in your dependency infrastructure.
I wouldn't mind so much if these micro-modules were written in a style of thoroughness; heavily commented, heavily documented with pre-conditions, post-conditions and all imaginable inputs and outputs explicitly anticipated and formally reasoned about. I don't mind over-engineering when it comes to quality assurance.
Looking at that left-pad module though - no comments, abbreviated variable names, no documentation except a readme listing the minimally intended usage examples. This is not good enough, in my opinion, to upload to a public repository with the objective that other people will use it. It is indistinguishable from something one could throw up in a couple of minutes; I certainly have no reason to believe that the future evolution of this code will conform to any "expectation" or honour any "commitment" that I might have hopefully ascribed to it.
[EDIT: I've just noticed that there are a handful of tests as well. I wouldn't exactly call it "well tested", as said elsewhere in this thread, but it's still more than I gave it credit for. Hopefully my general point still stands.]
The benefits of reusing other people's code, to a code reuser, are supposed to be something like:
(a) It'll increase the quality of my program to reuse this code - the writer already hardened and polished this function to a greater extent than I would be bothered to do myself if I tried right now
(b) It'll save me time to reuse this code - with the support of appropriate documentation, I shouldn't need to read the code myself, yet still be able to use it correctly and safely.
Neither of those things are true for this module. It's not that the module is small, it's that it is bad.
(True that npm's mutability is a problem too - this is just a side-track.)
Completely agree here - the problem isn't micro-modules. It's partly just a lacking standard library for javascript and largely just exposing issues in npm that the community was pretty ignorant of until just now.
The whole "amaga, it's a whole package for just ten lines of code" is just elitism. Given the number of downloads on things like left-pad, it's clearly useful code.
Agreed as well. In fact, I would posit that this wasn't even really a problem until npm@3 came out and made installing dependencies far, far slower. Yet it was necessary; a standard project using babel + webpack installs nearly 300MB (!!!) of dependencies under npm@2, and about 120MB under npm@3. Both are unacceptable, but at least npm3 helps.
1) JS is unique in that it is delivered over the wire, so there is a benefit in having micro-modules instead of a bigger "string helpers" module. Things like webpack are changing that now (you can require lodash, and use only lodash.padStart).
2) JS's "standard" library is so small, because it's the intersection of all of the browser implementations of JS dating as far back as you care to support. As pointed out in sibling, a proposal for padLeft is included in ECMA2016. But we'll still need Left-Pad for years after it's adopted.
Uh, no Left Padding is NOT built-in in JavaScript. The proposal to add `String.prototype.padLeft()` was just added to ECMAScript 2016.
JavaScript had a very minimal standard library, it's pretty asinine of you to compare it to C or any other language with a pretty extensive standard library.
Wow, I feel like I could have written this. Back when I used Python, I had a folder full of functions I would copy-paste between my projects. (And maybe some of the projects contained unit-tests for the function. I didn't always keep those tests in sync.) Updating them was a pain because inevitably each one would get slightly modified over time in each project separately. Eventually, I bundled all of them into a bundle of completely unrelated utility functions in a folder on my computer somewhere, and I would import the folder with an absolute path. Sharing the code I wrote was a pain because of how much it referenced files on my local computer outside of the project. I never considered publishing my utility module because all of the stuff was completely unrelated. I'd rather publish nothing than a horrifying random amalgram that no single project of mine was even related to all of the subject matter present in it.
With npm and the popularity of small modules, it was obvious that I could just cheaply publish each of my utility functions as separate modules. Some of them are about a few dozen lines, but have hundreds of lines of tests and have had significant bugfixes that I am very happy that I haven't had to manually port to dozens of projects. I don't miss copy-pasting code across projects, no matter how many claim I've "forgotten how to program".
What I see is that a module has a non-zero overhead in complexity in itself. That is, ten 10 line modules and twenty 5 line modules do not yield the same complexity. The modules themselves have a complexity overhead associated, and submodules have their own complexity overhead associated, albeit smaller than first party modules. That complexity is easily seen from the recent situation of unpublishing modules, which resulting in modules multiple steps removed having problems building.
So, when I read "It doesn't matter if the module is one line or hundreds." I call bullshit. There is overhead, it's usually just fairly small (at may event begin to rival the gains from using a module at that level), but that small amount adds up. Once you've had to deal with a dependency graph that's 10 levels deep and contains hundreds or thousands of modules, that small extra complexity imposed by using a module is no longer in total, and comes at a real cost, as we've just seen.
Other module ecosystems have gone through some of the same problems. There was a movement in Perl/CPAN a few years back to supply smaller, more tightly focused modules a while back, to combat the sprawling dependencies that were popping up. The module names were generally suffixed with "Tiny"[1] and the goals where multiple:
- Where possible, clean up APIs where consensus had generally been built over what the most convenient usage idioms were.
- Try to eliminate or reduce non-core dependencies where possible.
- Try to keep the modules themselves and their scope fairly small.
- Remove features in comparison to the "everything included" competitor modules.
This has yielded quite a few very useful and strong modules that are commonly includes in any project. They aren't always tiny, but they attack their problem space efficiently and concisely. Even so, I'm not sure there's ever a module that's a single line of code (or less than 10, given the required statements to namespace, etc), as the point is to serve a problem, not an action.
It doesn't handle edge cases, it doesn't perform well and it isn't well tested. There is also no documentation. Obviously 30 seconds wasn't enough for you to verify anything at all about this module (namely that it's complete garbage).
And just because some random guy didn't get something as trivial as this right the first time, doesn't mean nobody else can. Also the de facto standard library lodash already has padding utilities, made by people who have a proven track record.
I don't agree with the explosion of micro-modules. There's a reason the vast majority of languages doesn't have them, at least not at function level.
IMO in the Javascript world they're only there in order to minimize script size for front end work. See lodash & lodash-something1 / lodash-something2 / ..., where there's an option of using the whole module or just including 1-function long scripts, precisely to avoid the script size issue.
Is there a solution for this? I know that the Google Closure compiler can remove dead code, ergo making inclusion of large modules less costly in terms of code size. Am I missing some ES6 feature that also helps with this?
You're just trading in the complexity of the code you'd have to write for the delayed complexity of dealing with dependency issues down the line. It's a waste of a trade off for tiny things like this.
I agree with the points you've made, but I would also posit that adding a dependency using this mutable package manager is making a commitment to maintain the integrity of that dependency, which is arguably more work than maintaining the handful of lines of code.
Nobody has forgotten. These people never knew to begin with.
NPM/JS has subsumed the class of programmer who would have previously felt at home inside PHPs battery-included ecosystem. Before that, a similar set of devs would have felt at home with Visual Basic. Seriously, go visit the comments section on archived copies of the PHP documentation. You'll find code of a similar nature. If PHP had had a module system 10+ years ago you would have seen this phenomenon then. Instead it was copy and paste.
This isn't elitism, it's just the way it is. The cost of a low barrier to entry in to a software ecosystem is taking in those who don't yet have software engineering experience.
Nobody should be surprised that NPM, which I believe has more packages than any other platform, is 90% garbage. There are only so many problems to solve and so few who can solve them well, in any language. Put 100 programmers in a room, each with 10 years experience, and you'll be lucky to find 1 who has written a good library. Writing libraries is really hard.
This is the answer, 100%. All it takes to publish an npm package, is the command npm publish, and you're done. So of course it is no surprise that there are tons upon tons of seemingly useless or tiny projects (gotta pad out that github profile for those recruiters!), or that there are then plenty of packages that use them.
Add into that the fact that:
1) Javascript has a huge number of developers, and is often an entry-level language
2) The developers on this thread (I like to think of HN as at least slightly above average) are divided whether having small packages / large dependencies trees is a good or bad thing
3) Dependency management is something that matters mostly to long term (professional / enterprise / etc) applications, which is a subset of programming, and I wonder if not a minority subset of node.js projects in general.
4) If I'm writing a throwaway app or proof of concept, and therefore don't care about dependency maintenance, using as many dependencies as possible is a major time savor,
and of course you get this situation, and it seems to make perfect sense.
Personally, I wish there was an NPM Stable, where packages underwent much more scrutiny and security in order to get it, but nonetheless, nothing I've read so far about npm really scares me given the the above context. If you are a dev creating an unmanageable dependency tree for your enterprise app, you're a shitty dev. That doesn't necessarily mean that NPM is wrong for being so open in allowing others to publish their packages, or that smaller / more worthless packages shouldn't be allowed to publish.
That said, I would really like to hear a response to this post, as I have limited experience with different package management systems.
The huge difference is that PHP package manager support namespaces and dependencies ARE FLAT. You cannot import 2 versions of the same package under the same namespace. Which
1/ forces package authors to write stable libraries
2/ forces dependencies to narrow the versions of their dependencies
3/ prevents name squatting to some extent. You cannot have a package named "forms" and then sell the name for real money, like seen on NPM. your package needs to be "namespace"/"name". NPM made a huge mistake with its gems like global namespace and it explains half the problems it is having today.
For a post that is claiming to not be elitist, it reads pretty elitist.
Can you expand on how to identify the class of programmers you're referring to? Are they the type that copy / paste code directly from StackOverflow? They lack a classical computer science education? They haven't worked on a large, enterprise-grade project?
From what I've seen, there's one division between programmers that's hard to overcome. Some see it as a tool to get certain results for a job. Some of them are bad, some of them are lazy, but most of them are good enough to get to their objective.
Others see programming more as an art. They take care to make the code not only efficient but also elegant. They'll read up on new and interesting algorithms and incorporate them in novel ways. They might often be behind deadlines, but when they are, they create things like GNU Hurd that inspire a lot of interest and lead to interesting results, maybe even teach people a few things. Their code is interesting to read. They tend to write the libraries that the first group uses.
Both groups contribute a lot, but it's not easy to get them to understand that about each other.
Comparing NPM to PECL/PEAR doesn't make much sense when talking about PHP developers. With PECL, the overhead of building a module in C is waay too high to make it viable for micro packages. And PEAR didn't just accept any random stuff, they were shooting for the one-solution-fits-all libraries and not tons of user-defined micro libraries like ecosystems like NPM encourage.
Compare NPM to Composer/Packagist and you get a better comparision. I've personally seen only very few micro packages on Packagist, thankfully this never seemed to gain traction in the PHP world.
Going down the "lots of tiny modules" route is about these three things:
a) No standard lib in JS
b) JS is delivered over the internet to web pages in a time sensitive manner ... so we don't want to bundle huge "do everything" libs. Sometimes its convenient to just grab a tiny module that does one thing well. There isn't the same restriction on any other platform
c) Npm makes it really easy to publish/consume modules
d) And because of c) the community is going "all in" with the approach. It's a sort of experiment. I think that's cool ... if the benefits can be reaped, while the pitfalls understood and avoided then JS development will be in an interesting and unique place. Problems like today can help because they highlight the issues, and the community can optimise to avoid them.
Everyone likes to bash the JS community around, we know that. And this sort of snafu gives a good opportunity. But there many JS developers working happily every day with their lots of tiny modules and being hugely productive. These are diverse people from varied technical backgrounds getting stuff done. We're investigating an approach and seeing how far we can take it.
We don't use tiny modules because we're lazy or can't program, we use them because we're interested in a grand experiment of distributing coding effort across the community.
I can't necessarily defend some of the micro modules being cited as ridiculous in this thread, but you can't judge an entire approach by the most extreme examples.
I think b) is true only because JavaScript tooling cannot perform dead code elimination. Other languages have big grab-bag utility libraries like lodash that don't hinder performance because a linker or runtime can avoid loading unused portions.
Note for b): If you include libraries such as jQuery on you website via CDN, I believe browsers will be able to use the cached version even if they never visited your website before (given that they've cached this version from the same CDN before).
I don't see anything wrong with using a pre-made left pad function. Why waste time and lines of code implementing something so trivial when there is already a solution available?
However, I agree it is ridiculous to have a dedicated module for that one function. For most nontrivial projects I just include lodash, which contains tons and tons of handy utility functions that save time and provide efficient, fast implementations of solutions for common tasks.
I think the article's thesis is essentially that every dependency your project pulls in -- which includes all the dependencies your dependencies pull in -- is a point of potential failure. I understand the "don't re-invent the wheel" defense, but the Node/JavaScript ecosystem tacitly encourages its users to build vehicles by chaining together dozens of pre-made wheels, all of which depend on more wheels, and each and every one of those wheels has a small but non-zero chance of exploding the next time you type "npm update."
(And, y'know, maybe it's because I'm not a JS programmer, but the notion of looking for a module to implement a string padding function would never have even occurred to me.)
The problem is not that, the problem is depending on unreleased versions, instead of simply depending on the version that was written when u wrote your code.
a Git submodule like approach would be much better
> Why waste time and lines of code implementing something so trivial when there is already a solution available?
Because it's so trivial? I can't wrap my head around why this is an argument in the first place. It makes no sense to bring in a module from a third party adding yet another dependency and potential point of failure when reimplementing it yourself literally takes as long as it takes to find the module, add it to package.json and run npm install.
People should be trying to limit dependencies where possible. Reproducible builds are really important if it costs you almost no time you should have it in your code base IMO.
People taking the DRY principle to the most extreme degree always makes for the worst code to debug and maintain.
This entire comment thread is such a breath of fresh air. I was beginning to think that I was that guy who was crazy for thinking that all of the people doing this were crazy. This thread is like my new support group.
> It makes no sense to bring in a module from a third party adding yet another dependency and potential point of failure when reimplementing it yourself literally takes as long as it takes to find the module, add it to package.json and run npm install.
Even if it does take the same amount of time (which it shouldn't), a 1-line call to a standard module imposes less of a future maintenance burden than 14 lines of custom code.
> People should be trying to limit dependencies where possible. Reproducible builds are really important if it costs you almost no time you should have it in your code base IMO.
That's a non sequitur. Reproducible builds are important, but unless you write code with 0 external dependencies you already have a system in place for handling library dependencies in a reproducible way. So why not use it?
> People taking the DRY principle to the most extreme degree always makes for the worst code to debug and maintain.
> However, I agree it is ridiculous to have a dedicated module for that one function. For most nontrivial projects I just include lodash, which contains tons and tons of handy utility functions that save time and provide efficient, fast implementations of solutions for common tasks.
I think that was largely the OP's point tbh. Using something like lodash [a utility library] is fine while using a module [for a single function] is not.
It might have gotten lost in the ranting from on high but I don't think the author truly meant more than that.
Some of the libraries people were mentioning broke yesterday already have lodash as dependencies, I have no idea why they wouldn't have just been using this...
Someone mentioned below that lodash had some breaking changes related to the padding functions a couple times, which could be a totally valid reason to avoid using those. I was under the impression that the lodash API was more stable than to have that kind of thing happening.
> I don't see anything wrong with using a pre-made left pad function. Why waste time and lines of code implementing something so trivial when there is already a solution available?
I'll tell you why.
The least important ones is that downloading such trivial module wastes
bandwidth and resources in general (now multiply this by several hundred
times, because of dependency fractal JS sloshes in). I would also spend much
more time searching for such module than I would implementing the damn
function.
More important is that you give up the control over any and every bug you
could introduce in such trivial function or module. You don't make it less
probable to have those bugs (because battle-tested package! except, not so
much in JavaScript, or Ruby, for that matter), you just make it much harder to
fix them.
And then, dependencies have their own cost later. You actually need a longer
project, not a throw-away one, to see this cost. It manifests in much slower
bug fixing (make a fix, find the author or maintainer, send him/her an e-mail
with the fix, wait for upstream release, vs. make a fix and commit it), it
manifests when upstream unexpectedly introduces a bug (especially between you
making a change and you running `npm install' on production installation), it
manifests when upstream does anything weird to the module, and it manifests in
many, many other subtle and annoying ways.
> You don't make it less probable to have those bugs (because battle-tested package! except, not so much in JavaScript, or Ruby, for that matter)
Battle-tested still applies - if you have that many people using a line of code they're more likely to find any bugs. (Formal proof is better than any amount of testing, but no mainstream language requires formal proof on libraries yet)
> And then, dependencies have their own cost later. You actually need a longer project, not a throw-away one, to see this cost. It manifests in much slower bug fixing (make a fix, find the author or maintainer, send him/her an e-mail with the fix, wait for upstream release, vs. make a fix and commit it), it manifests when upstream unexpectedly introduces a bug (especially between you making a change and you running `npm install' on production installation), it manifests when upstream does anything weird to the module, and it manifests in many, many other subtle and annoying ways.
Large monolithic dependencies have this kind of problem - "we upgraded rails to fix our string padding bug and now database transactions are broken". But atomised dependencies like this avoid that kind of problem, since you can update (or not) each one independently. Regarding fixing upstream bugs, you need a good process around this in any case (unless you're writing with no dependencies at all).
Finding this module on NPM or npmsearch.com is pretty trivial compared to ensuring you implement this in a way that catches every edge case.
> It manifests in much slower bug fixing
I don't buy this at all, because I've done it myself many times. If you're waiting on a PR from the original repo owner to fix a Production bug, you're doing it wrong. It's trivial to copy the dependency out of node_modules and into your src, and then fix the bug yourself. Then when the owner accepts your PR, swap it back in. I don't understand the problem here.
I agree that Lodash would be a better choice because it seems like a well maintained project. There could be two counter args, in theory:
- if the programmer uses other functions included in Lodash his code will have a single larger point of failure. For example, if Lodash is unpublished (intentionally as in this case, or unintentionally) then the programmer will have a lot more work to redo.
- Lodash introduces a lot of code, while the programmer only needs one of its functions to pad a string.
Using a library like lodash makes a lot more sense once you use a module bundler that allows tree shaking (like Rollup or Webpack 2.0) along with the ES6 module syntax. Heck, even if you're just using babel with Browserify or Webpack 1.x, you can use babel-plugin-lodash [0] so it'll update your imports and you only pull in what you need.
I think it speaks to just how lacking the baseline Javascript standard library is. The libraries that come with node help, but all of this stuff seems like it should be built-in, or at least available in some sort of prelude-like standard addon library. The lack of either leads to all these (apparently ephemeral) dependencies for really simple functions like these.
That said, I work with Java, Clojure and Python mostly so I may be more used to having a huge standard library to lean on than is typical.
So many people use lodash as a drop-in standard addon library that I'm surprised people aren't just using the padding functions that are right in there... Some of the packages that broke yesterday even have lodash included as dependencies already!
Looking at the changelog, there have been more than 70 versions of Lodash in less than four years. The first was in April 2012. [1]
_.padleft
does not exist. It was added as part of version 3.0.0 January 26, last year and renamed to _.padstart in Version 4.0 on January 12, this year.
So in less than a year "padleft" came and went away because all strings don't start on the left and someone decided that "left" means "start" except that the reason that it doesn't is the reason that it was changed. Even worse, the 4.0 documentation does not document that _.padstart renamed _.padleft. It's hard to grok what cannot be grepped.
Why blame someone for depending on padleft in a world where libraries swap out abstractions in less than a year? Breaking changes are bad for other people. Semantic versioning doesn't change that.
What if lodash itself was unpublished?
I'm having a hard time drawing a line here, obviously a 10 line function is too far on the bad side of lazy, but I can't tell what is an acceptable dependency.
Probably they do use that if they need left padding but also have a dependency on another package that thinks 'lodash is too big when all I need is left padding' so we get to this situation
This seems like the right answer to me. It's not that we forgot how to program, it's that Javascript forgot a stdlib. You could easily write your own left-pad function in any language, but a stdlib (or this module) gives you a standard way to reference it, so you don't have to look up what you named it or which order the args go in.
Agreed the JavaScript standard library is poor and instead of addressing it they've mostly just added syntax changes to ECMAScript 6 and 7. It's incredibly disappointing.
For instance I added a utility to my own library (msngr.js) so I could make HTTP calls that work in node and the browser because even the fetch API isn't universal for some insane reason.
I think in some ways a good standard library is a measure of programming language maturity. I remember when C++ had a lot of these problems back before you had the STL etc. In the early 90's it was a dog's breakfast.
We have a large internal C++ app at my work of that vintage (~1992) it uses its own proprietary super library (called tools.h++) which is just different enough from how the C++ standard evolved that its not a simple task to migrate our codebase. So now every time we change hardware platforms (has happened a few times in last 30 years) we have to source a new version of this tools++ library as well.
I find it amusing Javascript hasn't learnt from this.
Usually, dependency hell doesn't bite you, until it does. Try to rebuild that thousand-dependencies app in three years from now and you'll see ;-)
I recently had to rebuild a large RoR app from circa 2011 and it took me longer to solve dependencies issues than to familiarise myself with the code base.
Excessive dependencies are a huge anti-pattern and, in our respective developers communities, we should try to circulate the idea that, while it's silly to reinvent the wheel, it's even worse to add unnecessary dependencies.
> Try to rebuild that thousand-dependencies app in three years from now and you'll see ;-)
Let's be honest though, in the current trendy javascript ecosystem these people will already be two or three jobs away before the consequences of their decisions become obvious. Most of the stuff built with this is basically disposable.
I never can believe how often frontend developers talk about "you're just going to rebuild it all in 2 years" anyway. I guess it's a good way to keep yourself employed.
The gemfile.lock must have been "gitignored" at some point, because it had much older packages than the ones in Gemfile. Background: all we had was a git repo and did not have access to any "living" installation.
> Try to rebuild that thousand-dependencies app in three years from now and you'll see ;-)
This is your fault for expecting free resources to remain free forever. If you care about build reproduction, dedicate resources to maintain a mirror for your dependencies. These are trivial to setup for any module system worth mentioning (and trivial to write if your module system is so new or esoteric that one wasn't already written for you). If you don't want to do this, you have no place to complain when your free resource disappears in the future.
I agree. But I find two problems with your proposal:
1- Maintaining a mirror of dependencies can be a non-trivial overhead. In this app that I was working on, the previous devs had forked some gems on github, and then added that specific github repo to the requirements. But they did not do it for every dependency, probably they did not have time/resources to do that.
2- As a corollary to the above, sometimes the problem is not the package itself but compatibility among packages. E.g. package A requires version <=2.5 of package B, but package C requires version >= 2.8 of package B. Now I hear you asking "then how did it compile in the frist place?" probably the requirement was for package A v.2.9 and package C latest version, so while A was frozen, C got updated. This kind of problems is not solved by forking on Github, unless you mantain a different fork of each library for each of your project, but that's even more problematic than maintaining dependencies themselves.
P.S. At least for once, it wasn't "my fault", I didn't build that app LOL ;-)
There's more to dependency hell than "oops, the package disappeared." Try updating one of those dependencies because of a security fix, and finding that it now depends on Gizmo 7.0 when one of your other dependencies requires Gizmo < 6.0.
So, in the context of this discussion... you should make use of micro-modules to reduce code duplication, avoid defects, etc. However, don't expect those micro-modules to be maintained or available in the future; so you need to set up your own package cache to be maintained in perpetuity.
Or, you can implement the functionality yourself (or copy/paste if the license allows) and avoid the hassle.
I've been in the same situation as OP many times (although in most cases I've been brought in fix someone else's code).
In the Ruby ecosystem, library authors didn't really start caring about semantic versioning and backwards compatability until a few years ago. Even finding a changelog circa 2011 was a godsend.
I think this was mainly caused by the language itself not caring about those either. 10 years ago upgrading between patch releases of Ruby (MRI) was likely to break something.
At least this is one thing JavaScript seems to do better.
I can't speak for him, but upgrading really old Rails apps can get complicated very quickly. Especially when you're going across multiple major versions and have to deal with significant changes in Rails behavior and broken gems. "Rebuild" might not be the most accurate way to describe the slow, steady incremental approach you're forced to take (you aren't redoing huge swaths of your domain logic, for instance), but it gets the gist across.
Long story... we had the git repo but lost access to any working installation. I had to rebuild a dev vagrant VM first, and later on a production-ish setup on a server.
I find large dependencies like RoR itself cause a lot more dependency hell than zillions of small dependencies like this one. What kind of dependency hell could possibly happen for a module like this?
I wanted to write this post after the left-pad debacle but I've been beaten to it.
I think we got to this state because everyone was optimizing js code for load time-- include only what you need, use closure compiler when it matters, etc. For front end development, this makes perfect sense.
Somewhere along the line, front end developers forgot about closure compiler, decided lodash was too big, and decided to do manual tree shaking by breaking code into modules. The close-contact between nodejs and front end javascript resulted in this silly idea transiting out of front-end land and into back-end land.
Long time developers easily recognize the stupidity of this, but since they don't typically work in nodejs projects they weren't around to prevent it from happening.
New developers: listen to your elders. Don't get all defensive about how this promised land of function-as-a-module is hyper-efficient and the be-all end-all of programming efficiency. It's not. Often times, you already know you're handing a string, you don't need to vary the character that you're using for padding and you know how many characters to pad. Write a for loop; it's easy.
Note that this is exactly the sort of question I ask in coding interviews: I expect a candidate to demonstrate their ability to solve a simple problems in a simple manner; I'm not going to ask for a binary search. Separately, I'll ask a candidate to break down a bigger problem into smaller problems. In my experience, a good programmer is someone who finds simple solutions to complex problems.
Note: rails is similarly pushing back against developers that have too many dependencies:
Everything in this article is categorically wrong and antithetical to every principle of good programming ever articulated. The only problem here, as others have already noted, is that NPM allows people to delete published packages.
Small modules are not evidence of a problem, and they certainly aren't evidence of an inability to implement these things on the part of the people depending on them. Why would I implement left-pad myself when there is already a well-tested implementation that I can install? Building up an ecosystem of tiny abstractions, bit by bit, iteratively and evolutionarily, is how we get robust, well-designed complex systems. We don't get there by everyone reinventing the left-pad function to sate some misplaced appetite for self-reliance.
The author seems to make some arbitrary distinction between things that are 'large enough' to be packaged and 'pure functions' which are 'too small' to be their own modules, and I just couldn't disagree more. Tiny, pure functions are ideal modules. They facilitate the greatest degree of re-use, most clearly articulate what they ought to be used for, and stateless things are, in general, more composable than stateful things. There is no better unit of re-use than a tiny, pure function.
I don't know why you were downvoted, but I agree with you 100%.
Pure functions are indeed a good target for modularity and deserve proper documention and proper testing, at the very least.
For the readers of this comment:
* How many functions did you not commented last time you wrote some code?
* How many functions do you leave untested?
* How many functions did you wrote more than once?
* How many "trivial" functions did you wrote that actually took you 3 hours, because it's actually tricky, so you checked other implementations and tried to wrap your mind around it.
Check haskell's hoogle to a small sample of this concept.
a) I think that's a fine number of tests for a module this simple, which is itself an argument in favor of small modules. And I think it's four more assertions than anyone implementing this inline in a project would have.
b) The details of this particular project are orthogonal to the philosophy of small modules generally. Whether or not this module is well implemented or well tested has no real relation to whether or not it is a good idea to compose tiny modules.
You are assuming well tested is referring to unit tests, rather than the wealth of developers depending on and running it in their projects daily.
Don't forget how many different targets there are for JavaScript, even if you had a test environment with limitless budget, it wouldn't compare to having a popular package on NPM.
I'm not sure if you're saying this of my comment or the OP, but I code professionally every day in this style and it's working out just fine. And the smaller my modules get the better it seems to work.
Functions are too small to make into a package and dependency. Pure functions don’t have cohesion; they are random snippets of code and nothing more. Who really wants a “cosine” dependency? We’d all really like a “trigonometry” dependency instead which encompasses many “tricky” functions that we don’t want to have to write ourselves.
This is a pretty weak argument. What is "cohesion" and why do we care that modules have it? Joe Armstrong, one of the creators of Erlang, has argued the opposite (http://erlang.org/pipermail/erlang-questions/2011-May/058768): that lots of small, individual-function modules are better than a "misc" module that grows endlessly and may overlap with other people's "misc" modules.
Calling a function instead of writing the code yourself doesn't mean you've forgotten how to program! The real problem here is the cost and risks associated with dependencies in general (both of which are actually lower for single-function modules), and the broken package removal policies of npm.
I would retort with: Packages are too big to make into a functional dependency.
In the end, in this functions-as-dependencies world, a trig package would be something like:
(name) =>
switch (name) {
case 'sin':
(x) => sin(x);
break;
case 'cos':
(x) => cos(x);
break;
}
While in general I agree with the article I must admit that I also strongly DISAGREE with the overall message. Especially with this:
"Finally, stringing APIs together and calling it programming doesn’t make it programming."
Stringing APIs together is what actually programming is. This is building software and for instance when i use .toString() method I can easily forget how it is done, focus on other high level things and don't care about dependencies, as long as everything works fine.
Let's admit that the main problem here is with broken npm, rather than packages themselves. If someone has written the "leftpad" function, it is so I don't have to write it again, and I can save probably 15-40 min programming and checking some corner cases.
Also please note that javascript can be really tricky down in the details. So if there's anything that can help, it's better that it exists, rather than not.
> Let's admit that the main problem here is with broken npm
It is absurd to have packages suddenly retracted and important parts of the ecosystem stop functioning. This never happened with other languages I have used. Maybe we need a way to make sure the packages are always going to exist. Checksumming and adding the checksum to the version number would be useful too.
That's why there are proposals for immutable and distributed packages managers. Look at gx for instance. This is probably the future for package managers maintained by community.
The funniest thing about this entire debacle is the thousand of self-assured programmers coming out to show the JS/NPM world how it's done, only to have their short, simple, no-nonsense functions fail miserably on some edge cases they didn't think about.
Yes, or more accurately a large new generation of coders is entering the workforce who know how to code only in a superficial sense and think this is a good thing.
Programming, and especially startup programming, is being taken over by people who are primarily technicians rather than engineers. They want to assemble prefab components in standardized ways rather than invent new things. They are plumbers who know how to install from a menu of standard components, rather than civil engineers desigining purpose built one-off aqueducts.
It is the inverse of the "not invented here syndrome." The technician-programmer is trained to minimize time spent thinking about or working on a problem, and to minimize the amount of in-house code that exists. The goal is to seek quick fix solutions in the form of copy/paste from StackOverflow, libraries, and external dependencies to the greatest extent possible.
In house coding should, they believe, ideally be limited to duct-taping together prebuilt 3rd party libraries and services. Those who want to reinvent the wheel are pompous showboating wankers (they believe); creating your own code when you don't absolutely have to is a self-indulgent waste of time for impractical people who just like to show off their hotshot skills and don't care about getting things done. Move fast and break things and all that.
This began with stuff like PHP but really got going with Rails, which preached convention over configuration as a religion, and supplied a standardized framework into which you could easily fit any generic CRUD app that shuttles data between HTML forms and a database (but is painful if you want to deviate from that template in any way.) Note that Rails doesn't use foreign keys and treats the relational database as little more than a glorified persistent hash table.
This set the stage for Node.js (why bother learning more than 1 programming language?) and NoSQL (why bother learning how database schemas work?)
an engineer is more like a plumber than an inventor though.
you don't need to design a new type of screw to make a one off aquaduct, you don't need to design new bearings to make a gear box, you don't need to design a new opamp to make an amplifier, nor do you need to design a new MVC framework to make a one off CRUD app.
You use off the shelf parts and combine them with your knowledge and skills to produce and effective solution for your constraints. If you can't do it with existing stuff, then you design something new that can do it.
The point is that for most things, there were no such off-the-shelf parts in software engineering until recently. You had to invent each screw and bearing yourself, because you couldn't just order a bunch of screws and bearings. Nobody sold them.
Moreover, the difference is that today's technician-programmers don't know how screws and bearings work at all. The engineer who uses prefab screws and bearings still understands them in detail and knows when to deploy them and when not to. Rather than understanding what bearings are for, the technician-programmer has a problem X, reads a blog post that says, "I solved Y by adding some bearings," and thinks, "gee, X is similar enough to Y that adding some bearings may work."
They add some random bearings, some screws, some cams and driveshafts here and there, without any deep understanding at all of why you ought to use these things in a design... and over time build a Rube Goldberg monolith with 10,000 moving parts sticking out randomly that nobody actually understands at all, so as a result everyone is terrified to work on any part of it for fear of inadvertently breaking the other 60 hidden dependencies of that piece.
For substance: Consider how this is all part of the effort by management to make programmers feel as interchangeable and insignificant as possible.
And it's not even in the name of quality. Plenty of software out there breaks because of its multiple single points of failure in the form of dependencies.
Yes, totally. I can understand why management does this. Coders have a very high turnover rate as a profession. When your genius Clojure or Rust programmer leaves, it's much harder and more expensive to find a replacement than when your PHP or Rails programmer quits.
This then incentivizes new coders to take up one of the popular languages, because that's what most of the advertised jobs are for, creating a self-reinforcing positive feedback loop.
The thing is, it actually doesn't matter. No one needs to be a certain way. Cool things will be done by people that can and will do those cool things. In the end the prescriptivity of this whole space is only ascribed by people internal to the whole system: externally nothing is 'supposed' to be done. This is just a symptom of more people programming.
The thing that will speak the loudest is actions and results. If people don't like depending on modules, don't. If you do, do. Eventually everything will be lost and forgotten like teardrops in the rain.
This "that's just like, your opinion, man" attitude may be fine if you have no worldly goal affinity and are planning on spending the rest of your days in a remote mountain monastery. If you don't care about achieving any particular goal, then yes, all possible attitudes are equivalent. But that sort of handwavey quietism is not actually relevant to almost all people's situations.
If you do have specific short term goals -- such as, say, "building a profitable software product before we run out of money" -- then all methods of approaching the task are demonstrably not equal. Some will work much better than others in terms of achieving that goal.
We can study the empirical results of the numerous attempts already made by other people and thus avoid repeating their mistakes.
Specialize. Pick a subfield of CS that you really enjoy, that really resonates with you -- machine learning, NLP, algorithms, whatever -- and become an absolute expert at it. Know your subfield inside and out.
Start thinking about what specialty you want now, today. Do some research, talk to professors and other students, etc. By the time you are a senior, you should know what you want to specialize in and be well on your way.
Then, you can get a job that actually requires deep knowledge. Since you actually care about this, you'll quickly become bored and frustrated if you take a standard startup code monkey job where you are expected to write Rails or PHP code all day, but then that's the only experience you'll have. You'll also be competing with less well educated programmers who are willing to work for less.
'"The Excel development team will never accept it," he said. "You know their motto? 'Find the dependencies -- and eliminate them.' They'll never go for something with so many dependencies."
In-ter-est-ing. I hadn't known that. I guess that explained why Excel had its own C compiler.'
Considering the fact that Excel had to run on multiple versions of Windows and also run on the Apple Mac I can understand why they would hate hidden dependencies.
I've experienced the joy of trying to make sure Win32 code runs on Windows 3.11 (with Win32s), Windows 95, Windows 98, Windows Me, NT, 2000 and XP.
Generally it all works, but sure enough there are times when a small code change breaks one or more of those platforms.
Luckily these days things are much better as Win32 is a lot more consistent.
The entire Javascript ecosystem is a huge catastrophe. It will collapse any time soon. It's complex, fragmented and no one really likes it. There are a dozen different tools to get started. No one even understands how to get started easily. There are no fundamental tools. Everything is changing every week. You can't just build a product and then rebuild it even a month later. Nothing works anymore a month later - your dependencies have changed their APIs, your tools have different flags that do different things, there are new data models that you never needed and shouldn't even care about.
The developers are in high stress. Devops engineers are in even higher stress because they get to see what developers don't.
It's a huge mess and my advice to "prefer core-language solutions to small abstractions to small helper libraries to general libraries to frameworks" (http://bit.ly/1UlQzcH) hasn't been more relevant than today.
Software should be developed using least amount of complexity, dependencies, effort and using fundamental tools that have been and will be here for the next 20 years. Cut those dependencies, you don't need them. They're here today and won't be here tomorrow.
So, I suppose you could do something like this instead.
function leftPad(str, width, pad = ' ') {
const actualWidth = Math.max(str.length, width);
return `${pad[0].repeat(actualWidth - str.length)}${str}`;
}
And that would do a leftPad pretty well, and be reasonably robust to stuff like the required width being less than the string width, the padding character being multiple characters long, and so forth. It doesn't do any type-checking of course.
It also doesn't work on older browsers - both string.repeat and template strings are new. You could fake it with string addition, but addition behaves oddly in the case your arguments are numerical, whereas template strings handle that. There's also a trick where you can say (new Array(desiredLength + 1)).join(' ') to make a string that is the appropriate length, but you've got OBOEs to worry about if you're not paying attention (Array.join puts the character between the elements, so you need an n+1 array for an n-length string). Also, at least on some browsers, Array.join is pretty cruddy, and you really ought to construct the string with an old-fashioned for loop.
Javascript has all kinds of weird corner cases and lots of browser compatibility problems. The fact that someone's written a decent implementation of something that should have been standard in the String object means I don't have to worry about it.
Of course, I do have to worry about stuff like losing access to left-pad when someone throws an npm tantrum, or dealing with future build issues if npm becomes untrustworthy. A cryptographically sound package manager seems like a reasonable want, especially after this week's issues.
But if your take-away from this whole problem is "meh, javascript devs are lazy", you're missing the point.
In a way this shows what a great job NPM did at making it easy to publish packages. It's so easy that people decide to package up extremely easy functions.
As a python developer I would never publish a small package, simply due to the overhead of setting up a PIP package.
Oh God yes. After being spoiled in npm-land for a couple of years, I went back and tried to re-release a very small python module I had written before. I simply couldn't do it. Oh well!
Very true. Although pypi is in a better position than npmjs.org on a few fronts (including security), it's also such a pain in the arse to deal with.
Packaging is by very far my least favourite part of python.
> In my opinion, if you cannot write a left-pad, is-positive-integer, or isArray function in 5 minutes flat (including the time you spend Googling), then you don’t actually know how to code.
Spoken like someone who writes functions in 5 minutes that I find bugs in later.
Just because a problem is simple to describe informally doesn't mean it is simple to implement without bugs.
It would be trivial if the language lent itself to trivial solutions and there wasn't a culture of trying to anticipate and catch really messed up error cases that results largely from user errors.
In Python I would write ´if num > 0´, if it matters that it is an integer I would cast it to int, and that handles everything you should be handling at that level. If your user passes in a list instead of a number type, the code should crash because then there's probably an input error somewhere. If a user has monkeypatched a base class to make a number into something else the code should also fail and you deserve whatever happens to you.
So, apparently some guys managed to build a system where it is very easy to re-use small parts of other people's code and now the author is complaining that "too much code re-use is happening" ?
I'm fairly old, so I remember the complaints a decade or two ago that people had where "We can compose hardware from IC's and you don't have to know what's going on inside and it's all standard and just works! Why can we not do that with software?!?! (of course that ended up with things like CORBA and DCOM, which was all wrong)"
aaaand here we are in a situation where code re-use is actually happening on a wide scale and now you're complainig about that?
28k lines in an empty project? ha, how many lines of code does the preprocessor generate for
#include <stdio.h>
I haven't actually measured, but I bet it isn't that far off from 28k lines.
The elephant in the room here is Javascript. No other scripting language has been so bloody awful in the past that a five-line module that saves you typing `array.prototype.forEach.call` or something that papers over the language's awful idea of equality comparison with three functions has been "useful" or "more than five minutes coding without references".
Granted, these modules don't do such useful things, but that's the environment their creators were immersed in.
What concerns me here is that so many packages took on a dependency for a simple left padding string function, rather than taking 2 minutes to write such a basic function themselves.
It takes more than two minutes. That little module has a test suite, if you're including it then you have some assurance it does what it says it does. If you write it you've got to worry about whether it works or not.
As a dot net developer I avoid adding packages and references at all costs due to all of these same reasons. Usually for something simple like this left pad example, I just copy the code and put it in to my project in some sort of class of helper functions or extension methods.
Seems like a lot of these basic Javascript functions need to be built in to javascript/node itself or consolidated down to a single package of common core functions that extend Javascript. Like padding, is array, etc. As others mentioned, these are fundamental things in other languages.
Have we forgotten how to program? No, we have changed the way we program. We leverage the work of others to solve ever more complex problems.
When we do this we get more than just the simple functionality we require.
We get all the testing that comes from a package being used by thousands of projects.
We get updates when people find a better faster way. We get to lower the number of lines of code we have to maintain.
Yes, these are simple tasks but do we gain anything doing these simple tasks ourselves?
I find there is a finite amount of focus I have when programming, and I'd rather spend that solving the bigger problem.
This reminds me of a discussion I overheard between two professors when I was an undergrad.
Prof 1 "This new language makes it so much easier to x, y and z"
Prof 2 "Yes but what's the point, I can do all these things in Java"
Prof 1 "I could do all these things in NAND gates, but I'll get more work done if I have this tool."
In the case of left-pad, 2538464 of its 2550569 downloads last month are attributed to dependents of the line-numbers package (https://www.npmjs.com/package/line-numbers). So it would appear that relatively few people directly rely on left-pad, which highlights the importance of vetting the dependencies of dependencies.
DEPRECATED. This is a rather silly package that I do not recommend using. It's easier to copy the ~20 lines of code of this package and customize that code, rather than downloading and learning how to use this package.
What's the problem of writing a function with a few lines of code and exports as a module?
I think it's totally fine. Like other people said, it's the mindset we borrow from Unix, do one thing and do one thing well. The function would be well tested, and could be reusable.
I don't understand why so many people just require lodash into their project (when they start project) while they only use one or only minimum set of the functions. I mean lodash is a very great library with clean and well tested code, but it's also quite bulky like a big utility lib, and for me most of the time I only need one or two of the functions I would just go to npm and find a module just do that thing.
Also, Node.JS modules can be used browser side. On the browser you only want to include code you're using so you're not making the client downloading excessive code. A large 500kb module with one useful function is worse than a one 10kb module with the one function you need.
When I was prevented from upgrading Xcode/Swift for a project last year because of podfile dependencies, that cemented in my mind that every time I include a dependency, I'm putting myself at risk.
If I can spend half a day writing the code myself, I will -- because I know it will prevent headaches down the road. Yes, yes, I know any code I write adds the probability of bugs down the road. But at least the roadblock would be of my own doing, and not dependent on if/when the package maintainer upgrades their stuff.
There are lots of useful titbits often left out of a languages corelib.
Back in the day we would have built up various pieces of library code, which we would have utilised rather than continually guessing best practices. For example, the isArray method cited may be trivial but is also non obvious. We'd probably have something like that in our library of useful snippets.
Sometimes we may have shared the code on forums and the like and people would copy and paste the code, sometimes into their own libraries. We would locate them by browsing known locations or inefficiently querying search engines
Now we utilise a central resource and simple tools to have a global library of code. Instead of searching far and wide we query a handful of tools that effectively does the copying and pasting for us.
How that can be considered a bad thing is beyond me. It's not a question of knowing how to code, it's a question of using your time effectively.
Granted, there is the problem of so-called modules being removed and dependencies breaking. This can be alleviated by vendoring your modules, a simple task with most dependency management tools.
Personally I think that published modules should persistent indefinitely based on the license the code utilises, although I'm not clear on the actual legalities of the recent npm issue (although if it's due to a trademark complaint, I don't see how it would ever be enforceable for completely unrelated code in any slightly sane country).
Maybe we've forgotten how to /just/ program. Everyone bangs the drum so hard of "let github be your resume." Incentivizing putting every brain fart you ever had out into the universe instead of just keeping it to yourself.
The problem is not that we forgot how to program, the problem is that we never actually learned how to create a good programming language which avoids these problems.
Just look at the whole build system and module hell of C and C++...
I totally like the language C++ but I hate the tooling around it with a passion. As long as you only need stuff from stdlib then you are fine, but as soon as you want to create some custom library for use in your other projects? Which should compile on Linux and Windows? Either you use make and Visual Studio solutions or you have to fight with cmake... Just thinking about it makes me angry.
For this reason I just don't use C++ very often. It is way easier to just use Python because most stuff is just an 'import library' away and you can concentrate on your actual program instead of fighting against the build system.
But god forbid you actually try to ship your Python program to your customers...
Why do I as a programmer have to deal with all this crap? I want to focus on programming and not on library management amd writing make files. My CPU is idle 95% of the time so there is enough processing power which could solve these problems.
I have high hopes for Rust and it's integrated build/module system!
It sounds like your problem with C/C++ tooling is that it's hard to support Windows. The reason for that is simply that microsoft bundles its own proprietary C++ toolchain with the platform-specific tools that Windows developers need. Windows developers don't want to install a second posix-like toolchain, and os x/bsd/linux developers don't want to install a Windows-like toolchain because in both cases the new toolchain wouldn't integrate as well with the rest of the system, and it would add unnecessary complexity to the operating system.
The problem with proper unix tools, the ones that do one thing well and integrate with other programs as a part of their design, is that they make doing cross platform work a problem, because those other programs might not exist on other platforms. GNU Make works just fine on Windows, but you're not using just make when you write a makefile. Makefiles typically depend on GNU Make, pkg-config, gcc/clang (or another compiler with the same CLI), coreutils, bourne shell, and an assortment of a few other utilities. That means that Makefiles by design will only work on a single platform.
The other operating systems have essentially made themselves into the same platform for these purposes through the use of system package managers. I've heard Windows is kind of getting one? More like homebrew than what we have in linux/bsd-land but that might still help. But maybe Windows developers will still be reluctant to install software with MinGW dependencies. That would be unfortunate, but fair; I wouldn't install a package that had Wine/Mono dependencies just because some asshole wanted to use their Windows build system on Linux! Hell, I avoid installing smaller things with a Qt or GTK3 dependency simply because none of the software I use needs those and I don't want to clutter my installed-packages list for something that isn't important.
I'm not a huge fan of Rust's Cargo. The list of requirements that the Rust team had for their package manager and build system made sense, and if I had to design a tool that met those requirements, it probably would have looked quite a bit like Cargo, but it makes me sad that the most reasonable design is that which abandons the unix-like simplicity of the python and C (and perl?) toolchains.
Of course we learned how to create good programming languages which deal with this problem. A very good and thorough approach was introduced with Modula-2, which, as the name hints to, had modules as a main focus point. Building Modula programs was an easy and quick process. The module system lives on today in the form of Golang - one of the founders was a student or Wirth.
I totally agree. The npm ecosystem and the generally accepted practices in js these days come close to the bash style of composing & chaining a bunch of solid, specific little programs together to solve problems trivially that would have been a bear to do in many other platforms like java, C++, etc. I cringe at the idea that you should find yourself implementing l-pad in 2016, at least for the vast majority of projects that a typical engineer will encounter in their career. A similar case can be made for something as small as "is positive integer".
While I agree, there are two counter arguments:
1) As noted in other comments, this is a reflection of the core libs of javascript not covering enough of the basics. This is a subjective thing when it comes to where to draw the line, but wherever the border lies between "core" and "community" libs, on the other side you start running into things like "L-pad" and "Lib-L-Pad" and "L-pad2". If there's a great enough fundamental need, you experience a lot of crowding where lots of people offer up their solution to this, and reconciling that across larger dep. chains that disagree on which one to use can become a real burden.
2) Have you ever had the conversation with an auditor (PCI, HIPAA, etc) that your application is actually built on top of a huge substrate of software written by unaffiliated 3rd parties? And that between your own iterations you could easily have different versions for any/all of them? It's a difficult conversation. Much less the explanations to QA about why a build failed because lib X was updated to a new hotfix version in the 14 hours since yesterday's build, after a couple hours of wasted time of initially suspecting your own diffs, and trying to navigate through all the indirection between the actual stack trace and what actually caused the blow-up...
The loudest people in the Node community have been evangelizing this practice for as long as I can remember. This shouldn't come as a surprise.
The argument, "If I didn't write it I don't have to think about it" is ludicrous. I just have to point at the left-pad incident disprove the premise of this argument.
The analogy of building things with a bunch of npm lego blocks is laughable. Those responsible for advocating the use of trivial functions by acquiring module dependencies are leading the masses astray.
"But, If I find that there's a bug in a module I can AUTOMATICALLY fix it everywhere!"
No.
You still need to assess how the change to that module impacts any code that depends on it. Just by updating a module and posting a "minor" bug fix can lead to other bugs that RELIED on the behavior as it was originally written.
It's simple, write your own trivial functions. Test them. Maintain them.
P.S.
Another module that can easily be in-lined to every code base you own. (3 million downloads this week).
Just because it's a one-liner doesn't mean that this code is not complicated. If you forgot something in this one-liner, you could have something that's not working in every case.
> There’s a package called isArray that has 880,000 downloads a day, and 18 million downloads in February of 2016. It has 72 dependent NPM packages. Here’s it’s entire 1 line of code: return toString.call(arr) == '[object Array]';
How anyone can deal with JavaScript for more than 5 minutes is absolutely beyond me
Sensationalist stuff. isArray doesn't depend on anything, but lots of other packages depend on it. Why? Because it's actually kind of hard to tell the difference between an array of things and an object with integer keys. The Array.isArray function wasn't added to the language until ES5, so you need a way to shim older tests.
If you're writing only for modern browsers, you don't need it (if you actually visit the code in question you'll see it defaults to Array.isArray - in that sense, it's a polyfill or whatever). But if your code might run on old browsers, it can't hurt to have it, and it cleanly encapsulates a trick that you no longer have to remember the syntax for.
...maybe it's time a committee of really smart people sit and sip through all the most used modules below N lines of code or smth, and just write an opensource JS-stdlib, hopefully merging in top 30% most used methods of Lodash too? Node/NPM is a great example of why too much democracy and decentralization is bad. Just gather some experts and have them centrally plan a "standard library" then impose it as "industry standard" have a recommended "no fragmentation policy" like "no forking" and "use it all or not at all", the hell with your web app's need for "performance"... even a few hundred Ks of code will not hurt anyone nowadays ff sake...
I even consider PHP a "more sane" language because you at least have most of the useful utility functions in a global namespace and everyone uses them. Of course, the real ideal on this is Python's solution: a nice set of standard libraries that you know are baked in, but most of them you still import explicitly - hence it's pretty easy to write even large Python applications that have a small and comprehensible number of dependencies!
(And more generally: our strike for "more efficiency" in programming is stupid imho! I'd always take a less efficient solution, even "less safe/tested", if it's more "understandable" and "explainable" and sometimes, paradoxically, making things a bit more monolithic and centrally planned makes then orders of magnitude easier to reason about for our tiny ape brains...)
There is also a big programmer efficiency to be gained by having batteries included library: The new person coming to a project has learned the same batteries already, and it need not to re-learn new set of minipackages the last person working on a project has used.
In fact this should be the key selling point for stdlib to JavaScript. It actually saves money from their employers if trivial functions are mostly in the same place.
However pushing a stdlib is difficult there is a lot of hate for Python's "kitchen sink" library in JavaScript world which is totally misplaced. It maybe founded on fact that you should then minify the whole package to use one function in it, but this could be solved with dead code elimination (with proper type checking) in the minifier.
> there is a lot of hate for Python's "kitchen sink" library in JavaScript world
Is there? Most "js people" have never had the pleasureful experience of working with a "pleasantly opinionated" language like Python and the similar feel of its libraries, even the 3rd party ones.
Really, there's something really awesome in the Python ecosystem that should be spread to others before node takes over the world: If done right, being opinionated ("one way to do it") and centrally planned (a BDFL or smth like that) is A GOOD THING for an open-source language/infrastructure project! And it's also good if you extend the philosophy to the libraries and frameworks. Don't ditch the baby with the bathwater!
But I have a really hard time translating this message to "node-speak" unfortunately... Thanks for the "saves money" tip though, never thought of that, will try it ;)
After working with JS/Coffeescript/Python/Ruby for years, Go has been an enormous breath of fresh air for me. Of course it's all a matter of preference.
Man, I have to quote: `In my opinion, if you cannot write a left-pad, is-positive-integer, or isArray function in 5 minutes flat (including the time you spend Googling), then you don’t actually know how to code.`
You would be surprised of how many developer these days have 'afraid' to write such functions, or how lazy they are, they found this thing and just add to a project, then push to some google list and the project got a lot of followers, and in the next day the project has changed about 90%.
I saw this happen over and over again in this ecosystem, this is insane dude.
A lesson I learn is: you _need_ to read every module source code before add to any project, the NPM ecosystem has so many "shits" out there. You cannot trust in any npm module, recently I tried to trust in a module that has more than 5k stars, but I found a such ugly bug on that, that I feel my soul die, and I swear I hear the angels cry, thats not how open source supposed to be.
These days, seems that people dont care about the 'bug free' as long as it work a half way.
A Java left pad implementing a Padder class instanciated by a PadderFactory that you can get by a PadderInjector from a PadderRegistry all that accepting an AbstractPaddingString.
Then one in Ruby where you monkey patch the string to that it pads, then add an AUTO_PAD settings set to True by default and a fluent API to chain padding.
Then one version PHP version containing pad_string, then real_pad_string that deals with unicode, then real_pad_string_i_swear_this_time that that call str() on parameters automatically.
Then a Haskell one with a monad and a trampolin and some |=> giberrish to mix it all.
Then a Go one that creates a string padding microservice. With a docker file, just in case.
And the last trend in Python, an aio_string_pad which gives you an asyncronous padding coroutine, but under the wood calling run_in_executor, and optionally auto start an event loop.
So any developer could make this in five minutes, but for some reason they can't verify whether or not it works? That doesn't make sense.
In reality it could take an hour to get this working properly, but it does take only a couple minutes to verify that the solution here is correct. There are certainly good reasons for not adding extra dependencies to your project, but trading a known amount of time to check that an existing project does what you want for an unknown amount of time to redo it yourself is probably not a great bet.
Unzipped, the source code for GNU coreutils is 30MB (zipped 4MB). This is a great example of a collection of single purpose functions you should never rewrite yourself. There's only one dependency if you want to use them because they're packaged together. With normal desktop code, 30MB doesn't really matter and you can link only what you need. Can you do that with the usual Javascript package managers/bundlers, or would you need to send the whole 30MB package to the client to use one function from it?
It's not that anyone has forgotten, it's that a lot of people never learned how to in the first place. Every programmer community is riddled with these problems but the NPM world seems to be the worst. The ruby gem "american_date" annoys me to no end. It's just a highly-specific implementation of Time#strptime. Gah
node is can be a good idea. But people don't take JavaScript programming seriously. Most libraries objectively suck.
Google, authors of V8 and #1 subject matter experts on V8, have published coding standard. Does someone use it in the node community? No. Everyone loves "standard", a lousy standard that allows everyone put a badge on their github page while still having a code base full of vomit.
JSDoc. A great solution for documentation. You would expect major libraries to adopt it, or a similar thing. But again, no. Major libraries such as busboy do not use them. The documentation resembles a napkin.
Then everything else: input validation, error handling, consistency... etc. Take "request" for instance, one of the most widely used libraries. The state machine it implements is inconsistent. You abort a request and get a timeout, you can abort a request without starting it and get an exception. Issues that will drive you insane while debugging.
Express, one of the most widely used web frameworks on node.js. Do this on a route: setTimeout(function(){ throw new Error(); });. Great, now you have broken out of the error handling context. Great job.
Node libraries suck all across the board. It's the PHP of the 21st century. There are exceptions, like: lodash, bluebird, and others.
We have. I spent some time optimizing [0] a String.repeat function over at Stackoverflow and I was surprised that many developers today don't know what they are doing, including core team members [1]. Specifically,
function repeatString(str, len) {
return Array.apply(null, {
length: len + 1
}).join(str).slice(0, len)
}
So I don't think this is an example of core team members who don't know what you're doing.
This function repeats a string up to a certain length. e.g.
repeatString(foo, 10) = foofoofoof
This could entirely be what the developers needed in their code base.
The fact that it doesn't repeat a string N times, is at best, an example of bad naming.
The issue here is that someone unrelated to the core team grepped the node codebase for the term they wanted and threw a function up as a SO answer without understanding what it did.
One of the things I've enjoyed the most while programming in Go for the past couple years is the standard library and how much the Go community emphasizes keeping external dependencies to a minimum. For most projects I find myself using few if any external packages.
Now of course there are times you'll want to reach for a library, say for something like an http router, and up until recently the dependency management side of Go has been lacking. But when a pattern or library arises that many find useful the core team is open to pulling that in to the standard library if a strong enough case is made, for example in the context package (https://github.com/golang/go/issues/14660).
The main reason there is a fetish for these micropackages is a fetish for github stars. The formula seems to be: Overly generic + Simple + Javascript = Lots of stars.
That being said, there is something to be said for using these micropackages. Left padding a string is easy, but you might just have forgotten about that one edge case where in browser X and language Y you have to do things different. It's not really the case here, but things that seem simple at first often turn out to be hard because of some edge cases. One might hope these edge cases are solved if they use a library.
I call this kind of attitude the "Tea Party" of JavaScript development. The reason why we currently have JavaScript tooling fatigue is exactly because Tea Party developers insist on writing everything themselves instead of trying to build a better abstraction. The lesson here isn't not fewer dependencies: it's managing dependencies. NPM should not allow someone to arbitrarily remove modules that other's may be depending on. It's like building a bridge and them deciding to remove it after a whole city now depends on it.
It's more like planting a bush in your front yard, and for some reason the metropolitan train system decides to use your bush's shadow to tell the time of day, and builds that into every aspect of the entire city's transportation network. Then you cut the bush down and everyone starves.
Here's the funny thing that gets forgotten: in a lot of commercial software, 3rd party dependencies need to go through legal to get properly vetted and attributed and so on. This also usually requires an engineer (to be able to answer things like if it's dynamically linked or not, etc.).
As staid and corporate as it might sound initially, it's a very smart thing to do. One screw-up with licenses could be catastrophic. Are you all really checking that carefully?
I can't even imagine how any sort of proper legal checks could be done with a trillion micro libraries.
I'm content and happy not knowing what the fuck this is all about. Pad left? Are you kidding me? If your tower og babel fell because of your reliance on some script kiddie's toy project, I am happy and content knowing you get what you deserve. Law of leaky abstractions, motherfucker.
If I engage in as much hyperbole as the author, where does "write it yourself" stop? If I'm working on a team of two, should we each write our own left-pad? How about a team of three? Four? Five? Fifty? At a certain point, it makes sense for that to be written once for the project. We spent 30 years in software engineering trying to figure out how to get code re-use, and now that's it common and widespread, we want to go back to NIH?
I think about this every time I write a UTF-8 decoder. It's a task which is simple enough that I can bang it out from scratch about as fast as I can find a library that does it, so I can't really stomach adding a full-blown dependency to deal with it - but it's also just complex enough that I have to think about it, write some tests to make sure I didn't forget something, and wonder whether this implementation is any different from the last one I wrote.
I pretty much came to the comments to post something similar. I'm sure there was a long comment thread here on HN discussing the perils of NIH this week. This seems like the end point of the alternative.
NPM modules can be used on browsers. On browsers, space is a premium.
Why would you want to install a 500kb dependency that has only one function you need, when you can install a 10kb dependency that has it?
Would you want each of your five 20kb dependencies to re-implement the same 5kb function, increasing the code you must send to the client by 20%, or would it be more optimal for each of those dependency to use the same 5kb function?
The author rants about practices of developers from different programming environment, without experience, without figuring how things came to be. If he did give an effort to think from the perspective from Node.JS developers he’d have addressed the previous two points.
This is like going to a friend’s house and complaining everything is put in the wrong place. It would have been wise to immerse in Node.JS conventions and observe for a while before making comment.
EDIT: Reply to scrollaway:
I've also understated the problem.
Let's look at the problem in the current Node.js environment, it's not uncommon for a web app to have 20 dependencies, each of those have 10, and each of those 10 have 5. That's a total of 20 times 10 times 5 = 1000 dependencies in total.
Let's say you were to remove a 10 line library function that's "standard library-like", used by 15% of those dependencies, and have each of the existing dependencies re-implement that in each of those dependencies that uses it.
15% times 1000 times 10 lines is 1500 lines of code.
So if you're going to troll a solid argument by nitpicking, do it properly and get the details right.
FYI, you can reply to my comments if you click on their timestamp, in case the reply box isn't showing up
> So if you're going to troll a solid argument by nitpicking, do it properly and get the details right.
First of all, I don't appreciate you calling me a troll. Someone who mentions "5kb functions" clearly has no idea of what 5kb represents, period.
And second, this is not a solid argument at all. There is logic behind it, which is why we usually have high quality stdlibs in popular languages. Javascript lacks that. So instead, javascript gets this absolute mess of an ecosystem, where things like this can happen.
Several people have brought up various issues with the way it's done now. Dependency hell, for one. Lack of discoverability, which in turn leads to duplicate libraries, which in turn leads to a system where you have 1000 dependencies, but despite your "modular" idealism you still have hundreds of duplicate dependencies. Not to mention all the duplicates of different versions of the same dependency.
This "saving lines of code" math is completely broken exactly because this stuff is not in the stdlib. The various issues with the JS ecosystem mean that the actual results are nowhere close to ideal, and have a net negative impact.
I also love when people mention how much "one time cost" is saved and back it up with similar math, completely forgetting the amount of continuous time wasted downloading and installing these dependencies every time.
You're mixing up development time & bandwidth used by client browser. I was addressing the latter. Not having to download a 500 kilobyte library in order to use a single function is a big reason why the node.js ecosystem is the way it is today. Yet it is completely unaccounted for in the article. The author did not go into how the Node.js community came about to this practice in the first place, before going on a rant about it. That is what I was criticising. Whether having minimal modules maximises utility in society is completely irrelevant, because I was addressing perspective and etiquette.
You are a troll because talking about the actual number I used and how good a grasp I have in my mind completely misses the point. You're criticising the javascript ecosystem, I get that, but you completely missed the point of my comment. If you're not going to try to understand how Node.js conventions came to be, what it's strengths and weaknesses are, what the trade-offs made were, then it's inappropriate to focus on particular weaknesses of its practices.
It's like going on about how Objective-C development sucks because it's hard to do on Linux. But you're missing the point about what Objective-C's strengths are that attracted Objective-C developers in the first place.
The author should spend some time acclimating to Node.js best practices before writing this article. And you should do the same before you start knocking down straw men to prove how right you are.
It's quite ironic how you're lecturing people about dependencies and bandwidth when you yourself don't seem to have a good grasp in your mind of how immense even 5 kilobytes is (you talk about 5kb functions). 5 kilobytes (well, KiB) is 5120 bytes. That's about 200-300 lines of code. If you're talking about minimized code, maybe up to 500 lines.
While a 5kb function is immense, I work on a legacy code base and I have seen functions up to 2000 lines long. You possibly overestimate the quality of code that people in business actually write. I've worked in a few places and all have parts of the code which I would thought were made up if submitted to the dailywtf.
Also, have you ever seen a commercial library require >2MB of minified JS? Here's an example:
That file is almost 2MB minified (5.6MB unminified).
And that is is only 1 of the few different files required for our dependency on it, there is a bunch more javascript/css also required. Would I use that library on a new product? I'd probably advise against it due to the size, but a legacy product has dependencies and you don't have the freedom to rewrite everything all the time.
A lot of the discussion here really isn't talking about the problem at hand.
From the perspective of Babelify users, a major bug was introduced into software they depended on. I don't know how much money in developer time was lost due to this but it would almost certainly be in the thousands of dollars.
And it could have been a lot worse. It could have been something more complicated than left-pad. The author could have introduced a vulnerability or outright malicious code, or been hacked and done the same, and millions of people would have downloaded it and run it.
Arguably, small modules are good if you control them. Maybe they are more composable, maybe they enable better testing, maybe they encourage code reuse. I am not going to argue for our against small modules.
But none of the positives of small modules matter if an unknown developer who you have no reason to trust can change or unpublish the module out from under you. It's irresponsible as developers to risk our employers' and clients' businesses in this way for a function we could write in five minutes.
Its interesting how everyone used this as a chance to attack the small modules approach. This approach definitely has downsides, but the problem caused by leftPad being unpublished wasn't one of them.
If jdalton declared a jihad on arrays tomorrow and decided to pull all array related functions from lodash, we would have the exact same problem.
If kriskowal decided that Q must mirror built in Promise and published a version that does this tomorrow, we would again have the exact same problem.
There is only one connection between this problem and the small module approach. As the size of a module decreases, the number of dependencies increases and so does the number of authors that produced your dependencies. With the number of authors increasing, the chances that some author decides to go rouge or protest for some reason also significantly increases.
Therefore, its irresponsible to use this approach with a package manager that allows an old, established module with many dependents to be unpublished so easily by the original author.
Would people automatically update to the new versions of the libraries you mentioned?
The problem with unpublishing is that it changes an existing version. If instead of depublishing a correctly versioned empty module named leftpad was pushed to npm (increment major, because a non-implementation is incompatible with an implementation), there would not be half as much pain.
As long as unpublishing exists, micromodules increase the "attack surface" to this specific method of changing the contents of a specific published version.
This is a non-issue and taking focus away from the real issue. The issue is the security hole that NPM opens up when a namespace can be grabbed up by anyone if the original developer pulls out.
Personally, it's not about the 'original' developer pulling out, but about the simple question: who or what do you trust?
1. The developer,
2. The content distributor,
3. The code (by git SHA ref perhaps),
4. The contract of the code (using formal verification),
5. The legal contract with the code supplier.
And if you trust 1, do you expect of him to sign the stable releases using GPG tags?
All the focus on OSS nowadays seems to be on 1, but as professional engineers, shouldn't we focus more on 4 and 5?
I know I wasn't interested in doing any JavaScript until they started modularizing it and npm was that on crack. Possibly once the kids had their shiny new npm, some went mad with power and took it too far. Take the good with the bad though as the JavaScript language is known for.
I'm not really sure whether or not I should to add my voice to the din, but I feel like this whole thing is more a problem with npm and what it allows vs. what it encourages (and the rather paltry standard libraries in Node & browsers), rather than a problem with developers feeling entitled to not have to write their sorters and padding functions.
npm actively encourages structuring projects as many tiny individual modules and dealing with the resultant dependency trees and deduplication. Both of these things (along with the ease of publication) combine to encourage people to share their packages.
They make it incredibly easy to consume code from other people, but but at the same time provide a similarly low-barrier mechanism to retroactively change published code. That combination seems like a way more deserving topic of criticism than the unending refrain of "developers these days are so lazy".
I find NPM packaging ridiculous. Awhile I go I used NPM on windows, where the folder hierarchy became so deep it broke windows file handling. I could not delete the modules folder. I had install a npm package which allowed me to delete it. I think this is fixed in new versions by flattening the hierarchy, but still.
There's so much wrong here, it can't even all be seen at once. The part we can perceive is only a projection from a higher dimension into our three-dimensional space.
The insanity is that JavaScript doesn't have a standard string library. More a case of forgetting how to design a programming language than how to program.
Agreed.
Most development groups should be building a local collection of utilities that contains all of these snippets, and most importantly, some documentation of what they do and some unit tests to demonstrate that they are correct.
No need to have global dependencies on small snippets that really should be in a core library anyway. C has libc, Java and C# have the creator's (Oracle or Microsoft) standard set of libraries, Python has the "batteries included" stuff in all the different distros. And so on. All of these snippets rightly belong elsewhere, not in packages.
And even if you did get them added to the right libraries, I guarantee you that you will not get rid of the need for a collection of small, and somewhat random, in-house functions, classes and libraries.
After reading this, i don’t think i’m capable of ever complain about OOP again. Whenever you think you’ve seen it all, web developers always manage to come up with something worse. The only thing more depressing than this is the comment section below the article.
Insight from this story: If it is important that a thing be done correctly, it should not be made so easy to do that it will end up being done by people who shouldn't be allowed to do it.
I may have just invented a rule for choosing programming language and systems there.
Another explanation for the flourishing of these one function modules may be the fact that some people feel a kind of high/achievement from being able to say they have a published module out there. A sort of bragging rights if you will.
"I may not be Mick Jagger, but I am playing in a band and we did have some gigs (before we had that 50% decline in audience when my mother got bored and stopped showing up)"
- just about every twentysomething before coding became a desirable career
It beats the alternative - pulling in some huge package with lots of little functions, all of which end up in the output. At least you're not loading huge amounts of unreachable code into the running system.
In languages with linkers, the better linkers would discard all unreachable functions. Then came DLLs. Use one function in a DLL/.so, and the whole thing has to be loaded. Go and Rust are usually statically linked, reflecting the fact that pulling in big DLLs was usually a lose. You rarely have two different programs on the same machine using the same DLLs, except for some standard low-level ones such as the C library.
I wouldn't even think of looking for a package that does something as simple as the ones mentioned. If I need to pad a string, my first thought would be to create a new function, not look for a package...
The problem with shared hyper-modularisation is that it assumes the name of a function is unambiguous with only one valid implementation. If that were true, it should be encouraged but given it isn't, the practice will crushed by ambiguity and unintended consequences.
My app might well have an is-positive-integer function but it will include a range of context dependent choices about e.g. floating point, infinities, zero, null, "9", "9 ", "09", boxed numbers, string representations exceeding js max int, etc. etc.
I can't take this author seriously at all, one of his most egregious cases is the is-positive-integer library which until today had around 30 downloads in the last month.... No one was really using this and furthermore of course you can find bad/iffy code on NPM for the same reason you can find bad/iffy code on github. ANYONE can publish code. I could write a similar module for any other library, publish it to their repo, then scream LOOK! python is stupid and python devs are stupid.
I firmly believe that building on what has already been done allows for much safer code written at a quicker pace. Why should we constantly repeat ourselves? Also by using npm modules we can abstract logic and prevent someone on the team from going in and modifying it for their own use. It is a documented/constant function that we can use knowing exactly what it does. Is it a better world where everyone just copies code out other's repos and then has to include the licence/docs/tests along with it? It's much easier to just pull in the repo which contains everything and make it trivial to see where a function came from.
People are blowing this whole thing way out of proportion just because it makes a good headline "11 lines of code broke node"... You can all try to shame people who build on what's come before and chant "Not invented here" but I'll opt to build on what is proven to work instead of rewriting everything. At the end of the day that's what ships products.
This isn't forgetting to program, it's a deliberate choice as to how library code needs to be organised. We can disagree about choices made, but let's not assume other people aren't able to code as well as we think we can code.
That being said, I think this is a perfect example of where a good concept (small, tightly scoped modules) is applied dogmatically at the cost of the codebase. It's the node.js equivalent of AbstractRequestFactoryFactoryFactory stuff you see in Java, and the Mock Messes you see in Ruby.
Every single dependency you import should make you nervous. You would look at it like hiring someone. You are letting go of some responsibility. You are letting go of some control. The end result might be better than you could do in-house, but it might not. But you are already hoping that it's better, and giving up control.
Save these for the things you can not do in house, like NaCL. Don't write that yourself.
But string padding, sorry. Any developer worth their salt would laugh at adding a dependency for this. It's irresponsible, and comes across as amateur hour.
This is a simple case of optimizing for responsibility. The inexperienced programmer does not know how to do this, because they have not spent enough time being responsible, and having to deal with the fallout of managing responsibility poorly.
An experienced programmer carefully manages responsibility. A simple function, that is easy to understand, easy to test, and easy to reason about, is something that makes more sense to either pull all the way into your codebase, or write yourself.
Years of doing this means that managing dependencies should never be just slap it into your packaging system and start using the function. If you are on the line for making the wheels turn for a large scale platform that is directly connected to a monetary transaction of some nature, you will quickly find yourself preferring to remain responsible for everything that you possibly can control. There is certainly enough that you can't control to keep you on your toes.
I think this is the fundamental thing people don't understand about NPM and JavaScript, and the web in general:
Nothing is included. And that's a feature. The web is not trying to be the kitchen sink. That's iOS. They provide high level APIs for everything. And as a result, the platform is architecturally only as vibrant as Apple can make it.
Now maybe you're happy with Apple, maybe you love the iOS APIs. But if you don't, you're stuck. There's not a rich bed of alternative view layers that you can draw from to build your own vision of how software should work.
Node and the web browser strive to be lowest common denominators. They provide just the very basics: a document format, a very simple programming language, and an http server. The rest is up to you.
That's pretty scary, and so the JavaScript world has dabbled in frameworks. In the end all-inclusive frameworks are antithetical to the spirit I'm talking about, so things trend towards small modules that do one thing pretty well. People go overboard sometimes. I would argue left-pad should just be a copy-pasted snippet rather than module. But that's not a sickness in the community, it's just developers feeling out how far to go.
If you like every application to look the same, and you don't mind being chained to enormous and restrictive standard libraries and frameworks, then you will hate JavaScript. If you like writing software from scratch, forming opinions about all of the different parts of the system, and allowing each application to be built out of different parts, out of the right parts, then you should give JavaScript a closer look.
This is flawed. Basic things like checking if something is an array or a integer should be part of a stdlib. No one is saying whole frameworks need to be included, just basic functions, partially because JS isn't sane in the first place. Nothing in C's stdlib forces apps to look the same.
You are talking about extremes here, but there are nice place somewhere in the middle, where open source libraries/modules exist to help you solving complex problems and you are free to write whatever you want.
The fact that anyone has to even think about code size is the real problem.
Yes, downloading a giant JS lib for one function is insane, hence the ton of tiny dependencies.
However, it is equally insane that basic SDK expectations found in every other language has yet to come to be pre-implemented by the JS engine in the web browser itself. Some basic code ought to already be on the localhost the moment your app arrives.
Have to put a little clarity over my "so sad yet so true".
When the developers of such a serious library as React start to depend on a third-party one-function module made by some Dick-from-a-mountain (a russian idiom that means a random person who did nothing significant but tries to show out in all possible means), that means React developers are even more to blame than that Dick-from-a-mountain himself.
If you make any really popular piece of software, you absolutely must have a failover plan. No excuses for not having it.
But what's even sadder is that this issue had spawned a new wave of pseudo-elitist attacks on the entire JS dev community. Calm down guys, things like that could have happened for any language that has a widely-used centralized package system (Perl's CPAN, Python's pip, Ruby's gems etc).
Let me repeat that again: languages don't make software bad, people do. Just don't let such Dicks-from-a-mountain rule over your own modules with elementary stuff like leftpad, and you'll be safe.
Here's a proposal (that I'm sure others have come up with in the past) -- why not create one big, community backed "batteries included"-type module that would implement all the small, commonly used functions. This could combine all these ridiculously small libraries and greatly reduce the number of necessary dependencies for a package. Extending the standard library should be just that: standardized. If the entire community focused on one project like that they could just as easily write the same code (but with smaller package.jsons, less require()s, and less time spent learning new libraries/searching for the right libraries. In fact, it would be great if something like that could be packaged as a standard node module so you'd get the same sort of quality assurance as you get with official projects.
<rant>
The problem here is JS developers have baked in the notion of having NPM as the alternative to Google + StackOverflow + own thoughts. It's really a no-brainer (literally) to just slap another package than to bother thinking about what a piece of code does, the edge cases and pitfalls. Move fast and break things, right?
Sure there was some argument about Unix philosophy, small module doing one thing and does it very well. Did anyone bother considering the quality of most NPM packages? Quality is not reflected with passed Travis CI or extensive community testing and feedbacks. Not at all. Look at the those packages on apt-get. They are modular and robust. They do what they were supposed to do.
Now take a long hard look at the state of NPM. What do we have? People clamoring for reusability and whatnots. Most of them don't even know what they're talking about, just reciting the latest hip statement from the Internet. Being mature about development means accountability for what you do, not pushing shit around that you don't even have knowledge off. As a self-proclaimed polyglot, I love JavaScript as a language but not the ecosystem. It's like watching a dog chasing its tails:
- Endless loops of discussion that help stroke the egos but not improve anything else.
- Craps for resume/repository padding, not for actual developers to use.
- Bandwagon mentality that just pushes the latest fad along, and the herd towards the cliff.
The notion that JS developers are kids playing grown-up, has been reinforced with this NPM incident. If we want to discard that notion slowly, we need to be more mature developers. It's that simple. Here's what I think we could do:
- Have a clear idea on what dependency you need. Browser, IDE, terminal etc are dependencies. Basic type checking is not.
- Be better craftsmen. Roll and maintain your own toolboxes. Only share a working hammer, not a broken nail or a wood chip.
- Note that for each package you publish, thousands more hours would be spent on learning, adapting, using and reporting mistakes. Collectively, the current community wastes so much time with finding the right things to use. Often times, we learn much more by playing with code, even posting on StackOverflow. That's hands-on, `npm i` is not.
- Own the code better. The idea that teams like Babel and React devs with all the brilliant developers choose to put their eggs in a private corp's whims is just scary. You can't hope to build robust software while playing Jenga tower.
I agree that some packages might be too much but I don't think `left-pad` is one of them.
I wrote my own left-pad for a project I'm working now and I had to revisit a few times for tiny problems and lack of time to write tests. I would definitely use `left-pad` module if I knew the existence at that time.
Separate from the discussion of whether super small modules and hundreds or thousands of deps are a good idea, is the point of operational stability.
Putting on your devops hat, whatever your dependencies, from a reliability and reproducibility point of view, you should control your reliance on unexpected decisions of npm or third-party developers. A lot of the panic with npm issues comes from people blindly using the npm registry and then seeing breakages, with no safety net. I hate to say "I told you so" but this is an issue we worried about a lot when considering Node productionization last year: https://medium.com/@ojoshe/fast-reproducible-node-builds-c02...
I am posting the article "No silver bullets" again, in the wake of the npm fiasco. I think it is an essential reading for every programmer, every year!
* If you're in favor of the micro-module approach, you shouldn't be relying directly on NPM, and should have something like Sinopia in place. After all, external code isn't the only thing you're vendoring, right?
* Micro modules are fine - but your application code should depend on a privately published utils module whose entry point is a prebuilt distribution of all your external micro-modules exposed through a facade. Your utils module deps are all installed as dev dependencies to avoid the Fractal Nightmare.
* Yay, now you have your own 'standard library' which still manages to leverage the NPM philosophy of distributed code. And if some twit decides to throw a tantrum, it will only impact future builds of your custom std lib - and you'll know about it at build time.
Yeah look at all these modern languages that do so much for you, like .empty() on an array -- have we forgotten how to do simple comparisons?! You could just take a second to consider the properties of an empty array, namely it contains no items (.count() == 0).
My point being, if something is a completely reusable and basic feature, a dependency is totally just. I remember a few years ago when I and all devs I knew (which weren't many, I was maybe 17) had our own libraries to include in all personal projects we made. It contained features we had to look up once and from then on just automated and imported, stuff like password hashing or input checking. This went out of style as single-programmer programs are going out of style, but the little useful features are still there.
One can argue that modules are good, but them depending blindly on newer versions like that was bad dependency management.
I say this because I strongly believe that reinventing the wheel is unnecessary and can bring more problems than not.
There are many examples, and I could come up with a made up one, but here's a very real bug that I debugged from another programmer not so long ago:
So, he came up with a JNI for SPSS's C library, applied it correctly, and got haunted for lots of months with an unsolvable bug. The problem? he miswrote the final copy of the file, and sometimes, some bytes where copied twice.
He tried to solve this problem for a long time (and eventually lived with it because SPSS was still resilient to this)
Is this concrete example of a 'module' ridiculously short? yes, but my logic still holds IMO.
This shows the beauty and success of npm in making it very easy and cheap to publish small modules.
It's not that we can't write a left pad function ourselves. It's that we might easily miss an edge case or make a mistake in doing so.
The author seems to be hung up on a preconceived idea of what a package "should" be without actually offering a compelling argument for why a single short function can't be a module.
Yes, every dependency you introduce is a liability. But so is every line of code you write. I'd much rather take the risk on a shared library which can be audited and battle tested by the entire community.
If a function is so easy to write, it's trivial to check out the module's code, review it does the right thing, and then lock the dependency.
One should think of these requires as like .h files and the underlying code as something like a .c file. They're public definitions with potentially changing underlying code.
It's good to have small packages. Don't forget that the underlying ECMA script is changing so the implementation of these 'libraries' are (or will be) different over time from what they used to be. If somebody finds a faster way to do the method, then it will be done.
Finally, anybody who has used js in the real world understands how many corner cases there are and how difficult it is to make durable methods (i.e. how to know if an array is empty - which requires like 4 different conditions).
On DigitalOcean instance I cannot even use browserify+minify+babel6 cause npm process is killed by host (it consumes > 512Mb of RAM). So I have to manually run browserify + babel then minify. Still it produces 500kb of bundle.js :D
The problem is not of small modules. The problem is lack of dependability. If the language patrons stand behind a set of modules and guarantee continuity and availability, it really doesn't matter what is in them and the world can continue regardless of how insane the module or the whims of any one author. This is not about the technical merits of having or not having a stdlib. The module in question could have been anything.
Making this about is-positive-integer misses the point that this is a social/political problem not a technical one. A language ecosystem must address concerns of business continuity as first class concerns.
My current programming project, my goal has been to do as much in-app as possible. Does that mean I'm more likely to have bugs in my own code? Yes. But I've learned a ton doing it, and I know that my code doesn't have a giant monster of bloat hidden behind some random dependency somewhere. And yeah, that means when I wanted to handle email, I learned a heck of a lot about how programs handle email. Did it take more time? Yup. Education is time well spent.
I've got two dependencies besides the basic framework my project is built on: A scheduling engine, and a database interface. I eventually hope to factor out both.
That's true, I don't. But if your deadline is too tight to do it right, maybe you need to reevaluate your project plan.
I mean, I can certainly see using more dependencies than I'm using. Mail is a great example of something that probably should be handled by dedicated, well-tested code. But to carelessly dump piles of third-party dependencies in your project to save a few minutes, as talked about in this article... just crazy.
Somehow, someone always forgets that engineering is about trade-offs, and so every few years we can an indignant series of articles about how stupid and ignorant today's programmers are and how we should all go back to the same processes that they called stupid and ignorant 4-5 years ago.
Relying on reimplementation, copy-paste, npm shrinkwrap, or various other ways of importing third-party code into your repository results in the following advantages:
1. You know exactly what goes into your product, and can audit everything for security, performance, or coding standards.
2. You often end up importing less, as third-party modules may have functionality that you don't need but other clients do.
3. You can modify the resulting code to add showstopper functionality, even if upstream doesn't want to.
4. You aren't subject to the whims of someone removing your dependency from the Internet or replacing it with a version that does something you don't want.
Relying on lots of little libraries installed via package manager gives you the following advantages:
1. You can easily install & try out modules that other people have written, letting you test out new features on users more quickly.
2. You can share code with other modules that have the same dependencies, often reducing the overall size of your system. This is important when there's a cost (eg. download size) to your total bundle.
3. You have less code for your engineers to read & maintain.
4. You can easily track licensing & contact information for your dependencies.
5. You automatically get any new features released by your upstream dependencies.
6. You automatically get security updates and performance enhancements released by your upstream dependencies.
The last is nothing to scoff at: imagine if the headline, instead of 'left-pad breaks the Internet!', had been a security vulnerability in left-pad which literally broke the Internet. Imagine how hard that would be to fix if everyone had copy/pasted the code or re-implemented it. This is not an academic scenario either: remember "Nearly all binary searches and mergesorts are broken", published by the guy who wrote the broken binary search implementation in the Java standard libraries?
Always copying your dependencies into your source tree is not the answer to this, no more than always relying on npm modules was the answer to updating your dependencies. They both have pluses and minuses, and if you really want to be a good programmer, you need to weigh both of them. For my projects, I tend to use whatever libraries I need when building them out (via npm, if possible), and then periodically audit the dependencies to make sure I'm still using them and they wouldn't be better off incorporated directly into the project. I wish more products did this, but I don't control what other programmers do.
Without addressing the wisdom or lack thereof of including dependencies for small functions, perhaps the problem of disappearing/changing small dependencies could be solved with an option along the lines of
Which would just copy the dependency verbatim to <destination>. Maybe have a "<dependency> is 100kb. Are you sure you wish to copy the entire source to <destination> instead of using a 'require' reference? y/n" prompt for the inevitable silly types who'd do it with Angular.
Dependencies are one side of the problem. Unavailability of binaries and dogmatic use of dynamic linkage are the other side.
When I installed a simple lines-of-code counting tool through Macports the other day I accidentally opened the door to dependency hell as gigabytes of not even remotely related stuff started to build [1].
Without a doubt something is going very wrong with Free Software and package managers. On the other hand, never look a gift horse in the mouth so I may not even be the right guy to complain here.
I'm hoping that proper support for ES6 modules to enable tree-shaking bundle builds (in the short term), and HTTP2 with client support for modules (in the long term), will allow us to head towards a world where we converge around a handful of large utility libraries.
In theory, tiny dependencies was supposed to allow us to only include code we actually needed in our bundled code. Bu the reality is that everyone uses different tiny dependencies for solving the same problem. So you end up with enormous bundles made up of different solutions to the same tiny problems.
What would happen to all those NPM projects if GitHub is destroyed? I don't think it will close anytime soon, but lets say, a meteor shower hits GitHub data center or something along the lines of this.
I still find it hilarious that, upon the bulk of SCM use shifting to a DVCS, the first[1] thing people did is run out to build a central hub.
[1] For the pedantic, it probably was not literally the first thing; I'm sure some folks went to the rest room, others ate something, still others took a nap.
The thing with DCVS is not that central hub does not exist. It is that you do not need specific central hub. That means if github dies, I can take any of my git repos I checked from github and continue working with them, and create a local git server that would allow others to work with it. Of course, there is a convenience value in having central hub. The point is not denying that, the point is turning it from necessity to option.
Smart people created software that lowered the barriers of entry to making software. It was inevitable that not-so-smart people would eventually be writing software that others would build upon.
The author brings up an excellent point, but I disagree with the solution. We should of course reuse existing, well-tested code if it available, even for simple things like left-padding. The real issue here is that there is a module for left-pad alone. If it were something like StringUtils module with a bunch of commonly used string functionality, it would have been great.
What is it about the node community that triggered this misunderstanding of package management and code reuse?
Hell, the java programmers I work with seem to never use each others simple functions and instead recreate the will every single time.
As for the issue with such a short piece of code being reused by many, why score on number of lines? If it works and is a useful function is more important to me. I am not familiar with bundling within the usage the article covers but we tend to bundle like functions together and the compiler drops unused ones
I guess this is the way one is supposed to use node, preventing one to write non-DRY code. (Golang takes the exact opposite approach, having it's own drawbacks of course.) However, when using React, I kind of trust that the maintainers don't include packages than require 12 star projects, and if, that they fork this stuff themselves.
BTW, isn't that a Facebook project, so aren't they supposed to use a CI? ;P
I couldn't agree more. I've been using the new ES6 style module syntax for a few days now because a co-worker forced me to, so he would use my library.
I'm not convinced its worth it compared to the simplicity of commonjs and module.exports. You have to pull in babel, which has over 40k files to do all this.
Why are people destroying the beautiful simplicity that is javascript? Can those people please go back to java?
Regardless of whether micro-modules are good or bad, I think that if you are the owner/manager of a project, you should be able, given its full flat list of dependencies, to explain why each one of them is useful for you.
Every project I've seen that uses npm always required 100s or 1000s of dependencies.
If building or running your project requires something and you can't explain why, I think there's a problem.
On the one hand, I ,love how the JS community is constantly coming up with new [versions of old] things. Even though most of it is creative waste, it's still creative and out of that churn we get some pretty awesome apps and tools.
On the other hand, there's a lot of bad practices disguised as "simple" and "efficient." Using NPM for one line functions is a great example of this.
So funny, just few weeks back I had an argument with somebody about writing a simple functions vs. importing libs when you need less then 5% of the functionality. I am more convinced than ever that it is better off to have the least amount of external dependencies. Of course I would not want to rewrite a 2M+ LOC library with very complex code, but left pad is not one of those use cases.
I'd just like to point out that React does not have a dependency on left-pad. React has no dependencies at all[0]. devDependencies have no impact on third party consumers.
The author missed the point about modularity in Javascript.
Small packages done right, well tested = maintenable, reusable, stable code.
The problem does NOT comes from packages. The problem comes from un-publishing public packages and centralized repository server.
I have a java project with a lot of dependencies. Does it mean it's bad ? No, but if maven repos are closing tomorrow, my project will not build as well.
Why was it removed anyway? I agree that the ability to unpublish something is the real problem, but I wonder why the author actually unpublished it. I wonder if the author knew about all the projects that depend(ed) on it. Maybe he/she actually did it as an evil experiment, though a very interesting and eye-opening experiment. Does anyone know?
I'm content and happy not knowing what the fuck this is all about. Pad left? Are you kidding me? If your tower og babel fell because of your reliance on some script kiddie's toy project, I am happy and content knowing you get what you deserve. Law of leaky abstractions, motherfucker. Spolsky...do you read it?
Not just making them feel... I think this may be targeted towards recreuiters, who may be impressed by the sheer amount of npm modules someone has published.
Well it has always been like this. In early 2000s most developers I knew would simply copy-paste code from all over the Internet.
Nowadays you use NPM search instead of Google search
The fact is that lazy programmers are lazy. The methods change but the principle remains the same.
In 90s people typed in code from magazines and books.
I just removed the tilde and caret of all my dependencies (in my package.json) and maybe that's the way to go. Seal a local version of your packages and don't update unless is completely needed. But I'm still worried about the fragility of the package environment.
The core issue here is that there is no included standard package in Javascript. It happened because Javascript did not have authoritative implementation when it first began. Next ECMA should require to pack some batteries in to avoid this micro-module hell.
It seems to me that the JavaScript VMs should get together and start including standard library. That would also give the benefit that those would be highly optimized. They can keep it small at first and focus on number and string manipulation.
Part of the problem, in my opinion, is that packages are not subject to any review process. These dependency chains get out of hand because they are rarely top-of-mind.
Here's a quick rule of thumb. If it's the kind of function you would ask a candidate to write at the start of a job interview, you shouldn't be importing a separate module to do it.
.... isn't this a problem with node, and not developers? Wouldn't you say this is a symptom of a systemic problem of the framework that it is lacking common features that everyone needs?
When am I developing some large project for a long time, sometimes I find, that I have reimplemented the same function at several places (few years between implementations).
Everyone keeps mentioning the lack of a standard library for JavaScript as an excuse for this shit show. IMO this is just a futile attempt to mask incompetence.
Any modern standrards-compliant implementation of JS (following ES5 and especially ES6/ES2015 standard) already has everything that a sane programmer would ever need.
What actually happens when you try to update a required package, but it is gone from upstream? Is there no way to keep the existing package you already have?
There's nuance to the discussion that both sides are missing. People argue forcefully whether these small modules are good or bad, but I'm not seeing much evidence that they understand the other side.
First: why small modules are bad. Lots of dependencies complicate your build, and you end up with the dreaded diamond dependency issue. Failures in a dependency become more likely to affect you. It gets you in the habit of using prebuilt modules even if maybe it's not quite what you need and it would have been better to write yourself. With `npm` specifically, we've seen how its mutability can break the build, though that's about `npm` and not the idea necessarily.
I think most software developers' gut responses are that something is wrong and crazy in the npm ecosystem.
That said, there are benefits that this blog post and others aren't mentioning, related to the javascript situation specifically.
The first one is that javascript is a surprisingly difficult language to get right. Sure, the final solution is only a few lines, but which lines are hard. You have to navigate the mindfield that are is V8 in nodejs, v8 in chrome, spidermonkey, chakra, etc. I've had code work in Chrome before but blow up in IE, and it's really hard to track down and test.
The comments in the blog post are illustrative:
One line of code package:
return toString.call(arr) == '[object Array]';
Crazy right? And my first stab probably wouldn't have been to implement it that way. Why not:
(testvar.constructor === Array)
that a commenter suggested, which should be faster? Well another commenter said:
The constructor comparison will fail if the array comes from a different context (window).
I've run into issues before with cross-browser compatibility stuff, and it's frustrating and hard to test. If there's some de facto standard package that implements it for you, hopefully the community can iron out edge cases.
The other thing that people don't bring up, is that there's not much JS standard library, and in the browser context you have to send all your code to the front end.
So maybe you write these 11 lines yourself, and then another package writes these 11 lines, and another... it adds up. But if everyone uses the same package, the code only gets sent once and they all share it.
Lastly, people talk about how `sin` should be a part of a "trigonometry" package and not by itself. Well, again you're faced with sending a bunch of unnecessary code to the frontend. With webpack2 and tree shaking, or e.g. Google's Closure compiler, it can strip out dead code and so this issue will go away in the future, but we're not quite there yet. So package authors still bundle all these things separately.
That is what PHP has, and it received countless attacks on that approach during the years; although most of those attacks were due to the inconsistent function naming.
This is why one guy can now compete with say Google or Microsoft, because that guy uses code written and managed by more engineers then both Google and Microsoft have combined. Instead of paying hundreds of dollars to said companies, you can just NPM install "what you need".
Libraries/modules should help to solve complex problems where writing your own version is not optimal. Single line functions as modules, on the other hand, won't help you compete with Google or Microsoft and result in more problems than they solve. If a simple padding or "is this int positive?" is a complex problem, well, there are different professions than programmer's..
Seems that we need a tool to crawl all the repos that a developer owns and report lines of code in each package that they wrote. If there are lots of small packages, then this is not the kind of person you want to hire, except maybe to do PR.
I don't have an issue with modules or the Unix philosophy, I have an issue with using NPM for all these tiny modules. Hint: you can make your own modules, store them within the project, and add them to your package.json to be required anywhere.
This is acceptable because web developers expect a certain level of coddling. Many developers are quick to defend this insanity because they simply cannot see their own level of entitlement.
What's next? A hello-world package and Node tutorial about hello world program being include this package as dependency and call the exported function.
`std::min` isn't a library of its own, it's a function in the `<algorithms>` header of the Standard Template Library. The STL also encompasses a wide range of other things. There are some efforts to provide "the missing standard library" for JS, e.g. lodash.
Then the problem isn't that people don't know how to program, rather that js is missing standard functions. In the C++ world there is a large push to avoid rolling your own things, indeed even simple constructs like `for` loops are starting to be frowned upon.
"Immutability at the centralized authority level and more decentralization of package distribution is the solution, not 'write more functions yourself'."
what the fuck does that mean?? they just don't give up, do they... Fucking retards.
I remember how I have been downvoted to oblivion for comparing JavaScript madness with Java EE "packer's" paradise years ago.
The Programmer's Stone first essay is actual as it never been before.
Actually, it is a common pattern. When some activity becomes popular due to very low barrier to enter it will end up in a such kind of mess. It seems like nowadays everyone is either a programmer or a scientist and researcher.
This is the quality of their software and research.
There has been a reason why good schools taught principles (algorithms and data structures) not particulars (objects and classes). But since MIT and Berkeley dropped Scheme-based courses in favor of "pragmatic" Python (thank god not JavaScript) based courses we are heading to a disaster. Java madness taught us nothing.
History is full of examples where assault by mediocrity ruined the whole branches of philosophy, arts and crafts. Instead we have fastfood, mass media, social media and now this mass coding, which combines worst from mass and social.
Just try to compare things like Smalltalk or Plan9 or R4RS Scheme or Zeta LISP of Symbolics or with this stuff.
What's with these kids and their "calculators". Back in my day we used a slide rule and we liked it!
But seriously this is stupid. Programming shouldn't be the goal. Just because you can write a function doesn't mean you should. Every line of code you write is overhead that must be tested and maintained. I guarantee that if the author chose to hand roll code instead of using packages he'd have a lot more bugs. But he wouldn't know that until he hit some mundane edge case scenario in production.
"Every line of code you write is overhead that must be tested and maintained."
The problem is, that modules doesn't solve problems magically. Every dependency should be tested, maintained and kept online as well, by somebody else, whom you have no control of. In turn, you have a bunch of black boxes that can blow up your application anytime. Of course, another extreme of write everything yourself is also bad, so it's a trade-off that must be considered carefully. I've build many systems during my career and one of the biggest nightmare I encounter is managing dependencies.
> What concerns me here is that so many packages took on a dependency for a simple left padding string function, rather than taking 2 minutes to write such a basic function themselves.
> if you cannot write a left-pad, is-positive-integer, or isArray function in 5 minutes flat (including the time you spend Googling), then you don’t actually know how to code.
I'd probably get 2 of those wrong in some weird way, but I blame javascript. I mean without Google of course.
Good programmers understand the risks of making your system depend on something you don't have control really well; They know how keeping system complexity low is like an good investment which makes your life later easier (low maintaining costs).
Bad programmers stacks up technical debts such as including unnecessary dependencies until the system no longer works.
If you're new to technical discussions, people normally provide supporting arguments. Both of those handle sequences of tasks that need to complete in order. "No you don't understand" without any further info is not a response.
This post is too dismissive and confuses some topics.
"Packages", "modules", "functions", and other words can mean different things within different contexts. Well-known and tested functions are useful. Putting them in a "package" is just a consequence of how node/npm work. There should certainly be a much, much better implementation of code sharing, but sharing and using well-known, singular functions should be exactly what we are going for.
This is just more evidence that the unit of code is a function, not a module.
Eventually, given a pure enough language, every "package" could contain only a single function, and every function in a project could be published as an independently reusable unit.
Only if your modules are little more than textual inclusion and perhaps a mechanism to deal with name clashes. ML style modules are way more substantial than that, they can be used as units of abstraction.
Holy moly-- is-positive-integer/index.js:
I retract my previous statements that Javascript programmers are going down the same enterprise-y mess that Java programmers went down a decade ago.
They've already taken it to an entirely different level of insanity.
This comes up a lot when people discuss anything related to npm modules. It's easy to simply dismiss these trivial one-line modules as "insanity" and move on, but there's actually plenty of good reasons as to why many prefer to work with multiple small modules in this manner. This GitHub comment by Sindre Sorhus (author of over 600 modules on npm) is my favorite writeup on the topic:
https://github.com/sindresorhus/ama/issues/10#issuecomment-1...
TL;DR: Small modules are easy to reason about, and encourage code reuse and sharing across the entire community. This allows these small modules to get a tremendous amount of real world testing under all sorts of use cases, which can uncover many corner cases that an alternative naive inlined solution would never have covered (until it shows up as a bug in production). The entire community benefits from the collective testing and improvements made to these modules.
I also wanted to add that widespread use of these small modules over inlining everything makes the new module-level tree-shaking algorithms (that have been gaining traction since the advent of ES6 modules) much more effective in reducing overall code size, which is an important consideration in production web applications.
Small modules are easy to reason about
Yes they are, in the same way that a book in which every page consists of a single word is easier to understand than one with more content per page.
By focusing on the small-scale complexity to such an extreme, you've managed to make the whole system much harder to understand, and understanding the big picture is vital to things like debugging and making systems which are efficient overall.
IMHO this hyperabstraction and hypermodularisation (I just made these terms up, but I think they should be used more) is a symptom of a community that has mainly abandoned real thought and replaced it with dogmatic cargo-cult adherence to "best practices" which they think will somehow magically make their software awesome if taken to extremes. It's easy to see how advice like "keep functions short" and "don't implement anything yourself" could lead to such absurdity when taken to their logical conclusions. The same mentality with "more OOP is better" is what lead to Enterprise Java.
Related article that explains this phenomenon in more detail: http://countercomplex.blogspot.ca/2014/08/the-resource-leak-...
78 replies →
They really aren't, when I'm reading is-positive-integer(x) and wonder if 0 is positive I need to hunt down the definition of positive through two packages and as many files. And it gets wrose if both your code and one of your dependencies required 'is-positive-integer' and I have to also figure out which version each part of the code base is using.
If you had written (x > 0) I would have known immediately, it also wouldn't be doing the same thing as is-positive-integer(x) but how many calls to is-positive-integer are actually correct in all the corner cases that is-positive-integer covers?
And then there's the other problem with dependencies: you are trusting some unknown internet person to not push a minor version that breaks your build because you and Dr. is-positive-integer had different definitions of 'backwards compatibility'.
14 replies →
>TL;DR: Small modules are easy to reason about, and encourage code reuse and sharing across the entire community.
How does that even remotely applies to the "is positive integer" test, and even more so to "it's positive" and "it's integer"?
What's next? is-bigger-than-5? word-starts-with-capital-letter? add-1-to-a-number?
45 replies →
You'd have a good point, if all of those tiny but probably useful modules were given the use you're describing.
Discoverability though is so poor that most of those modules are most likely just used by the author and the author's co-workers.
If a typical npm user writes hundreds of packages, how the hell am I supposed to make use of them, when I can't even find them? Npm's search is horrendous, and is far from useful when trying to get to "the best/most supported module that does X" (assuming that random programmers rely on popularity to make their choice, which in itself is another problem...).
28 replies →
> This GitHub comment by Sindre Sorhus (author of over 600 modules on npm) is my favorite writeup on the topic:
https://github.com/sindresorhus/ama/issues/10#issuecomment-1....
If anything looking at sindresorhus's activity feed: (https://github.com/sindresorhus) perfectly supports the author's point. Maybe some people have so little to do that they can author or find a relevant package for every single ~10 line function they need to use in their code and then spend countless commits bumping project-versions and updating package.json files. I have no idea how they get work done though..
17 replies →
How is that superior to copypasting from a stackoverflow answer?
If it's a popular issue, lots of people had the same issue, many will be nice enough to add their edge cases and make the answer better, most will not. Same goes for contributing to a package.
With a package you would be able to update when someone adds an edge case, but it might break your existent code and that edge case may be something that is not particular to your system.
If you don't want to get too deep in the issue, you can copy paste from SO, just the same you can just add a package.
If you want to understand the problem, you can read the answers, comments, etc. With the package you rely on reading code, I don't know how well those small packages are documented but I wouldn't count on it.
The only arguments that stands are code reuse and testability. But code reuse at the cost of the complexity the dependencies add, which IMO is not worth the time it'll take you to copy and paste some code from SO. Testability is cool but with an endless spiral of dependencies that quite often use one or more of the different (task|package|build) (managers|tools) that the node ecosystem has, I find it hard to justify adding a dependency for something trivial.
1 reply →
The simple rebuttal to that is that modules that are collections of small functions are easy to reason about as well, and don't the downside of increasing the metadata management to useful function ratio nearly as much. Why have just an average (mean) function, when it makes sense to provide a median and mode as well? Even then, you might find that there's a bunch of extra math operations that are useful, and you might want to include some of those.
7 replies →
Just sugar-coated kool aid I'm hearing. Community benefits? First of all, I'm coding to get paid and this recent madness proved that JS ecosystem is semi-garbage. Back to the original question - do people really can't program that they need packages like left-pad or is-integer which had their own dependencies? Before writing those cool toolchains (which would likely work in a specific machine with a specific setting with all the real world testing the community has) can we at least pretend that we know the computer science basics?
9 replies →
They do NOT encourage code re-use because the effort required to understand your need, refrain from writing code, and hunt down a module in npm, far outweighs the effort to just write the code and stick it in your in-house utils library.
4 replies →
I think that there's a certain wishfulness bordering on naïveté to this pursuit. We tell ourselves that we are managing complexity by having small, well-tested units that are reused by an entire community. But software complexity continues to exist. We think we are mitigating the complexity of our own software. But the complexity really just shifts to the integration of components.
Anyone that has been around long enough has war stories about getting two relatively simple pieces of software working with each other. In my experience, integration problems are often the most difficult to deal with.
But this doesn't address the author's point that each module, each dependency is an additional point of failure.
4 replies →
author of over 600 modules and 1200 lines of javascript.
5 replies →
> Small modules are easy to reason about
Flip-side: it isn't easy to reason about a large and complicated graph of dependencies.
I'm not at all clear why this blog post is touted as evidence that the tiny modules approach is correct. I think it might be all the people after it congratulating him.
"It's all about containing complexity." - this completely ignores the complexity of maintaining dependencies. The more dependencies I have the more complicated my project is to maintain.
Dependencies are pieces of software managed by separate entities. They have bugs and need updates. It's hard to keep up to date.
When I update a piece of software I read the CHANGELOG, how am I expected to read the CHANGELOG for 1,000 packages?
Depending on a bigger package (handled by the same entities, who write one changelog, in the same form) is more straight forward.
I'm not saying this is wrong - but there's a balance here, and you must not ignore the complexity of increasing your number of dependencies. It does make things harder.
My problem with this, as an occasional JavaScript developer, is "discoverability" (as many others have mentioned). If I decide I need a left-pad function, and search on NPM, how do I choose which one is best? The one with the most downloads? Not always the best indicator of quality; perhaps it's just the oldest.
Not to mention the cognitive overhead of stopping programming, going to NPM, searching/finding/installing the module, then reading the documentation to understand its API. Isn't it simpler to `while (str.length < endLength) str = padChar + str;`? How can there be a bug in that "alternative naive inlined solution"? Either it works or it doesn't!
I don't see how your linked comment brings more to the table than the basic arguments for code reuse.
But naturally, with any code reuse there's a benefit and a cost to instance of internal or external reuse.
The Benefits for external reuse include ideal reliability as your describe as well as not having to create the code. The costs for external reuse include having your code tied to not just an external object but also the individuals and organizations creating that object.
I think that means that unless someone takes their hundreds of modules from the same person or organization and is capable of monitoring that person, that someone is incorporating a layer of risk to their code that they don't anticipate at all.
Percentage of module owners who you can't trust to not screw up their module: H
Risk of indirectly hosing a project with N module owners providing dependencies: 1-((1-H)^N)
Let's say H is very small, like 0.05% of module owners being the type who'd hose their own packages.
3 module owners: 0.15% chance your own project gets hosed
30 module owners: 1.49% chance your own project gets hosed
300 module owners: 13.93% chance your own project gets hosed
Keep in mind it's not just your dependencies, but your entire dependency chain. And if you think a module owner might hose some modules but not others, maybe H is actually the number of modules in which case 300 starts getting pretty attainable.
Upshot:
Not everyone is trustworthy enough to hang your project on. The more packages you include, the more risk you incur. And the more module owners you include, definitely more risk.
The micromodule ecosystem is wonderful for all the reasons described, but it's terrible for optimizing against dependency risk.
Takeaways:
Host your own packages. That makes you the module owner for the purposes of your dependency chain.
If you're not going to do that, don't use modules dynamically from module owners you don't trust directly with the success of your project.
I love collaborative ecosystems, but some people suck and some perfectly non-sucky people make sucky decisions, at least from your perspective. The ecosystem has to accommodate that. Trust is great...in moderation.
I agree with you, besides the tree-shaking (nice word btw). It's like NPM shaking the Christmas tree and then say "Have fun cleaning up the floor". Remember that NPM is not like apt-get, where the packages are Managed for you by an expert. In NPM you have to manage the packages! And where you can't have NPM and build dependencies, like in production, maintenance now becomes much harder!
My problem is one of productivity. There's already a standard library, and if it's a language I've been using for a while, I probably remember most of it. I can pretty much go straight from my thought of what I want done to typing it out, much like typing English. If you force a 'cache miss' and force me out of my head and into a documentation search, well, that's going to have a significant effect on my performance. If the function is well-named, it has less of a cost in reading the code, but there's still a cost, because what if it's slightly different? I have a pretty good feel for the gotchas in much of the standard library of the languages I use. I have to stop and check what the gotchas of your function are.
Yes, at some point the complexity cost of gluing together the standard library functions to do something becomes greater than the lookup cost of finding a function that does what I want; but I am saying that adding more functions is not costless.
Small modules are also often the result of dealing with different javascript implementations over the years. I've recently seen a simpler version of the left pad that would not have worked on multiple Safari versions or <IE6
The derision is unwarranted, due to a failure in critical thinking from otherwise smart people.
It's interesting to me that people find this convincing. I find it to be complete insanity. People need their libraries, but putting everything in tiny buckets is just not working. Why aren't people working on good utility libraries instead?
There's even some guy calling for a "micro-lodash". To me, as a Python engineer, lodash [1] is already a tiny utility library.
I guess it's also about the fact that JS is a pretty bad language. That you need a one-line `isArray` dependency to `toString.call(arr) == '[object Array]'` is crazy.
[1] https://lodash.com/docs
2 replies →
I get that author's point, but do you REALLY need a dependency module that tells you the type of something?
That's not a Lego block; that's an excuse.
1 reply →
The alternative to small modules is not no modules at all, it's libraries.
Why dont they put everything together and call it "commons" like in Java? (commons-lang, commons-io, etc).
That argument sounds like an enterprise architect explaining why the AbstractReusableBeanFactoryGenerator is going to increase productivity.
it reminds an apt proverb: missing forest for the trees.
Too many modules do not necessarily become a good thing. They may appear to get rid of complexity but in reality, you will have to face the complexity some level above and in fact, the sheer number of small modules will most probably add more complexity of themselves.
The reasoning makes sense for small modules that might change in the future, but as he says himself, most of his modules are finished and will never change. That makes many arguments in his post moot and the modules should probably be snippets instead that are implemented directly.
I get that author's point, but do you REALLY need a dependency module that tells you the type of something?
That's not a Lego block; that's an excuse.
> author of over 600 modules on npm
Nope.
Author of "is-positive-integer" here. I will admit the implementation is pretty funny, but I move-out all single-purpose utils out of projects to modules for a bunch of reasons. DRY is the most obvious one, but one that may be less obvious is for better testing.
I move out modules so I can write really nice tests for the independent of the projects I am using them in. Also, I tend to write projects w/ 100% test coverage, breaking out utils allows me to test projects easier and faster.
Also note, the implementation of this module changed a few times today. With it being open source and having the collaboration of other engineers, we ended up with a very performant version, and discovered interesting quirks about "safe integers" in JS.
"DRY is the most obvious one, but one that may be less obvious is for better testing."
Isn't that why we just write functions? Turning simple functions into entire modules just adds an unnecessary level of abstraction that helps nobody.
2 replies →
Two points:
- Breaking out is-positive-integer hasn't reduced the number of paths to test. You have not gained anything, you've added overhead.
- 100% test coverage is rarely a good thing. It is required for safety critical areas like avionics. I can guarantee that your JS code is not making into any safety critical environment!
6 replies →
Just to pick at a nit with you, it's a little meaningless to say "100% test coverage", without specifying whether you're talking about line coverage, branch coverage, path coverage...
This is especially true for one-liner modules in js, where any test at all might let you claim 100% statement coverage, without accounting for branches or loops within method calls.
With how weird javascript can be about numbers, it actually didn't surprise me that your module existed.
Actually, thats a good reason to use trivial functions like the one described. Hopefully the author has discovered all of the quirks in Javascript that might affect this. It will likely be a lot more tested than any version I would write.
As someone who spends 80% on the back end, I often get bit on the ass from JavaScripts quirks when I need to add some front end stuff.
That's interesting. JavaScript has enough quirks to warrant an is-positive-integer module.
1 reply →
It would be really, really great if this function was not in its own module, but was part of a larger library in which all such functions of related modules were captured, without the (cognitive and other) overhead of the separate packaging.
A fun r/programming subthread about this package: https://www.reddit.com/r/programming/comments/4bjss2/an_11_l...
Those are just the three top-level dependencies. The package has 9 recursive dependencies.
There's also a nice table explaining how a package called "is-positive" managed to reach version 3.1.0.
This is just baffling...
Similarly, the `average` package on NPM is one that I came across:
https://www.npmjs.com/package/average
It's hard to not stare at that in complete disbelief; someone thought that it was worthwhile to create a package for determining the mean of an array of numbers.
You know what's worse? Javascript numbers are all floating point numbers, which means integers are 53 bits long. So, you might think this library would try to address issues this can cause, but nope, this is the average statement you'd write if you didn't know was a mantissa was and had never heard of big.js, bignumber.js, decimal.js, crunch.js or even strint (which represents integers as strings because wtf not).
26 replies →
Well a bunch of standard libraries have sum defined, right?
JavaScript has suffered from a lack of a standard library for a while. Having a small package like this means that (in theory) everyone is using the same bug free version of summing instead of writing their own.
Honestly having JS start building a standard library at this point would be wonderful.
6 replies →
Sounds like the author was aiming for something to put on his resume: e.g., "Author of 25 libraries on NPM, some with more than 500K downloads." etc...
12 replies →
This sounds like a symptom of an inadequate standard library. I do expect to be able to call "average" on a list of numbers without writing it myself, but I expect that to be part of the language not a 3rd party package.
18 replies →
I think the scarier thing there is this line: "0.0.1 is the latest of 2 releases"
I wonder if there will be a 3rd release with an updated averaging function.
Edit: to be fair, it was an optimization from reduce to for loop [https://github.com/bytespider/average/commit/7d1e2baa8b8304d...]
I'm thinking that someone wanted to learn about building and publishing a package and the ecosystem so they made this computationally trivial thing as a practical exercise.
Pretty much every package management system gets cruft in it like this. Example: for a long time someone had uploaded a random Wordpress core install into Bower.
And 54 people found it a worthwhile package to install _this week alone_.
2 replies →
While it demonstrates the problem of npm lacking namespaces (such that the word "average" is wasted on such a trivial implementation)...it doesn't seem anyone was using that library
1 reply →
That package is, at best, average...
ducks
(Edit: typo)
1 reply →
Yes, but it's MIT licensed. We also need average_GPL to go with it.
1 reply →
It's a fucking nightmare, is what it is.
I wanted to use a javascript tool that would make my life easier, and when I looked at the npm dependency tree it had 200+ dependencies in total.
If I used that javascript tool, I'd be trusting hundreds of strangers, lots of which had absolutely no clout in github (low number of stars, single contributor projects) with my stuff.
And not just them, I'd be trusting that no one steals their github credentials and commits something harmful (again, these projects are not very popular).
It doesn't help that npm doesn't (AFAIK) implement code signing for packages which at least would let me manage who I choose to trust
YES YES YES!
In all the debate about this, why is the trust-dependency-fuck-show not getting more attention?
Every dependency you take is another degree of trust in someone else not getting compromised then suddenly finding all sorts of horribleness making it into your production environment.
It beggars belief!
This is more a reflection of how bad the JS language is than anything. Real programming languages have means of standardizing the most common UX and simple patterns into a standard library. Javascript is a consortium hell that never gets updated sufficiently, and has no good standard library, so NPM basically replaces the standard library with a thousand micropackages.
Also it is a lot easier to get it wrong in JS. Is it null? Is it undefined? Is it bird? Is it a plane? No it's a string!(but sometimes a number). Good programming languages make it easy to write is negative e.g. isNegative = (<0) where implicitly by the 0 and < it will take a Num and return a bool and this is type checked at compile time.
4 replies →
Yeah, there are a few shitty examples on npm. It's an open system and anyone can upload anything. The market speaks on how valuable those are. Cherry picking poor modules says nothing about the rest.
Plus, if you think that's too small, write your own broader module that does a bunch of stuff. If people find it valuable, they'll use it. If they find it more valuable than a bunch of smaller modules, you'll get 10,000 downloads and they'll get 10.
The module you roundly ridicule has had 86 downloads in the last month, 53 of which were today (at the time of this writing). I imagine most of those 53 were after you posted. So that's 40 downloads in a month, as compared to the express framework which has had 5,653,990 downloads in the last month.
The wailing and gnashing of teeth over this module is ridiculous.
DRY taken to the dogmatic extreme where everything is made up and the github stars are the only thing that matters.
This article touches on things that are wrong in the javascript culture. I always had this nagging feeling when working with NPM, this article brings it to light. For what it's worth I never felt this while writing Ruby, C# or Go.
It's the -culture- that needs to change here, not the tools.
The recursive folder structure in npm-modules was the first indication. At least Java had a single tree with com.bigco.division.application.framework.library.submodule.NIHObject.java
That recursive node_modules or whatever it is called was what made me hate this whole npm thing, specially because it is not centralized somewhere in my computer. And that means the same files a few times repeated on my drive just eating space. Being a Java developer I don't understand why the approach was not more like maven.
3 replies →
"This isn't your average everyday darkness.... this is Advanced Darkness." - S. Squarepants
You should then go dive into the passAll function:
https://github.com/tjmehta/101/blob/master/pass-all.js
It’s written in such a way that every time you call...
... there are something like 5 + 2n attribute accesses, 5 + 3n function calls, 3 + n new functions created, as well as some packing and unpacking of arguments, not including the actual application of the functions to the arguments that we care about. That’s in addition to the several functions defined in the dependent submodules, which you only have to pay for constructing once.
[From my quick eyeball count. These numbers could be a bit off.]
I'm more disappointed that it doesn't short-circuit the operation at all. It applies all the functions, THEN it determines whether all of them passed. Even worse, it uses `every` (which does short-circuit) to determine that all the functions are, indeed, functions, but apparently the ability to use that function to determine whether every predicate passes was missed.
1 reply →
Lol, nice, I wrote "pass-any" first. I then copied the code and replaced "or" w/ "and" to create "pass-all".
I will probably have go back and change this now that I know about it. In general though, not gonna a lie, I am not very concerned about micro performance optimizations.
1 reply →
Yeah, I thought the dependency thing was a joke (I mean, to have a package for is positive integer is already a joke, but come on)
Really
I think it actually is not. That's from some years ago and my memory of it is fuzzy, but at that time it was surprisingly hard to check whether a variable is a positive integer – maybe it was a negative one though and that was harder? You'd think it is just checking whether it is an integer and bigger than 0, or just checking whether it is bigger than 0. And it is. But to get that code to work reliably, regardless of whether it gets a string or float or an undefined, with the JS type system of that time and in multiple browsers, even the crappy ones, that took some time. There was one specific edge case involved.
Not that it was impossible, but I still remember having to search for it and being astonished that that was necessary.
Sure, should not apply anymore like that.
2 replies →
Yea. I don't think many argue against abstracting complexity into more easily understood orthogonal modules. But some of these modules aren't abstracting complexity. They are ludicrous. They are nonsense. They are a symptom of a very real problem with how JS is programmed, how people expect to program in JS, and the difficulty people have with decomposing problems.
So many people on this page have written about how these are well tested, performant, and correct modules. But, the modules aren't even correct in many cases, let alone providing incomplete coverage over edge cases or the slow performance and horrendous dependencies of many of the modules.
it's a module with exactly one dependent module -- by the same author. Daily WTF material but who cares beyond that?
Formerly 9 dependent modules... but who cares maybe it was a homework assignment or a joke.
I don't use NPM so maybe it doesn't really matter aside from the level of abstraction being implemented being relatively ridiculous.
However, if my build system had to go out and grab build files for every X number of basic functions I need to use, grab Y number of dependencies for those functions, run X * Y number of tests for all those dependent packages, AND then also fell apart if someone threw a tantrum and removed any one of those packages basically shutting me down for a day... then I'd question every single thing about my decisions to use that technology.
[Quick Edit] Basically I'm saying "Get off my lawn ya kids!"
2 replies →
This implementation reads almost as parody, although I don't suspect that the author meant it as such. If you didn't have a sense of what lurked behind the abstraction, it would be kinda beautiful.
I can't decide what's crazier to me: that such a package exists, or that JavaScript is such a ridiculously bad language that an "is positive integer" function is non-trivial to write.
I end up spending most of my working life working on other peoples code, rather than new features I end up debugging and fixing bad code. (I actually rather like it)
The majority of code I have ever seen is awful (20 years across large + small companies) but that is why I am hired to fix awful code so I am skewed. The amount of times I have seen people implement something simple, in a convoluted error prone was is unbelievable.
I know this seems ridiculous but when you see time and time again how people fail to do the simplest things it seems like a good idea.
I had several arguments about JS and I was shocked how many developers consider this platform great. I am not sure why these extremely bad practices are defended by devs, what are they getting out of it? I am only hoping we are moving towards more sensible development environment, there are many of them with better best practices and more sane libraries.
Sure, because just writing
is boring. Need more requires to require the requires...
Keep in mind though that absolutely not all JS programmers are like that. Not everyone wants to be an aforementioned Dick-from-a-mountain.
> They've already taken it to an entirely different level of insanity.
Why is this insane? What alternatives would be better?
You create unit tested functions and expose them to client code with a factory pattern, how is that so ridiculous?
I have an extremely hard time believing this module is not a joke, ditto for many other NPM modules.
What I want to know is how many systems does the author of is-positive-integer have implicit root access to?
Sounds to me like publishing oneliners on NPM is a trivial way to build a botnet.
Seems like the answer is "none" by default, from https://docs.npmjs.com/misc/scripts#user
> If npm was invoked with root privileges, then it will change the uid to the user account or uid specified by the user config, which defaults to nobody. Set the unsafe-perm flag to run scripts with root privileges.
1 reply →
I think this is some sort of cargo cult UNIX minimalism - do one thing and one thing only.
No, its a byproduct of programming-interview-via-GitHub-repo
For some reason I misread that as "do one thing and do one thing ugly". Either way, it applies.
Lodash does this (and versioning, and clean code, and tests) really well though.
require the ultimate
Philistine! You should just be doing i things.
I can't stop laughing. I think you have to admire the elegance of the concept as performance art though: this is cheap insanity. In fact, I've got to hand it to them, this is the most fun I've had looking at something programming related in a while. I recall the opening lines of SICP,
> I think that it's extraordinarily important that we in computer science keep fun in computing. When it started out, it was an awful lot of fun. Of course, the paying customers got shafted every now and then, and after a while we began to take their complaints seriously. We began to feel as if we really were responsible for the successful, error-free perfect use of these machines. I don't think we are. I think we're responsible for stretching them, setting them off in new directions, and keeping fun in the house. I hope the field of computer science never loses its sense of fun. Above all, I hope we don't become missionaries. Don't feel as if you're Bible salesmen. The world has too many of those already. What you know about computing other people will learn. Don't feel as if the key to successful computing is only in your hands. What's in your hands, I think and hope, is intelligence: the ability to see the machine as more than when you were first led up to it, that you can make it more.
Quoted in The Structure and Interpretation of Computer Programs by Hal Abelson, Gerald Jay Sussman and Julie Sussman (McGraw-Hill, 2nd edition, 1996).
It really is the fault of the language that something this fundamentally horrible is allowed to exist in itself.
When people ask why Javascript is terrible, show them this.
The same could be said for your post. When people ask why commenting on message boards is terrible, show them this.
If you don't want to write modules this way, don't. Nothing about javascript requires that you even read articles about modules that you don't want to use. Or read articles and then follow up by posting on message boards about articles about modules you aren't going to use.
Bang some code out instead. Your opinion of javascript is about as valuable as the opinion of the person who wrote the modules.
1 reply →
Counter-argument:
A good micro-module removes complexity. It has one simple purpose, is tested, and you can read the code yourself in less than 30 seconds to know what's happening.
Take left-pad, for example. Super simple function, 1 minute to write, right? Yes.
But check out this PR that fixes an edge case: https://github.com/azer/left-pad/pull/1
The fact of the matter is: every line of code I write myself is a commitment: more to keep in mind, more to test, more to worry about.
If I can read left-pad's code in 30 seconds, know it's more likely to handle edge cases, and not have to write it myself, I'm happy.
The fault in this left-pad drama is not "people using micro-modules". The fault is in npm itself: all of this drama happened only because npm is mutable. We should focus on fixing that.
> every line of code I write myself is a commitment
That's true. However:
Every dependency you add to your project is also a commitment.
When you add a dependency, you're committing to deal with the fallout if the library you're pulling in gets stale, or gets taken over by an incompetent dev, or conflicts with something else you're using, or just plain disappears. If you add a dependency for just a few lines of code, you're making a way bigger commitment than if you'd just copy/pasted the code and maintained it yourself. That's why so many people are shaking our heads at a 17-line dependency. It's way more risk than it's worth. If you need a better stdlib for your language (some of us write PHP and feel your pain) then find one library that fills in the gaps and use that.
> If you add a dependency for just a few lines of code, you're making a way bigger commitment than if you'd just copy/pasted the code and maintained it yourself.
This is a problem with NPM, not with dependencies. With different package management systems with stable builds and lockfiles, then you pin to a specific version and there is no way upstream can cause problems. A lockfile is a pure win over vendoring.
7 replies →
OR: you depend on a specific version of the library, that you know that works, and you have none of those problems.
20 replies →
Maintaining a dependency on a library should be much less effort than maintaining 17 lines of code. If it isn't that's a deficiency in your dependency infrastructure.
1 reply →
I wouldn't mind so much if these micro-modules were written in a style of thoroughness; heavily commented, heavily documented with pre-conditions, post-conditions and all imaginable inputs and outputs explicitly anticipated and formally reasoned about. I don't mind over-engineering when it comes to quality assurance.
Looking at that left-pad module though - no comments, abbreviated variable names, no documentation except a readme listing the minimally intended usage examples. This is not good enough, in my opinion, to upload to a public repository with the objective that other people will use it. It is indistinguishable from something one could throw up in a couple of minutes; I certainly have no reason to believe that the future evolution of this code will conform to any "expectation" or honour any "commitment" that I might have hopefully ascribed to it.
[EDIT: I've just noticed that there are a handful of tests as well. I wouldn't exactly call it "well tested", as said elsewhere in this thread, but it's still more than I gave it credit for. Hopefully my general point still stands.]
The benefits of reusing other people's code, to a code reuser, are supposed to be something like:
(a) It'll increase the quality of my program to reuse this code - the writer already hardened and polished this function to a greater extent than I would be bothered to do myself if I tried right now
(b) It'll save me time to reuse this code - with the support of appropriate documentation, I shouldn't need to read the code myself, yet still be able to use it correctly and safely.
Neither of those things are true for this module. It's not that the module is small, it's that it is bad.
(True that npm's mutability is a problem too - this is just a side-track.)
Completely agree here - the problem isn't micro-modules. It's partly just a lacking standard library for javascript and largely just exposing issues in npm that the community was pretty ignorant of until just now.
The whole "amaga, it's a whole package for just ten lines of code" is just elitism. Given the number of downloads on things like left-pad, it's clearly useful code.
Agreed as well. In fact, I would posit that this wasn't even really a problem until npm@3 came out and made installing dependencies far, far slower. Yet it was necessary; a standard project using babel + webpack installs nearly 300MB (!!!) of dependencies under npm@2, and about 120MB under npm@3. Both are unacceptable, but at least npm3 helps.
I want to plug ied (https://github.com/alexanderGugel/ied) here, which both installs about 5x faster than npm@2, yet deduplicates as well as npm@3.
No, no, no
Left padding is (almost in all languages) built-in, even C can do it with printf (edited)
The problem is not having a library that offers that, but having this micro-module thing as a whole NPM module. No other language does that.
If it was inside a string helpers, that's great.
But don't make me one single module for just left-padding (or is-integer-number)
I think there's two points related to that:
1) JS is unique in that it is delivered over the wire, so there is a benefit in having micro-modules instead of a bigger "string helpers" module. Things like webpack are changing that now (you can require lodash, and use only lodash.padStart).
2) JS's "standard" library is so small, because it's the intersection of all of the browser implementations of JS dating as far back as you care to support. As pointed out in sibling, a proposal for padLeft is included in ECMA2016. But we'll still need Left-Pad for years after it's adopted.
12 replies →
JavaScript much to my dismay has no sprintf. It's one of the most annoying omissions ever for server side work.
6 replies →
Uh, no Left Padding is NOT built-in in JavaScript. The proposal to add `String.prototype.padLeft()` was just added to ECMAScript 2016.
JavaScript had a very minimal standard library, it's pretty asinine of you to compare it to C or any other language with a pretty extensive standard library.
4 replies →
Another interesting argument for micro-modules:
https://github.com/sindresorhus/ama/issues/10#issuecomment-1...
Wow, I feel like I could have written this. Back when I used Python, I had a folder full of functions I would copy-paste between my projects. (And maybe some of the projects contained unit-tests for the function. I didn't always keep those tests in sync.) Updating them was a pain because inevitably each one would get slightly modified over time in each project separately. Eventually, I bundled all of them into a bundle of completely unrelated utility functions in a folder on my computer somewhere, and I would import the folder with an absolute path. Sharing the code I wrote was a pain because of how much it referenced files on my local computer outside of the project. I never considered publishing my utility module because all of the stuff was completely unrelated. I'd rather publish nothing than a horrifying random amalgram that no single project of mine was even related to all of the subject matter present in it.
With npm and the popularity of small modules, it was obvious that I could just cheaply publish each of my utility functions as separate modules. Some of them are about a few dozen lines, but have hundreds of lines of tests and have had significant bugfixes that I am very happy that I haven't had to manually port to dozens of projects. I don't miss copy-pasting code across projects, no matter how many claim I've "forgotten how to program".
15 replies →
What I see is that a module has a non-zero overhead in complexity in itself. That is, ten 10 line modules and twenty 5 line modules do not yield the same complexity. The modules themselves have a complexity overhead associated, and submodules have their own complexity overhead associated, albeit smaller than first party modules. That complexity is easily seen from the recent situation of unpublishing modules, which resulting in modules multiple steps removed having problems building.
So, when I read "It doesn't matter if the module is one line or hundreds." I call bullshit. There is overhead, it's usually just fairly small (at may event begin to rival the gains from using a module at that level), but that small amount adds up. Once you've had to deal with a dependency graph that's 10 levels deep and contains hundreds or thousands of modules, that small extra complexity imposed by using a module is no longer in total, and comes at a real cost, as we've just seen.
Other module ecosystems have gone through some of the same problems. There was a movement in Perl/CPAN a few years back to supply smaller, more tightly focused modules a while back, to combat the sprawling dependencies that were popping up. The module names were generally suffixed with "Tiny"[1] and the goals where multiple:
- Where possible, clean up APIs where consensus had generally been built over what the most convenient usage idioms were.
- Try to eliminate or reduce non-core dependencies where possible.
- Try to keep the modules themselves and their scope fairly small.
- Remove features in comparison to the "everything included" competitor modules.
This has yielded quite a few very useful and strong modules that are commonly includes in any project. They aren't always tiny, but they attack their problem space efficiently and concisely. Even so, I'm not sure there's ever a module that's a single line of code (or less than 10, given the required statements to namespace, etc), as the point is to serve a problem, not an action.
1: https://metacpan.org/search?size=20&q=%3A%3Atiny&search_type...
I'd say you're playing fast and loose with the term "edge case" there. I call that a bugfix, for a feature that does not work.
It doesn't handle edge cases, it doesn't perform well and it isn't well tested. There is also no documentation. Obviously 30 seconds wasn't enough for you to verify anything at all about this module (namely that it's complete garbage).
And just because some random guy didn't get something as trivial as this right the first time, doesn't mean nobody else can. Also the de facto standard library lodash already has padding utilities, made by people who have a proven track record.
I don't agree with the explosion of micro-modules. There's a reason the vast majority of languages doesn't have them, at least not at function level.
IMO in the Javascript world they're only there in order to minimize script size for front end work. See lodash & lodash-something1 / lodash-something2 / ..., where there's an option of using the whole module or just including 1-function long scripts, precisely to avoid the script size issue.
Is there a solution for this? I know that the Google Closure compiler can remove dead code, ergo making inclusion of large modules less costly in terms of code size. Am I missing some ES6 feature that also helps with this?
You're just trading in the complexity of the code you'd have to write for the delayed complexity of dealing with dependency issues down the line. It's a waste of a trade off for tiny things like this.
Padding a string is a one liner. I'm perfectly happy to write it, test it, convince myself that it works, and then forget about it and just use it.
Completely agree.
Combining this with isomorphic code, cross-browser development and micro-services.
I even wrote a blog post couple of days ago about it [1].
1 : http://www.drinchev.com/blog/increase-your-dependencies/
I agree with the points you've made, but I would also posit that adding a dependency using this mutable package manager is making a commitment to maintain the integrity of that dependency, which is arguably more work than maintaining the handful of lines of code.
What's wrong with copy/paste?
Why do you need an external dependency on something so small?
Nobody has forgotten. These people never knew to begin with.
NPM/JS has subsumed the class of programmer who would have previously felt at home inside PHPs battery-included ecosystem. Before that, a similar set of devs would have felt at home with Visual Basic. Seriously, go visit the comments section on archived copies of the PHP documentation. You'll find code of a similar nature. If PHP had had a module system 10+ years ago you would have seen this phenomenon then. Instead it was copy and paste.
This isn't elitism, it's just the way it is. The cost of a low barrier to entry in to a software ecosystem is taking in those who don't yet have software engineering experience.
Nobody should be surprised that NPM, which I believe has more packages than any other platform, is 90% garbage. There are only so many problems to solve and so few who can solve them well, in any language. Put 100 programmers in a room, each with 10 years experience, and you'll be lucky to find 1 who has written a good library. Writing libraries is really hard.
This is the answer, 100%. All it takes to publish an npm package, is the command npm publish, and you're done. So of course it is no surprise that there are tons upon tons of seemingly useless or tiny projects (gotta pad out that github profile for those recruiters!), or that there are then plenty of packages that use them.
Add into that the fact that:
1) Javascript has a huge number of developers, and is often an entry-level language
2) The developers on this thread (I like to think of HN as at least slightly above average) are divided whether having small packages / large dependencies trees is a good or bad thing
3) Dependency management is something that matters mostly to long term (professional / enterprise / etc) applications, which is a subset of programming, and I wonder if not a minority subset of node.js projects in general.
4) If I'm writing a throwaway app or proof of concept, and therefore don't care about dependency maintenance, using as many dependencies as possible is a major time savor,
and of course you get this situation, and it seems to make perfect sense.
Personally, I wish there was an NPM Stable, where packages underwent much more scrutiny and security in order to get it, but nonetheless, nothing I've read so far about npm really scares me given the the above context. If you are a dev creating an unmanageable dependency tree for your enterprise app, you're a shitty dev. That doesn't necessarily mean that NPM is wrong for being so open in allowing others to publish their packages, or that smaller / more worthless packages shouldn't be allowed to publish.
That said, I would really like to hear a response to this post, as I have limited experience with different package management systems.
The huge difference is that PHP package manager support namespaces and dependencies ARE FLAT. You cannot import 2 versions of the same package under the same namespace. Which
1/ forces package authors to write stable libraries
2/ forces dependencies to narrow the versions of their dependencies
3/ prevents name squatting to some extent. You cannot have a package named "forms" and then sell the name for real money, like seen on NPM. your package needs to be "namespace"/"name". NPM made a huge mistake with its gems like global namespace and it explains half the problems it is having today.
For a post that is claiming to not be elitist, it reads pretty elitist.
Can you expand on how to identify the class of programmers you're referring to? Are they the type that copy / paste code directly from StackOverflow? They lack a classical computer science education? They haven't worked on a large, enterprise-grade project?
From what I've seen, there's one division between programmers that's hard to overcome. Some see it as a tool to get certain results for a job. Some of them are bad, some of them are lazy, but most of them are good enough to get to their objective.
Others see programming more as an art. They take care to make the code not only efficient but also elegant. They'll read up on new and interesting algorithms and incorporate them in novel ways. They might often be behind deadlines, but when they are, they create things like GNU Hurd that inspire a lot of interest and lead to interesting results, maybe even teach people a few things. Their code is interesting to read. They tend to write the libraries that the first group uses.
Both groups contribute a lot, but it's not easy to get them to understand that about each other.
I think the languages themselves give rise to such code because of the lack of consistency and all the weird quirks everywhere.
When you can't even reason about the truthiness of a variable because the language coerces everything, of course things end up screwy.
PHP has had an extension module repository since 1999
Comparing NPM to PECL/PEAR doesn't make much sense when talking about PHP developers. With PECL, the overhead of building a module in C is waay too high to make it viable for micro packages. And PEAR didn't just accept any random stuff, they were shooting for the one-solution-fits-all libraries and not tons of user-defined micro libraries like ecosystems like NPM encourage.
Compare NPM to Composer/Packagist and you get a better comparision. I've personally seen only very few micro packages on Packagist, thankfully this never seemed to gain traction in the PHP world.
Going down the "lots of tiny modules" route is about these three things:
a) No standard lib in JS
b) JS is delivered over the internet to web pages in a time sensitive manner ... so we don't want to bundle huge "do everything" libs. Sometimes its convenient to just grab a tiny module that does one thing well. There isn't the same restriction on any other platform
c) Npm makes it really easy to publish/consume modules
d) And because of c) the community is going "all in" with the approach. It's a sort of experiment. I think that's cool ... if the benefits can be reaped, while the pitfalls understood and avoided then JS development will be in an interesting and unique place. Problems like today can help because they highlight the issues, and the community can optimise to avoid them.
Everyone likes to bash the JS community around, we know that. And this sort of snafu gives a good opportunity. But there many JS developers working happily every day with their lots of tiny modules and being hugely productive. These are diverse people from varied technical backgrounds getting stuff done. We're investigating an approach and seeing how far we can take it.
We don't use tiny modules because we're lazy or can't program, we use them because we're interested in a grand experiment of distributing coding effort across the community.
I can't necessarily defend some of the micro modules being cited as ridiculous in this thread, but you can't judge an entire approach by the most extreme examples.
I think b) is true only because JavaScript tooling cannot perform dead code elimination. Other languages have big grab-bag utility libraries like lodash that don't hinder performance because a linker or runtime can avoid loading unused portions.
Webpack 2 will support this: http://www.2ality.com/2015/12/webpack-tree-shaking.html
Yes it can https://github.com/rollup/rollup. Webpack 2.x can also do this.
Note for b): If you include libraries such as jQuery on you website via CDN, I believe browsers will be able to use the cached version even if they never visited your website before (given that they've cached this version from the same CDN before).
I don't see anything wrong with using a pre-made left pad function. Why waste time and lines of code implementing something so trivial when there is already a solution available?
However, I agree it is ridiculous to have a dedicated module for that one function. For most nontrivial projects I just include lodash, which contains tons and tons of handy utility functions that save time and provide efficient, fast implementations of solutions for common tasks.
Lodash includes `padStart` by the way (https://lodash.com/docs#padStart).
I think the article's thesis is essentially that every dependency your project pulls in -- which includes all the dependencies your dependencies pull in -- is a point of potential failure. I understand the "don't re-invent the wheel" defense, but the Node/JavaScript ecosystem tacitly encourages its users to build vehicles by chaining together dozens of pre-made wheels, all of which depend on more wheels, and each and every one of those wheels has a small but non-zero chance of exploding the next time you type "npm update."
(And, y'know, maybe it's because I'm not a JS programmer, but the notion of looking for a module to implement a string padding function would never have even occurred to me.)
Worse yet, a backdoor or legitimate bug in any of these module could leave huge exploits in the entire Node.js ecosystem.
7 replies →
The problem is not that, the problem is depending on unreleased versions, instead of simply depending on the version that was written when u wrote your code.
a Git submodule like approach would be much better
> Why waste time and lines of code implementing something so trivial when there is already a solution available?
Because it's so trivial? I can't wrap my head around why this is an argument in the first place. It makes no sense to bring in a module from a third party adding yet another dependency and potential point of failure when reimplementing it yourself literally takes as long as it takes to find the module, add it to package.json and run npm install.
People should be trying to limit dependencies where possible. Reproducible builds are really important if it costs you almost no time you should have it in your code base IMO.
People taking the DRY principle to the most extreme degree always makes for the worst code to debug and maintain.
This entire comment thread is such a breath of fresh air. I was beginning to think that I was that guy who was crazy for thinking that all of the people doing this were crazy. This thread is like my new support group.
> It makes no sense to bring in a module from a third party adding yet another dependency and potential point of failure when reimplementing it yourself literally takes as long as it takes to find the module, add it to package.json and run npm install.
Even if it does take the same amount of time (which it shouldn't), a 1-line call to a standard module imposes less of a future maintenance burden than 14 lines of custom code.
> People should be trying to limit dependencies where possible. Reproducible builds are really important if it costs you almost no time you should have it in your code base IMO.
That's a non sequitur. Reproducible builds are important, but unless you write code with 0 external dependencies you already have a system in place for handling library dependencies in a reproducible way. So why not use it?
> People taking the DRY principle to the most extreme degree always makes for the worst code to debug and maintain.
This is the opposite of my experience.
6 replies →
> However, I agree it is ridiculous to have a dedicated module for that one function. For most nontrivial projects I just include lodash, which contains tons and tons of handy utility functions that save time and provide efficient, fast implementations of solutions for common tasks.
I think that was largely the OP's point tbh. Using something like lodash [a utility library] is fine while using a module [for a single function] is not.
It might have gotten lost in the ranting from on high but I don't think the author truly meant more than that.
For what it's worth, that's what I read in to it too.
Some of the libraries people were mentioning broke yesterday already have lodash as dependencies, I have no idea why they wouldn't have just been using this...
Someone mentioned below that lodash had some breaking changes related to the padding functions a couple times, which could be a totally valid reason to avoid using those. I was under the impression that the lodash API was more stable than to have that kind of thing happening.
1 reply →
> I don't see anything wrong with using a pre-made left pad function. Why waste time and lines of code implementing something so trivial when there is already a solution available?
I'll tell you why.
The least important ones is that downloading such trivial module wastes bandwidth and resources in general (now multiply this by several hundred times, because of dependency fractal JS sloshes in). I would also spend much more time searching for such module than I would implementing the damn function.
More important is that you give up the control over any and every bug you could introduce in such trivial function or module. You don't make it less probable to have those bugs (because battle-tested package! except, not so much in JavaScript, or Ruby, for that matter), you just make it much harder to fix them.
And then, dependencies have their own cost later. You actually need a longer project, not a throw-away one, to see this cost. It manifests in much slower bug fixing (make a fix, find the author or maintainer, send him/her an e-mail with the fix, wait for upstream release, vs. make a fix and commit it), it manifests when upstream unexpectedly introduces a bug (especially between you making a change and you running `npm install' on production installation), it manifests when upstream does anything weird to the module, and it manifests in many, many other subtle and annoying ways.
TIL Java/.NET/Perl/Python/PHP/Ruby are all wasting bandwidth forcing people to download there leftPad bundled in their SDK.
> You don't make it less probable to have those bugs (because battle-tested package! except, not so much in JavaScript, or Ruby, for that matter)
Battle-tested still applies - if you have that many people using a line of code they're more likely to find any bugs. (Formal proof is better than any amount of testing, but no mainstream language requires formal proof on libraries yet)
> And then, dependencies have their own cost later. You actually need a longer project, not a throw-away one, to see this cost. It manifests in much slower bug fixing (make a fix, find the author or maintainer, send him/her an e-mail with the fix, wait for upstream release, vs. make a fix and commit it), it manifests when upstream unexpectedly introduces a bug (especially between you making a change and you running `npm install' on production installation), it manifests when upstream does anything weird to the module, and it manifests in many, many other subtle and annoying ways.
Large monolithic dependencies have this kind of problem - "we upgraded rails to fix our string padding bug and now database transactions are broken". But atomised dependencies like this avoid that kind of problem, since you can update (or not) each one independently. Regarding fixing upstream bugs, you need a good process around this in any case (unless you're writing with no dependencies at all).
Finding this module on NPM or npmsearch.com is pretty trivial compared to ensuring you implement this in a way that catches every edge case.
> It manifests in much slower bug fixing
I don't buy this at all, because I've done it myself many times. If you're waiting on a PR from the original repo owner to fix a Production bug, you're doing it wrong. It's trivial to copy the dependency out of node_modules and into your src, and then fix the bug yourself. Then when the owner accepts your PR, swap it back in. I don't understand the problem here.
3 replies →
Every function in Lodash is also available as a dedicated module, for the same reason left-pad existed: https://www.npmjs.com/package/lodash.padstart
If it was a good implementation, I'd agree with you, but it's not: https://github.com/azer/left-pad/pulls
I agree that Lodash would be a better choice because it seems like a well maintained project. There could be two counter args, in theory:
- if the programmer uses other functions included in Lodash his code will have a single larger point of failure. For example, if Lodash is unpublished (intentionally as in this case, or unintentionally) then the programmer will have a lot more work to redo.
- Lodash introduces a lot of code, while the programmer only needs one of its functions to pad a string.
I'd be amazed if lodash were unpublished.
Using a library like lodash makes a lot more sense once you use a module bundler that allows tree shaking (like Rollup or Webpack 2.0) along with the ES6 module syntax. Heck, even if you're just using babel with Browserify or Webpack 1.x, you can use babel-plugin-lodash [0] so it'll update your imports and you only pull in what you need.
[0] https://github.com/lodash/babel-plugin-lodash
I think it speaks to just how lacking the baseline Javascript standard library is. The libraries that come with node help, but all of this stuff seems like it should be built-in, or at least available in some sort of prelude-like standard addon library. The lack of either leads to all these (apparently ephemeral) dependencies for really simple functions like these.
That said, I work with Java, Clojure and Python mostly so I may be more used to having a huge standard library to lean on than is typical.
So many people use lodash as a drop-in standard addon library that I'm surprised people aren't just using the padding functions that are right in there... Some of the packages that broke yesterday even have lodash included as dependencies already!
Looking at the changelog, there have been more than 70 versions of Lodash in less than four years. The first was in April 2012. [1]
does not exist. It was added as part of version 3.0.0 January 26, last year and renamed to _.padstart in Version 4.0 on January 12, this year.
So in less than a year "padleft" came and went away because all strings don't start on the left and someone decided that "left" means "start" except that the reason that it doesn't is the reason that it was changed. Even worse, the 4.0 documentation does not document that _.padstart renamed _.padleft. It's hard to grok what cannot be grepped.
Why blame someone for depending on padleft in a world where libraries swap out abstractions in less than a year? Breaking changes are bad for other people. Semantic versioning doesn't change that.
[1]: https://github.com/lodash/lodash/wiki/Changelog
4 replies →
What if lodash itself was unpublished? I'm having a hard time drawing a line here, obviously a 10 line function is too far on the bad side of lazy, but I can't tell what is an acceptable dependency.
3 replies →
Probably they do use that if they need left padding but also have a dependency on another package that thinks 'lodash is too big when all I need is left padding' so we get to this situation
This seems like the right answer to me. It's not that we forgot how to program, it's that Javascript forgot a stdlib. You could easily write your own left-pad function in any language, but a stdlib (or this module) gives you a standard way to reference it, so you don't have to look up what you named it or which order the args go in.
Agreed the JavaScript standard library is poor and instead of addressing it they've mostly just added syntax changes to ECMAScript 6 and 7. It's incredibly disappointing.
For instance I added a utility to my own library (msngr.js) so I could make HTTP calls that work in node and the browser because even the fetch API isn't universal for some insane reason.
I think in some ways a good standard library is a measure of programming language maturity. I remember when C++ had a lot of these problems back before you had the STL etc. In the early 90's it was a dog's breakfast.
We have a large internal C++ app at my work of that vintage (~1992) it uses its own proprietary super library (called tools.h++) which is just different enough from how the C++ standard evolved that its not a simple task to migrate our codebase. So now every time we change hardware platforms (has happened a few times in last 30 years) we have to source a new version of this tools++ library as well.
I find it amusing Javascript hasn't learnt from this.
Usually, dependency hell doesn't bite you, until it does. Try to rebuild that thousand-dependencies app in three years from now and you'll see ;-)
I recently had to rebuild a large RoR app from circa 2011 and it took me longer to solve dependencies issues than to familiarise myself with the code base.
Excessive dependencies are a huge anti-pattern and, in our respective developers communities, we should try to circulate the idea that, while it's silly to reinvent the wheel, it's even worse to add unnecessary dependencies.
> Try to rebuild that thousand-dependencies app in three years from now and you'll see ;-)
Let's be honest though, in the current trendy javascript ecosystem these people will already be two or three jobs away before the consequences of their decisions become obvious. Most of the stuff built with this is basically disposable.
Not to mention a good chunk of the companies they were writing this type of code for won't exist by then either...
I never can believe how often frontend developers talk about "you're just going to rebuild it all in 2 years" anyway. I guess it's a good way to keep yourself employed.
Shouldnt the Gemfile.lock have prevented that? It takes your gemfile and makes a specific-version locked file of those gems.
The gemfile.lock must have been "gitignored" at some point, because it had much older packages than the ones in Gemfile. Background: all we had was a git repo and did not have access to any "living" installation.
> Try to rebuild that thousand-dependencies app in three years from now and you'll see ;-)
This is your fault for expecting free resources to remain free forever. If you care about build reproduction, dedicate resources to maintain a mirror for your dependencies. These are trivial to setup for any module system worth mentioning (and trivial to write if your module system is so new or esoteric that one wasn't already written for you). If you don't want to do this, you have no place to complain when your free resource disappears in the future.
I agree. But I find two problems with your proposal:
1- Maintaining a mirror of dependencies can be a non-trivial overhead. In this app that I was working on, the previous devs had forked some gems on github, and then added that specific github repo to the requirements. But they did not do it for every dependency, probably they did not have time/resources to do that.
2- As a corollary to the above, sometimes the problem is not the package itself but compatibility among packages. E.g. package A requires version <=2.5 of package B, but package C requires version >= 2.8 of package B. Now I hear you asking "then how did it compile in the frist place?" probably the requirement was for package A v.2.9 and package C latest version, so while A was frozen, C got updated. This kind of problems is not solved by forking on Github, unless you mantain a different fork of each library for each of your project, but that's even more problematic than maintaining dependencies themselves.
P.S. At least for once, it wasn't "my fault", I didn't build that app LOL ;-)
4 replies →
There's more to dependency hell than "oops, the package disappeared." Try updating one of those dependencies because of a security fix, and finding that it now depends on Gizmo 7.0 when one of your other dependencies requires Gizmo < 6.0.
1 reply →
Most of the time this is something you inherit. Not something you wrote.
3 replies →
So, in the context of this discussion... you should make use of micro-modules to reduce code duplication, avoid defects, etc. However, don't expect those micro-modules to be maintained or available in the future; so you need to set up your own package cache to be maintained in perpetuity.
Or, you can implement the functionality yourself (or copy/paste if the license allows) and avoid the hassle.
1 reply →
I've been in the same situation as OP many times (although in most cases I've been brought in fix someone else's code).
In the Ruby ecosystem, library authors didn't really start caring about semantic versioning and backwards compatability until a few years ago. Even finding a changelog circa 2011 was a godsend.
I think this was mainly caused by the language itself not caring about those either. 10 years ago upgrading between patch releases of Ruby (MRI) was likely to break something.
At least this is one thing JavaScript seems to do better.
Why did you have to rebuild it?
I can't speak for him, but upgrading really old Rails apps can get complicated very quickly. Especially when you're going across multiple major versions and have to deal with significant changes in Rails behavior and broken gems. "Rebuild" might not be the most accurate way to describe the slow, steady incremental approach you're forced to take (you aren't redoing huge swaths of your domain logic, for instance), but it gets the gist across.
The same goes for almost any legacy app.
4 replies →
PS I did not downvote you and I find your question legitimate...
Long story... we had the git repo but lost access to any working installation. I had to rebuild a dev vagrant VM first, and later on a production-ish setup on a server.
I find large dependencies like RoR itself cause a lot more dependency hell than zillions of small dependencies like this one. What kind of dependency hell could possibly happen for a module like this?
I wanted to write this post after the left-pad debacle but I've been beaten to it.
I think we got to this state because everyone was optimizing js code for load time-- include only what you need, use closure compiler when it matters, etc. For front end development, this makes perfect sense.
Somewhere along the line, front end developers forgot about closure compiler, decided lodash was too big, and decided to do manual tree shaking by breaking code into modules. The close-contact between nodejs and front end javascript resulted in this silly idea transiting out of front-end land and into back-end land.
Long time developers easily recognize the stupidity of this, but since they don't typically work in nodejs projects they weren't around to prevent it from happening.
New developers: listen to your elders. Don't get all defensive about how this promised land of function-as-a-module is hyper-efficient and the be-all end-all of programming efficiency. It's not. Often times, you already know you're handing a string, you don't need to vary the character that you're using for padding and you know how many characters to pad. Write a for loop; it's easy.
Note that this is exactly the sort of question I ask in coding interviews: I expect a candidate to demonstrate their ability to solve a simple problems in a simple manner; I'm not going to ask for a binary search. Separately, I'll ask a candidate to break down a bigger problem into smaller problems. In my experience, a good programmer is someone who finds simple solutions to complex problems.
Note: rails is similarly pushing back against developers that have too many dependencies:
https://www.mikeperham.com/2016/02/09/kill-your-dependencies...
Everything in this article is categorically wrong and antithetical to every principle of good programming ever articulated. The only problem here, as others have already noted, is that NPM allows people to delete published packages.
Small modules are not evidence of a problem, and they certainly aren't evidence of an inability to implement these things on the part of the people depending on them. Why would I implement left-pad myself when there is already a well-tested implementation that I can install? Building up an ecosystem of tiny abstractions, bit by bit, iteratively and evolutionarily, is how we get robust, well-designed complex systems. We don't get there by everyone reinventing the left-pad function to sate some misplaced appetite for self-reliance.
The author seems to make some arbitrary distinction between things that are 'large enough' to be packaged and 'pure functions' which are 'too small' to be their own modules, and I just couldn't disagree more. Tiny, pure functions are ideal modules. They facilitate the greatest degree of re-use, most clearly articulate what they ought to be used for, and stateless things are, in general, more composable than stateful things. There is no better unit of re-use than a tiny, pure function.
I don't know why you were downvoted, but I agree with you 100%.
Pure functions are indeed a good target for modularity and deserve proper documention and proper testing, at the very least.
For the readers of this comment:
* How many functions did you not commented last time you wrote some code?
* How many functions do you leave untested?
* How many functions did you wrote more than once?
* How many "trivial" functions did you wrote that actually took you 3 hours, because it's actually tricky, so you checked other implementations and tried to wrap your mind around it.
Check haskell's hoogle to a small sample of this concept.
> Why would I implement left-pad myself when there is already a well-tested implementation that I can install?
You call 4 basic assertions "well-tested"?
https://github.com/azer/left-pad/blob/master/test.js
a) I think that's a fine number of tests for a module this simple, which is itself an argument in favor of small modules. And I think it's four more assertions than anyone implementing this inline in a project would have.
b) The details of this particular project are orthogonal to the philosophy of small modules generally. Whether or not this module is well implemented or well tested has no real relation to whether or not it is a good idea to compose tiny modules.
1 reply →
You are assuming well tested is referring to unit tests, rather than the wealth of developers depending on and running it in their projects daily.
Don't forget how many different targets there are for JavaScript, even if you had a test environment with limitless budget, it wouldn't compare to having a popular package on NPM.
What if it was well-tested?
What if it was well-typed?
What if it was awesomely documented?
Would that be ok? I'm asking, because a lot of the discussion in this post comes a single simple function being a dependency.
I wish I could say you were joking. At best it comes off as ivory tower idealism, at worst the complete naivete of someone who doesnt code.
I'm not sure if you're saying this of my comment or the OP, but I code professionally every day in this style and it's working out just fine. And the smaller my modules get the better it seems to work.
3 replies →
"Sometimes, the elegant implementation is just a function. Not a method. Not a class. Not a framework. Just a function."
--John Carmack
He wouldn't put that small function in a package (or even a separate file) though.
Functions are too small to make into a package and dependency. Pure functions don’t have cohesion; they are random snippets of code and nothing more. Who really wants a “cosine” dependency? We’d all really like a “trigonometry” dependency instead which encompasses many “tricky” functions that we don’t want to have to write ourselves.
This is a pretty weak argument. What is "cohesion" and why do we care that modules have it? Joe Armstrong, one of the creators of Erlang, has argued the opposite (http://erlang.org/pipermail/erlang-questions/2011-May/058768): that lots of small, individual-function modules are better than a "misc" module that grows endlessly and may overlap with other people's "misc" modules.
Calling a function instead of writing the code yourself doesn't mean you've forgotten how to program! The real problem here is the cost and risks associated with dependencies in general (both of which are actually lower for single-function modules), and the broken package removal policies of npm.
There's a big difference between a "misc" module and a "trigonometry" module.
I would retort with: Packages are too big to make into a functional dependency. In the end, in this functions-as-dependencies world, a trig package would be something like:
Packages are only too big if your infrastructure makes them too big.
A trig package like yours could be implemented on top of sin and cos packages if it's something you find useful.
While in general I agree with the article I must admit that I also strongly DISAGREE with the overall message. Especially with this: "Finally, stringing APIs together and calling it programming doesn’t make it programming."
Stringing APIs together is what actually programming is. This is building software and for instance when i use .toString() method I can easily forget how it is done, focus on other high level things and don't care about dependencies, as long as everything works fine.
Let's admit that the main problem here is with broken npm, rather than packages themselves. If someone has written the "leftpad" function, it is so I don't have to write it again, and I can save probably 15-40 min programming and checking some corner cases.
Also please note that javascript can be really tricky down in the details. So if there's anything that can help, it's better that it exists, rather than not.
> Let's admit that the main problem here is with broken npm
It is absurd to have packages suddenly retracted and important parts of the ecosystem stop functioning. This never happened with other languages I have used. Maybe we need a way to make sure the packages are always going to exist. Checksumming and adding the checksum to the version number would be useful too.
That's why there are proposals for immutable and distributed packages managers. Look at gx for instance. This is probably the future for package managers maintained by community.
The funniest thing about this entire debacle is the thousand of self-assured programmers coming out to show the JS/NPM world how it's done, only to have their short, simple, no-nonsense functions fail miserably on some edge cases they didn't think about.
This discussion about the "isarray" package is probably my favorite: https://www.reddit.com/r/programming/comments/4bjss2/an_11_l...
Yes, or more accurately a large new generation of coders is entering the workforce who know how to code only in a superficial sense and think this is a good thing.
Programming, and especially startup programming, is being taken over by people who are primarily technicians rather than engineers. They want to assemble prefab components in standardized ways rather than invent new things. They are plumbers who know how to install from a menu of standard components, rather than civil engineers desigining purpose built one-off aqueducts.
It is the inverse of the "not invented here syndrome." The technician-programmer is trained to minimize time spent thinking about or working on a problem, and to minimize the amount of in-house code that exists. The goal is to seek quick fix solutions in the form of copy/paste from StackOverflow, libraries, and external dependencies to the greatest extent possible.
In house coding should, they believe, ideally be limited to duct-taping together prebuilt 3rd party libraries and services. Those who want to reinvent the wheel are pompous showboating wankers (they believe); creating your own code when you don't absolutely have to is a self-indulgent waste of time for impractical people who just like to show off their hotshot skills and don't care about getting things done. Move fast and break things and all that.
This began with stuff like PHP but really got going with Rails, which preached convention over configuration as a religion, and supplied a standardized framework into which you could easily fit any generic CRUD app that shuttles data between HTML forms and a database (but is painful if you want to deviate from that template in any way.) Note that Rails doesn't use foreign keys and treats the relational database as little more than a glorified persistent hash table.
This set the stage for Node.js (why bother learning more than 1 programming language?) and NoSQL (why bother learning how database schemas work?)
an engineer is more like a plumber than an inventor though.
you don't need to design a new type of screw to make a one off aquaduct, you don't need to design new bearings to make a gear box, you don't need to design a new opamp to make an amplifier, nor do you need to design a new MVC framework to make a one off CRUD app.
You use off the shelf parts and combine them with your knowledge and skills to produce and effective solution for your constraints. If you can't do it with existing stuff, then you design something new that can do it.
The point is that for most things, there were no such off-the-shelf parts in software engineering until recently. You had to invent each screw and bearing yourself, because you couldn't just order a bunch of screws and bearings. Nobody sold them.
Moreover, the difference is that today's technician-programmers don't know how screws and bearings work at all. The engineer who uses prefab screws and bearings still understands them in detail and knows when to deploy them and when not to. Rather than understanding what bearings are for, the technician-programmer has a problem X, reads a blog post that says, "I solved Y by adding some bearings," and thinks, "gee, X is similar enough to Y that adding some bearings may work."
They add some random bearings, some screws, some cams and driveshafts here and there, without any deep understanding at all of why you ought to use these things in a design... and over time build a Rube Goldberg monolith with 10,000 moving parts sticking out randomly that nobody actually understands at all, so as a result everyone is terrified to work on any part of it for fear of inadvertently breaking the other 60 hidden dependencies of that piece.
Bravo. Saving this comment.
For substance: Consider how this is all part of the effort by management to make programmers feel as interchangeable and insignificant as possible.
And it's not even in the name of quality. Plenty of software out there breaks because of its multiple single points of failure in the form of dependencies.
Yes, totally. I can understand why management does this. Coders have a very high turnover rate as a profession. When your genius Clojure or Rust programmer leaves, it's much harder and more expensive to find a replacement than when your PHP or Rails programmer quits.
This then incentivizes new coders to take up one of the popular languages, because that's what most of the advertised jobs are for, creating a self-reinforcing positive feedback loop.
The thing is, it actually doesn't matter. No one needs to be a certain way. Cool things will be done by people that can and will do those cool things. In the end the prescriptivity of this whole space is only ascribed by people internal to the whole system: externally nothing is 'supposed' to be done. This is just a symptom of more people programming.
The thing that will speak the loudest is actions and results. If people don't like depending on modules, don't. If you do, do. Eventually everything will be lost and forgotten like teardrops in the rain.
This "that's just like, your opinion, man" attitude may be fine if you have no worldly goal affinity and are planning on spending the rest of your days in a remote mountain monastery. If you don't care about achieving any particular goal, then yes, all possible attitudes are equivalent. But that sort of handwavey quietism is not actually relevant to almost all people's situations.
If you do have specific short term goals -- such as, say, "building a profitable software product before we run out of money" -- then all methods of approaching the task are demonstrably not equal. Some will work much better than others in terms of achieving that goal.
We can study the empirical results of the numerous attempts already made by other people and thus avoid repeating their mistakes.
1 reply →
What would you recommend to a junior CS major to avoid falling into this pattern?
Specialize. Pick a subfield of CS that you really enjoy, that really resonates with you -- machine learning, NLP, algorithms, whatever -- and become an absolute expert at it. Know your subfield inside and out.
Start thinking about what specialty you want now, today. Do some research, talk to professors and other students, etc. By the time you are a senior, you should know what you want to specialize in and be well on your way.
Then, you can get a job that actually requires deep knowledge. Since you actually care about this, you'll quickly become bored and frustrated if you take a standard startup code monkey job where you are expected to write Rails or PHP code all day, but then that's the only experience you'll have. You'll also be competing with less well educated programmers who are willing to work for less.
Relevant:
'"The Excel development team will never accept it," he said. "You know their motto? 'Find the dependencies -- and eliminate them.' They'll never go for something with so many dependencies."
In-ter-est-ing. I hadn't known that. I guess that explained why Excel had its own C compiler.'
http://www.joelonsoftware.com/articles/fog0000000007.html
Considering the fact that Excel had to run on multiple versions of Windows and also run on the Apple Mac I can understand why they would hate hidden dependencies.
I've experienced the joy of trying to make sure Win32 code runs on Windows 3.11 (with Win32s), Windows 95, Windows 98, Windows Me, NT, 2000 and XP.
Generally it all works, but sure enough there are times when a small code change breaks one or more of those platforms.
Luckily these days things are much better as Win32 is a lot more consistent.
Thanks for a great post.
Yes, we have.
The entire Javascript ecosystem is a huge catastrophe. It will collapse any time soon. It's complex, fragmented and no one really likes it. There are a dozen different tools to get started. No one even understands how to get started easily. There are no fundamental tools. Everything is changing every week. You can't just build a product and then rebuild it even a month later. Nothing works anymore a month later - your dependencies have changed their APIs, your tools have different flags that do different things, there are new data models that you never needed and shouldn't even care about.
The developers are in high stress. Devops engineers are in even higher stress because they get to see what developers don't.
It's a huge mess and my advice to "prefer core-language solutions to small abstractions to small helper libraries to general libraries to frameworks" (http://bit.ly/1UlQzcH) hasn't been more relevant than today.
Software should be developed using least amount of complexity, dependencies, effort and using fundamental tools that have been and will be here for the next 20 years. Cut those dependencies, you don't need them. They're here today and won't be here tomorrow.
So, I suppose you could do something like this instead.
And that would do a leftPad pretty well, and be reasonably robust to stuff like the required width being less than the string width, the padding character being multiple characters long, and so forth. It doesn't do any type-checking of course.
It also doesn't work on older browsers - both string.repeat and template strings are new. You could fake it with string addition, but addition behaves oddly in the case your arguments are numerical, whereas template strings handle that. There's also a trick where you can say (new Array(desiredLength + 1)).join(' ') to make a string that is the appropriate length, but you've got OBOEs to worry about if you're not paying attention (Array.join puts the character between the elements, so you need an n+1 array for an n-length string). Also, at least on some browsers, Array.join is pretty cruddy, and you really ought to construct the string with an old-fashioned for loop.
Javascript has all kinds of weird corner cases and lots of browser compatibility problems. The fact that someone's written a decent implementation of something that should have been standard in the String object means I don't have to worry about it.
Of course, I do have to worry about stuff like losing access to left-pad when someone throws an npm tantrum, or dealing with future build issues if npm becomes untrustworthy. A cryptographically sound package manager seems like a reasonable want, especially after this week's issues.
But if your take-away from this whole problem is "meh, javascript devs are lazy", you're missing the point.
In a way this shows what a great job NPM did at making it easy to publish packages. It's so easy that people decide to package up extremely easy functions.
As a python developer I would never publish a small package, simply due to the overhead of setting up a PIP package.
Oh God yes. After being spoiled in npm-land for a couple of years, I went back and tried to re-release a very small python module I had written before. I simply couldn't do it. Oh well!
Very true. Although pypi is in a better position than npmjs.org on a few fronts (including security), it's also such a pain in the arse to deal with. Packaging is by very far my least favourite part of python.
> In my opinion, if you cannot write a left-pad, is-positive-integer, or isArray function in 5 minutes flat (including the time you spend Googling), then you don’t actually know how to code.
Spoken like someone who writes functions in 5 minutes that I find bugs in later.
Just because a problem is simple to describe informally doesn't mean it is simple to implement without bugs.
It would be trivial if the language lent itself to trivial solutions and there wasn't a culture of trying to anticipate and catch really messed up error cases that results largely from user errors.
In Python I would write ´if num > 0´, if it matters that it is an integer I would cast it to int, and that handles everything you should be handling at that level. If your user passes in a list instead of a number type, the code should crash because then there's probably an input error somewhere. If a user has monkeypatched a base class to make a number into something else the code should also fail and you deserve whatever happens to you.
Catching every possible error is a mistake.
So, apparently some guys managed to build a system where it is very easy to re-use small parts of other people's code and now the author is complaining that "too much code re-use is happening" ?
I'm fairly old, so I remember the complaints a decade or two ago that people had where "We can compose hardware from IC's and you don't have to know what's going on inside and it's all standard and just works! Why can we not do that with software?!?! (of course that ended up with things like CORBA and DCOM, which was all wrong)"
aaaand here we are in a situation where code re-use is actually happening on a wide scale and now you're complainig about that?
28k lines in an empty project? ha, how many lines of code does the preprocessor generate for #include <stdio.h> I haven't actually measured, but I bet it isn't that far off from 28k lines.
minor nitpick: it's not 28k lines, it is 28k FILES
Which, given what I understand about most packages on npm, corresponds closely to 28k lines :P
The elephant in the room here is Javascript. No other scripting language has been so bloody awful in the past that a five-line module that saves you typing `array.prototype.forEach.call` or something that papers over the language's awful idea of equality comparison with three functions has been "useful" or "more than five minutes coding without references".
Granted, these modules don't do such useful things, but that's the environment their creators were immersed in.
^^ Nice, sane point.
What concerns me here is that so many packages took on a dependency for a simple left padding string function, rather than taking 2 minutes to write such a basic function themselves.
It takes more than two minutes. That little module has a test suite, if you're including it then you have some assurance it does what it says it does. If you write it you've got to worry about whether it works or not.
As a dot net developer I avoid adding packages and references at all costs due to all of these same reasons. Usually for something simple like this left pad example, I just copy the code and put it in to my project in some sort of class of helper functions or extension methods.
Seems like a lot of these basic Javascript functions need to be built in to javascript/node itself or consolidated down to a single package of common core functions that extend Javascript. Like padding, is array, etc. As others mentioned, these are fundamental things in other languages.
Have we forgotten how to program? No, we have changed the way we program. We leverage the work of others to solve ever more complex problems. When we do this we get more than just the simple functionality we require. We get all the testing that comes from a package being used by thousands of projects. We get updates when people find a better faster way. We get to lower the number of lines of code we have to maintain.
Yes, these are simple tasks but do we gain anything doing these simple tasks ourselves? I find there is a finite amount of focus I have when programming, and I'd rather spend that solving the bigger problem.
This reminds me of a discussion I overheard between two professors when I was an undergrad. Prof 1 "This new language makes it so much easier to x, y and z" Prof 2 "Yes but what's the point, I can do all these things in Java" Prof 1 "I could do all these things in NAND gates, but I'll get more work done if I have this tool."
In the case of left-pad, 2538464 of its 2550569 downloads last month are attributed to dependents of the line-numbers package (https://www.npmjs.com/package/line-numbers). So it would appear that relatively few people directly rely on left-pad, which highlights the importance of vetting the dependencies of dependencies.
This is in the description of line-numbers:
DEPRECATED. This is a rather silly package that I do not recommend using. It's easier to copy the ~20 lines of code of this package and customize that code, rather than downloading and learning how to use this package.
Committed five hours ago: https://github.com/lydell/line-numbers/commit/711f6ad0eb1771...
This mess appears to have convinced the author of that.
About 2/3 of those (1.7M) were from babel-code-frame, which is a core dependency of babel:
https://www.npmjs.com/package/babel-code-frame
And the fix was just to drop the functionality and inline the function as a no-op:
https://github.com/babel/babel/commit/09287643c712bcd203bbd6...
Nearly everyone has had it drilled into them the "Don't reinvent the wheel." nonsense for better or worse.
I use lodash in almost every javascript project I start, big or small because it makes my life easier.
I'd rather use the lodash isArray than roll my own.
https://lodash.com/docs#isArray
Do you support IE 8? If not, why not use the one provided by the language, Array.isArray()?
The latest version of lodash (4.0.0) doesn't support IE 8, so you would need to polyfill isArray() yourself or use a pre-modern lodash version.
1 reply →
Yeah but you're loading an entire library to use one function?
Tree shaking.
What's the problem of writing a function with a few lines of code and exports as a module?
I think it's totally fine. Like other people said, it's the mindset we borrow from Unix, do one thing and do one thing well. The function would be well tested, and could be reusable.
I don't understand why so many people just require lodash into their project (when they start project) while they only use one or only minimum set of the functions. I mean lodash is a very great library with clean and well tested code, but it's also quite bulky like a big utility lib, and for me most of the time I only need one or two of the functions I would just go to npm and find a module just do that thing.
Also, Node.JS modules can be used browser side. On the browser you only want to include code you're using so you're not making the client downloading excessive code. A large 500kb module with one useful function is worse than a one 10kb module with the one function you need.
When I was prevented from upgrading Xcode/Swift for a project last year because of podfile dependencies, that cemented in my mind that every time I include a dependency, I'm putting myself at risk.
If I can spend half a day writing the code myself, I will -- because I know it will prevent headaches down the road. Yes, yes, I know any code I write adds the probability of bugs down the road. But at least the roadblock would be of my own doing, and not dependent on if/when the package maintainer upgrades their stuff.
There are lots of useful titbits often left out of a languages corelib.
Back in the day we would have built up various pieces of library code, which we would have utilised rather than continually guessing best practices. For example, the isArray method cited may be trivial but is also non obvious. We'd probably have something like that in our library of useful snippets.
Sometimes we may have shared the code on forums and the like and people would copy and paste the code, sometimes into their own libraries. We would locate them by browsing known locations or inefficiently querying search engines
Now we utilise a central resource and simple tools to have a global library of code. Instead of searching far and wide we query a handful of tools that effectively does the copying and pasting for us.
How that can be considered a bad thing is beyond me. It's not a question of knowing how to code, it's a question of using your time effectively.
Granted, there is the problem of so-called modules being removed and dependencies breaking. This can be alleviated by vendoring your modules, a simple task with most dependency management tools.
Personally I think that published modules should persistent indefinitely based on the license the code utilises, although I'm not clear on the actual legalities of the recent npm issue (although if it's due to a trademark complaint, I don't see how it would ever be enforceable for completely unrelated code in any slightly sane country).
Have we forgotten how to program?
Maybe we've forgotten how to /just/ program. Everyone bangs the drum so hard of "let github be your resume." Incentivizing putting every brain fart you ever had out into the universe instead of just keeping it to yourself.
Just a thought.
The problem is not that we forgot how to program, the problem is that we never actually learned how to create a good programming language which avoids these problems.
Just look at the whole build system and module hell of C and C++...
I totally like the language C++ but I hate the tooling around it with a passion. As long as you only need stuff from stdlib then you are fine, but as soon as you want to create some custom library for use in your other projects? Which should compile on Linux and Windows? Either you use make and Visual Studio solutions or you have to fight with cmake... Just thinking about it makes me angry.
For this reason I just don't use C++ very often. It is way easier to just use Python because most stuff is just an 'import library' away and you can concentrate on your actual program instead of fighting against the build system.
But god forbid you actually try to ship your Python program to your customers...
Why do I as a programmer have to deal with all this crap? I want to focus on programming and not on library management amd writing make files. My CPU is idle 95% of the time so there is enough processing power which could solve these problems.
I have high hopes for Rust and it's integrated build/module system!
It sounds like your problem with C/C++ tooling is that it's hard to support Windows. The reason for that is simply that microsoft bundles its own proprietary C++ toolchain with the platform-specific tools that Windows developers need. Windows developers don't want to install a second posix-like toolchain, and os x/bsd/linux developers don't want to install a Windows-like toolchain because in both cases the new toolchain wouldn't integrate as well with the rest of the system, and it would add unnecessary complexity to the operating system.
The problem with proper unix tools, the ones that do one thing well and integrate with other programs as a part of their design, is that they make doing cross platform work a problem, because those other programs might not exist on other platforms. GNU Make works just fine on Windows, but you're not using just make when you write a makefile. Makefiles typically depend on GNU Make, pkg-config, gcc/clang (or another compiler with the same CLI), coreutils, bourne shell, and an assortment of a few other utilities. That means that Makefiles by design will only work on a single platform.
The other operating systems have essentially made themselves into the same platform for these purposes through the use of system package managers. I've heard Windows is kind of getting one? More like homebrew than what we have in linux/bsd-land but that might still help. But maybe Windows developers will still be reluctant to install software with MinGW dependencies. That would be unfortunate, but fair; I wouldn't install a package that had Wine/Mono dependencies just because some asshole wanted to use their Windows build system on Linux! Hell, I avoid installing smaller things with a Qt or GTK3 dependency simply because none of the software I use needs those and I don't want to clutter my installed-packages list for something that isn't important.
I'm not a huge fan of Rust's Cargo. The list of requirements that the Rust team had for their package manager and build system made sense, and if I had to design a tool that met those requirements, it probably would have looked quite a bit like Cargo, but it makes me sad that the most reasonable design is that which abandons the unix-like simplicity of the python and C (and perl?) toolchains.
2 replies →
Of course we learned how to create good programming languages which deal with this problem. A very good and thorough approach was introduced with Modula-2, which, as the name hints to, had modules as a main focus point. Building Modula programs was an easy and quick process. The module system lives on today in the form of Golang - one of the founders was a student or Wirth.
If your package manager is so cumbersome that it makes a 14-line package not worth it, get a better package manager.
We haven't forgotten how to program. We've got better at it.
I totally agree. The npm ecosystem and the generally accepted practices in js these days come close to the bash style of composing & chaining a bunch of solid, specific little programs together to solve problems trivially that would have been a bear to do in many other platforms like java, C++, etc. I cringe at the idea that you should find yourself implementing l-pad in 2016, at least for the vast majority of projects that a typical engineer will encounter in their career. A similar case can be made for something as small as "is positive integer".
While I agree, there are two counter arguments:
1) As noted in other comments, this is a reflection of the core libs of javascript not covering enough of the basics. This is a subjective thing when it comes to where to draw the line, but wherever the border lies between "core" and "community" libs, on the other side you start running into things like "L-pad" and "Lib-L-Pad" and "L-pad2". If there's a great enough fundamental need, you experience a lot of crowding where lots of people offer up their solution to this, and reconciling that across larger dep. chains that disagree on which one to use can become a real burden.
2) Have you ever had the conversation with an auditor (PCI, HIPAA, etc) that your application is actually built on top of a huge substrate of software written by unaffiliated 3rd parties? And that between your own iterations you could easily have different versions for any/all of them? It's a difficult conversation. Much less the explanations to QA about why a build failed because lib X was updated to a new hotfix version in the 14 hours since yesterday's build, after a couple hours of wasted time of initially suspecting your own diffs, and trying to navigate through all the indirection between the actual stack trace and what actually caused the blow-up...
"There are no small parts, only small actors". - Constantin Stanislavski
If there's a flaw to this debacle, it's that packages can be un-published. That is some grade A+ BS.
But no, there is no such thing as a package too small. Coding is hard. Collaboration should be default-on, not default-off.
I completely agree with the author.
The loudest people in the Node community have been evangelizing this practice for as long as I can remember. This shouldn't come as a surprise.
The argument, "If I didn't write it I don't have to think about it" is ludicrous. I just have to point at the left-pad incident disprove the premise of this argument.
The analogy of building things with a bunch of npm lego blocks is laughable. Those responsible for advocating the use of trivial functions by acquiring module dependencies are leading the masses astray.
"But, If I find that there's a bug in a module I can AUTOMATICALLY fix it everywhere!"
No.
You still need to assess how the change to that module impacts any code that depends on it. Just by updating a module and posting a "minor" bug fix can lead to other bugs that RELIED on the behavior as it was originally written.
It's simple, write your own trivial functions. Test them. Maintain them.
P.S.
Another module that can easily be in-lined to every code base you own. (3 million downloads this week).
https://www.npmjs.com/package/escape-string-regexp
Just because it's a one-liner doesn't mean that this code is not complicated. If you forgot something in this one-liner, you could have something that's not working in every case.
> There’s a package called isArray that has 880,000 downloads a day, and 18 million downloads in February of 2016. It has 72 dependent NPM packages. Here’s it’s entire 1 line of code: return toString.call(arr) == '[object Array]';
How anyone can deal with JavaScript for more than 5 minutes is absolutely beyond me
Sensationalist stuff. isArray doesn't depend on anything, but lots of other packages depend on it. Why? Because it's actually kind of hard to tell the difference between an array of things and an object with integer keys. The Array.isArray function wasn't added to the language until ES5, so you need a way to shim older tests.
If you're writing only for modern browsers, you don't need it (if you actually visit the code in question you'll see it defaults to Array.isArray - in that sense, it's a polyfill or whatever). But if your code might run on old browsers, it can't hurt to have it, and it cleanly encapsulates a trick that you no longer have to remember the syntax for.
This was probably grotesquely naive of me, but I literally had no idea how jenga-tower like the javascript ecosystem was. Eye opener!
...maybe it's time a committee of really smart people sit and sip through all the most used modules below N lines of code or smth, and just write an opensource JS-stdlib, hopefully merging in top 30% most used methods of Lodash too? Node/NPM is a great example of why too much democracy and decentralization is bad. Just gather some experts and have them centrally plan a "standard library" then impose it as "industry standard" have a recommended "no fragmentation policy" like "no forking" and "use it all or not at all", the hell with your web app's need for "performance"... even a few hundred Ks of code will not hurt anyone nowadays ff sake...
I even consider PHP a "more sane" language because you at least have most of the useful utility functions in a global namespace and everyone uses them. Of course, the real ideal on this is Python's solution: a nice set of standard libraries that you know are baked in, but most of them you still import explicitly - hence it's pretty easy to write even large Python applications that have a small and comprehensible number of dependencies!
(And more generally: our strike for "more efficiency" in programming is stupid imho! I'd always take a less efficient solution, even "less safe/tested", if it's more "understandable" and "explainable" and sometimes, paradoxically, making things a bit more monolithic and centrally planned makes then orders of magnitude easier to reason about for our tiny ape brains...)
There is also a big programmer efficiency to be gained by having batteries included library: The new person coming to a project has learned the same batteries already, and it need not to re-learn new set of minipackages the last person working on a project has used.
In fact this should be the key selling point for stdlib to JavaScript. It actually saves money from their employers if trivial functions are mostly in the same place.
However pushing a stdlib is difficult there is a lot of hate for Python's "kitchen sink" library in JavaScript world which is totally misplaced. It maybe founded on fact that you should then minify the whole package to use one function in it, but this could be solved with dead code elimination (with proper type checking) in the minifier.
> there is a lot of hate for Python's "kitchen sink" library in JavaScript world
Is there? Most "js people" have never had the pleasureful experience of working with a "pleasantly opinionated" language like Python and the similar feel of its libraries, even the 3rd party ones.
Really, there's something really awesome in the Python ecosystem that should be spread to others before node takes over the world: If done right, being opinionated ("one way to do it") and centrally planned (a BDFL or smth like that) is A GOOD THING for an open-source language/infrastructure project! And it's also good if you extend the philosophy to the libraries and frameworks. Don't ditch the baby with the bathwater!
But I have a really hard time translating this message to "node-speak" unfortunately... Thanks for the "saves money" tip though, never thought of that, will try it ;)
Rob Pike: "A little copying is better than a little dependency"
After working with JS/Coffeescript/Python/Ruby for years, Go has been an enormous breath of fresh air for me. Of course it's all a matter of preference.
Man, I have to quote: `In my opinion, if you cannot write a left-pad, is-positive-integer, or isArray function in 5 minutes flat (including the time you spend Googling), then you don’t actually know how to code.`
You would be surprised of how many developer these days have 'afraid' to write such functions, or how lazy they are, they found this thing and just add to a project, then push to some google list and the project got a lot of followers, and in the next day the project has changed about 90%. I saw this happen over and over again in this ecosystem, this is insane dude.
A lesson I learn is: you _need_ to read every module source code before add to any project, the NPM ecosystem has so many "shits" out there. You cannot trust in any npm module, recently I tried to trust in a module that has more than 5k stars, but I found a such ugly bug on that, that I feel my soul die, and I swear I hear the angels cry, thats not how open source supposed to be.
These days, seems that people dont care about the 'bug free' as long as it work a half way.
The Python community has a proper response: https://pypi.python.org/pypi/left-pad/ /s
Then you can do variants.
A Java left pad implementing a Padder class instanciated by a PadderFactory that you can get by a PadderInjector from a PadderRegistry all that accepting an AbstractPaddingString.
Then one in Ruby where you monkey patch the string to that it pads, then add an AUTO_PAD settings set to True by default and a fluent API to chain padding.
Then one version PHP version containing pad_string, then real_pad_string that deals with unicode, then real_pad_string_i_swear_this_time that that call str() on parameters automatically.
Then a Haskell one with a monad and a trampolin and some |=> giberrish to mix it all.
Then a Go one that creates a string padding microservice. With a docker file, just in case.
And the last trend in Python, an aio_string_pad which gives you an asyncronous padding coroutine, but under the wood calling run_in_executor, and optionally auto start an event loop.
This is 99 bottles of beer all over again.
So any developer could make this in five minutes, but for some reason they can't verify whether or not it works? That doesn't make sense.
In reality it could take an hour to get this working properly, but it does take only a couple minutes to verify that the solution here is correct. There are certainly good reasons for not adding extra dependencies to your project, but trading a known amount of time to check that an existing project does what you want for an unknown amount of time to redo it yourself is probably not a great bet.
Unzipped, the source code for GNU coreutils is 30MB (zipped 4MB). This is a great example of a collection of single purpose functions you should never rewrite yourself. There's only one dependency if you want to use them because they're packaged together. With normal desktop code, 30MB doesn't really matter and you can link only what you need. Can you do that with the usual Javascript package managers/bundlers, or would you need to send the whole 30MB package to the client to use one function from it?
It's not that anyone has forgotten, it's that a lot of people never learned how to in the first place. Every programmer community is riddled with these problems but the NPM world seems to be the worst. The ruby gem "american_date" annoys me to no end. It's just a highly-specific implementation of Time#strptime. Gah
node is can be a good idea. But people don't take JavaScript programming seriously. Most libraries objectively suck.
Google, authors of V8 and #1 subject matter experts on V8, have published coding standard. Does someone use it in the node community? No. Everyone loves "standard", a lousy standard that allows everyone put a badge on their github page while still having a code base full of vomit.
JSDoc. A great solution for documentation. You would expect major libraries to adopt it, or a similar thing. But again, no. Major libraries such as busboy do not use them. The documentation resembles a napkin.
Then everything else: input validation, error handling, consistency... etc. Take "request" for instance, one of the most widely used libraries. The state machine it implements is inconsistent. You abort a request and get a timeout, you can abort a request without starting it and get an exception. Issues that will drive you insane while debugging.
Express, one of the most widely used web frameworks on node.js. Do this on a route: setTimeout(function(){ throw new Error(); });. Great, now you have broken out of the error handling context. Great job.
Node libraries suck all across the board. It's the PHP of the 21st century. There are exceptions, like: lodash, bluebird, and others.
We have. I spent some time optimizing [0] a String.repeat function over at Stackoverflow and I was surprised that many developers today don't know what they are doing, including core team members [1]. Specifically,
[0]: http://stackoverflow.com/questions/202605/repeat-string-java...
[1]: http://stackoverflow.com/questions/202605/repeat-string-java...
So I don't think this is an example of core team members who don't know what you're doing.
This function repeats a string up to a certain length. e.g.
repeatString(foo, 10) = foofoofoof
This could entirely be what the developers needed in their code base.
The fact that it doesn't repeat a string N times, is at best, an example of bad naming.
The issue here is that someone unrelated to the core team grepped the node codebase for the term they wanted and threw a function up as a SO answer without understanding what it did.
Allocate and join an array of blanks just to repeat a string is acceptable?!
4 replies →
One of the things I've enjoyed the most while programming in Go for the past couple years is the standard library and how much the Go community emphasizes keeping external dependencies to a minimum. For most projects I find myself using few if any external packages.
Now of course there are times you'll want to reach for a library, say for something like an http router, and up until recently the dependency management side of Go has been lacking. But when a pattern or library arises that many find useful the core team is open to pulling that in to the standard library if a strong enough case is made, for example in the context package (https://github.com/golang/go/issues/14660).
To me, this looks like a symptom not of bad programmers but of a terrible standard library.
Well put. When a hammer requires endless accessories to drive a nail, is it the right tool for the job?
Formula for a top HN article:
1. Make an observation about a popular thing. 2. Blindly extrapolate. 3. Make one or more broad, controversial statements. 4. (Optional) Nuance.
The main reason there is a fetish for these micropackages is a fetish for github stars. The formula seems to be: Overly generic + Simple + Javascript = Lots of stars.
That being said, there is something to be said for using these micropackages. Left padding a string is easy, but you might just have forgotten about that one edge case where in browser X and language Y you have to do things different. It's not really the case here, but things that seem simple at first often turn out to be hard because of some edge cases. One might hope these edge cases are solved if they use a library.
I call this kind of attitude the "Tea Party" of JavaScript development. The reason why we currently have JavaScript tooling fatigue is exactly because Tea Party developers insist on writing everything themselves instead of trying to build a better abstraction. The lesson here isn't not fewer dependencies: it's managing dependencies. NPM should not allow someone to arbitrarily remove modules that other's may be depending on. It's like building a bridge and them deciding to remove it after a whole city now depends on it.
It's more like planting a bush in your front yard, and for some reason the metropolitan train system decides to use your bush's shadow to tell the time of day, and builds that into every aspect of the entire city's transportation network. Then you cut the bush down and everyone starves.
Here's the funny thing that gets forgotten: in a lot of commercial software, 3rd party dependencies need to go through legal to get properly vetted and attributed and so on. This also usually requires an engineer (to be able to answer things like if it's dynamically linked or not, etc.).
As staid and corporate as it might sound initially, it's a very smart thing to do. One screw-up with licenses could be catastrophic. Are you all really checking that carefully?
I can't even imagine how any sort of proper legal checks could be done with a trillion micro libraries.
I'm content and happy not knowing what the fuck this is all about. Pad left? Are you kidding me? If your tower og babel fell because of your reliance on some script kiddie's toy project, I am happy and content knowing you get what you deserve. Law of leaky abstractions, motherfucker.
If I engage in as much hyperbole as the author, where does "write it yourself" stop? If I'm working on a team of two, should we each write our own left-pad? How about a team of three? Four? Five? Fifty? At a certain point, it makes sense for that to be written once for the project. We spent 30 years in software engineering trying to figure out how to get code re-use, and now that's it common and widespread, we want to go back to NIH?
I think about this every time I write a UTF-8 decoder. It's a task which is simple enough that I can bang it out from scratch about as fast as I can find a library that does it, so I can't really stomach adding a full-blown dependency to deal with it - but it's also just complex enough that I have to think about it, write some tests to make sure I didn't forget something, and wonder whether this implementation is any different from the last one I wrote.
I have 'tools' repository on my local gitolite server just for that. Saved a lot of time already.
I pretty much came to the comments to post something similar. I'm sure there was a long comment thread here on HN discussing the perils of NIH this week. This seems like the end point of the alternative.
NPM modules can be used on browsers. On browsers, space is a premium.
Why would you want to install a 500kb dependency that has only one function you need, when you can install a 10kb dependency that has it?
Would you want each of your five 20kb dependencies to re-implement the same 5kb function, increasing the code you must send to the client by 20%, or would it be more optimal for each of those dependency to use the same 5kb function?
The author rants about practices of developers from different programming environment, without experience, without figuring how things came to be. If he did give an effort to think from the perspective from Node.JS developers he’d have addressed the previous two points.
This is like going to a friend’s house and complaining everything is put in the wrong place. It would have been wise to immerse in Node.JS conventions and observe for a while before making comment.
EDIT: Reply to scrollaway:
I've also understated the problem.
Let's look at the problem in the current Node.js environment, it's not uncommon for a web app to have 20 dependencies, each of those have 10, and each of those 10 have 5. That's a total of 20 times 10 times 5 = 1000 dependencies in total.
Let's say you were to remove a 10 line library function that's "standard library-like", used by 15% of those dependencies, and have each of the existing dependencies re-implement that in each of those dependencies that uses it.
15% times 1000 times 10 lines is 1500 lines of code.
So if you're going to troll a solid argument by nitpicking, do it properly and get the details right.
FYI, you can reply to my comments if you click on their timestamp, in case the reply box isn't showing up
> So if you're going to troll a solid argument by nitpicking, do it properly and get the details right.
First of all, I don't appreciate you calling me a troll. Someone who mentions "5kb functions" clearly has no idea of what 5kb represents, period.
And second, this is not a solid argument at all. There is logic behind it, which is why we usually have high quality stdlibs in popular languages. Javascript lacks that. So instead, javascript gets this absolute mess of an ecosystem, where things like this can happen.
Several people have brought up various issues with the way it's done now. Dependency hell, for one. Lack of discoverability, which in turn leads to duplicate libraries, which in turn leads to a system where you have 1000 dependencies, but despite your "modular" idealism you still have hundreds of duplicate dependencies. Not to mention all the duplicates of different versions of the same dependency.
This "saving lines of code" math is completely broken exactly because this stuff is not in the stdlib. The various issues with the JS ecosystem mean that the actual results are nowhere close to ideal, and have a net negative impact.
I also love when people mention how much "one time cost" is saved and back it up with similar math, completely forgetting the amount of continuous time wasted downloading and installing these dependencies every time.
You're mixing up development time & bandwidth used by client browser. I was addressing the latter. Not having to download a 500 kilobyte library in order to use a single function is a big reason why the node.js ecosystem is the way it is today. Yet it is completely unaccounted for in the article. The author did not go into how the Node.js community came about to this practice in the first place, before going on a rant about it. That is what I was criticising. Whether having minimal modules maximises utility in society is completely irrelevant, because I was addressing perspective and etiquette.
You are a troll because talking about the actual number I used and how good a grasp I have in my mind completely misses the point. You're criticising the javascript ecosystem, I get that, but you completely missed the point of my comment. If you're not going to try to understand how Node.js conventions came to be, what it's strengths and weaknesses are, what the trade-offs made were, then it's inappropriate to focus on particular weaknesses of its practices.
It's like going on about how Objective-C development sucks because it's hard to do on Linux. But you're missing the point about what Objective-C's strengths are that attracted Objective-C developers in the first place.
The author should spend some time acclimating to Node.js best practices before writing this article. And you should do the same before you start knocking down straw men to prove how right you are.
2 replies →
It's quite ironic how you're lecturing people about dependencies and bandwidth when you yourself don't seem to have a good grasp in your mind of how immense even 5 kilobytes is (you talk about 5kb functions). 5 kilobytes (well, KiB) is 5120 bytes. That's about 200-300 lines of code. If you're talking about minimized code, maybe up to 500 lines.
While a 5kb function is immense, I work on a legacy code base and I have seen functions up to 2000 lines long. You possibly overestimate the quality of code that people in business actually write. I've worked in a few places and all have parts of the code which I would thought were made up if submitted to the dailywtf.
Also, have you ever seen a commercial library require >2MB of minified JS? Here's an example:
http://cdn.wijmo.com/jquery.wijmo-pro.all.3.20153.84.min.js
That file is almost 2MB minified (5.6MB unminified).
And that is is only 1 of the few different files required for our dependency on it, there is a bunch more javascript/css also required. Would I use that library on a new product? I'd probably advise against it due to the size, but a legacy product has dependencies and you don't have the freedom to rewrite everything all the time.
1 reply →
A lot of the discussion here really isn't talking about the problem at hand.
From the perspective of Babelify users, a major bug was introduced into software they depended on. I don't know how much money in developer time was lost due to this but it would almost certainly be in the thousands of dollars.
And it could have been a lot worse. It could have been something more complicated than left-pad. The author could have introduced a vulnerability or outright malicious code, or been hacked and done the same, and millions of people would have downloaded it and run it.
Arguably, small modules are good if you control them. Maybe they are more composable, maybe they enable better testing, maybe they encourage code reuse. I am not going to argue for our against small modules.
But none of the positives of small modules matter if an unknown developer who you have no reason to trust can change or unpublish the module out from under you. It's irresponsible as developers to risk our employers' and clients' businesses in this way for a function we could write in five minutes.
Its interesting how everyone used this as a chance to attack the small modules approach. This approach definitely has downsides, but the problem caused by leftPad being unpublished wasn't one of them.
If jdalton declared a jihad on arrays tomorrow and decided to pull all array related functions from lodash, we would have the exact same problem.
If kriskowal decided that Q must mirror built in Promise and published a version that does this tomorrow, we would again have the exact same problem.
There is only one connection between this problem and the small module approach. As the size of a module decreases, the number of dependencies increases and so does the number of authors that produced your dependencies. With the number of authors increasing, the chances that some author decides to go rouge or protest for some reason also significantly increases.
Therefore, its irresponsible to use this approach with a package manager that allows an old, established module with many dependents to be unpublished so easily by the original author.
Would people automatically update to the new versions of the libraries you mentioned?
The problem with unpublishing is that it changes an existing version. If instead of depublishing a correctly versioned empty module named leftpad was pushed to npm (increment major, because a non-implementation is incompatible with an implementation), there would not be half as much pain.
As long as unpublishing exists, micromodules increase the "attack surface" to this specific method of changing the contents of a specific published version.
Which means unpublishing is the problem, not micromodules (in this particular case).
1 reply →
This is a non-issue and taking focus away from the real issue. The issue is the security hole that NPM opens up when a namespace can be grabbed up by anyone if the original developer pulls out.
Personally, it's not about the 'original' developer pulling out, but about the simple question: who or what do you trust?
1. The developer, 2. The content distributor, 3. The code (by git SHA ref perhaps), 4. The contract of the code (using formal verification), 5. The legal contract with the code supplier.
And if you trust 1, do you expect of him to sign the stable releases using GPG tags?
All the focus on OSS nowadays seems to be on 1, but as professional engineers, shouldn't we focus more on 4 and 5?
The OP is deliberately talking about a completely different issue that just happened to be spawned from the recent drama.
He is talking about learning to code vs asking google for implementations of even the most trivial things and about what a useful library is.
I know I wasn't interested in doing any JavaScript until they started modularizing it and npm was that on crack. Possibly once the kids had their shiny new npm, some went mad with power and took it too far. Take the good with the bad though as the JavaScript language is known for.
This culture of tiny one-function modules sounds like Joe Armtrong's proposal about the "Key-Value database of all functions".
http://erlang.org/pipermail/erlang-questions/2011-May/058768...
I'm not really sure whether or not I should to add my voice to the din, but I feel like this whole thing is more a problem with npm and what it allows vs. what it encourages (and the rather paltry standard libraries in Node & browsers), rather than a problem with developers feeling entitled to not have to write their sorters and padding functions.
npm actively encourages structuring projects as many tiny individual modules and dealing with the resultant dependency trees and deduplication. Both of these things (along with the ease of publication) combine to encourage people to share their packages.
They make it incredibly easy to consume code from other people, but but at the same time provide a similarly low-barrier mechanism to retroactively change published code. That combination seems like a way more deserving topic of criticism than the unending refrain of "developers these days are so lazy".
Joe Armstrong had an interesting related comment here:
http://erlang.org/pipermail/erlang-questions/2011-May/058768...
Maybe the unit of modularity should be a single function and we can do away with modules?
I find NPM packaging ridiculous. Awhile I go I used NPM on windows, where the folder hierarchy became so deep it broke windows file handling. I could not delete the modules folder. I had install a npm package which allowed me to delete it. I think this is fixed in new versions by flattening the hierarchy, but still.
I've been there. If I recall correctly, I found that a "rmdir /s" from a command prompt does the trick.
There's so much wrong here, it can't even all be seen at once. The part we can perceive is only a projection from a higher dimension into our three-dimensional space.
The insanity is that JavaScript doesn't have a standard string library. More a case of forgetting how to design a programming language than how to program.
Contrast with the Scheme Request for Implementation (SRFI) process, see <http://srfi.schemers.org/>.
Agreed. Most development groups should be building a local collection of utilities that contains all of these snippets, and most importantly, some documentation of what they do and some unit tests to demonstrate that they are correct.
No need to have global dependencies on small snippets that really should be in a core library anyway. C has libc, Java and C# have the creator's (Oracle or Microsoft) standard set of libraries, Python has the "batteries included" stuff in all the different distros. And so on. All of these snippets rightly belong elsewhere, not in packages.
And even if you did get them added to the right libraries, I guarantee you that you will not get rid of the need for a collection of small, and somewhat random, in-house functions, classes and libraries.
I started out really thinking Kik was wrong to sue this guy but like with all things, the longer this goes on the less sympathetic I grow.
Write your own goddamn utility classes, people. Or learn how to suspend package releases, include them in your projects, and smoke test your releases.
These issues are ortogonal.
After reading this, i don’t think i’m capable of ever complain about OOP again. Whenever you think you’ve seen it all, web developers always manage to come up with something worse. The only thing more depressing than this is the comment section below the article.
For a less sensational review of this situation, NPM published a post-mortem:
http://blog.npmjs.org/post/141577284765/kik-left-pad-and-npm
Insight from this story: If it is important that a thing be done correctly, it should not be made so easy to do that it will end up being done by people who shouldn't be allowed to do it.
I may have just invented a rule for choosing programming language and systems there.
Another explanation for the flourishing of these one function modules may be the fact that some people feel a kind of high/achievement from being able to say they have a published module out there. A sort of bragging rights if you will.
"I may not be Mick Jagger, but I am playing in a band and we did have some gigs (before we had that 50% decline in audience when my mother got bored and stopped showing up)" - just about every twentysomething before coding became a desirable career
It beats the alternative - pulling in some huge package with lots of little functions, all of which end up in the output. At least you're not loading huge amounts of unreachable code into the running system.
In languages with linkers, the better linkers would discard all unreachable functions. Then came DLLs. Use one function in a DLL/.so, and the whole thing has to be loaded. Go and Rust are usually statically linked, reflecting the fact that pulling in big DLLs was usually a lose. You rarely have two different programs on the same machine using the same DLLs, except for some standard low-level ones such as the C library.
I know the discussion revolves around the amount of dependencies, but I want to add a comment about semver, which has a part in this mess:
In my opinion it will not be done right in 80% of cases.
Every breaking change requires updating the major versions, but developer are hesitating to go from 1.0.0 to 6.0.0 in a month.
The way out is staying in the 0.x range therefore abandoning semver alltogether.
A nice write up about how packages are not following semver in java-land:
http://avandeursen.com/2014/10/09/semantic-versioning-in-mav...
I wouldn't even think of looking for a package that does something as simple as the ones mentioned. If I need to pad a string, my first thought would be to create a new function, not look for a package...
I think we missed the point here. The fact that it's 11 lines is meaningless, this could have been babel itself.
The problem with shared hyper-modularisation is that it assumes the name of a function is unambiguous with only one valid implementation. If that were true, it should be encouraged but given it isn't, the practice will crushed by ambiguity and unintended consequences.
My app might well have an is-positive-integer function but it will include a range of context dependent choices about e.g. floating point, infinities, zero, null, "9", "9 ", "09", boxed numbers, string representations exceeding js max int, etc. etc.
I can't take this author seriously at all, one of his most egregious cases is the is-positive-integer library which until today had around 30 downloads in the last month.... No one was really using this and furthermore of course you can find bad/iffy code on NPM for the same reason you can find bad/iffy code on github. ANYONE can publish code. I could write a similar module for any other library, publish it to their repo, then scream LOOK! python is stupid and python devs are stupid.
I firmly believe that building on what has already been done allows for much safer code written at a quicker pace. Why should we constantly repeat ourselves? Also by using npm modules we can abstract logic and prevent someone on the team from going in and modifying it for their own use. It is a documented/constant function that we can use knowing exactly what it does. Is it a better world where everyone just copies code out other's repos and then has to include the licence/docs/tests along with it? It's much easier to just pull in the repo which contains everything and make it trivial to see where a function came from.
People are blowing this whole thing way out of proportion just because it makes a good headline "11 lines of code broke node"... You can all try to shame people who build on what's come before and chant "Not invented here" but I'll opt to build on what is proven to work instead of rewriting everything. At the end of the day that's what ships products.
This isn't forgetting to program, it's a deliberate choice as to how library code needs to be organised. We can disagree about choices made, but let's not assume other people aren't able to code as well as we think we can code.
That being said, I think this is a perfect example of where a good concept (small, tightly scoped modules) is applied dogmatically at the cost of the codebase. It's the node.js equivalent of AbstractRequestFactoryFactoryFactory stuff you see in Java, and the Mock Messes you see in Ruby.
> What concerns me here is that so many packages took on a dependency for a simple left padding string function
Clearly to mitigate such a tightly coupled dependency, Left-Pad should be a micro-service. :-\
Every single dependency you import should make you nervous. You would look at it like hiring someone. You are letting go of some responsibility. You are letting go of some control. The end result might be better than you could do in-house, but it might not. But you are already hoping that it's better, and giving up control.
Save these for the things you can not do in house, like NaCL. Don't write that yourself.
But string padding, sorry. Any developer worth their salt would laugh at adding a dependency for this. It's irresponsible, and comes across as amateur hour.
This is a simple case of optimizing for responsibility. The inexperienced programmer does not know how to do this, because they have not spent enough time being responsible, and having to deal with the fallout of managing responsibility poorly.
An experienced programmer carefully manages responsibility. A simple function, that is easy to understand, easy to test, and easy to reason about, is something that makes more sense to either pull all the way into your codebase, or write yourself.
Years of doing this means that managing dependencies should never be just slap it into your packaging system and start using the function. If you are on the line for making the wheels turn for a large scale platform that is directly connected to a monetary transaction of some nature, you will quickly find yourself preferring to remain responsible for everything that you possibly can control. There is certainly enough that you can't control to keep you on your toes.
I think this is the fundamental thing people don't understand about NPM and JavaScript, and the web in general:
Nothing is included. And that's a feature. The web is not trying to be the kitchen sink. That's iOS. They provide high level APIs for everything. And as a result, the platform is architecturally only as vibrant as Apple can make it.
Now maybe you're happy with Apple, maybe you love the iOS APIs. But if you don't, you're stuck. There's not a rich bed of alternative view layers that you can draw from to build your own vision of how software should work.
Node and the web browser strive to be lowest common denominators. They provide just the very basics: a document format, a very simple programming language, and an http server. The rest is up to you.
That's pretty scary, and so the JavaScript world has dabbled in frameworks. In the end all-inclusive frameworks are antithetical to the spirit I'm talking about, so things trend towards small modules that do one thing pretty well. People go overboard sometimes. I would argue left-pad should just be a copy-pasted snippet rather than module. But that's not a sickness in the community, it's just developers feeling out how far to go.
If you like every application to look the same, and you don't mind being chained to enormous and restrictive standard libraries and frameworks, then you will hate JavaScript. If you like writing software from scratch, forming opinions about all of the different parts of the system, and allowing each application to be built out of different parts, out of the right parts, then you should give JavaScript a closer look.
This is flawed. Basic things like checking if something is an array or a integer should be part of a stdlib. No one is saying whole frameworks need to be included, just basic functions, partially because JS isn't sane in the first place. Nothing in C's stdlib forces apps to look the same.
You are talking about extremes here, but there are nice place somewhere in the middle, where open source libraries/modules exist to help you solving complex problems and you are free to write whatever you want.
The fact that anyone has to even think about code size is the real problem.
Yes, downloading a giant JS lib for one function is insane, hence the ton of tiny dependencies.
However, it is equally insane that basic SDK expectations found in every other language has yet to come to be pre-implemented by the JS engine in the web browser itself. Some basic code ought to already be on the localhost the moment your app arrives.
Need more of this WTF stuff? Have a look at the docker registry/repository/hub and those pearls of wisdom of the new DevOps guilde.
Have to put a little clarity over my "so sad yet so true".
When the developers of such a serious library as React start to depend on a third-party one-function module made by some Dick-from-a-mountain (a russian idiom that means a random person who did nothing significant but tries to show out in all possible means), that means React developers are even more to blame than that Dick-from-a-mountain himself.
If you make any really popular piece of software, you absolutely must have a failover plan. No excuses for not having it.
But what's even sadder is that this issue had spawned a new wave of pseudo-elitist attacks on the entire JS dev community. Calm down guys, things like that could have happened for any language that has a widely-used centralized package system (Perl's CPAN, Python's pip, Ruby's gems etc).
Let me repeat that again: languages don't make software bad, people do. Just don't let such Dicks-from-a-mountain rule over your own modules with elementary stuff like leftpad, and you'll be safe.
This is literally one of the funniest things I've heard about in months. Look for my new python library isDict, coming soon.
Here's a proposal (that I'm sure others have come up with in the past) -- why not create one big, community backed "batteries included"-type module that would implement all the small, commonly used functions. This could combine all these ridiculously small libraries and greatly reduce the number of necessary dependencies for a package. Extending the standard library should be just that: standardized. If the entire community focused on one project like that they could just as easily write the same code (but with smaller package.jsons, less require()s, and less time spent learning new libraries/searching for the right libraries. In fact, it would be great if something like that could be packaged as a standard node module so you'd get the same sort of quality assurance as you get with official projects.
because everyone would stop using it in a week when the next one arrived
<rant> The problem here is JS developers have baked in the notion of having NPM as the alternative to Google + StackOverflow + own thoughts. It's really a no-brainer (literally) to just slap another package than to bother thinking about what a piece of code does, the edge cases and pitfalls. Move fast and break things, right?
Sure there was some argument about Unix philosophy, small module doing one thing and does it very well. Did anyone bother considering the quality of most NPM packages? Quality is not reflected with passed Travis CI or extensive community testing and feedbacks. Not at all. Look at the those packages on apt-get. They are modular and robust. They do what they were supposed to do.
Now take a long hard look at the state of NPM. What do we have? People clamoring for reusability and whatnots. Most of them don't even know what they're talking about, just reciting the latest hip statement from the Internet. Being mature about development means accountability for what you do, not pushing shit around that you don't even have knowledge off. As a self-proclaimed polyglot, I love JavaScript as a language but not the ecosystem. It's like watching a dog chasing its tails:
- Endless loops of discussion that help stroke the egos but not improve anything else.
- Craps for resume/repository padding, not for actual developers to use.
- Bandwagon mentality that just pushes the latest fad along, and the herd towards the cliff.
The notion that JS developers are kids playing grown-up, has been reinforced with this NPM incident. If we want to discard that notion slowly, we need to be more mature developers. It's that simple. Here's what I think we could do: - Have a clear idea on what dependency you need. Browser, IDE, terminal etc are dependencies. Basic type checking is not. - Be better craftsmen. Roll and maintain your own toolboxes. Only share a working hammer, not a broken nail or a wood chip. - Note that for each package you publish, thousands more hours would be spent on learning, adapting, using and reporting mistakes. Collectively, the current community wastes so much time with finding the right things to use. Often times, we learn much more by playing with code, even posting on StackOverflow. That's hands-on, `npm i` is not. - Own the code better. The idea that teams like Babel and React devs with all the brilliant developers choose to put their eggs in a private corp's whims is just scary. You can't hope to build robust software while playing Jenga tower.
I agree that some packages might be too much but I don't think `left-pad` is one of them.
I wrote my own left-pad for a project I'm working now and I had to revisit a few times for tiny problems and lack of time to write tests. I would definitely use `left-pad` module if I knew the existence at that time.
Separate from the discussion of whether super small modules and hundreds or thousands of deps are a good idea, is the point of operational stability.
Putting on your devops hat, whatever your dependencies, from a reliability and reproducibility point of view, you should control your reliance on unexpected decisions of npm or third-party developers. A lot of the panic with npm issues comes from people blindly using the npm registry and then seeing breakages, with no safety net. I hate to say "I told you so" but this is an issue we worried about a lot when considering Node productionization last year: https://medium.com/@ojoshe/fast-reproducible-node-builds-c02...
I am posting the article "No silver bullets" again, in the wake of the npm fiasco. I think it is an essential reading for every programmer, every year!
https://news.ycombinator.com/item?id=11350728
This problem speaks volumes about the myriad shortcomings of the JavaScript standard library, in my opinion.
Couple of things:
* If you're in favor of the micro-module approach, you shouldn't be relying directly on NPM, and should have something like Sinopia in place. After all, external code isn't the only thing you're vendoring, right?
* Micro modules are fine - but your application code should depend on a privately published utils module whose entry point is a prebuilt distribution of all your external micro-modules exposed through a facade. Your utils module deps are all installed as dev dependencies to avoid the Fractal Nightmare.
* Yay, now you have your own 'standard library' which still manages to leverage the NPM philosophy of distributed code. And if some twit decides to throw a tantrum, it will only impact future builds of your custom std lib - and you'll know about it at build time.
Yeah look at all these modern languages that do so much for you, like .empty() on an array -- have we forgotten how to do simple comparisons?! You could just take a second to consider the properties of an empty array, namely it contains no items (.count() == 0).
My point being, if something is a completely reusable and basic feature, a dependency is totally just. I remember a few years ago when I and all devs I knew (which weren't many, I was maybe 17) had our own libraries to include in all personal projects we made. It contained features we had to look up once and from then on just automated and imported, stuff like password hashing or input checking. This went out of style as single-programmer programs are going out of style, but the little useful features are still there.
One can argue that modules are good, but them depending blindly on newer versions like that was bad dependency management.
I say this because I strongly believe that reinventing the wheel is unnecessary and can bring more problems than not.
There are many examples, and I could come up with a made up one, but here's a very real bug that I debugged from another programmer not so long ago:
So, he came up with a JNI for SPSS's C library, applied it correctly, and got haunted for lots of months with an unsolvable bug. The problem? he miswrote the final copy of the file, and sometimes, some bytes where copied twice.
He tried to solve this problem for a long time (and eventually lived with it because SPSS was still resilient to this)
Is this concrete example of a 'module' ridiculously short? yes, but my logic still holds IMO.
This shows the beauty and success of npm in making it very easy and cheap to publish small modules.
It's not that we can't write a left pad function ourselves. It's that we might easily miss an edge case or make a mistake in doing so.
The author seems to be hung up on a preconceived idea of what a package "should" be without actually offering a compelling argument for why a single short function can't be a module.
Yes, every dependency you introduce is a liability. But so is every line of code you write. I'd much rather take the risk on a shared library which can be audited and battle tested by the entire community.
If a function is so easy to write, it's trivial to check out the module's code, review it does the right thing, and then lock the dependency.
One should think of these requires as like .h files and the underlying code as something like a .c file. They're public definitions with potentially changing underlying code.
It's good to have small packages. Don't forget that the underlying ECMA script is changing so the implementation of these 'libraries' are (or will be) different over time from what they used to be. If somebody finds a faster way to do the method, then it will be done.
Finally, anybody who has used js in the real world understands how many corner cases there are and how difficult it is to make durable methods (i.e. how to know if an array is empty - which requires like 4 different conditions).
On DigitalOcean instance I cannot even use browserify+minify+babel6 cause npm process is killed by host (it consumes > 512Mb of RAM). So I have to manually run browserify + babel then minify. Still it produces 500kb of bundle.js :D
The problem is not of small modules. The problem is lack of dependability. If the language patrons stand behind a set of modules and guarantee continuity and availability, it really doesn't matter what is in them and the world can continue regardless of how insane the module or the whims of any one author. This is not about the technical merits of having or not having a stdlib. The module in question could have been anything.
Making this about is-positive-integer misses the point that this is a social/political problem not a technical one. A language ecosystem must address concerns of business continuity as first class concerns.
My current programming project, my goal has been to do as much in-app as possible. Does that mean I'm more likely to have bugs in my own code? Yes. But I've learned a ton doing it, and I know that my code doesn't have a giant monster of bloat hidden behind some random dependency somewhere. And yeah, that means when I wanted to handle email, I learned a heck of a lot about how programs handle email. Did it take more time? Yup. Education is time well spent.
I've got two dependencies besides the basic framework my project is built on: A scheduling engine, and a database interface. I eventually hope to factor out both.
Spoken like someone who doesn't have a tight deadline.
That's true, I don't. But if your deadline is too tight to do it right, maybe you need to reevaluate your project plan.
I mean, I can certainly see using more dependencies than I'm using. Mail is a great example of something that probably should be handled by dedicated, well-tested code. But to carelessly dump piles of third-party dependencies in your project to save a few minutes, as talked about in this article... just crazy.
1 reply →
Spoken like someone who isn't on a tight deadline.
Somehow, someone always forgets that engineering is about trade-offs, and so every few years we can an indignant series of articles about how stupid and ignorant today's programmers are and how we should all go back to the same processes that they called stupid and ignorant 4-5 years ago.
Relying on reimplementation, copy-paste, npm shrinkwrap, or various other ways of importing third-party code into your repository results in the following advantages:
1. You know exactly what goes into your product, and can audit everything for security, performance, or coding standards.
2. You often end up importing less, as third-party modules may have functionality that you don't need but other clients do.
3. You can modify the resulting code to add showstopper functionality, even if upstream doesn't want to.
4. You aren't subject to the whims of someone removing your dependency from the Internet or replacing it with a version that does something you don't want.
Relying on lots of little libraries installed via package manager gives you the following advantages:
1. You can easily install & try out modules that other people have written, letting you test out new features on users more quickly.
2. You can share code with other modules that have the same dependencies, often reducing the overall size of your system. This is important when there's a cost (eg. download size) to your total bundle.
3. You have less code for your engineers to read & maintain.
4. You can easily track licensing & contact information for your dependencies.
5. You automatically get any new features released by your upstream dependencies.
6. You automatically get security updates and performance enhancements released by your upstream dependencies.
The last is nothing to scoff at: imagine if the headline, instead of 'left-pad breaks the Internet!', had been a security vulnerability in left-pad which literally broke the Internet. Imagine how hard that would be to fix if everyone had copy/pasted the code or re-implemented it. This is not an academic scenario either: remember "Nearly all binary searches and mergesorts are broken", published by the guy who wrote the broken binary search implementation in the Java standard libraries?
http://googleresearch.blogspot.com/2006/06/extra-extra-read-...
Always copying your dependencies into your source tree is not the answer to this, no more than always relying on npm modules was the answer to updating your dependencies. They both have pluses and minuses, and if you really want to be a good programmer, you need to weigh both of them. For my projects, I tend to use whatever libraries I need when building them out (via npm, if possible), and then periodically audit the dependencies to make sure I'm still using them and they wouldn't be better off incorporated directly into the project. I wish more products did this, but I don't control what other programmers do.
7. You automatically get malwares and backdoors released by your upstream dependencies.
Without addressing the wisdom or lack thereof of including dependencies for small functions, perhaps the problem of disappearing/changing small dependencies could be solved with an option along the lines of
Which would just copy the dependency verbatim to <destination>. Maybe have a "<dependency> is 100kb. Are you sure you wish to copy the entire source to <destination> instead of using a 'require' reference? y/n" prompt for the inevitable silly types who'd do it with Angular.
Dependencies are one side of the problem. Unavailability of binaries and dogmatic use of dynamic linkage are the other side.
When I installed a simple lines-of-code counting tool through Macports the other day I accidentally opened the door to dependency hell as gigabytes of not even remotely related stuff started to build [1].
Without a doubt something is going very wrong with Free Software and package managers. On the other hand, never look a gift horse in the mouth so I may not even be the right guy to complain here.
[1] http://pastebin.com/cAZgbaFN
This is from a client-side web dev perspective:
I'm hoping that proper support for ES6 modules to enable tree-shaking bundle builds (in the short term), and HTTP2 with client support for modules (in the long term), will allow us to head towards a world where we converge around a handful of large utility libraries.
In theory, tiny dependencies was supposed to allow us to only include code we actually needed in our bundled code. Bu the reality is that everyone uses different tiny dependencies for solving the same problem. So you end up with enormous bundles made up of different solutions to the same tiny problems.
It's not that we have forgotten how to program. It's that everybody and their dog is now a programmer.
Professional racers don't need an automatic transmission, but it's quite helpful for an unprofessional driver.
What would happen to all those NPM projects if GitHub is destroyed? I don't think it will close anytime soon, but lets say, a meteor shower hits GitHub data center or something along the lines of this.
I still find it hilarious that, upon the bulk of SCM use shifting to a DVCS, the first[1] thing people did is run out to build a central hub.
[1] For the pedantic, it probably was not literally the first thing; I'm sure some folks went to the rest room, others ate something, still others took a nap.
The thing with DCVS is not that central hub does not exist. It is that you do not need specific central hub. That means if github dies, I can take any of my git repos I checked from github and continue working with them, and create a local git server that would allow others to work with it. Of course, there is a convenience value in having central hub. The point is not denying that, the point is turning it from necessity to option.
That's because central hubs are very useful...
Well npm doesn't use GitHub for hosting of modules so honestly npm would be fine. Bower on the other hand would be completely screwed.
It is the nature of git that most repos on GitHub also exist in many other places.
Smart people created software that lowered the barriers of entry to making software. It was inevitable that not-so-smart people would eventually be writing software that others would build upon.
The author brings up an excellent point, but I disagree with the solution. We should of course reuse existing, well-tested code if it available, even for simple things like left-padding. The real issue here is that there is a module for left-pad alone. If it were something like StringUtils module with a bunch of commonly used string functionality, it would have been great.
What is it about the node community that triggered this misunderstanding of package management and code reuse?
Hell, the java programmers I work with seem to never use each others simple functions and instead recreate the will every single time.
As for the issue with such a short piece of code being reused by many, why score on number of lines? If it works and is a useful function is more important to me. I am not familiar with bundling within the usage the article covers but we tend to bundle like functions together and the compiler drops unused ones
I guess this is the way one is supposed to use node, preventing one to write non-DRY code. (Golang takes the exact opposite approach, having it's own drawbacks of course.) However, when using React, I kind of trust that the maintainers don't include packages than require 12 star projects, and if, that they fork this stuff themselves.
BTW, isn't that a Facebook project, so aren't they supposed to use a CI? ;P
I couldn't agree more. I've been using the new ES6 style module syntax for a few days now because a co-worker forced me to, so he would use my library.
I'm not convinced its worth it compared to the simplicity of commonjs and module.exports. You have to pull in babel, which has over 40k files to do all this.
Why are people destroying the beautiful simplicity that is javascript? Can those people please go back to java?
Regardless of whether micro-modules are good or bad, I think that if you are the owner/manager of a project, you should be able, given its full flat list of dependencies, to explain why each one of them is useful for you.
Every project I've seen that uses npm always required 100s or 1000s of dependencies.
If building or running your project requires something and you can't explain why, I think there's a problem.
On the one hand, I ,love how the JS community is constantly coming up with new [versions of old] things. Even though most of it is creative waste, it's still creative and out of that churn we get some pretty awesome apps and tools.
On the other hand, there's a lot of bad practices disguised as "simple" and "efficient." Using NPM for one line functions is a great example of this.
Of course I would never create a dependency on a small module like left-pad. I would simply copy the function wholesale into the body of my code!
So funny, just few weeks back I had an argument with somebody about writing a simple functions vs. importing libs when you need less then 5% of the functionality. I am more convinced than ever that it is better off to have the least amount of external dependencies. Of course I would not want to rewrite a 2M+ LOC library with very complex code, but left pad is not one of those use cases.
I'd just like to point out that React does not have a dependency on left-pad. React has no dependencies at all[0]. devDependencies have no impact on third party consumers.
[0] https://github.com/facebook/react/blob/master/package.json
The author missed the point about modularity in Javascript.
Small packages done right, well tested = maintenable, reusable, stable code.
The problem does NOT comes from packages. The problem comes from un-publishing public packages and centralized repository server.
I have a java project with a lot of dependencies. Does it mean it's bad ? No, but if maven repos are closing tomorrow, my project will not build as well.
Why was it removed anyway? I agree that the ability to unpublish something is the real problem, but I wonder why the author actually unpublished it. I wonder if the author knew about all the projects that depend(ed) on it. Maybe he/she actually did it as an evil experiment, though a very interesting and eye-opening experiment. Does anyone know?
https://medium.com/@azerbike/i-ve-just-liberated-my-modules-... discussed here in this thread : https://news.ycombinator.com/item?id=11340510
I'm content and happy not knowing what the fuck this is all about. Pad left? Are you kidding me? If your tower og babel fell because of your reliance on some script kiddie's toy project, I am happy and content knowing you get what you deserve. Law of leaky abstractions, motherfucker. Spolsky...do you read it?
Maybe some people just want to claim they have a published module. Making them feel some sort of achievement or glory.
Not just making them feel... I think this may be targeted towards recreuiters, who may be impressed by the sheer amount of npm modules someone has published.
Look at these left pads...
https://github.com/search?utf8=%E2%9C%93&q=left+pad
https://github.com/nameoverflow/left-pads
Well it has always been like this. In early 2000s most developers I knew would simply copy-paste code from all over the Internet.
Nowadays you use NPM search instead of Google search
The fact is that lazy programmers are lazy. The methods change but the principle remains the same. In 90s people typed in code from magazines and books.
If the code is open source, what difference does it make if the code is in my module or someone else's?
I just removed the tilde and caret of all my dependencies (in my package.json) and maybe that's the way to go. Seal a local version of your packages and don't update unless is completely needed. But I'm still worried about the fragility of the package environment.
For more fun, my commentary on leftpad code: http://www.win-vector.com/blog/2016/03/more-on-npm-leftpad/
The core issue here is that there is no included standard package in Javascript. It happened because Javascript did not have authoritative implementation when it first began. Next ECMA should require to pack some batteries in to avoid this micro-module hell.
It seems to me that the JavaScript VMs should get together and start including standard library. That would also give the benefit that those would be highly optimized. They can keep it small at first and focus on number and string manipulation.
Packaging is the new programming.
Clearly this is insane - but what the solution?
Part of the problem, in my opinion, is that packages are not subject to any review process. These dependency chains get out of hand because they are rarely top-of-mind.
By the way, ES6 syntax (works in modern foxes and chromes):
leftpad = (str, len, pd = ' ') => Array(len > str.length ? 1+len-str.length : 0).join(pd) + str
WTF are you talking about? Making this into a module?!
Here's a quick rule of thumb. If it's the kind of function you would ask a candidate to write at the start of a job interview, you shouldn't be importing a separate module to do it.
Should you rewrite it for every project you need it in, or copy and paste from the last one?
Should tests also be rewritten or copied each time you do this?
What do you do when you've used it in multiple projects and you find a bug or performance issue?
.... isn't this a problem with node, and not developers? Wouldn't you say this is a symptom of a systemic problem of the framework that it is lacking common features that everyone needs?
That's what you get when you let JavaScript into the server side.
What's wrong with reusing small fragments of code?
The usual complains about many dependencies are mostly void (it's not bloat if you only depend on single functions you actually use).
When am I developing some large project for a long time, sometimes I find, that I have reimplemented the same function at several places (few years between implementations).
"Small modules are easy to reason about"
No... "appropriately sized modules are easy to reason about"
In this case... "Appropriate" has gone out of the window!
Everyone keeps mentioning the lack of a standard library for JavaScript as an excuse for this shit show. IMO this is just a futile attempt to mask incompetence.
This.
Any modern standrards-compliant implementation of JS (following ES5 and especially ES6/ES2015 standard) already has everything that a sane programmer would ever need.
What actually happens when you try to update a required package, but it is gone from upstream? Is there no way to keep the existing package you already have?
I find it interesting that blaming code reuse is a valid thing, but blaming a lack of test coverage and CI build testing is not.
The problem is the lack of a test culture.
Check out the work on Morte for a more reasoned approach for how to take micro-modularization to it's natural (or insane) conclusion.
So sad yet so true.
We haven't. React developers probably have.
It's up to you how to go about it on server side you for example can go with express or hapi (minimal external dependencies).
There's nuance to the discussion that both sides are missing. People argue forcefully whether these small modules are good or bad, but I'm not seeing much evidence that they understand the other side.
First: why small modules are bad. Lots of dependencies complicate your build, and you end up with the dreaded diamond dependency issue. Failures in a dependency become more likely to affect you. It gets you in the habit of using prebuilt modules even if maybe it's not quite what you need and it would have been better to write yourself. With `npm` specifically, we've seen how its mutability can break the build, though that's about `npm` and not the idea necessarily.
I think most software developers' gut responses are that something is wrong and crazy in the npm ecosystem.
That said, there are benefits that this blog post and others aren't mentioning, related to the javascript situation specifically.
The first one is that javascript is a surprisingly difficult language to get right. Sure, the final solution is only a few lines, but which lines are hard. You have to navigate the mindfield that are is V8 in nodejs, v8 in chrome, spidermonkey, chakra, etc. I've had code work in Chrome before but blow up in IE, and it's really hard to track down and test.
The comments in the blog post are illustrative:
One line of code package:
Crazy right? And my first stab probably wouldn't have been to implement it that way. Why not:
that a commenter suggested, which should be faster? Well another commenter said:
I've run into issues before with cross-browser compatibility stuff, and it's frustrating and hard to test. If there's some de facto standard package that implements it for you, hopefully the community can iron out edge cases.
The other thing that people don't bring up, is that there's not much JS standard library, and in the browser context you have to send all your code to the front end.
So maybe you write these 11 lines yourself, and then another package writes these 11 lines, and another... it adds up. But if everyone uses the same package, the code only gets sent once and they all share it.
Lastly, people talk about how `sin` should be a part of a "trigonometry" package and not by itself. Well, again you're faced with sending a bunch of unnecessary code to the frontend. With webpack2 and tree shaking, or e.g. Google's Closure compiler, it can strip out dead code and so this issue will go away in the future, but we're not quite there yet. So package authors still bundle all these things separately.
So pros and cons.
It suddenly feels like the 90s all over again.
You mean like the sad story of PKzip?
May you share the story ?
1 reply →
Surely there is a need to standardise on a set of well-maintained "batteries included" packages.
That is what PHP has, and it received countless attacks on that approach during the years; although most of those attacks were due to the inconsistent function naming.
npm had the solution one year ago (namespacing) https://docs.npmjs.com/getting-started/scoped-packages if only developer can embrace the "change"
I'm generally not a big fan of IPFS, but IPFS seems like the perfect solution to this problem.
when you go out for food or order in, do you ask yourself "have I forgotten how to cook?"
Sure in some ways NPM has packages that don't deserve the title of a package, but isn't the convenience of not having to reinvent every code worth it?
Isn't the problem that there is a lack of a good standard library in Javascript?
This is why one guy can now compete with say Google or Microsoft, because that guy uses code written and managed by more engineers then both Google and Microsoft have combined. Instead of paying hundreds of dollars to said companies, you can just NPM install "what you need".
Libraries/modules should help to solve complex problems where writing your own version is not optimal. Single line functions as modules, on the other hand, won't help you compete with Google or Microsoft and result in more problems than they solve. If a simple padding or "is this int positive?" is a complex problem, well, there are different professions than programmer's..
The less code you write, the less code you have to maintain. Even if it's just one less thing, it's a win.
The comonJS module pattern also helps tremendously, as your requires are block/function scoped. It makes it trivial to refactor (remove/rewrite) code.
1 reply →
IMHO, this is all due to lack of a standard library in the Javascript world.
Seems that we need a tool to crawl all the repos that a developer owns and report lines of code in each package that they wrote. If there are lots of small packages, then this is not the kind of person you want to hire, except maybe to do PR.
Real developers do not do stuff like this.
> Even if correct, is it the most optimal solution possible?
"most optimal"?
JavaScript was a mistake.
I don't have an issue with modules or the Unix philosophy, I have an issue with using NPM for all these tiny modules. Hint: you can make your own modules, store them within the project, and add them to your package.json to be required anywhere.
It's not about programming, it's about fast progression.
It does seem pretty insane, but how many of these are polyfills?
Forgotten? Most people who develop on Node.js never knew...
Re-usability is a good concept but can be overused easily.
This is acceptable because web developers expect a certain level of coddling. Many developers are quick to defend this insanity because they simply cannot see their own level of entitlement.
What's next? A hello-world package and Node tutorial about hello world program being include this package as dependency and call the exported function.
All the comments here have answered the question quite well. Yes we have forgotten how to program and we want to argue about it.
I don't see nobody criticizing 'std::min()'. Perhaps what we really need is a 'std' for js?
`std::min` isn't a library of its own, it's a function in the `<algorithms>` header of the Standard Template Library. The STL also encompasses a wide range of other things. There are some efforts to provide "the missing standard library" for JS, e.g. lodash.
Then the problem isn't that people don't know how to program, rather that js is missing standard functions. In the C++ world there is a large push to avoid rolling your own things, indeed even simple constructs like `for` loops are starting to be frowned upon.
No saner alternative presented.
What I don't get about Left-Pad, shouldn't they have used Arrays.join for better perormance?
Array performance isn't always better than string concatenation.
In this test case it comes out slower: https://jsperf.com/left-pad-arrays/3
But ultimately it doesn't matter in the least for 99.9% of use cases:
https://blog.codinghorror.com/the-sad-tragedy-of-micro-optim...
Interesting - well I thought if it is being used by thousands of libraries it might be worth optimizing.
I just tried in NodeJS, you could also just instantiate an Array of a given length and join it with spaces, like this.
var x = new Array(5);x[4] = "abc";x.join(" ");
No idea if it could be faster - but the test you linked to pushes spaces into the Array.
here is one of the comments to original post:
"Immutability at the centralized authority level and more decentralization of package distribution is the solution, not 'write more functions yourself'."
what the fuck does that mean?? they just don't give up, do they... Fucking retards.
I remember how I have been downvoted to oblivion for comparing JavaScript madness with Java EE "packer's" paradise years ago.
The Programmer's Stone first essay is actual as it never been before.
Actually, it is a common pattern. When some activity becomes popular due to very low barrier to enter it will end up in a such kind of mess. It seems like nowadays everyone is either a programmer or a scientist and researcher.
This is the quality of their software and research.
There has been a reason why good schools taught principles (algorithms and data structures) not particulars (objects and classes). But since MIT and Berkeley dropped Scheme-based courses in favor of "pragmatic" Python (thank god not JavaScript) based courses we are heading to a disaster. Java madness taught us nothing.
History is full of examples where assault by mediocrity ruined the whole branches of philosophy, arts and crafts. Instead we have fastfood, mass media, social media and now this mass coding, which combines worst from mass and social.
Just try to compare things like Smalltalk or Plan9 or R4RS Scheme or Zeta LISP of Symbolics or with this stuff.
This is, finally, the "Fractal Of Bad Design" moment for JavaScript.
LOOL..I agree with this one.
What's with these kids and their "calculators". Back in my day we used a slide rule and we liked it!
But seriously this is stupid. Programming shouldn't be the goal. Just because you can write a function doesn't mean you should. Every line of code you write is overhead that must be tested and maintained. I guarantee that if the author chose to hand roll code instead of using packages he'd have a lot more bugs. But he wouldn't know that until he hit some mundane edge case scenario in production.
"Every line of code you write is overhead that must be tested and maintained."
The problem is, that modules doesn't solve problems magically. Every dependency should be tested, maintained and kept online as well, by somebody else, whom you have no control of. In turn, you have a bunch of black boxes that can blow up your application anytime. Of course, another extreme of write everything yourself is also bad, so it's a trade-off that must be considered carefully. I've build many systems during my career and one of the biggest nightmare I encounter is managing dependencies.
> What concerns me here is that so many packages took on a dependency for a simple left padding string function, rather than taking 2 minutes to write such a basic function themselves.
Wait -- code reuse is bad now??
YES... it is... its a trade off... code reuse has a upside and a downside...
Isn't that the least efficient way to do that function? Prepending a string has always been very expensive operation.
Calculating needed length. Using repeat and just concatenating 2 strings would be faster.
V8 internally uses cons-style strings for these operations, and only flattens the string when necessary for other operations.
> if you cannot write a left-pad, is-positive-integer, or isArray function in 5 minutes flat (including the time you spend Googling), then you don’t actually know how to code.
I'd probably get 2 of those wrong in some weird way, but I blame javascript. I mean without Google of course.
Programmers haven't forgotten how to program. JavaScript script-kiddies never knew how to program in the first place.
I see two kinds of programmers here.
Good programmers understand the risks of making your system depend on something you don't have control really well; They know how keeping system complexity low is like an good investment which makes your life later easier (low maintaining costs).
Bad programmers stacks up technical debts such as including unnecessary dependencies until the system no longer works.
> Good programmers understand the risks of making your system depend on something you don't have control really well
For example, the quality of your own code.
Technical debt doesn't stop being technical debt just because it's your code.
It's js .. they never learned to program.
No it's because they don't do the same thing at all. This comparison shows you clearly don't understand the first thing about promises.
This comment as well as https://news.ycombinator.com/item?id=11351657 and marked it off-topic.
If you're new to technical discussions, people normally provide supporting arguments. Both of those handle sequences of tasks that need to complete in order. "No you don't understand" without any further info is not a response.
I think if you don't even know the basics of something, you should go learn them before discussing it.
Ahh job security :D
Ahhh job security :D
This post is too dismissive and confuses some topics.
"Packages", "modules", "functions", and other words can mean different things within different contexts. Well-known and tested functions are useful. Putting them in a "package" is just a consequence of how node/npm work. There should certainly be a much, much better implementation of code sharing, but sharing and using well-known, singular functions should be exactly what we are going for.
npm install goldmansachs
Functions Are Not Packages
Cracked me up :D
I don't think it is bad at all. For a lot of project where saving even a minute matters.
This is just more evidence that the unit of code is a function, not a module.
Eventually, given a pure enough language, every "package" could contain only a single function, and every function in a project could be published as an independently reusable unit.
Only if your modules are little more than textual inclusion and perhaps a mechanism to deal with name clashes. ML style modules are way more substantial than that, they can be used as units of abstraction.