Comment by torginus
5 months ago
Very interesting article and good points.
But how about we flip the original statement around:
Many problems thought to require giant software packages/libraries are very solvable locally if you know what you are doing
This is why LC is actually meaningful - imagine if you faced the coin challenge problem IRLin prod and you decided to pull in a constraint solver - what would've been a 25 line function now is a giant python library that carries 30MB of native dependencies around.
These solutions are often lacking in many ways, since mentally offloading some of the work means you don't have to understand the problem in depth or how the designated tool solves it, which can lead to nasty surprised in production.
We see this everywhere - people pulling in questionable npm packages to save themselves 30 mins of thinking or 20 mins of reading docs - people convinced that you do need that huge 3D framework that comes with an editor to make a WebGL widget on your landing page etc.
You are more willing to accept bloat of others, just because they pulled in Electron, because they were too afraid to learn how native UI works.
People need to be more curious, and strive to be more knowledgeable.
You are right to be sceptical of “dependency bloat” - many many systems could be made simpler to debug and maintain if that advice was followed.
But: I urge you to give constraint solvers a try. This is something very different from some hipster ORM library. The good ones will be a single dependency containing extremely carefully optimised matrix math code that let you solve optimisation problems faster and with much more maintainable code.
Try some toy problem with the HiGHS solver for instance, it has good python bindings and even ships as a wasm package you can run in browsers. “Find the optimal times to run this heat pump given electricity prices”, “find the ideal train departure times that optimise for when people need to travel and minimises transfers”
Wait are you making the opposite claim? That one should eschew the "correct" formulation in favor of a bespoke one? Despite the stated (and hopefully obvious) difficulties that brings with maintenance, generalization, etc?
You probably haven't see front-end projects that pulls tons of library for a simple sorting or grouping task. Sometimes even solvable with build-in array function alone. It's a true nightmare when you have to deal with that kind of projects.
What, like lodash? Some of those hail from a time that we didn't have a good set of native methods. So the library is just legacy'd in. But I do agree, lodash performance compared to native functions is crazy bad.
5 replies →
I'm sorry but after reading your comment I can't seem to be able to decide if you favor writing the dynamic programming version or pulling in the constraint solver.
Both are correct in the sense that they give the right output, and I don't think pulling in a huge library (maintained by who knows and for how long) is going to be beneficial for maintenace. And having a good understanding of both the precise requirements, and the algorithms used to solve them also helps maintenance.
It's just having two dozen lines tucked of code away in a function in the repo seems infinitely more maintainable to me then using some giant framework (of possibly unknown quality) to solve the issue.
This is a general argument I'm making, not just applying to this constraint solver/
There are other aspects to maintenance, like requirements change. In this case it's trivial to change or add new constraints to a constraint solver, whereas even small changes to a typical DP problem can require a total rethink of the approach. Extending the analogy to other kinds of dependencies left as an exercise for the reader.
Point being that software has many dimensions. Reducing the use of dependencies to fear of learning or thinking is a bit reductive in my opinion, even for stuff that seems simple initially.
2 replies →
I agree with essentially everything you said. While reinventing the wheel isn't always to most efficient solution, such actions have sometimes paid dividends in my past.
> People need to be more curious, and strive to be more knowledgeable.
Absolutely, though I am not sure lack of curiosity nor strive is always the issue. It's one thing to green field one's own solution in an attempt to better one's abilities, but it's also a bit idealistic in a working world of impending deadlines and death-march Agile sprints.
In this case, a 25 line function to solve a constraint would likely take me far less time than grokking an external library's documentation, but in many cases, I feel developers jump on external dependencies because time is of the essence.
hard disagree:
Given ANY problem: first ask yourself have others solved this before? that is a hard question to answer, since we don't know in which context a similar puzzle was solved before (structures in different domains can boil down to the same mathematical puzzle). A literature search would be very time costly.
The most important point is the flexibility in being able to change the puzzle (a small change in the puzzle can result in a large change in the type of solution), as the author of the article points out. The bespoke algorithm is brittle. Description of the problem itself is less brittle (you can reuse most of the problem statement).
It may sound incredibly expensive to pull in a constraint solver, but if the application warrants constraint-solver-quality results, it should afford either the dependency, or the data structures and solvers used in the solver dependency, to optimize for the application,
Its just bizarre to ask people to beat the feats of those standing on the shoulders of giants, without allowing them to stand on the shoulders of giants too.
Think about why one is recruiting employees with data structure skills (so more than just information plumbers). Is it really so strange that the most qualified people understand the reality that the state of the art is constantly changing, but understand at least the basics of how these solvers work internally?
Viewed through this lens, the ideal job candidate are those who design, implement and maintain... constraint solvers! Assuming their familiarity with the constraint-solver code-base they could profile the solver package while its solving the puzzle. Do this on many instances of the puzzle, and keep track of the dead-ends and optimal solutions, to figure out which functionality can be ripped out of the solver, and which must be kept.
So in order of preference:
1. Programmers or mathematicians (or equivalent, think physicists, etc.) familiar with 1 or more constraint solver source code bases.
2. End-users of a constraint solver package, with sufficient familiarity (as a user) with constraint solvers. WITH data structure and algorithms experience.
3. End-users of a constraint solver package as above, but without data structure and algorithms knowledge
4. People with data structure and algorithms knowledge.
If 1. is too expensive you'll have to combine skills over multiple hires.
If industry is really interested in good profiles for these criteria, it should sponsor universities / students to get familiar with constraint solver usage, and if possible constraint solver development and contributions. After a few years the candidates will pop into existence.
>>It may sound incredibly expensive to pull in a constraint solver
Like this is actually kinda the point of the article. On every constraint programming article half the comments insist that they could do the (example) problem in a dozen lines of C so bringing in SCIP or OR-Tools is "too much" for the (example) problem. Wayne's point (here and in other articles he's written) is that actually constraint problems are fucking everywhere.
This is one of the real lessons of learning Prolog: data queries, scheduling, SAT, LP, integer programming, optimization... etc; it's all constraint programming, hell parsing is a constraint problem if you do it right. No one sees it like that though so they see their entire business application logic as containing at most one knapsack.