Comment by hsn915
3 years ago
Relying on configuration files to glue together a huge stack of tools and services in order to produce an end product is a huge mistake. I say it's a mistake fully knowing that nearly everyone in the industry is doing it.
Configuration is opaque. Undebuggable. Unmaintainable. By its very nature. No matter what "language" you use for it.
You should strive to keep configurable things to an absolute minimum.
Although it's worth making a distinction between "configurations" and "settings": when something is configured wrong, the application basically fails to operate as intended. For settings, there can't be "wrong" settings, all settings are valid, and if the input for a specific setting was invalid, the application can simply ignore it and use the default value. Example of a "setting": font size for a text editor. Example of configuration: the connection information for a database.
Wholeheartedly agree - convention before configuration wins with tooling even at modest scale.
Larger team(s) using free form yaml configuration and templating quickly becomes messy.
Setting a boundary at “configuration” and exposing configuration options to dev teams through settings works really well.
It requires you to have a tooling team with a few developers, but it scales!
Convention over configuration is really the future, but not everything follows that philosophy. For example, OCaml codebases are basically a free for all and you can organize things however you want. So with that in mind, you need something to organize and build.
Convention seems to be nice in the beginning. But it's very hard to maintain.
Much better is default configuration. I.e. all "conventions" are explicitly generated into a default-configuration file which you can then change/overwrite/update in whatever way you want.
1 reply →
One problem with convention is that it isn't discoverable. At least config files guide you to where to look for stuff.
But even so I think you're probably right that convention is better because it forces everyone to use the same structure for stuff.
Convention before configuration doesn't solve the core problem.
Why do you need configuration? Because you build your application by gluing together a multitude of different applications that need to configured properly to be able to talk to each other.
Once you understand this, the solution is obvious: don't build your application this way.
Build it in one language. If you want to "reuse" existing solutions, reuse them as libraries, not as separate programs.
Say sql database:
Teams have agreed to use postgres for sql, this is by agreed convention.
A tooling team implement hashicorp vault and build a pg module/deployment that is packaged and accessible through whatever tooling you build.
This makes a datasource of type pg sql available to all teams in a couple of minutes.
These modules/deployments are governed by a bunch of conventions, but those are of no special concern to consuming dev teams.
Use “something” to keep track of service upstream/downstreams: consul, a database, or something else.
When a service is connected to a database, say using consul upstreams you have the job gen api inject the appropriate env vars according to an agreed upon convention.
Now dev teams have an option to deploy incredibly secure pg databases, and by simply setting a service as downstream it will automatically receive rotating connection strings.
There’s no configuration to be managed by dev teams except keeping a service graph up-to-date.
Lot of conventions above, and a tooling dev team with a lot of infra and tooling expertise.
I believe that for scaling dev teams you should look at tooling like a product domain and treat it as such.
> Configuration is opaque. Undebuggable. Unmaintainable. By its very nature. No matter what "language" you use for it.
What makes you think so? How do you define configuration?
I mean, in the extreme case you can see Python as a configuration language for the behaviour of the Python interpreter. Does that make Python a configuration language? Where do you draw the line?
Configuration is some sort of information that is stored in a text file, for reading by one or more programs, where these programs cannot meaningfully operate without this critical information in the configuration files.
If you have ever worked in the web startup industry, their use is endemic in everything. You often cannot deploy any web applciation without properly understanding tons of configuration files.
Because everyone builds applications out of services that must be glued together, and these services often glue themselves together via configuration files. The python side of the application code needs to figure out how to connect to all the databases (postgres, redis, elastic, memcached ... etc). Each of these databases sometimes needs its own configuration before it can meaninfully operate. And because everyone is using Docker, you need docker files for all of your services. The list of things that need text based configuration fiels goes on and on.
Go to any web company and ask around to see who can meaningfully understand and edit or maintain these configuration files, and you will find the majority of the people have no idea what's going on inside them. It's very common that only two people have a decent understanding of all the configs.
Now to answer your core question:
Why are they opaque and undebuggable?
Because to understand the configuration files, you have to understand the programs they are inteded for and the environments they end up in, and the language they are written in.
For example, to understand a three-line docker compose configuration for ha_proxy, you have to understand the language of docker compose, and the way configuration files are aggregated. For example, you can define some VARIABLES in one config files and have them be available for reading by another docker compose config file in a totally different place. But you have to understand under what conditions which files are included together. For instance, you can have 10 files define different values for one VARIABLE and each of them runs in a different environment. You have to understand all of this. And we haven't even gotten to ha_proxy yet. Now you have to understand how ha_proxy is influenced by the configurations you have specified. Let's say it's telling it about another server to connect to, or a directory to serve static assets from. The path of this directory depends entirely on how other docker images are composed together, which in turn is usually influenced by the contents of many other docker compose config files that are - again - spread out across many files. You need to figure out _which_ config file is used to make a certain directory available at a certain location, etc etc.
There's no tool that can debug this mess. There's no substitute for deep and intimiate knowledge about all the details of the system.
It's very possible for the whole system to collapse when you make one tiny small mistake in one configuration file. There's no tool that can help you debug what's going on.
> Because to understand the configuration files, you have to understand the programs they are inteded for and the environments they end up in, and the language they are written in.
Seems like the right approach. To use something, one has to understand how to use it. Why is this unusual?
1 reply →