← Back to context

Comment by dham

5 years ago

Yes let's please do this. I'm tried of this book being brought up at work.

My clean code book:

* Put logic closest to where it needs to live (feature folders)

* WET (Write everything twice), figure out the abstraction after you need something a 3rd time

* Realize there's no such thing as "clean code"

> WET (Write everything twice), figure out the abstraction after you need something a 3rd time

so much this. it is _much_ easier to refactor copy pasta code, than to entangle a mess of "clean code abstractions" for things that isn't even needed _once_. Premature Abstraction is the biggest problem in my eyes.

Write Code. Mostly functions. Not too much.

  • I think where DRY trips people up is when you have what I call "incidental repetition". Basically, two bits of logic seem to do exactly the same thing, but the contexts are slightly different. So you make a nice abstraction that works well until you need to modify the functionality in one context and not the other...

    • If you mostly deduplicate by writing functions, fixing this problem is never very hard: duplicate the function, rename it and change the call-site.

      The interesting thing about DRY is that opinions about it seem to depend on what project you’ve worked on most recently: I inherited a codebase written by people skeptical of DRY, and we had a lot of bugs that resulted from “essential duplication”. Other people inherit code written by “architecture astronauts”, and assume that the grass is greener on the WET side of the fence.

      Personally, having been in both situations, I’d almost always prefer to untangle a bad abstraction rather than maintain a WET codebase.

      13 replies →

    • Yep. Re-use is difficult. When you overdo it you cause a whole new set of problems. I once watched a person write a python decorator that tried unify HTTP header based caching with ad hoc application caching done using memcached.

      When I asked exactly what they were trying to accomplish they kept saying "they didn't want caching in two places". I think anyone with experience can see that these items are unrelated beyond both having the word "cache"

      This is what was actually hanging them up ... the language. Particularly the word "cache". I've seen this trap walked into over and over. Where a person tries to "unify" logic simply because two sets of logic contain some of the same words.

    • So much this. Especially early in project's life when you aren't sure what the contexts/data model/etc really need to be, so much logic looks the same. It becomes so hard to untangle later.

    • Then you copy function and modify one place? I don't get what is so hard about it. The IDE will even tell you about all places where function is called, there is no way to miss one.

  • > it is _much_ easier to refactor copy pasta code

    So long as it remains identical. Refactoring almost identical code requires lots of extremely detailed staring to determine whether or not two things are subtly different. Especially if you don't have good test coverage to start with.

    • I personally love playing the game called "reconcile these very-important-but-utterly-without-tests sequences of gnarly regexes that were years ago copy-pasted in seven places and since then refactored, but only in three of the seven places, and in separate efforts each time".

  • There's a problem with being overly zealous. It's entirely possible to write bad code, either being overly dry or copy paste galore. I think we are prone to these zealous rules because they are concrete. We want an "objective" measure to judge whether something is good or not.

    DRY and WET are terms often used as objective measures of implementations, but that doesn't mean that they are rock solid foundations. What does it mean for something to be "repeated"? Without claiming to have TheGreatAnswer™, some things come to mind.

    Chaining methods can be very expressive, easy to follow and maintain. They also lead to a lot of repetition. In an effort to be "DRY", some might embark on a misguided effort to combine them. Maybe start replacing

      `map(x => x).reduce(y, z => v)` 
    

    with

      `mapReduce(x => x, (y,z) => v)`
    

    This would be a bad idea, also known as Suck™.

    But there may equally be situations where consolidation makes sense. For example, if we're in an ORM helper class and we're always querying the database for an object like so

      `objectContext.Orders.Select(e => e.id = y).Include(e => e.Customers).Include(e => e.Bills).Include(e => e.AwesomeDogs)...`
    

    then it with make sense to consolidate that into

      `orderIncludingCustomersBillsAndDogs(id) => ...`
    

    My $0.02:

    Don't needlessly copy-pastes that which is abstractable.

    Don't over abstract at the cost of simplicity and flexibility.

    Don't be a zealot.

  • >it is _much_ easier to refactor copy pasta code

    I totally agree assuming that there will be time to get to the second pass of the "write everything twice" approach...some of my least favorite refactoring work has been on older code that was liberally copy-pasted by well-intentioned developers expecting a chance to come back through later but who never get the chance. All too often the winds of corporate decision making will change and send attention elsewhere at the wrong moment, and all those copy pasted bits will slowly but surely drift apart as unfamiliar new developers come through making small tweaks.

  • I worked on a small team with a very "code bro" culture. No toxic, but definitely making non-PC jokes. We would often say "Ask your doctor about Premature Abstractuation" or "Bad news, dr. says this code has abstractual dysfunction" in code reviews when someone would build an AbstractFactoryFactoryTemplateConstructor for a one-off item.

    When we got absorbed by a larger team and were going to have to "merge" our code review / git history into a larger org's repos, we learned that a sister team had gotten in big trouble with the language cops in HR when they discovered similar language in their git commit history. This brings back memories of my team panicked over trying to rewrite a huge amount of git history and code review stuff to sanitize our language before we were caught too.

    • Wait whaat? I am not from USA and hail from a much more blunt culture, what's wrong with those 2 statements?

      To me they're harmless jokes but maybe someone will point out they're ableist?

      2 replies →

    • If you were "caught", what would happen? You probably have a couple of zoom calls and get forced to watch sensitivity training videos. Who cares.

  • > it is _much_ easier to refactor copy pasta code,

    Its easy to refactor if its nondivergent copypasta and you do it everywhere it is used not later than the third iteration.

    If the refactoring gets delayed, the code diverges because different bugs are noticed and fixed (or thr same bug is noticed and fixed different ways) in different iterations, and there are dozens of instances across the code base (possibly in different projects because it was copypastad across projects rather than refactored into a reusable library), the code has in many cases gotten intermixed with code addressing other concerns...

  • > Write Code. Mostly functions. Not too much.

    Think about data structures (types) first. Mostly immutable structures. Then add your functions working on those structures. Not too many.

    • OMG. This is exactly my experience after trying to write code first for 10+ years. (Yes, I am a terrible [car] driver, and a totally average programmer!)

      "Bad programmers worry about the code. Good programmers worry about data structures and their relationships." - Linus Torvalds

      He wasn't kidding!

      And the bit about "immutable structures". I doubted for infinity-number-of-years ("oh, waste of memory/malloc!"). Then suddenly your code needs to be multi-threaded. Now, immutable structures looks genius!

I think this ties in to something I've been thinking, though it might be project specific.

Good code should be written to be easy to delete.

'Clever' abstractions work against this. We should be less precious about our code and realise it will probably need to change beyond all recognition multiple times. Code should do things simply so the consequences of deleting it are immediately obvious. I think your recommendations fit with this.

  • Aligns with my current meta-principle, which is that good code is malleable (easily modified, which includes deletion). A lot of design principles simply describe this principle from different angles. Readable code is easy to modify because you can understand it. Terse code is more easily modified because there’s less of it (unless you’ve sacrificed readability for terseness). SRP limits the scope of changes and thus enhances modifiability. Code with tests is easier to modify because you can refactor with less fear. Immutability makes code easier to modify because you don’t have to worry about state changes affecting disparate parts of the program.

    Etc... etc...

    (Not saying that this is the only quality of good code or that you won’t have to trade some of the above for performance or whatnot at times).

  • The unpleasant implication of this is that code has a tendency towards becoming worse over time. Because the code that is good enough to be easy to delete or change is, and the code that is too bad to be touched remains.

> * WET (Write everything twice), figure out the abstraction after you need something a 3rd time

There are two opposite situations. One is when several things are viewed as one thing while they're actually different (too much abstraction), and another, where a thing is viewed as different things, when it's actually a single one (when code is just copied over).

In my experience, the best way to solve this is to better analyse and understand the requirements. Do these two pieces of code look the same because they actually mean thing in the meaning of the product? Or they just happen to look the same at this particular moment in time, and can continue to develop in completely different directions as the product grows?

  • Solving the former is generally way uglier/more obnoxious IMO than solving the latter, esp. if you were not the person who designed the former.

I read Clean Code in 2010 and trying out and applying some of the principles really helped to make my code more maintainable. Now over 10 years later I have come to realize that you cannot set up too many rules on how to structure and write code. It is like forcing all authors to apply the same writing style or all artists to draw their paintings with the exact same technique. With that analogy in mind, I think that one of the biggest contributors to messy code is having a lot of developers, all with different preferences, working in the same code base. Just imagine having 100 different writers trying to write a book, this is the challenge we are trying to address.

  • I'm not sure that's really true. Any publication with 100 different writers almost certainly has some kind of style guide that they all have to follow.

> WET (Write everything twice)

In practice (time pressure) you might end up duplicating it many times, at which point it becomes difficult to refactor.

  • If it's really abstractable it shouldn't be difficult to refactor. It should literally be a substitution. If it's not, then you have varied cases that you'd have to go back and tinker with the abstraction to support.

    It's a similar design and planning principle to building sidewalks. You have buildings but you don't know exactly the best paths between everything and how to correctly path things out. You can come up with your own design but people will end up ignoring them if they don't fit their needs. Ultimately, you put some obvious direct connection side walks and then wait to see the paths people take. You've now established where you need connections and how they need to be formed.

    I do a lot of prototyping work and if I had to sit down and think out a clean abstraction everytime I wanted to get for a functional prototype, I'd never have a functional prototype--plus I'd waste a lot of cognitive capacity on an abstraction instead of solving the problem my code is addressing. It's best, from my experience, to save that time and write messy code but tuck in budget to refactor later (the key is you have to actually refactor later not just say you will).

    Once you've built your prototype, iterated on it several times had people break designs forcing hacked out solutions, and now have something you don't touch often, you usually know what most the product/service needs to look like. You then abstract that out and get 80-90% of what you need if there's real demand.

    The expanded features beyond that can be costly if they require significant redesign but at that point, you hopefully have a stable enough product it can warrant continued investment to refactor. If it doesn't, you saved yourself a lot of time and energy worrying trying to create a good abstract design that tends to fail multiple times at early stages. There's a balance point of knowing when to build technical debt, when to pay it off, and when to nullify it.

    Again, the critical trick is you have to actually pay off the tech debt if that time comes. The product investor can't look bright eyed and linearly extrapolate progress so far thinking they saved a boatload of capital, they have to understand shortcuts were taken and the rationale was to fix them if serious money came along or chuck them in the bin if not.

    • > If it's really abstractable it shouldn't be difficult to refactor. It should literally be a substitution.

      This is overstated. Not all abstractions are obvious substitutions. To elaborate: languages vary in their syntax, typing, and scope. So what might be an 'easy' substitution and one language might not be easy in another.

WET is great until JSON token parsing breaks and a junior dev fixes it in one place and then I am fixing the same exact problem somewhere else and moving it into a shared file. If it's the exact same functionality, move it into a service/helper.

How do you deal with other colleagues that have all the energy and time to push for these practices and I feel makes things worse than the current state?

  • Explain that the wrong abstraction makes code more complicated than copy-paste and that before you can start factoring out common code you need to be sure the relationship is fundamental and not coincidental.

> figure out the abstraction after you need something a 3rd time

That's still too much of a "rule".

Whenever I feel (or know) two functions are similar, the factors that determine if I should merge them:

- I see significant benefit too doing so, usually the benefit of a utility that saves writing the same thing in the future, or debugging the same/similar code repeatedly.

- How likely the code is to diverge. Sometimes I just mark things for de-duping, but leave it around a while to see if one of the functions change.

- The function is big enough it cannot just be in-lined where it is called, and the benefit of de-duplication is not outweighed by added complexity to the call stack.

repeat after me:

Document.

Your.

Shit.

everything else can do one. Just fucking write documentation as if you're the poor bastard trying to maintain this code with no context or time.

  • Documentation is rarely adequately maintained, and nothing enforces that it stay accurate and maintained.

    Comments in code can lie (they're not functional); can be misplaced (in most languages, they're not attached to the code they document in any enforced way); are most-frequently used to describe things that wouldn't require documenting if they were just named properly; are often little more than noise. Code comments should be exceedingly-rare, and only used to describe exception situations or logic that can't be made more clear through the use of better identifiers or better-composed functions.

    External documentation is usually out-of-sight, out-of-mind. Over time, it diverges from reality, to the point that it's usually misleading or wrong. It's not visible in the code (and this isn't an argument in favor of in-code comments). Maintaining it is a burden. There's no agreed-upon standard for how to present or navigate it.

    The best way to document things is to name identifiers well, write functions that are well-composed and small enough to understand, stick to single-responsibility principles.

    API documentation is important and valuable, especially when your IDE can provide it readily at the point of use. Whenever possible, it should be part of the source code in a formal way, using annotations or other mechanisms tied to the code it describes. I wish more languages would formally include annotation mechanisms for this specific use case.

    • > Documentation is rarely adequately maintained,

      yes, and the solution is to actually document.

      > wouldn't require documenting if they were just named properly

      I mean not really. Having decent names for things tells us what you are doing, but not why.

      > only used to describe exception situations

      Again, just imagine if that wasn't the case. Imagine people had the empathy to actually do a professional job?

      > The best way to document things is to name identifiers well, write functions that are well-composed and small enough to understand, stick to single-responsibility principles.

      No, thats the best way to code.

      The best way to document is to imagine you are a new developer to the code base, and any information they should know is where they need it. Like your code, you need to test your documentation.

      I know you don't _like_ documenting, but thats not the point. Its about being professional, just imagine if an engineer didn't _like_ doing certification of their stuff? they'd loose their license. Sure you could work it out later, but thats not the point. You are paid to be professional, not a magician.

      6 replies →

> WET

One of the most difficult to argue comments in code reviews: “let’s make it generic in case we need it some place else”. First of all, chances that we need it some place else aren’t exactly high, unless you are writing a library and code explicitly design to be shared. And even if such need arises, chances of getting it right generalizing from one example are slim.

Regarding the book though, I have participated in one of the workshops with the author and he seemed to be in favor of WER and against “architecting” levels of abstraction before having concrete examples.

> Realize there's no such thing as "clean code"

You can disagree over what exactly is clean code. But you will learn to distinguish what dirty code is when you try to maintain it.

As a person that has had to maintain dirty code over the years, hearing someone saying dirty code doesn't exist is really frustrating. Noone wants to clean up your code, but doing it is better than allowing the code to become unmaintainable, that's why people bring up that book. If you do not care about what clean code is, stop making life difficult for people that do.

  • > hearing someone saying dirty code doesn't exist is really frustrating

    Not sure why you're being downvoted, but as an unrelated aside, the quote you're responding to literally did not say this.

    > that's why people bring up that book

    I think the point is that following that book does not really lead to Clean Code.

    • > I think the point is that following that book does not really lead to Clean Code.

      And that is why I started saying "You can disagree over what exactly is clean code". Different projects have different requirements. Working on some casual website is not the same as working on a pacemaker firmware.

      1 reply →

  • I think it's more that clean code doesn't exist because there's no objective measure of this (and those services that claim there are are just as dangerous as Clean Code, the book); anyone can come along and find something about the code that could be tidied up. And legacy is legacy, it's a different problem space to the one a greenfield project exists in.

    > As a person that has to maintain dirty code

    This is a strange credential to present and then use as a basis to be offended. Are you saying that you have dirty code and have to keep it dirty?

    • The counterintuitive aspect of this problem that acts as a trap of the less pragmatic people, is that an objective measure is not always necessary.

      Let's say you are a feeding a dog. You can estimate what amount of food is dog too little, and what amount of dog food is too much... but now, some jerk comes around and tells you they're going to feed the dog the next time. You agree.

      You check the plate, and there's very little food in it. So you say: "hey, you should add more dog food".

      Then, the jerk reacts by putting an excessive amount of food in it, just to fuck with you. Then you say "that's too much food!"... So then the jerk reacts saying "you should tell me exactly how many dog pellets I should put on the plate".

      Have you ever had to count dog pellets individually to feed a dog? no. You haven't, you have never done it, yet you have fed dogs for years without problems just using a good enough estimate of how much a dog can eat.

      Just to please the fucking jerk, you take the approximate amount dog food you regularly feed the dog every day, count the pellets, and say: "there, you fuck, 153 dog pellets".

      But the jerk is not happy yet. Just to really fuck with you, the guy will challenge you and say: "so what happens if I feed the dog 152 pellets, or 154... see? you are full of shit". Then you have to explain the jerk that 153 was never important, what's important is the approximate amount of dog food. But the jerk doesn't think that way, the jerk wants a fucking exact number so they can keep fighting you...

      Then the jerk will probably say that a dog pellet is not a proper unit of mass, and then the jerk will say that nutrients are not equally distributed in dog pellets, and the bullshit will go on and on and on.

      And if you are ever done discussing the optimal number of pellets then there will be another discussion about the frequency in which you feed the dog, and you will probably end up talking about spacetime and atomic clocks and the NTP protocol and relativity and inertial frames just to please the jerk whose objective is just to waste your time until you give up trying to enforce good practices.

      And this is how the anti-maintainability jerks operate, by getting into endless debates about how an objective measure of things is required, when in reality, it's not fucking needed and it never was. Estimation is fine.

      Just like you won't feed a dog a fucking barrel of dog food you won't create a function that is 5 thousand lines of code long because it's equally nonsensical.

      So in the end, what do you do? you say: this many lines of code in a function is too fucking much, don't write functions this long. Why? because I say so, end of debate, fuck it.

      There's a formal concept for this in philosophy, but that's up to you to figure out.

      8 replies →

This is what I'm doing even while creating new code. There's a few instances for example where the "execution" is down to a single argument - one of "activate", "reactivate" and "deactivate". But I've made them into three distinct, separate code paths so that I can work error and feedback messages into everything without adding complexity via arguments.

I mean yes it's more verbose, BUT it's also super clear and obvious what things do, and they do not leak the underlying implementation.

I’ve never heard the term WET before but that’s exactly what I do.

The other key thing I think is not to over-engineer abstractions you don’t need yet. But to try and leave ‘seams’ where it’s obvious how to tease code about if you need to start building abstractions.

My experience interviewing recently a number of consultants with only a few years experience was the more they mumbled clean code the less they knew what they were doing.

Ah hahha. I love WET. I always say the only way to write something correctly is to re-write it.

  • That's not what WET means. The GP is saying you shouldn't isolate logic in a function until you've cut-and-pasted the logic in at least two places and plan to do so in a third.

> Put logic closest to where it needs to live (feature folders)

Can you say more about this?

I think I may have stumbled on a similar insight myself. In a side project (a roguelike game), I've been experimenting with a design that treats features as first-class, composable design units. Here is a list of the subfolder called game-features in the source tree:

  actions
  collision
  control
  death
  destructibility
  game-feature.lisp
  hearing
  kinematics
  log
  lore
  package.lisp
  rendering
  sight
  simulation
  transform

An extract from the docstring of the entire game-feature package:

  "A Game Feature is responsible for providing components, events,
  event handlers, queries and other utilities implementing a given
  aspect of the game. It's primarily a organization tool for gameplay code.
  
  Each individual Game Feature is represented by a class inheriting
  from `SAAT/GF:GAME-FEATURE'. To make use of a Game Feature,
  an object of such class should be created, preferably in a
  system description (see `SAAT/DI').
  This way, all rules of the game are determined by a collection of
  Game Features loaded in a given game.
  
  Game Features may depend on other Game Features; this is represented
  through dependencies of their classes."

The project is still very much work-in-progress (procrastinating on HN doesn't leave me much time to work on it), and most of the above features are nowhere near completion, but I found the design to be mostly sound. Each game feature provides code that implements its own concerns, and exports various functions and data structures for other game features to use. This is an inversion of traditional design, and is more similar to the ECS pattern, except I bucket all conceptually related things in one place. ECS Components and Systems, utility code, event definitions, etc. that implement a single conceptual game aspect live in the same folder. Inter-feature dependencies are made explicit, and game "superstructure" is designed to allow GFs to wire themselves into appropriate places in the event loop, datastore, etc. - so in game startup code, I just declare which features I want to have enabled.

(Each feature also gets its set of integration tests that use synthetic scenarios to verify a particular aspect of the game works as I want it to.)

One negative side effect of this design is that the execution order of handlers for any given event is hard to determine from code. That's because, to have game features easily compose, GFs can request particular ordering themselves (e.g. "death" can demand its event handler to be executed after "destructibility" but before "log") - so at startup, I get an ordering preference graph that I reconcile and linearize (via topological sorting). I work around this and related issues by adding debug utilities - e.g. some extra code that can, after game startup, generate a PlantUML/GraphViz picture of all events, event handlers, and their ordering.

(I apologize for a long comment, it's a bit of work I always wanted to talk about with someone, but never got around to. The source of the game isn't public right now because I'm afraid of airing my hot garbage code.)

  • I'd be interested in how you attempt this. Is it all in lisp?

    It might be hard to integrate related things, e.g. physical simulation/kinematics <- related to collisions, and maybe sight/hearing <- related to rendering; Which is all great if information flows one way, as a tree, but maybe complicated if it's a graph with intercommunication.

    I thought about this before, and figured maybe the design could be initially very loose (and inefficient), but then a constraint-solver could wire things up as needed, i.e. pre-calculate concerns/dependencies.

    Another idea, since you mention "logs" as a GF: AOP - using " join points" to declaratively annotate code. This better handles code that is less of a "module" (appropriate for functions and libraries) and more of a cross-cutting "aspect" like logging. This can also get hairy though: could you treated "(bad-path) exception handling" as an aspect? what about "security"?

    • > I'd be interested in how you attempt this. Is it all in lisp?

      Yes, it's all in Common Lisp.

      > It might be hard to integrate related things, e.g. physical simulation/kinematics <- related to collisions, and maybe sight/hearing <- related to rendering;

      It is, and I'm cheating a bit here. One simplification is that I'm writing a primarily text-based roguelike, so I don't have to bother with a lot of issues common to real-time 3D games. I can pick and choose the level of details I want to go (e.g. whether to handle collisions at a tile granularity, or to introduce sub-tile coordinates and maybe even some kind of object shape representation).

      > Which is all great if information flows one way, as a tree, but maybe complicated if it's a graph with intercommunication.

      The overall simulation architecture I'm exploring in this project is strongly event-based. The "game frame" is basically pulling events from a queue and executing them until the queue is empty, at which point the frame is considered concluded, and simulation time advances forward. It doesn't use a fixed time step - instead, when a simulation frame starts, the code looks through "actions" scheduled for game "actors" to find the one that will complete soonest, and moves the simulation clock to the completion time of that action. Then the action completion handler fires, which is the proper start of frame - completion handler will queue some events, and handlers of those events will queue those events, and the code just keeps processing events until the queue empties again, completing the simulation frame.

      Structure-wise, simulation GF defines the concept of "start frame" and "end frame" (as events), "game clock" (as query) and ability to shift it (as event handler), but it's the actions GF that contains the computation of next action time. So, simulation GF knows how to tell and move time, but actions GF tells it where to move it to.

      This is all supported by an overcomplicated event loop that lets GFs provide hints for handler ordering, but also separates each event handling process into four chains: pre-commit, commit, post-commit and abort. Pre-commit handlers fire first, filling event structure with data and performing validation. Then, commit handlers apply direct consequences of event to the real world - they alter the gameplay state. Then, post-commit handlers process further consequences of an event "actually happening". Alternatively, abort handlers process situations when an event was rejected during earlier chains. All of them can enqueue further events to be processed this frame.

      So, for example, when you fire a gun, pre-commit handlers will ensure you're able to do it, and reject the event if you can't. If the event is rejected, abort chain will handle informing you that you failed to fire. Otherwise, the commit handlers will instantiate an appropriate projectile. Post-commit handlers may spawn events related to the weapon being fired, such as informing nearby enemies about the discharge.

      This means that e.g. if I want to implement "ammunition" feature, I can make an appropriate GF that attaches a pre-commit handler to fire event - checking if you have bullets left and rejecting the event if you don't (events rejected in pre-commit stage are considered to "have never happened"), and a post-commit handler on the same event to decrease your ammo count. The GF is also responsible for defining appropriate components that store ammo count, so that (in classical ECS style) your "gun" entity can use it to keep track of ammunition. It also provides code for querying the current count, for other GFs that may care about it for some reason (and the UI rendering code).

      > I thought about this before, and figured maybe the design could be initially very loose (and inefficient), but then a constraint-solver could wire things up as needed, i.e. pre-calculate concerns/dependencies.

      I'm halfway there and I could easily do this, but for now opted against it, on the "premature optimization" grounds. That is, since all event handlers are registered when the actual game starts, I "resolve constraints" (read: convert sorting preferences into a DAG and toposort it; it's dumb and very much incorrect, but works well enough for now) and linearize handlers - so that during gameplay, each event handler chain (e.g. "fire weapon", pre-commit) is just a sequence of callbacks executed in order. It would be trivial to take such sequence, generate a function that executes them one by one (with the loop unrolled), and compile it down to metal - Common Lisp lets you do stuff like that - but I don't need it right now.

      > Another idea, since you mention "logs" as a GF

      FWIW, logs GF is implementing the user-facing logs typical for roguelike games - i.e. the bit that says "You open the big steel doors". Diagnostic logs I do classically.

      > AOP - using " join points" to declaratively annotate code

      In a way, my weird multi-chain event loop is a reimplementation of AOP. Method combinations in Common Lisp are conceptually similar too, but I'm not making big use of them in game feature-related code.

      > This can also get hairy though: could you treated "(bad-path) exception handling" as an aspect? what about "security"?

      Yeah, I'm not sure if this pattern would work for these - particularly in full-AOP, "inject anything anywhere" mode. I haven't tried it. Perhaps, with adequate tooling support, it's workable? Common Lisp is definitely not a language to try this in, though - it's too dynamic, so tooling would not be able to reliably tell you about arbitrary pointcuts.

      In my case, I restricted the "feature-oriented design" to just game features - I feel it has a chance of working out, because in my mind, quite a lot of gameplay mechanics are conceptually additive. This project is my attempt at experimentally verifying if one could actually make a working game this way.

  • I've gone down roads similar to this. Long story short - the architecture solves for a lower priority class of problem, w/r to games, so it doesn't pay a great dividend, and you add a combination of boilerplate and dynamism that slows down development.

    Your top issue in the runtime game loop is always with concurrency and synchronization logic - e.g. A spawns before B, if A's hitbox overlaps with B, is the first frame that a collision event occurs the frame of spawning or one frame after? That's the kind of issue that is hard to catch, occurs not often, and often has some kind of catastrophic impact if handled wrongly. But the actual effect of the event is usually a one-liner like "set a stun timer" - there is nothing to test with respect to the event itself! The perceived behavior is intimately coupled to when its processing occurs and when the effects are "felt" elsewhere in the loop - everything's tied to some kind of clock, whether it's the CPU clock, the rendered frame, turn-taking, or an abstracted timer. These kinds of bugs are a matter of bad specification, rather than bad implementation, so they resist automated testing mightily.

    The most straightforward solution is, failing pure functions, to write more inline code(there is a John Carmack posting on inline code that I often use as a reference point). Enforce a static order of events as often as possible. Then debugging is always a matter of "does A happen before B?" It's there in the source code, and you don't need tooling to spot the issue.

    The other part of this is, how do you load and initialize the scene? And that's a data problem that does call for more complex dependency management - but again, most games will aim to solve it statically in the build process of the game's assets, and reduce the amount of game state being serialized to save games, reducing the complexity surface of everything related to saves(versioning, corruption, etc). With a roguelike there is more of an impetus to build a lot of dynamic assets(dungeon maps, item placements etc.) which leads to a larger serialization footprint. But ultimately the focus of all of this is on getting the data to a place where you can bring it back up and run queries on it, and that's the kind of thing where you could theoretically use SQLite and have a very flexible runtime data model with a robust query system - but fully exploiting it wouldn't have the level of performance that's expected for a game.

    Now, where can your system make sense? Where the game loop is actually dynamic in its function - i.e. modding APIs. But this tends to be a thing you approach gradually and grudgingly, because modders aren't any better at solving concurrency bugs and they are less incentivized to play nice with other mods, so they will always default to hacking in something that stomps the state, creating intermittent race conditions. So in practice you are likely to just have specific feature points where an API can exist(e.g. add a new "on hit" behavior that conditionally changes the one-liner), and those might impose some generalized concurrency logic.

    The other thing that might help is to have a language that actually understands that you want to do this decoupling and has the tooling built in to do constraint logic programming and enforce the "musts" and "cannots" at source level. I don't know of a language that really addresses this well for the use case of game loops - it entails having a whole general-purpose language already and then also this other feature. Big project.

    I've been taking the approach instead of aiming to develop "little languages" that compose well for certain kinds of features - e.g. instead of programming a finite state machine by hand for each type of NPC, devise a subcategory of state machines that I could describe as a one-liner, with chunks of fixed-function behavior and a bit of programmability. Instead of a universal graphics system, have various programmable painter systems that can manipulate cursors or selections to describe an image. The concurrency stays mostly static, but the little languages drive the dynamic behavior, and because they are small, they are easy to provide some tooling for.

    • Thanks for the detailed evaluation. I'll start by reiterating that the project is a typical tile-based roguelike, so some of the concerns you mention in the second paragraph don't apply. Everything runs sequentially and deterministically - though the actual order of execution may not be apparent from the code itself. I mitigate it to an extent by adding introspection features, like e.g. code that dumps PlantUML graphs showing the actual order of execution of event handlers, or their relationship with events (e.g. which handlers can send what subsequent events).

      I'll also add that this is an experimental hobby project, used to explore various programming techniques and architecture ideas, so I don't care about most constraints under which commercial game studios operate.

      > The perceived behavior is intimately coupled to when its processing occurs and when the effects are "felt" elsewhere in the loop - everything's tied to some kind of clock, whether it's the CPU clock, the rendered frame, turn-taking, or an abstracted timer. These kinds of bugs are a matter of bad specification, rather than bad implementation, so they resist automated testing mightily.

      Since day one of the project, the core feature was to be able to run headless automated gameplay tests. That is, input and output are isolated by design. Every "game feature" (GF) I develop comes with automated tests; each such test starts up a minimal game core with fake (or null) input and output, the GF under test, and all GFs on which it depends, and then executes faked scenarios. So far, at least for minor things, it works out OK. I expect I might hit a wall when there are enough interacting GFs that I won't be able to correctly map desired scenarios to actual event execution orders. We'll see what happens when I reach that point.

      > that's the kind of thing where you could theoretically use SQLite and have a very flexible runtime data model with a robust query system - but fully exploiting it wouldn't have the level of performance that's expected for a game.

      Funny you should mention that.

      The other big weird thing about this project is that it uses SQLite for runtime game state. That is, entities are database rows, components are database tables, and the canonical gameplay state at any given point is stored in an in-memory SQLite database. This makes saving/loading a non-issue - I just use SQLite's Backup API to dump the game state to disk, and then read it back.

      Performance-wise, I tested this approach extensively up front, by timing artificial reads and writes in expected patterns, including simulating a situation in which I pull map and entities data in a given range to render them on screen. SQLite turned out to be much faster than I expected. On my machine, I could easily get 60FPS out of that with minimum optimization work - but it did consume most of the frame time. Given that I'm writing a ASCII-style, turn(ish) roguelike, I don't actually need to query all that data 60 times per second, so this is quite acceptable performance - but I wouldn't try that with a real-time game.

      > The other thing that might help is to have a language that actually understands that you want to do this decoupling and has the tooling built in to do constraint logic programming and enforce the "musts" and "cannots" at source level. I don't know of a language that really addresses this well for the use case of game loops - it entails having a whole general-purpose language already and then also this other feature. Big project.

      Or a Lisp project. While I currently do constraint resolution at runtime, it's not hard to move it to compile time. I just didn't bother with it yet. Nice thing about Common Lisp is that the distinction between "compilation/loading" and "runtime" is somewhat arbitrary - any code I can execute in the latter, I can execute in the former. If I have a function that resolves constraints on some data structure and returns a sequence, and that data structure can be completely known at compile time, it's trivial to have the function execute during compilation instead.

      > I've been taking the approach instead of aiming to develop "little languages" that compose well for certain kinds of features

      I'm interested in learning more about the languages you developed - e.g. how your FSMs are encoded, and what that "programmable painter system" looks like. In my project, I do little languages too (in fact, the aforementioned "game features" are a DSL themselves) - Lisp makes it very easy to just create new DSLs on the fly, and to some extent they inherit the tooling used to power the "host" language.

      2 replies →