Comment by akoboldfrying

How to encode all flags in a minimum number of bits:

1. Quantise colours to, say, 8 colours that you can confidently distinguish. Use a scheme that prefers "most commonly used" colours that actually appear in the flags.

2. Render each colour-quantised flag to a fixed-size bitmap, e.g., 100x50.

3. We seek a minimum-size subset of pixel locations P such that every pair of flags differs in colour at at least one of these pixel locations. This is the NP-complete problem Minimum Test Set [0] -- in fact, a slight generalisation, because the answer to each "test" (pixel location) is not yes or no but one of 10 colours. You could try to solve this by growing an exactly minimal solution using branch and bound, but this is likely to be too slow for such a large bitmap. Alternatively, I expect repeatedly running a heuristic that builds solutions by randomly adding any pixel location until all flags become distinguishable to be highly effective as there will likely be many equal-size optimal solutions, though of course you won't get an optimality guarantee this way.

4. At this point, since the 8 colours can be represented by 3 bits each, you basically have a 3|P|-bit "hash" that distinguishes all flags. If that is still bigger than log2(nFlags), you could shrink it further with standard minimal perfect hashing techniques.

ETA: There are a few ways to improve this. One thing you want is to choose relatively "stable" pixel locations that are not close to boundaries between colours on any flag, to avoid the problem of slightly different rasterisations of the same flag giving different answers (imagine if you were applying this to scanned photos of flags). To achieve this, you could compute, for each pixel location, a "stability value": The minimum distance in pixels to any differently-coloured pixel, across all flags. Then instead of considering all 100x50 pixel locations, you might consider only the 30 with the highest stability values. With such a small set of pixel locations to consider, it's feasible to consider all ~1 billion subsets of them, giving you a known-optimal solution.

[0]: https://cs.stackexchange.com/a/81189