New Ultra Fast Lossless Audio Codec (HALAC)

2 years ago (hydrogenaud.io)

I love how some announcements of developments still occur in good old-fashioned forums; it really warms my heart to see that. Kudos to the author for the hard work!

  • Honestly this is one of the best technical audio forums. I always really appreciated how they take their rules seriously, like

    > TOS 8. All members that put forth a statement concerning subjective sound quality, must - to the best of their ability - provide objective support for their claims. Acceptable means of support are double blind listening tests (ABX or ABC/HR) demonstrating that the member can discern a difference perceptually, together with a test sample to allow others to reproduce their findings. Graphs, non-blind listening tests, waveform difference comparisons, and so on, are not acceptable means of providing support.

There's no source? It's very hard to take this seriously without source code...

He explicitly says it's not SIMD, which is nice because it rules out a way of cheesing it, but still...

  • Yeah, I was about to click and read and it occurred to me that this is only of possibly theoretical interest and I'll know about later if it comes to matter.

    There are some areas that I follow as-it-develops, but codecs and data compression is one that I'll use when ready. Still awaiting widespread av1 support/adoption.

    The area most needing improvements IMO is with Bluetooth, especially Apple's support of codecs (where they're dropping support that worked in older macOS versions).

    • Pretty much any low level tech is at most a theoretical curiosity to me. I'll use it when it works in every browser and OS and average people recognize the file extension. Unusual tech seems to attract unusual bugs!

      Still really really impressive to beat an established standard as an individual, that doesn't happen much.

      1 reply →

  • At this stage, what is taken seriously is quite personal. If the source code is a requirement to take it seriously; MS Windows, MS Office, Adobe, Autodesk, Oracle, Winrar and thousands of more wonderful software should not be taken seriously.

    Some of the SIMD optimization make compilers automatically. The others can be manually. These speeds can be obtained without using SIMD. Then there may be more with manual SIMD. I think this is what should be loved.

    • > MS Windows, MS Office, Adobe, Autodesk, Oracle, Winrar and thousands of more wonderful software should not be taken seriously.

      Apples and oranges. I don't need word processing software to be open source to understand how it works. A proportedly novel compression algorithm is a different story...

      I can be totally honest with you: FLAC being open source is more valuable to me than any performance benefit you could ever possibly offer over it. It only becomes interesting if I can actually read the code and see what you did.

      I am genuinely interested in what you've done here, and I sincerely hope you publish it.

      6 replies →

Yall should really have a look at his lossless image format. That has a spectacular compression:speed ratio. https://encode.su/threads/4025-HALIC-(High-Availability-Loss...

What interests me the most is the memory usage.

  • Yes, the HALIC uses quite low memory. Because independent small blocks work. Of course, this situation negatively affects the compression rate. It is a fact that I compromise on the compression rate to ensure less memory consumption.

Hakan; Ignore the open source stuff for now. I speak as someone from within the industry. You may not listen to me and be upset later. If those who say this had done even half the work you did, be sure they wouldn't be talking like this.

It seems that there is a serious effort. And your work clearly crushes other codecs. This will cause serious discomfort in some circles. I don't know much about HALIC, but it is said that it is also a very serious work.

There's no point in rushing for open source. First of all, improve your work as much as you can. If serious offers come later, you can evaluate them. Don't give up control.

Cool toy and a nice piece for the CV perhaps, but it is difficult to take it seriously if you refuse to offer source code or a implementable specification.

I would give you the benefit of the doubt that it might just be code shyness or perfectionism about something in its early stages, but it looks like the last codec you developed (“HALIC”) is still only available as Windows binaries after a year.

I struggle to see an upside to withholding source code in a world awash with performant open source media codecs.

  • Maybe it’s just me, but every lossless codec that’s:

    1. Not FLAC

    2. Not as open-source as FLAC

    comes across as a patent play.

    FLAC is excellent and widely supported (and where it’s not supported some new at-least-open-enough codec will also not be supported). I have yet to see a compelling argument for lossless audio encoders that are not FLAC.

    • FLAC’s compression algorithm was pretty much garbage when it came out, and is much worse now compared to the state of the art. Even mp3 + gzip residuals would probably compress better.

      FLAC doesn’t support more modern sampling formats (e.g. floating point for mastering), or complex multi channel compression for surround sound formats.

      There just isn’t something better (and free) to replace it yet.

      18 replies →

  • You are right about this. But there are things I should add to Halic and Halac. When I complete them and realize that it will really be used by someones, it will of course be open source.

    • One of the cool things about open source is that other people can do that for you! I've released a few bits of (rarely-used) software to open-source and been pleasantly surprised when people contribute. It helps to have a visible todo list so that new contributors know what to aim for.

      By the way, there will always be things to add! That feeling should not stop you from putting the source out there - you will still own it (you can license the code any way you like!) and you can choose what contributions make it in to your source.

      From the encode.su thread and now the HA thread, you've clearly gotten people excited, and I think that by itself means that people will be eager to try these out. Lossless codecs have a fairly low barrier for entry: you can use them without worrying about data loss by verifying that the decoder returns the original data, then just toss the originals and keep the matching decoder. So, it should be easy to get people started using the technology.

      Open-sourcing your projects could lead to some really interesting applications: for example, delivering lossless images on the internet is a very common need, and a WASM build of your decoder could serve as a very convenient way to serve HALIC images to web browsers directly. Some sites are already using formats like BPG in this way.

      12 replies →

    • > When I complete them and realize that it will really be used by someones, it will of course be open source

      There is a chicken and egg problem with this strategy: Few people will want to, or even be able to, use this unless it’s open source and freely licensed.

      The alternatives are mature, open or mostly open, and widely implemented. Minor improvements in speed aren’t enough to get everyone to put up with any difficulties in getting it to work. The only way to get adoption would be to make it completely open and as easy as possible to integrate everywhere.

      It’s a cool project, but the reality is that keeping it closed until it’s “completed” will prevent adoption.

      5 replies →

    • When you do decide to open the codec, you should talk to xiph.org about patent defensibility. If you want it open, but don’t build a large enough moat (multichannel, other bitrates, bit depth, echo and phase control, etc then the licensing org will offensively patent or extend your creation.

      1 reply →

    • I understand a forward compatibility concern, but have you considered to put an attention-grabbing alert in the encoder and clearly state that official releases in the future won't be able to decompress the output? Also your concern may have been too overblown; there have been more than 100 PAQ versions with mutually incompatible formats but such issues didn't happen too often. (Not to say that ZPAQ was pointless, of course.)

    • You may be trying to kill all criticisms, this is not possible. Not everyone will like you and not everyone will like your code. Fortunatly people irl that have personal differences tend to be a but more tactful than the software crowd can be online but something like this bound to get overwhelming amounts of love.

      No great project started out great and the best open source projects got to their state because of the open sourcing.

      Consider the problems you might be spending a lot of time solving might be someone else's trivial issue, so unless this is an enjoyable academic excercise for you (which i fully support), why suffer?

      1 reply →

    • You could open it now and crowd-source the missing pieces. I really see nothing to lose by making this oss-first.

    • You got that backwards buddy. Nobody will use them so long as they remain closed source like this.

For what looks at first glance to be a potentially impactful development, this post and the "encode.su" one linked from it are extremely sparse on details.

Where is the source code? A detailed description of what the codec actually does? References to relevant publications?

All I see are two mystery Windows binaries, hosted on a forum I've never heard about. The fact that "encode.su" uses the world's most notorious domain extension doesn't inspire confidence, to put it mildly.

  • > The fact that "encode.su" uses the world's most notorious domain extension doesn't inspire confidence, to put it mildly.

    Encode.su (formerly encode.ru) is indeed the most known forum dedicated for data compression in general. So much that many if not most notable data compression projects posted to HN are often first advertised to that forum first (for example, Zstandard [1]).

    [1] https://encode.su/threads/2119-Zstandard

  • The fact that "encode.su" uses the world's most notorious domain extension doesn't inspire confidence, to put it mildly.

    The leaders in data compression and information theory are all from the former Soviet Union, so that's no cause for concern.

    I think Andrey Markov started it all.

  • Hydrogenaudio is well known in this area and many new prototype codecs are announced there first. Also, the lack of source control and Windows-only binaries are very much congruent to the style of development there. See it as your confrontation with a new world, because small it is not!

    And later, you will learn to understand the depth of the contribution that the ffmpeg project provides :)

  • Also, what does the High Availability in the name of the codec refer to?

    I've searched the web and found not a single mention of "high availability" in the context of audio codecs. In fact, the top-ranked result was this very post, which doesn't explain what the term is intended to mean.

  • >>>uses the world's most notorious domain extension doesn't inspire confidence, to put it mildly.

    What does this mean? Why or how is this TLD the world’s most notorious?

I'd like to test this but mum said not to download unknown .exe files from the internet.

  • It’s not like the usual pipe-curl-to-bash installation instructions are much better.

    • And if the author of your parent comment saw a random forum user ask people to curl | bash from some random .su domain I'm sure they'd have no aversion to that! Great argument!

  • Honestly, if they provided 100mb source code, would you read it and then compile it? Source code alone doesn't make it secure.

    • Something like this doesn't require 100 MB of source code. I'd expect a few thousand LoC at most.

      And I absolutely do at least a quick visual "sanity check" of the code before compiling and running newly announced software.

      1 reply →

    • Irrelevant. Convenient or probable doesn't matter. What matters is possible vs not possible.

      All it takes is one person somewhere who wants to look something over, and they heads-up the rest, and then many others do verify.

      And that initial one does exist even though it's not you or me, the same way the author exists, the same way that at least once in a while for some things it is you or me.

      4 replies →

This doesn’t seem very interesting: there’s no source, the benchmarks are self reported, and the details in the forum post (??) are light at best.

It’s basically just a forum post with a self-signed .exe download…

Tip for those who are looking at the compression ratios thinking "so what?": look at the run-times. It's a minimum of 3x faster than its contemporaries.

  • IMHO that's still a "so what?". I see audio compression as having two primary purposes: realtime and archival, and speed is only vaguely relevant for the former.

    In realtime applications, any processor within the last decade is far more than powerful enough to encode and decode in realtime. The first set of tracks in the results is 2862s long and even the slowest WAVPACK manages to encode it in >113x realtime and close to 250x realtime for decode.

    For archival, high compression is important, but this codec doesn't compress better than WAVPACK either.

    • I am still at the beginning of the road in terms of compression rate and speed. There may be changes in subsequent versions.

    • > IMHO that's still a "so what?"

      This is addressed in another response in this same thread regarding electricity usage.

  • FLAC level 5 encode times should be WAY longer than decode times. FLAC level 0 to FLAC level 5 is a huge step up and encoding should be way longer (like factor of 3). Under no circumstances should FLAC level 5 decode be faster than FLAC level 0 decode.

    These are basic sanity checks.

    Something is wrong in the benchmark.

    • > Under no circumstances should FLAC level 5 decode be faster than FLAC level 0 decode.

      I don't know about FLAC, but from my knowledge of compression, this result seems sensible to me.

      Smaller file = less bits to process = faster. FLAC level 5 is expected to give a smaller file than level 0, so it makes sense that decoding it will be faster. Of course, it's possible that some codecs enable more codec features at higher compression levels, which makes decoding slower, so it's not always a given, but higher compression giving faster decode doesn't seem unreasonable.

    • I was very surprised to this situation, but Flac Level 5 (Default mode) results gave quick results in all my tests. I've tried this hundreds of times. You can try it with any converter and see it. I think much more improvements have been made on this mode.

  • It seems the world made a collective yawn about flac. Hard to imagine that would change much with a new format.

    I personally keep everything in flac, but Bandcamp is seemingly the only service where that is a given.

  • According to the author, right? Those results aren't backed up by Porcus's comments in that thread.

    • Maybe not x3-5, but Porcus confirms that it is still very fast:

      > Though it is fast indeed! The decoding speeds are outright impressive given how FLAC is the fastest thing we ever saw ...

      1 reply →

  • >It's a minimum of 3x faster than its contemporaries.

    Ok, but what would make that useful?

    • > Ok, but what would make that useful?

      Lower run-time means less electricity and less tying up of the CPU, making it available for other things. As a real-life example: i frequently use my Raspberry Pi 4 to convert videos from one format to another. This past week i got a Pi 5 and moved the conversion to that machine: it takes maybe 1/4th as much time. The principle with a faster converter, as opposed to faster hardware, is the same: the computer isn't tied up for as long, and not draining as much power.

      11 replies →

Novel, but until it's open source, it will never be taken seriously.

  • There are things I should add to Halic and Halac. When I complete them and realize that it will really be used by someones, it will of course be open source.

    • This is a bad take man, "when its done" will never come having read through your comments on this you seem to really be going for some perfection in your hobby and until thats released won't release, but the issue is that rarely if ever happens, and instead of getting insights on your code and the community and yourself actually improving the landscape it's basically just a fancy exe, that no one will use actually because theres no where for it to go except in your head.

    • Open source it anyway. Things don't need to be perfect to be useful. The spirit of OSS has a lot of iteration, anyway. :)

    • What most people do is they trademark the name, that way even though someone might fork, they have to use a different name.

      Something else you can do is use (A)GPL3. This means you automatically grant patent licenses, and anyone building on your work also has to release their source. You can then separately sell proprietary licenses without any of these restrictions.

      1 reply →

[flagged]

  • there are things I should add to Halic and Halac. When I complete them and realize that it will really be used by someones, it will of course be open source.

    • until you release the source, it will never be used by someone, so I guess you are stuck in a loop and it will enver be opened.

[flagged]

  • My works are fully cross-platform. There is no restrictive situation. I also prepared Linux and ARM versions for HALIC, but I didn't compile new versions because it didn't get much attention. When my Linux test machine is ready(crashed), I compile the Linux version of HALAC.

    • > When my Linux test machine is ready(crashed)

      In 2024, you don't need a separate machine to run different OSes

      qemu is your friend