F-Droid build servers can't build modern Android apps due to outdated CPUs

> This means their servers are very old ones that do not support x86-64-v2. Intel Core 2 Duo days?

This is not always a given. In our virtualization platform, we have upgraded a vendor supplied VM recently, and while it booted, some of the services on it failed to start despite exposing a x86_64v2 + AES CPU to the said VM. Minimum requirements cited "Pentium and Celeron", so it was more than enough.

It turned out that one of the services used a single instruction added in a v3 or v4 CPU, and failed to start. We changed the exposed CPU and things have returned to normal.

So, their servers might be capable and misconfigured, or the binary might require more that what it states, or something else.

$2-3k ? That’s barely the price of a lower end Threadripper bare cpu not a full Epyc server ???

Perhaps the servers run Coreboot / Libreboot?

I'm not even sure mainline Linux supports machines this old at this point. The cmpxchg16b instruction isn't that old, and I believe it's required now.

> about to ask people to donate, but they have $80k in their coffers

I'd still ask folks to donate. £80k isn't much at all given the time and effort I've seen their volunteers spend on keeping the lights on.

From what I recall, they do want to modernize their build infrastructure, but it is as big as an investment they can make. If they had enough in their "coffers", I'm sure they'd feel more confident about it.

It isn't like they don't have any other things to fix or address.

>they have $80k in their coffers but I am curious why they haven't spent $2-3k on one of those Zen4 or Zen5 matx consumer Epyc servers

I would also like to know this.

I agree it’s a bit concerning but please keep in mind F-Droid is a volunteer-run community project. Especially with some EU countries moving to open source software, it would be nice to see some public funding for projects like F-Droid.

Maybe if f-droid is important to you, donate, so they can buy newer build server?

> FDroid is by far the largest non-google android store at the moment

Not even sure it's in the top 10

> Are google looking into rolling back the requirement? (this last one sounds unlikely)

That's apparently what they did last time. From the ticket:

"Back in 2021 developers complained that AAPT2 from Gradle Plugin 4.1.0 was throwing errors while the older 4.0.2 worked fine. \n The issue was that 4.1.0 wanted a CPU which supports SSSE3 and on the older CPUs it would fail. \n This was fixed for Gradle Plugin 4.2.0-rc01 / Gradle 7.0.0 alpha 9"

>FDroid is by far the largest non-google android store at the moment

Samsung Galaxy Store is much much bigger.

why you read "google build tools cannot be built from source and it was compiled with an optional optimizations as required" and assume the right thing to do is to buy newer servers?

Have you tried building AOSP from available sources?

Binaries everywhere. Tried to rebuild some of them with the available sources and noped the f out because that breaks the build so bad it's ridiculous.

Using Docker with QEMU CPU emulation would be a more maintainable solution than recompiling aapt2, as it would handle future binary updates automatically without requiring custom patches for each release.

> However at the same time, not even offering a fallback path in non-assembly?

There's probably not any hand-written assembly at issue here, just a compiler told to target x86_64-v2. Among others, RHEL 9 and derivatives were built with such options. (RHEL 10 bumped up the minimum spec again to x86_64-v3, allowing use of AVX.)

Looking at the issue their builders seem to be Opterons G3 (K10?)[0]

[0] https://en.wikipedia.org/wiki/AMD_10h

The problem with offering fallbacks is testing -- there isn't any reasonable hardware which you could use, because as you say it's all very old and slow.

I'm sure theyll appreciate your old desktop donation

SDK binaries provided by Google are still used, see https://forum.f-droid.org/t/call-for-help-making-free-softwa...

I agree, this should be the case, but Gradle specifically relies on downloading prebuilt java libraries and such to build itself and anything you build with it, and sometimes these have prebuilt native code inside. Unlike buildroot and any linux distribution, there's no metadata to figure out how to build each library, and the process for them is different between each library (no standards like make, autotools and cmake), so building the gradle ecosystem from source is very tedious and difficult.

It is not fixed.

In the thread you linked to people are confusing a typo correction ("mas fixed" => "was fixed") as a claim about this new issue being fixed.

The one that was fixed is this similar old issue from years ago: https://issuetracker.google.com/issues/172048751

Still haven't. Currently, most of the devs aren't aware of this underlying issue!

It has been introduced in Intel Penryn, in November 2007.

However the AMD CPUs did not implement it until Bulldozer, in mid 2011.

While they lacked the many additional instructions provided by Bulldozer, also including AVX and FMA, for many applications the older Opteron CPUs were significantly faster than the Bulldozer-based CPUs, so there were few incentives for upgrading them, before the launch of AMD Epyc in mid 2017.

SSE 4.1 is a cut point in supporting old CPUs for many software packages, because older CPUs have a very high overhead for divergent computations (e.g. with if ... else ...) inside loops that are parallelized with SIMD instructions.

> I'd assume that a modern CPU would do the same amount of work with a fraction of energy so that it does not even make economical sense to run such outdated hardware.

There are 8,760 hours in a non-leap year. Electricity in the U.S. averages 12.53 cents per kilowatt hour[1]. A really power-hungry CPU running full-bore at 500 W for a year would thus use about $550 of electricity. Even if power consumption dropped by half, that’s only about 10% of the cost of a new computer, so the payoff date of an upgrade is ten years in the future (ignoring the cost of performing the upgrade, which is non-negligible — as is the risk).

And of course buying a new computer is a capital expense, while paying for electricity is an operating expense.

1: https://www.eia.gov/electricity/monthly/epm_table_grapher.ph...

I haven’t seen the real answer that I suspect here - the build servers are that one dual socket AMD board which runs open firmware and has no ME/PSP .

On the server side, probably not, but I'd like to point out that old hardware is not uncommon, and it's going to be more and more likely as time passes especially in the desktop space.

I was hit by this scenario in the 2000s with an old desktop pc I had, also in the 10ys range, I was using just for boring stuff and random browsing, which was old, but perfectly adequate for the purpose. With time programs got rebuilt with some version of SSE it didn't support. When even firefox switched to the new instruction set, I had to essentially trash a perfectly working desktop pc as it became useless for the purpose.

I was going to say that I assume that the reason for such old CPUs is the ability to use Canoeboot/GNU Boot. But you absolutely can put an SSE4.2 CPU in a KGPE-D16 motherboard. So IDK.

Because setting up servers is an annoying piece of grunt-work that people avoid doing more than absolutely necessary, there's an reason the expensive options of AWS,Azure and Google cloud make money because much "just works" when focusing on applications rather than the infra (until you actually need to do something advanced and the obscure commands or clicking bites you in the ass).

Hardware after the first couple of generations of x86_64 muliticore processors are perfectly capable machines to use as servers, even for tasks you want to put off to a build farm.

A few months ago Adobe finally updated Lightroom Classic to require these processor extensions. To squeeze all of the matrix mults it can for AI features also in CPU mode.

It's amazing how long of a run top end hardware from ~2011 has had (just missed the cutoff by a few months). It's taken this long for stuff to really require these features.

The Catima thread makes FDroid sound like a really difficult commmunity to work with. Although I'm basing this on one person's comment and other people agreeing, not on any knowledge or experience.

> But this is like everything with F-Droid: everything always falls on a deaf man's ears. So I would rather not waste more time talking to a brick wall. If I had the feeling it was possible to improve F-Droid by raising issues and trying to discuss how to solve them I wouldn't have left the project out of frustration after years of putting so much time and energy into it.

Relatedly, we don't really have any up to date free software build of the Android SDK AFAIK. To build Android apps, we all rely on the Google binaries, which are non-free.

https://forum.f-droid.org/t/call-for-help-making-free-softwa...

> Their servers are so old

When I read this, pop culture has trained me to expect an insult, like: “Their servers are so old, they sat next to Ben Franklin in kindergarten.”

There is no point for Google to push planned obsolescence on the PC or server space. They don't have a market there.

"If I had the time, I'd try to compile a binary of it that will run on Win95 just to give my fuckings to the planned obsolescence crowd"

The idea that not supporting a 20+ year old system is "planned obsolescence" is a bit shallow

Like it is a one-off thing to support some system. You must maintain it and account it for all the features you bring in going forward.

The Win95 API is pretty incomplete. That was actually a terrible OS. The oldest I'd go playing this game with anything serious is probably XP.

But you don't, so you won't, scoring one for the planned obsolescence crowd.

And so won't anyone else who has time to complain about planned obsolescence, and that includes myself.

> Our machines run older server grade CPUs

So a bit of both of older hardware, and not-matched-with-consumer-featureset hardware. I'd imagine some server hardware vendors supported SSE4 way earlier than most, and some probably supported it way later than most too.

SSE4.1 can legally buy lightly alcoholic beverages in various European countries already. Next year, it can buy strong spirits.

Using AMD hardware that's "only" 13 years old can also cause this problem, though.

Yeah I was kind of shocked too. Core 2 could do both of those instruction sets. A used Dell Precision can be had for very little and probably would be grossly more efficient than whatever they're using.

That F-Droid requires to do the build ensures all apps provided by F-Droid are free software (as in freedom) and proven to be buildable by someone other than the app developer

So I should take a binary from a random stranger because trust me bro?

I stood by Oracle, because in the long term as it has been proven, Android is Google's J++, and Kotlin became Google's C#.

Hardly any different from what was in the genesis of .NET.

Nowadays they support up to Java 17 LTS, a subset only as usual, mostly because Android was being left behind accessing the Java ecosystem on Maven central.

And even though now ART is updatable via PlayStore, all the way down to Android 12, they see no need to move beyond Java 17 subset, until most likely they start again missing on key libraries that decided to adopt newer features.

Also stuff like Panama, Loom, Vector, Valhala (if ever), don't count them ever being supported on ART.

At least, they managed to push into mainstream the closest idea of OSes like Oberon, Inferno, Java OS and co, where regardless of what think about the superiotity of UNIX clones, here they have to contend themselves with a managed userspace, something that Microsoft failed at with Longhorn, Singularity and Midori due to their internal politics.

ARM phones didn't have virtualisation back in the day so that would've been impossible.

Modern Android has virtual machines on devices with supported hardware+bootloader+kernels: https://source.android.com/docs/core/virtualization

JVM??? hell no, native FTW

The build servers appear to be AMD Opteron G3s, which only support part of SSE4 (SSE4a). Full SSE4 support didn't land until Bulldozer (late 2011).

Yes, SSE4.1 and SSSE3 have been introduced in ~2006. The F-Droid build server still uses that to build modern and some of the most popular FOSS apps.

I've said this before, but I'll say it again. Running on donations is not a viable strategy for any long-term goal. FOSS needs to passively invest the donations. That is a viable long-term strategy. Now when things like this happen, it becomes a major line item moment, and not a limp-along situation, with yet another WE NEED YOUR HELP banner blocking off 1/2 their website.

The problem is that your "good enough" is someone else's "woefully inadequate", and sticking to the old feature sets is going to make the software horribly inefficient - or just plain unusable.

I'm sure there's someone out there who believe their 8086 is still "good enough", so should we restrict all software to the features supported by an 8086: 16-bit computations only, 1 MB of memory, no multithreading, no SIMD, no floats, no isolation between OS and user processes? That would obviously be ludicrous.

At a certain point it just doesn't make any sense to support hardware that old anymore. When it is cheaper to upgrade than to keep running the old stuff, and only a handful of people are sticking with the ancient hardware for nostalgic reasons, should that tiny group really be holding back basically your entire user base?

OTOH, if software wants to take advantage of modern features, it becomes hell to maintain if you have to have flags for every possible feature supported by CPUID. It's also unreasonable to expect maintainers to package dozens of builds for software that is unlikely to be used.

There's some guidelines[1][2] for developers to follow for a reasonable set of features, where they only need to manage ~4 variants. In this proposal the lowest set of features include SSE4.1, which is basically includes nearly any x86_64 CPU from the past 15 years. In theory we could use a modern CPU to compile the 4 variants and ship them all in a FatELF, so we only need to distribute one set of binaries. This of course would be completely impractical if we had to support every possible CPU's distinct features, and the binaries would be huge.

[1]:https://lists.llvm.org/pipermail/llvm-dev/2020-July/143289.h...

[2]:https://en.wikipedia.org/wiki/X86-64#Microarchitecture_level...

The F-Drois builds have been slow for years and with how old their servers apparently are that isn't even surprising in retrospective.

Exactly. Everything should be compiled to target i386.

/s (should be obvious but probably not for this audience)

control the universe

Guess what the company behind Android wants to do...

Am I missing something, or does SIMDe only help for cases where a program is using instruction intrinsics, and it doesn't do anything to address cases where the compiler decides to use SIMD as a result of auto-vectorization?

IzzyOnDroid can publish updates even if it's not reproducible, this is not an "app publishing" issue at all. IzzyOnDroid can deal with AGP 8.12 fine.

Also "not necessarily your fault" means "probably not your fault", the opposite of "your fault"

Either they want to run on ideologically pure hardware too, without pesky management bits in it (or even indeed UEFI), or they are just "it used to work perfectly" guys.

In the former case, I fail to see how ME or its absence is relevant to building Android apps, which they do using Google-provided binaries that have even more opportunity to inject naughty bits into the software. In the latter case, I better forget they exist.

It's not the target that is now requiring new instructions, but one of the components in the build tools.

There are very few 17+ years old build servers at this point. Or laptops and desktops for that matter.

Half the point is that I trust this middleman more than the app devs. When app developers turn evil ( https://news.ycombinator.com/item?id=38505229 ), I explicitly want someone reviewing things and blocking software that works against my interests before it gets to me.

Obtainium assumes that the app developer is a trustworthy entity, when the reality behind the mobile ecosystem being as fucked up as it is primarily comes from the app developer. (Due to bad incentives made by mobile platform makers, mainly Apple.)

You need a middleman in place in case the app developer goes bad.

I have it installed. But the only thing I get updates for is Obtainium itself. There's no catalogue of apps, so I haven't installed anything via Obtainium.

this seems to be a general app finder and tracker. useful, but entirely different from what f-droid does, namely verify that apps are actually Free Software or Open Source and buildable from source.

How is this not another middleman (with a political banner in its README no less)?

A shitton of people, not to mention including all F-Droid users, would take FOSS ideology over new fangled bloated "non-decrepit" development tools _any day_.

But in any case, this is false dichotomy, and likely exaggerated one to begin with.

I think it's extremely useful to have more strict requirements on how programs are built, to make sure that developers don't do stupid things that makes code harder for others to compile.

The tools in question in OP should be easy to build from source and not rely on the host's architecture, to be usable on platforms like ARM and RISCV. It's clear that in the android ecosystem, people don't care, so F-Droid can't do miracles (the java/gradle ecosystem is just really bad at this), but this would not happen if the build tools had proper build recipes themselves.

As a user, i'm glad when devs use old tools so that my battery has a chance of lasting the whole day and my apps don't take 10 seconds just to open.