Comment by brentroose
4 days ago
A month ago, I went on a performance quest trying to optimize a PHP script that took 5 days to run. Together with the help of many talented developers, I eventually got it to run in under 30 seconds. This optimization process with so much fun, and so many people pitched in with their ideas; so I eventually decided I wanted to do something more.
That's why I built a performance challenge for the PHP community
The goal of this challenge is to parse 100 million rows of data with PHP, as efficiently as possible. The challenge will run for about two weeks, and at the end there are some prizes for the best entries (amongst the prize is the very sought-after PhpStorm Elephpant, of which we only have a handful left).
I hope people will have fun with it :)
Pitch this to whoever is in charge of performance at Wordpress.
A Wordpress instance will happily take over 20 seconds to fully load if you disable cache.
Are you talking about a new, empty WordPress instance running the default theme? Because if so, that doesn't match my anecdotal experience.
If you're talking about a WordPress instance with arbitrary plugins running an arbitrary theme, then sure — but that's an observation about those plugins and themes, not core.
As someone who has to work with WordPress, I have all kinds of issues with it, but "20 seconds to load core with caching disabled" isn't one of them.
Can concur. I bought a plugin a few years ago after using the free version for many years, wanting to support the devs for making such a useful plugin. I installed it on a few sites, and left my PC running overnight with a tab open to the plugin and woke up the next day to a lovely rebooted Windows (I hate how default Windows behavior after BSOD is a reboot with ZERO indication that it crashed or if it was an update that rebooted). Re-opened all my tabs, and queue the same waking up the next day to a freshly rebooted Windows, which made me suspicious. I assumed at that point it must have been a BSOD, so dug into Windows event logs, eventually realizing it was Firefox. Restored tabs yet again, left browser open over night, while installing more and more debugging tools for Firefox, none of which helped me track down the culprit. What pissed me off the most was that Firefox even allowed a process to consume > 30GB of RAM and cause my PC to crash! I finally caught it one night after > 10 BSODs - the tab had been open for 20+ hours, and right as it started to spiral out of control and my PC was about to crash, as programs were starting to error out and Windows was madly paging things to disk. I got lucky, and was able to open about:memory to see the culprit - this plugin had some kind of memory leak that wasn't noticeable but then suddenly went nuts. I emailed the devs multiple times with the full debug output, and was ignored for weeks until finally they responded, which pissed me off even more having finally paid for the pro version, only to be greeted with this. The free version didn't seem to have this issue either, which was like an extra slap in the face.
Naked Wordpress is plenty fast, but as soon as you start adding sketchy plugins and Themes, things can spiral out of control.
1 reply →
Microbenchmarks are very different from optimizing performance in real applications in wide use though, they could do great on this specific benchmark but still have no clue about how to actually make something large like Wordpress to perform OK out of the box.
Wordpress is something that I cannot believe hasn't been displaced by a service that uses a separate application for editing and delivery.
It seems like something like vercel/cloudflare could host the content-side published as a worker for mostly-static content from a larger application and that would be more beneficial and run better with less risk, for that matter. Having the app editing and auth served from the same location is just begging for the issues WP and plugins have seen.
As someone who built full ecommerce websites on wordpress over 15 years ago, I can tell you exactly why it hasn't been replaced - the plugin/theme ecosystem.
There are tens of thousands of plugins and themes to make a Wordpress website do whatever you want and look however you want, either for free or a very low fee. You have to replace that entire ecosystem for the same price to replace Wordpress.
No matter how many times people get hacked, the perceived value of getting something for nothing outweighs the eventual cost.
3 replies →
That's often a skill issue.
skill issue being they only know php
Much like anything else your performance is going to vary a lot based on architecture of implementation. You really shouldn't deploying anything into production without some kind of caching. Whether that's done in the application itself or with memcached/redis or varnish or OPcache.
Either you use a slow language and deal with caching or you use a fast language and just put Cloudflare/Bunny/etc in front.
1 reply →
> You really shouldn't deploying anything into production without some kind of caching.
Citation needed? You only need cache if a render is expensive to produce.
I’ve long since abandoned WP but this seems like an SQL resource issue as it bogs up against the oom reaper dealing with no swap. WordPress is like a mid level V6 Kia with all the options and a huge aftermarket.
> A month ago, I went on a performance quest trying to optimize a PHP script that took 5 days to run. Together with the help of many talented developers, I eventually got it to run in under 30 seconds
That's a huge improvement! How much was low hanging fruit unrelated to the PHP interpreter itself, out of curiosity? (E.g. parallelism, faster SQL queries etc)
Almost all, actually. I wrote about it here: https://stitcher.io/blog/11-million-rows-in-seconds
A couple of things I did:
- Cursor based pagination - Combining insert statements - Using database transactions to prevent fsync calls - Moving calculations from the database to PHP - Avoiding serialization where possible
Aren’t these optimizations less about PHP, and more about optimizing how your using the database.
10 replies →
In general, it is bad practice to touch transaction datasets in php script space. Like all foot-guns it leads to Read-modify-write bugs eventually.
Depending on the SQL engine, there are many PHP Cursor optimizations that save moving around large chunks of data.
Clean cached PHP can be fast for REST transactional data parsing, but it is also often used as a bodge language by amateurs. PHP is not slow by default or meant to run persistently (low memory use is nice), but it still gets a lot of justified criticism.
Erlang and Elixir are much better for clients/host budgets, but less intuitive than PHP =3
Fun challenge, but running the benchmark on Apple hardware is a weird decision as Apple doesn't even have server hardware. Would make much more sense to run it on a dedicated Linux box as that is more accessible and more realistic.
Hehe. Optimization ... it's a good way to learn. Earlier in my career I did a lot of PHP. Usually close to bare.
Other than the obvious point that writing an enormous JSON file is a dubious goal in the first place (really), while PHP can be very fast this is probably faster to implement in shell with sed/grep, or ... almost certainly better ... by loading to sqlite then dumping out from there. Your optimization path then likely becomes index specification and processing, and after the initial load potentially query or instance parallelization.
The page confirms sqlite is available.
If the judges whinge and shell_exec() is unavailable as a path, as a more acceptable path that's whinge-tolerant, use PHP's sqlite feature then dump to JSON.
If I wanted to achieve this for some reason in reality, I'd have the file on a memory-backed blockstore before processing, which would yield further gains.
Frankly, this is not much of a programming problem, it's more a system problem, but it's not being specced as such. This shows, in my view, immaturity of conception of the real problem domain (likely IO bound). Right tool for the job.
5 days to 30 seconds? What kind of factor/order of magnitude is that damn
What takes 5 days to run
Poorly made analytics/datawarehouse stuff.
One query per column per row
Using a language that is 100x slower than naive native programs to do a "speed challenge" is like spending your entire day speed walking to run errands when you can just learn how to drive a car.
Do not update the leaderboard.... at all.
exec(‘c program that does the parsing’);
Where do I get my prize? ;)
The FAQ states that solutions like FFI are not allowed because the goal is to solve it with PHP :)
What about using the filesystem as an optimized dict implementation?
3 replies →
A month ago, I went on a performance quest trying to optimize a PHP script that took 5 days to run. Together with the help of many talented developers, I eventually got it to run in under 30 seconds.
When people say leetcode interviews are pointless I might share a link to this post. If that sort of optimization is possible there is a structures and algorithms problem in the background somewhere.
I find that these kind of optimizations are usually more about technical architecture than leetcode. Last time I got speedups this crazy the biggest win was reducing the number of network/database calls. There were also optimisations around reducing allocations and pulling expensive work out of hot loops. But leetcode interview questions don't tend to cover any of that.
They tend to be about the implementation details of specific algorithms and data structures. Whereas the important skill in most real-world scenarios would be to understand the trade-offs between different algorithms and data structures so that you pick an appropriate off-the-shelf implementation to use.
I agree. The "advanced" leetcode is about those last % of optimization. But when network latency is involved in a flow, it is usually the most obvious low hanging fruit.
Well leetcode asks you to implement the data structure, not how and when to use which data structure. I don’t need to know how to implement a bloom filter on a whiteboard off the top of my head to know when to use it.
Hell, the number of times I've used a lot of the data structures that come up in leetcode exercises without at least looking at some reference material is pretty small. I usually assume I'm going to misremember it, and go double check before I write it so I don't waste ages debugging later.
Do you think they achieved that performance optimisation with a networked service because they switched from insertion sort to quicksort?
I did the same thing in PHP before. Issue was a foreach over a foreach with a search. The fix was to build the result set as you populate the main array of objects. Basically as you add stuff to the array if there's a value that matches you throw it into another "results" array for any cardinality (unique value) that exists. Since in PHP objects are always just pointers your results arrays are relatively painless. Just a series of int32 integers basically. Then when you need an answer your result is instant. I ended up getting a 80-90% speed up. This is not just a php thing either. Folks end up doing this type of stuff in every language.