Redhat perl. What a tragedy.
At my new startup, Slaant, we use a lot of perl. We use perl for parsing massive amount of HTML/XML documents, we’ve written a homegrown RDF store in perl and we have a set of web applications built on Catalyst. We use OS X boxes for development and Centos 5.2 for production. Last week we deployed new hardware - over 150 cores of CPU - expecting much higher data processing throughput along the pipeline of 30 or so perl sub-systems - but the performance boost seemed marginal and perl stood out as the culprit. This was over a year of work, running on production scale hardware for the first time, and it wasn’t measuring up. We almost threw perl out of the window.
We have one lone FreeBSD box in our production environment and I happened to notice that a perl program that read JSON structures from disk files was over 100x faster on this box compared to our Centos boxes. It should’ve been bottlenecked on disk I/O, but strace showed it was burning userland CPU. Surprised, I ran the fantastic Devel::NYTProf and discovered the most expensive call, by a big margin, was a “bless”. bless!?! Perl will happily do millions of blesses a second on my 2Ghz macbook. And this was a dual-2.5gz quad core server. What the hell?
Some investigation revealed that there’s a long standing bug in Redhat Perl that causes *severe* performance degradation on code that uses the bless/overload combo. The thread on this is here: https://bugzilla.redhat.com/show_bug.cgi?id=379791
In the thread, ritz posted the following snippet. Try it. It should take under a second if the perl is not broken and a lot longer if it is.
#!/usr/bin/perl
use overload q(<) => sub {};
my %h;
for (my $i=0; $i<50000; $i++) {
$h{$i} = bless [ ] => ‘main’;
print STDERR ‘.’ if $i % 1000 == 0;
}
There isn’t official fix yet, but there’s a patch in the thread. We applied the patch. However, it did not make the problem go away, just delayed it - perl processes using bless/overload start slowing down (and continue to do so exponentially) after a while. At this point, I decided to recompile perl from source. The bug was gone. And the difference was appalling. Everything got seriously fast. CPUs were chilling at a loadavg below 0.10 and we were processing data 100x to 1000x faster! I was giddy. This was insane. We’d given up on one of the processes - to parse about 25M HTML documents using a HTML::TreeBuilder::XPath parser - because calculations showed that it would take over a year to parse them all. We assumed the Tree::XPathEngine was somehow intrinsically slow - so we’d rewrite our parsers using regexen at some point. With the new perl, we parsed these documents in 2 days. 2 days, instead of 365 days.
Rather massively blown away by this, I started sending the snippet above to various companies and projects I am involved with that use a lot of perl on Redhat or related distributions. It turns out many of them are running the broken perl and some of them had spent considerable amount of money and time in optimizing their perl code and infrastructure to work around the performance issue. It also turns out the issue exists on perl that comes with Fedora 9 - even if you compile perl Fedora 9 source package.
So, wow. How many people might be affected by this?
I realized that anyone running perl code with the distribution perl interpretter on Redhat 5.2, Centos 5.2 or Fedora 9 is likely a victim. Yes, even if your code doesn’t use the fancy bless/overload idiom, many CPAN modules do! This google search shows 1500+ modules use the bless/overload idiom and they include some really popular ones like URI, JSON.
According to this google trends analysis Redhat, Centos and Fedora make up the majority of linux distributions used in production. All these have a broken perl. How much time and money has been lost because of this? I have a sinking feeling that it is a staggering number. I also have a sinking feeling that many people have moved away from perl to python/ruby/java/C because this bug caused them to assume “perl is slow”. I am hoping this issue will get more visibility because it’s silently killing perl’s reputation and resulting in some very serious wastage of resources.
August 26, 2008: Nicholas Clark, perl core developer, explains the background and points out that fixes have been available since November 2007, they just haven’t made it into RedHat packages.
September 8, 2008: Karanbir Singh has published a fixed perl 5.8.8 RPM for Centos 5.2. See his post here with details on how to upgrade.
September 17, 2008: Redhat has released RPMs the fix this issue! The details and upgrade instructions are available in the bug fix advisory, RHBA-2008:0876-3.
January 20, 2009: This is the official Redhat fix - RHBA-2009:0117-3.
Wow, pasting in that code proved very tedious (given the single quote replacements, etc.). I recommend a simple tag next time :^)
That would be a simple <pre> tag.
Wow. Thanks for posting this.
Yep, it’s so bad that DBIx::Class throws a warning if you try to load it under a bad Red Hat perl. Since I’m using DBIC with my Catalyst apps, it’s the first thing I noticed when I pushed to production.
http://search.cpan.org/src/ASH/DBIx-Class-0.08010/lib/DBIx/Class/StartupCheck.pm
Using your own perl is a cinch as long as you’re not using mod_perl, and with Catalyst, FCGI is a much better option in the first place.
Wordpress screwed up the single quotes in the example: print STDERR ‘.’
There’s been problems with RedHat packaged Perl for quite a while now (at least back to RHEL 3.0 which had horrible unicode issues with RH packaged 5.8.0*).
As a matter of course I never rely on vendors compiled languages and always (if possible) opt for compiling relevant dev language from source code.
Thus this….
1) avoids these vendor back porting issues!
2) and allows me to patch language in my development cycle and not when a vendor upgrade/patch decides to do it!!
/I3az/
It is also another argument for organizations to compile their entire stack themselves: Apache, OpenSSL, Perl, etc.
It seems that the only value that distributions are adding to most of this software, is just new bugs. Remeber the Debian OpenSSL fiasco?
There is an old quote: “The only value you can add to a banana is a bruise”.
Wow.
I always build my own perl for Slashdot, and now I have even more reason to do so! Compare RH perl versus the one I built for it, a few months ago:
$ time /usr/bin/perl -e ‘use overload q( sub {}; my %h; for (my $i=0; $i “main” }’
real 0m5.338s
user 0m5.295s
sys 0m0.012s
$ time /usr/local/bin/perl -e ‘use overload q( sub {}; my %h; for (my $i=0; $i “main” }’
real 0m0.097s
user 0m0.084s
sys 0m0.010s
Now, we don’t use overload, but some of our many installed packages do. Building perl is not too hard. I always do it, on all my production/development machines.
[...] http://blog.vipul.net/2008/08/24/redhat-perl-what-a-tragedy/ [...]
Hey…. great post. I’m constantly amazed at how much BAD and broken code ships by default and stays in production for YEARS without anyone fixing it.
Unfortunately, people developing apps on more than 100 cores are rare.
Also, we’ve done similar tuning on our architecture obviously…….
We have some of the code using hand written finite state machine parsers.
One of our parsers is a custom written aho-corasick style parser written by hand to normalize html…….
I think our numbers were showing a 5-15x performance over our regex code…
… and I should note that Java’s regex performance is competitive with Perl.
Kevin
The brilliance of different distros of linux. It is a double-edged sword.
Perl definitely is a serious tool. It should be default on all OSes (hint hint MS)
I tested your example code. Your blog changed the apostrophes into ’smart quotes’ so perl throws an error.
Works fine on the Debian installs that Dreamhost uses, anyway.
Good catch! Note that the default Perl on all Red Hat-based Amazon EC2 VMIs has this bug, so if you’re using those instances for XML or JSON batch processing you’re probably spending $1 for every 1 cent worth of processing you’re getting done…
And now anyone wonders why Red Hat is not doing any innovation these days? Even Bill Gates owns portions of Red Hat.
Serious hackers should stop relying on packages by companies.
You are not alone, vipul. We just upgraded to 5.2 from 3.0 and encountered the exact same issue (i.e. we do LOTS of xml processing with libxml). My organization happens to have a real support contract with RH and we have an open ticket with their staff to address this issue. They delivered pre-release binaries to us last week that only half solved the problem. Instead of being 60x slower we are only 30x slower than we used to be. We haven’t heard from them since. Hopefully there will be answer soon.
-Chad
Thomas, ok, I think I fixed the quotes. It had a
I got bit by this about three monthbs ago doing some XML and SOAP stuff. Luckily our Red Hat support paid for itself by getting us a fix. Phew.
Are you saying that compiling your own perl from source gets you the speedup period, or do you have to also apply the patch first? And what version of perl did you end up using?
It’s cool that Perl manages to blow away other scripting languages with regard to speed, even with this bug in place. Thanks for sharing your discovery.
@Mark - The fix is to compile perl yourself. The patch does not solve the issue.
[...] interesting article I found over at Balance Through Extremism. Apparently RedHat has had a long time bug in the bless/overload function of Perl. It causes [...]
For what it is worth, this problem is gone in the latest updates for at least Fedora 9 and rawhide (they’re both on 5.10.0):
Running the test described here:
https://bugzilla.redhat.com/show_bug.cgi?id=379791#c5
[spot@localhost ~]$ time ./stour-test.pl
…………………………………………..
real 0m0.220s
user 0m0.154s
sys 0m0.047s
[spot@localhost ~]$ perl -v
This is perl, v5.10.0 built for x86_64-linux-thread-multi
….
[spot@localhost ~]$ rpm -q perl
perl-5.10.0-40.fc10.x86_64
[zhilliard@Colt45 ~]$ time perl ./testperl
…………………………………………..
real 0m0.455s
user 0m0.409s
sys 0m0.023s
[zhilliard@Colt45 ~]$ cat /etc/redhat-release
Red Hat Enterprise Linux ES release 4 (Nahant Update 5)
[zhilliard@Colt45 ~]$
[zhilliard@Colt45 ~]$ perl -v
This is perl, v5.8.8 built for i386-linux-thread-multi
Copyright 1987-2006, Larry Wall
So, does anyone know what RedHat is doing that breaks Perl?
we had the same problem with perl 5.8.8 on centos, an upgrade to perl5.10 fixed this problem.
@kevin that sounds highly neat. it’s hard to optimize regular expressions for large pages, not to mention that they are hard to maintain. we’ve had good success with HTML::TreeBuilder and friends - which, aside from this issue, are pretty fast and provide a reasonable speed / parser complexity tradeoff. after writing a whole bunch of different parsers i am thinking the ideal would be a LALR(1) parser that worked at the XML tag level.
Please, don’t jump the gun. Are your test case based on clean install?
On my clean install;
Red Hat 4
time /usr/bin/perl -e ‘use overload q( sub {}; my %h; for (my $i=0; $i<50000; $i++) { “main” }’
real 0m0.021s
user 0m0.017s
sys 0m0.004s
Red Hat 5
time /usr/bin/perl -e ‘use overload q( sub {}; my %h; for (my $i=0; $i<50000; $i++) { “main” }’
real 0m0.019s
user 0m0.016s
sys 0m0.002s
@jacob - yes, i posted this after several people repro’ed the issue on Redhat 5.2 and Centos 5.2. See my perl -V below. Can you show us yours? Another interesting thing is the patch mentioned in the thread above causes the repro script to work well but our code continues to exhibit the issue (slightly delayed onset as I mention). I will try to create a repro script for the patched version as well.
cargo2:~ vipul$ perl -V
configuration:
Summary of my perl5 (revision 5 version 8 subversion
Platform:
osname=linux, osvers=2.6.18-92.1.10.el5.centos.plus, archname=x86_64-linux-thread-multi
uname=’linux cargo1.mb.upperbeyond.com 2.6.18-92.1.10.el5.centos.plus #1 smp thu aug 7 12:20:08 edt 2008 x86_64 x86_64 x86_64 gnulinux ‘
config_args=’-des -Doptimize=-O2 -g -Dversion=5.8.8 -Dmyhostname=localhost -Dperladmin=root@localhost -Dcc=gcc -Dcf_by=Red Hat, Inc. -Dinstallprefix=/usr -Dprefix=/usr -Dlibpth=/usr/local/lib64 /lib64 /usr/lib64 -Dprivlib=/usr/lib/perl5/5.8.8 -Dsitelib=/usr/lib/perl5/site_perl/5.8.8 -Dvendorlib=/usr/lib/perl5/vendor_perl/5.8.8 -Darchlib=/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi -Dsitearch=/usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi -Dvendorarch=/usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi -Darchname=x86_64-linux -Dvendorprefix=/usr -Dsiteprefix=/usr -Duseshrplib -Dusethreads -Duseithreads -Duselargefiles -Dd_dosuid -Dd_semctl_semun -Di_db -Ui_ndbm -Di_gdbm -Di_shadow -Di_syslog -Dman3ext=3pm -Duseperlio -Dinstallusrbinperl=n -Ubincompat5005 -Uversiononly -Dpager=/usr/bin/less -isr -Dd_gethostent_r_proto -Ud_endhostent_r_proto -Ud_sethostent_r_proto -Ud_endprotoent_r_proto -Ud_setprotoent_r_proto -Ud_endservent_r_proto -Ud_setservent_r_proto -Dinc_version_list=5.8.7 5.8.6 5.8.5 -Dscriptdir=/usr/bin’
hint=recommended, useposix=true, d_sigaction=define
usethreads=define use5005threads=undef useithreads=define usemultiplicity=define
useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
use64bitint=define use64bitall=define uselongdouble=undef
usemymalloc=n, bincompat5005=undef
Compiler:
cc=’gcc’, ccflags =’-D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -pipe -Wdeclaration-after-statement -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -I/usr/include/gdbm’,
optimize=’-O2 -g’,
cppflags=’-D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -pipe -Wdeclaration-after-statement -I/usr/local/include -I/usr/include/gdbm’
ccversion=”, gccversion=’4.1.2 20071124 (Red Hat 4.1.2-42)’, gccosandvers=”
intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
ivtype=’long’, ivsize=8, nvtype=’double’, nvsize=8, Off_t=’off_t’, lseeksize=8
alignbytes=8, prototype=define
Linker and Libraries:
ld=’gcc’, ldflags =”
libpth=/usr/local/lib64 /lib64 /usr/lib64
libs=-lresolv -lnsl -lgdbm -ldb -ldl -lm -lcrypt -lutil -lpthread -lc
perllibs=-lresolv -lnsl -ldl -lm -lcrypt -lutil -lpthread -lc
libc=/lib/libc-2.5.so, so=so, useshrplib=true, libperl=libperl.so
gnulibc_version=’2.5′
Dynamic Linking:
dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags=’-Wl,-E -Wl,-rpath,/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE’
cccdlflags=’-fPIC’, lddlflags=’-shared -O2 -g’
Characteristics of this binary (from libperl):
Compile-time options: MULTIPLICITY PERL_IMPLICIT_CONTEXT
PERL_MALLOC_WRAP USE_64_BIT_ALL USE_64_BIT_INT
USE_ITHREADS USE_LARGE_FILES USE_PERLIO
USE_REENTRANT_API
Built under linux
Compiled at Aug 14 2008 22:48:25
@INC:
/usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi
/usr/lib64/perl5/site_perl/5.8.7/x86_64-linux-thread-multi
/usr/lib64/perl5/site_perl/5.8.6/x86_64-linux-thread-multi
/usr/lib64/perl5/site_perl/5.8.5/x86_64-linux-thread-multi
/usr/lib/perl5/site_perl/5.8.8
/usr/lib/perl5/site_perl/5.8.7
/usr/lib/perl5/site_perl/5.8.6
/usr/lib/perl5/site_perl/5.8.5
/usr/lib/perl5/site_perl
/usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi
/usr/lib64/perl5/vendor_perl/5.8.7/x86_64-linux-thread-multi
/usr/lib64/perl5/vendor_perl/5.8.6/x86_64-linux-thread-multi
/usr/lib64/perl5/vendor_perl/5.8.5/x86_64-linux-thread-multi
/usr/lib/perl5/vendor_perl/5.8.8
/usr/lib/perl5/vendor_perl/5.8.7
/usr/lib/perl5/vendor_perl/5.8.6
/usr/lib/perl5/vendor_perl/5.8.5
/usr/lib/perl5/vendor_perl
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi
/usr/lib/perl5/5.8.8
.
A true WTF.
Fedora 8’s latest Perl package doesn’t exhibit this problem.
I tested it on both x86_64 and i386 (respectively)
[vector@smoke tmp]$ time ./slow-redhat-perl.pl
…………………………………………..
real 0m0.461s
user 0m0.346s
sys 0m0.026s
[vector@smoke ~]$ uname -a
Linux smoke 2.6.25.11-60.fc8 #1 SMP Mon Jul 21 01:40:51 EDT 2008 x86_64 x86_64 x86_64 GNU/Linux
[vector@smoke ~]$ cat /etc/redhat-release
Fedora release 8 (Werewolf)
[vector@smoke ~]$ perl -v
This is perl, v5.8.8 built for x86_64-linux-thread-multi
>———————————————-<
[vector@heat ~]$ time ./slow-redhat-perl.pl
…………………………………………..
real 0m9.125s
user 0m9.086s
sys 0m0.022s
[vector@heat ~]$ uname -a
Linux heat 2.6.18-92.1.10.el5 #1 SMP Tue Aug 5 07:41:53 EDT 2008 i686 i686 i386 GNU/Linux
[vector@heat ~]$ cat /etc/redhat-release
CentOS release 5.2 (Final)
[vector@heat ~]$ perl -v
This is perl, v5.8.8 built for i386-linux-thread-multi
So far, I haven’t found any commonality ASIDE from one being Fedora, one being CentOS. Good thing we’re planning on rolling our own packages for production soon.
Poo, it seems half of my comment was deleted. Might have been user errror.
At any rate, both of my Fedora 8 machines (both patched to latest, one i386, one x86_64) do not exhibit the problem, but the two RHEL 5 boxes I have (one is i386, one x86_64, both patched to latest) do exhibit the problem rather dramatically.
This is partly your fault. Every perl application developer worth his grain of salt knows to use a build policy for their application.
http://search.cpan.org/~rgarcia/perl/INSTALL#Site-wide_Policy_settings
So, a wtf, but trwtf is that you don’t have a build policy for your application.
@vipul,
I would give you audit of all of our x86 RH 3/4/5 with last stable update (mainly security critical update) and Fedora 5/6/7/8 (Fedora 9 is still unstable with my nVidia card…) with all of latest yum update, but my result would be same.
It seems your test cases only involve x64_86 arch binary. Can you load x86 kernel on same architecture machine and test against x86 compiled perl? It may be that *sigh* another x64_86 package build related issue.
$ cat /proc/cpuinfo
…
model name : Intel(R) Xeon(TM) CPU 3.00GHz
…
—————————————-
$ time /usr/bin/perl -e ‘use overload q( sub {}; my %h; for (my $i=0; $i<50000; $i ) { “main” }’
real 0m0.021s
user 0m0.019s
sys 0m0.002s
—————————————-
$ rpm -q perl
perl-5.8.5-36.2
—————————————-
$ cat /etc/redhat-release
Red Hat Enterprise Linux ES release 4 (Nahant Update 6)
—————————————-
$ perl -V
Summary of my perl5 (revision 5 version 8 subversion 5) configuration:
Platform:
osname=linux, osvers=2.6.9-34.elsmp, archname=i386-linux-thread-multi
uname=’linux builder 2.6.9-34.elsmp #1 smp tue apr 25 18:00:49 edt 2006 i686 i686 i386 gnulinux ‘
config_args=’-des -Doptimize=-O2 -g -pipe -m32 -march=i386 -mtune=pentium4 -Dversion=5.8.5 -Dmyhostname=localhost -Dperladmin=root@localhost -Dcc=gcc -Dcf_by=Red Hat, Inc. -Dinstallprefix=/usr -Dprefix=/usr -Darchname=i386-linux -Dvendorprefix=/usr -Dsiteprefix=/usr -Duseshrplib -Dusethreads -Duseithreads -Duselargefiles -Dd_dosuid -Dd_semctl_semun -Di_db -Ui_ndbm -Di_gdbm -Di_shadow -Di_syslog -Dman3ext=3pm -Duseperlio -Dinstallusrbinperl -Ubincompat5005 -Uversiononly -Dpager=/usr/bin/less -isr -Dinc_version_list=5.8.4 5.8.3 5.8.2 5.8.1 5.8.0′
hint=recommended, useposix=true, d_sigaction=define
usethreads=define use5005threads=undef useithreads=define usemultiplicity=define
useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
use64bitint=undef use64bitall=undef uselongdouble=undef
usemymalloc=n, bincompat5005=undef
Compiler:
cc=’gcc’, ccflags =’-D_REENTRANT -D_GNU_SOURCE -DDEBUGGING -fno-strict-aliasing -pipe -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -I/usr/include/gdbm’,
optimize=’-O2 -g -pipe -m32 -march=i386 -mtune=pentium4′,
cppflags=’-D_REENTRANT -D_GNU_SOURCE -DDEBUGGING -fno-strict-aliasing -pipe -I/usr/local/include -I/usr/include/gdbm’
ccversion=”, gccversion=’3.4.6 20060404 (Red Hat 3.4.6-9)’, gccosandvers=”
intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
ivtype=’long’, ivsize=4, nvtype=’double’, nvsize=8, Off_t=’off_t’, lseeksize=8
alignbytes=4, prototype=define
Linker and Libraries:
ld=’gcc’, ldflags =’ -L/usr/local/lib’
libpth=/usr/local/lib /lib /usr/lib
libs=-lresolv -lnsl -lgdbm -ldb -ldl -lm -lcrypt -lutil -lpthread -lc
perllibs=-lresolv -lnsl -ldl -lm -lcrypt -lutil -lpthread -lc
libc=/lib/libc-2.3.4.so, so=so, useshrplib=true, libperl=libperl.so
gnulibc_version=’2.3.4′
Dynamic Linking:
dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags=’-Wl,-E -Wl,-rpath,/usr/lib/perl5/5.8.5/i386-linux-thread-multi/CORE’
cccdlflags=’-fPIC’, lddlflags=’-shared -L/usr/local/lib’
Characteristics of this binary (from libperl):
Compile-time options: DEBUGGING MULTIPLICITY USE_ITHREADS USE_LARGE_FILES PERL_IMPLICIT_CONTEXT
Built under linux
Compiled at Jan 31 2008 22:15:51
@INC:
/usr/lib/perl5/5.8.5/i386-linux-thread-multi
/usr/lib/perl5/5.8.5
/usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi
/usr/lib/perl5/site_perl/5.8.4/i386-linux-thread-multi
/usr/lib/perl5/site_perl/5.8.3/i386-linux-thread-multi
/usr/lib/perl5/site_perl/5.8.2/i386-linux-thread-multi
/usr/lib/perl5/site_perl/5.8.1/i386-linux-thread-multi
/usr/lib/perl5/site_perl/5.8.0/i386-linux-thread-multi
/usr/lib/perl5/site_perl/5.8.5
/usr/lib/perl5/site_perl/5.8.4
/usr/lib/perl5/site_perl/5.8.3
/usr/lib/perl5/site_perl/5.8.2
/usr/lib/perl5/site_perl/5.8.1
/usr/lib/perl5/site_perl/5.8.0
/usr/lib/perl5/site_perl
/usr/lib/perl5/vendor_perl/5.8.5/i386-linux-thread-multi
/usr/lib/perl5/vendor_perl/5.8.4/i386-linux-thread-multi
/usr/lib/perl5/vendor_perl/5.8.3/i386-linux-thread-multi
/usr/lib/perl5/vendor_perl/5.8.2/i386-linux-thread-multi
/usr/lib/perl5/vendor_perl/5.8.1/i386-linux-thread-multi
/usr/lib/perl5/vendor_perl/5.8.0/i386-linux-thread-multi
/usr/lib/perl5/vendor_perl/5.8.5
/usr/lib/perl5/vendor_perl/5.8.4
/usr/lib/perl5/vendor_perl/5.8.3
/usr/lib/perl5/vendor_perl/5.8.2
/usr/lib/perl5/vendor_perl/5.8.1
/usr/lib/perl5/vendor_perl/5.8.0
/usr/lib/perl5/vendor_perl
.
Whoa, the comments on your first post exploded. Not sure if it’s because you now HAVE comments, or because you picked a touchy subject - but keep going!
[...] perl under most RedHat builds is broken (via [...]
The real tradegy is that people still use perl…
@jacob
I’ll try that. Also, you are running perl 5.8.5 - and all these reports are with 5.8.8. It’s more likely that’s the reason for your not seeing the issue.
[wchoi@wchoi tmp]$ cat /etc/redhat-release
Red Hat Enterprise Linux Client release 5.2 (Tikanga)
[wchoi@wchoi tmp]$ time /usr/bin/perl -e ‘use overload q( sub {}; my %h; for (my $i=0; $i<50000; $i ) { “main” }’
Can’t find string terminator “)” anywhere before EOF at -e line 1.
real 0m0.222s
user 0m0.001s
sys 0m0.004s
[wchoi@wchoi tmp]$ time /usr/bin/perl -e ‘use overload q( sub {}; my %h; for (my $i=0; $i<50000; $i ) { “main” })’
real 0m0.105s
user 0m0.011s
sys 0m0.007s
[wchoi@wchoi tmp]$
[wchoi@wchoi tmp]$ perl -V
configuration:
Summary of my perl5 (revision 5 version 8 subversion
Platform:
osname=linux, osvers=2.6.9-67.0.7.elsmp, archname=i386-linux-thread-multi
uname=’linux hs20-bc2-4.build.redhat.com 2.6.9-67.0.7.elsmp #1 smp wed feb 27 04:47:23 est 2008 i686 i686 i386 gnulinux ‘
config_args=’-des -Doptimize=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector –param=ssp-buffer-size=4 -m32 -march=i386 -mtune=generic -fasynchronous-unwind-tables -Dversion=5.8.8 -Dmyhostname=localhost -Dperladmin=root@localhost -Dcc=gcc -Dcf_by=Red Hat, Inc. -Dinstallprefix=/usr -Dprefix=/usr -Darchname=i386-linux -Dvendorprefix=/usr -Dsiteprefix=/usr -Duseshrplib -Dusethreads -Duseithreads -Duselargefiles -Dd_dosuid -Dd_semctl_semun -Di_db -Ui_ndbm -Di_gdbm -Di_shadow -Di_syslog -Dman3ext=3pm -Duseperlio -Dinstallusrbinperl=n -Ubincompat5005 -Uversiononly -Dpager=/usr/bin/less -isr -Dd_gethostent_r_proto -Ud_endhostent_r_proto -Ud_sethostent_r_proto -Ud_endprotoent_r_proto -Ud_setprotoent_r_proto -Ud_endservent_r_proto -Ud_setservent_r_proto -Dinc_version_list=5.8.7 5.8.6 5.8.5 -Dscriptdir=/usr/bin’
hint=recommended, useposix=true, d_sigaction=define
usethreads=define use5005threads=undef useithreads=define usemultiplicity=define
useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
use64bitint=undef use64bitall=undef uselongdouble=undef
usemymalloc=n, bincompat5005=undef
Compiler:
cc=’gcc’, ccflags =’-D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -pipe -Wdeclaration-after-statement -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -I/usr/include/gdbm’,
optimize=’-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector –param=ssp-buffer-size=4 -m32 -march=i386 -mtune=generic -fasynchronous-unwind-tables’,
cppflags=’-D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -pipe -Wdeclaration-after-statement -I/usr/local/include -I/usr/include/gdbm’
ccversion=”, gccversion=’4.1.2 20071124 (Red Hat 4.1.2-42)’, gccosandvers=”
intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
ivtype=’long’, ivsize=4, nvtype=’double’, nvsize=8, Off_t=’off_t’, lseeksize=8
alignbytes=4, prototype=define
Linker and Libraries:
ld=’gcc’, ldflags =’ -L/usr/local/lib’
libpth=/usr/local/lib /lib /usr/lib
libs=-lresolv -lnsl -lgdbm -ldb -ldl -lm -lcrypt -lutil -lpthread -lc
perllibs=-lresolv -lnsl -ldl -lm -lcrypt -lutil -lpthread -lc
libc=/lib/libc-2.5.so, so=so, useshrplib=true, libperl=libperl.so
gnulibc_version=’2.5′
Dynamic Linking:
dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags=’-Wl,-E -Wl,-rpath,/usr/lib/perl5/5.8.8/i386-linux-thread-multi/CORE’
cccdlflags=’-fPIC’, lddlflags=’-shared -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector –param=ssp-buffer-size=4 -m32 -march=i386 -mtune=generic -fasynchronous-unwind-tables -L/usr/local/lib’
Characteristics of this binary (from libperl):
Compile-time options: MULTIPLICITY PERL_IMPLICIT_CONTEXT
PERL_MALLOC_WRAP USE_ITHREADS USE_LARGE_FILES
USE_PERLIO USE_REENTRANT_API
Built under linux
Compiled at Jun 5 2008 08:05:05
@INC:
/usr/lib/perl5/site_perl/5.8.8/i386-linux-thread-multi
/usr/lib/perl5/site_perl/5.8.7/i386-linux-thread-multi
/usr/lib/perl5/site_perl/5.8.6/i386-linux-thread-multi
/usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi
/usr/lib/perl5/site_perl/5.8.8
/usr/lib/perl5/site_perl/5.8.7
/usr/lib/perl5/site_perl/5.8.6
/usr/lib/perl5/site_perl/5.8.5
/usr/lib/perl5/site_perl
/usr/lib/perl5/vendor_perl/5.8.8/i386-linux-thread-multi
/usr/lib/perl5/vendor_perl/5.8.7/i386-linux-thread-multi
/usr/lib/perl5/vendor_perl/5.8.6/i386-linux-thread-multi
/usr/lib/perl5/vendor_perl/5.8.5/i386-linux-thread-multi
/usr/lib/perl5/vendor_perl/5.8.8
/usr/lib/perl5/vendor_perl/5.8.7
/usr/lib/perl5/vendor_perl/5.8.6
/usr/lib/perl5/vendor_perl/5.8.5
/usr/lib/perl5/vendor_perl
/usr/lib/perl5/5.8.8/i386-linux-thread-multi
/usr/lib/perl5/5.8.8
Great article Vipul, thanks.
I wonder if you have benchmarked this problem with perl 5.10 and if there are any differences there? From what I understand perl 5.10 should be available in Red Hat, or at least Fedora as a package. I know it is available in debian testing and will be released when lenny gets released in September.
@Tom Packaging and linux distributions are quite useful, especially for users which make up the bulk of linux installations. Distributions often take care of things like security which is a very difficult task for a single user. They also solve problems like dependencies. In the perl world there is cpan and cpanp which can install any CPAN module you need but if you want to use a software package where you are unfamiliar with the language and the consequent dependencies needed, then something like apt-get is extremely useful and time saving. So your specious quote does not apply here.
Importante bug de rendimiento de Perl en RedHat…
Un bug de rendimiento en las versiones de perl instaladas con RedHat y afines (Fedora, CentOS), padecen un grave problema de rendimiento que llega a ralentizar 100 o hasta 1000 veces algunas aplicaciones. Via Reddit: http://www.reddit.com/r/programming/commen...
So instead of complaining, why aren’t you contributing a fix?
vk> The real tradegy is that people still use perl…
Ah, Red Hat themselves weigh in.
> we had the same problem with perl 5.8.8 on centos, an upgrade to
> perl5.10 fixed this problem.
WARNING:
Do not “upgrade” your system’s Perl. It belongs to the system and many system tools rely on it. You will break, many applications by “upgrading” your system Perl to Perl 5.10, if you don’t take speical care.
See:
http://www.perlmonks.org/?node_id=691399
Why this happened:
http://use.perl.org/~nicholas/journal/37274
FYI, having the same issue with cPanel’s 5.8.8 Perl build on CentOS.
http://forums.cpanel.net/showthread.php?t=90925
So you developed something on platform A and without testing it on platform B purchased “over 150 cores of CPU”. Now this bug may be a real issue, but you could have made your life a whole lot easier with some proper planning.
I have tested this on Debian etch 4.0 and it does not have this problem but a patched CentOS 4.6 (Final) has this problem. Both running on 64_x86 Intel Xeon.
Very intresting article thanks alot.
rjd, your comment about mod_perl being difficult to build against your own perl is not correct. It’s simple to build mod_perl with any perl you have on your system — just use the perl binary you want to build against when running “perl Makefile.PL”. It’s the same as building any other XS module.
*shrug*
Tested the posted code under Fedora 9 32bit, Debian 32 bit (Dreamhost), RHEL 4.6 32 and 64, and RHEL 5.2 64 bit.
The only one which had any issues was the RHEL 5.2 64 bit. And, my contact at RH said they are working on fixing that release, but he did not give any details on what the issue was.
I think that this may be a once off issue, and that the tone of the post may be a bit alarmist.
If you are looking to eek out ever last bit of performance from a box, then you need to to tune the stack, and that may include custom compiled tools. However, if you are supporting a large pile of hosts, then using the vendor tools is a better way to go.
Don
“We applied the patch. However, it did not make the problem go away, just delayed it - perl processes using bless/overload start slowing down (and continue to do so exponentially) after a while. At this point, I decided to recompile perl from source. The bug was gone. ”
Strange that recompiling from source would fix this issue. The only way this could happen is if the source code doesn’t match up with the perl build you had in the distribution. Must’ve been a later version that worked better than the official patch, I guess.
@jeremiah Fedora 9’s 5.10.0 is clean. Though some folks on #perl think it is not entirely production ready due to some memory leak issues.
1) AMD Turion64
Scientific Linux SL release 5.0 (Boron)
time ./t
…………………………………………..
real 0m29.992s
user 0m29.342s
sys 0m0.053s
This is perl, v5.8.8 built for x86_64-linux-thread-multi
2) HP i686
time ./t
…………………………………………..
real 0m1.593s
user 0m0.546s
sys 0m0.093s
This is perl, v5.10.0 built for cygwin-thread-multi-64int
3) SunBlade 150 Solaris 10
time ./t
…………………………………………..
real 0m2.147s
user 0m1.517s
sys 0m0.135s
This is perl, v5.8.4 built for sun4-solaris-64int
4) Sun-Fire-V490 Solaris 9
time ./t
…………………………………………..
real 0m0.394s
user 0m0.360s
sys 0m0.030s
This is perl, v5.8.2 built for sun4-solaris-multi
Your sysadmins should know about this issue; if they don’t, they’re not doing their job.
Generally, we prefer to take the SRPM and recompile fresh RPMS with this issue removed. This way you’re not having to worry about your own perl installs in non-standard locations, messing with binary and library paths for different applications and you still get the advantages of many RH customisations that *do* help performance/stability/security.
You guys should have used Debian from the beginning
Known Perl issues…
Do not use the perl that comes with Red Hat; it has serious performance problems…
[...] Джизас факинг Крайст! Внезапно они как бы говорят нам:
[OSUNIX-61] Install new perl on logviewer-wtl-000…
Install perl 5.10 or a non-Red Hat build of perl 5.8.8 on logviewer-wtl-000 so that I can test the performance of log-process.pl to see if it is impacted by the Red Hat 5.8.8 performance bug described here: http://blog.vipul.net/2008/08/24/redhat-perl….
Gentoo doesn’t appear affected (good for me, bad for you):
test@nst ~ $ time perl speedtest.pl
…………………………………………..
real 0m0.238s
user 0m0.200s
sys 0m0.040s
test@nst ~ $ cat /etc/gentoo-release
Gentoo Base System release 2.0.0
test@nst ~ $ emerge -vp perl | grep perl
[ebuild R ] dev-lang/perl-5.8.8-r5 USE=”berkdb doc* gdbm -build -debug -ithreads -perlsuid” 0 kB
There is another lesson available — the virtue of a truly heterogeneous environment.
Dunno if they still do it, but at one time, NASA had 4 computers controlling flight systems — 3 built by one vendor, and the 4th, using a different architecture, built from the ground up by another vendor. When an obscure bug devastates a platform, the more different your “plan-B,” the less likely it will fail too.
IF I ever find myself deploying to a large number of commodity servers, I’m going to try to ensure that we have at least two major variants of UNIX/LINUX represented. (My current environment is Sun servers with eleventy-four processors.)
If you had not had that one, out-of-place FreeBSD box, you would have proceeded to solve the wrong problem.
[...] ‘the slowest thing’ is the code supplied by your vendor? That was exactly the situation Vipul Ved Prakash discovered when he tinkered with a company Linux box on which Perl code was running at least 100 times slower [...]
How!
Nice Post.
On my Debian box:
$time perl test.pl
…………………………………………..
real 0m0.184s
user 0m0.168s
sys 0m0.012s
All my Perl code run fast
We gave up on the RedHat compiled perl in the AS2.1 days. Why? Because their package was not compiled with largefile support. When we questioned RedHat support about that, they only replied with “If we change that, we are changing the ABI” … wtf…
@vipul
No, it does not affect perl 5.8.8 on any of our x86 RHEL5 box.
$ cat /etc/redhat-release
Red Hat Enterprise Linux Server release 5 (Tikanga)
$ rpm -q perl
perl-5.8.8-10
$ time /usr/bin/perl -e ‘use overload q( sub {}; my %h; for (my $i=0; $i<50000; $i ) { “main” }’
real 0m0.020s
user 0m0.015s
sys 0m0.005s
I’m running perl 5.10.0-31.fc9, on Fedora 9 (Athlon X2 BE-2400). I just ran the snippet from “ritz”, and it took 1/4 second.
So I’m skeptical of the claim that “the issue exists on perl that comes with Fedora 9″.
Hey, looking at the official Bugzilla page for this bug :
https://bugzilla.redhat.com/show_bug.cgi?id=379791
(scroll down to the bottom)
It’s looks like RedHat is using the exact test package I and a co-worker of mine put together that illustrates this issue. And clearly RH has a Perl build in house that completely fixes this issue.
Look forward to a complete fix soon folks!!!
-CJ
[...] ./test_perl_setup.pl Si el item 3 tarda masomenos UN segundo, todo esta bien… pero si todo esta mal, va a tardar MUCHO MUCHO mas de un segundo. Para más detalles, pasen por aquí. [...]
On a Dell PowerEdge 1950, running CentOS 5.0:
…………………………………………..
real 0m3.744s
user 0m3.733s
sys 0m0.011s
On a 400Mhz PPC running yellow dog linux:
…………………………………………..
real 0m1.254s
user 0m0.836s
sys 0m0.104s
On FreeBSD 6.1 running in a virtual machine on a MacBook Pro:
real 0.17
user 0.04
sys 0.12
SLES 10, running in a virtual machine on a MacBook Pro:
real 0m0.338s
user 0m0.028s
sys 0m0.144s
[...] does this perl packaging bug affect Z? See this post and also this one for more details. If Z doesn’t bring along its own perl, this could be an [...]
[...] ./test_perl_setup Si el item 3 tarda masomenos UN segundo, todo esta bien… pero si todo esta mal, va a tardar MUCHO MUCHO mas de un segundo. Para más detalles, pasen por aquí. [...]
Links - 30.08.2008…
Zeit für den Ausklang des Monats. Als erstes eine kleine Anfrage. Falls jemand Lust auf ein kleines Einsteigerkompatibles Grails Projekt hat (mit Fokus auf Reporting und PDF Erstellung, nachdem eine kleine Logik zur Datenerfassung entwickelt hat), ka…
This is the same for OpenLDAP.
See our partners post at:
http://www.symas.com/blog/?p=20
Gavin.
Hey Vipul, It is a good find.
Cent OS 5.2 On a Dual Intel(R) Xeon(R) CPU Quad Core E5430 @ 2.66GHz .
time perl ~/tbless.pl
…………………………………………..
real 0m4.680s
user 0m4.668s
sys 0m0.007s
FC9 On AMD Athlon(tm) 64 X2 Dual Core Processor 4200+ 2210.190MHz
time perl tbless.pl
…………………………………………..
real 0m0.210s
user 0m0.193s
sys 0m0.017s
It is astonishing to see a such a performance degrade on a powerful machine.
Er… hang on… you are blaming RedHat for a perl issue,… but… whilst using CentOS…
if you want to whinge, you should at least go get some RedHat, put some $$ towards support, and THEN you have the right to complain about it… but even then, you should give them a chance to fix it first, (or explain why they cant) and after a reasonable period you then have the moral right to complain about their responsiveness…
i could be wrong here, but put your money where your mouth is and use the right channels.
Wow, this many comments and no one said it?
StumbleUpon’d
Fedora 9 here, yet several benchmarks (including the one above) show that the bug is fixed. Sorry to see Red Hat attracting public attention in such unpleasant way.
[...] of you may have heard about the kerfuffle surrounding poor Perl performance on Red Hat 5/CentOS 5 - it got a bit of attention on reddit/digg. Red Hat had been issuing hot [...]
FYI,
RH just delivered fresh rpm’s to my organization today. They fixed the problem 100%. I have no idea when they are going to integrate this fix into the mainline though.
-CJ
@chad
Excellent! Really hoping this gets merged into the mainline.
cheers,
vipul
Bug im Perl-RPM von RHEL5…
Ich betreue mehrere Installationen vom OTRS auf einem Red Hat Enterprise Linux 5 (RHEL5). OTRS wird mit mod_perl im Apache ausgeführt. Vor fast drei Monaten ist das System zum ersten Mal komplett abgestürzt: Es lief out-of-memory und konnte keine weite…