[Keepalived-devel] 100% CPU usage

unknown

1970-01-01 00:00:00 UTC

--bcaec52162f3383fe504de52d060
Content-Type: text/plain; charset=ISO-8859-1

HI,

When 100% CPU present, you can use "perf" to monitor where is hot spot.

Post by Rimbalza Rimbalza
Hi
I read about the problem in several posts but have not found no
resolutive answer.
The problem is 100% CPU usage, both by irq and keepalived process.
I use keepalived to have a VIP alive between 2 hosts to receive requests.
There are several VIPs, some are active (MASTER) on host A and some are
active on host B, to share the load. At each VIP is bound one haproxy
process to deliver the request to the real separated hosts.
No ipvs configuration here, just plain old vrrp IPs.
When there is a failover, CPU spikes to the top and doesn't get back
again. Sometimes reloading the cfg settles it down, last time (in
production ...) it didn't.
I read about onbackup transition scripts to "flush" and "reload"
something, but found no relevant information for me (if any at all I must
say).
Is there a solution to this problem or keepalived must be abandoned? In
this case, any suggestion for a replacement?
Using 1.2.7 compiled from source, under Ubuntu 12.04.2
x64 3.2.0-38-generic
Thanks
------------------------------------------------------------------------------
1. A cloud service to automate IT design, transition and operations
2. Dashboards that offer high-level views of enterprise services
3. A single system of record for all IT processes
http://p.sf.net/sfu/servicenow-d2d-j
_______________________________________________
Keepalived-devel mailing list
https://lists.sourceforge.net/lists/listinfo/keepalived-devel

--
-- Cz.

--bcaec52162f3383fe504de52d060
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable <div dir="ltr">HI,<div><br></div><div style>When 100% CPU present, you can use "perf" to monitor where is hot spot.</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">2013/6/4 Rimbalza Rimbalza <span dir="ltr"><<a href="mailto:***@gmail.com" target="_blank">***@gmail.com</a>></span><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi<div> I read about the problem in several posts but have not found no resolutive answer.</div><div>The problem is 100% CPU usage, both by irq and keepalived process.</div>
<div>I use keepalived to have a VIP alive between 2 hosts to receive requests. There are several VIPs, some are active (MASTER) on host A and some are active on host B, to share the load. At each VIP is bound one haproxy process to deliver the request to the real separated hosts.</div>

<div>No ipvs configuration here, just plain old vrrp IPs.</div><div><br></div><div>When there is a failover, CPU spikes to the top and doesn't get back again. Sometimes reloading the cfg settles it down, last time (in production ...) it didn't.</div>

<div><br></div><div>I read about onbackup transition scripts to "flush" and "reload" something, but found no relevant information for me (if any at all I must say).</div><div>Is there a solution to this problem or keepalived must be abandoned? In this case, any suggestion for a replacement?</div>

<div><br></div><div>Using 1.2.7 compiled from source, under Ubuntu 12.04.2 x64 3.2.0-38-generic</div><div>Thanks</div></div>
<br>------------------------------------------------------------------------------<br>
How ServiceNow helps IT people transform IT departments:<br>
1. A cloud service to automate IT design, transition and operations<br>
2. Dashboards that offer high-level views of enterprise services<br>
3. A single system of record for all IT processes<br>
<a href="http://p.sf.net/sfu/servicenow-d2d-j" target="_blank">http://p.sf.net/sfu/servicenow-d2d-j</a><br>_______________________________________________<br>
Keepalived-devel mailing list<br>
<a href="mailto:Keepalived-***@lists.sourceforge.net">Keepalived-***@lists.sourceforge.net</a><br>
<a href="https://lists.sourceforge.net/lists/listinfo/keepalived-devel" target="_blank">https://lists.sourceforge.net/lists/listinfo/keepalived-devel</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br>-- Cz.
</div>

--bcaec52162f3383fe504de52d060--

Paul Slootman

2013-06-04 12:26:09 UTC

Post by Rimbalza Rimbalza
I read about the problem in several posts but have not found no resolutive
answer.
The problem is 100% CPU usage, both by irq and keepalived process.

Just a wild guess: has this system been up since the previous leap
second was added?

If so, doing:
date -s "`date`"
might help. We've noticed a lot of different things eating CPU due to
the leap second problem. Java and MySQL were mentioned a lot at the
time, but other things also may suffer.

Paul

Rimbalza Rimbalza

2013-06-04 12:34:10 UTC

"This" is really several couples of servers I'm working with.
They range from machines online for days to machines online from seconds,
since I rebooted them several times to study the 100% behavior.
In the lists the problem seems known from ages, but never solved. Just
google for keepalived 100% cpu.
Here is the one that gave me some hope:
http://marc.info/?l=keepalived-devel&m=114040573028656
The link to the "remediation" is flawed, I contacted the author (very kind
in responding) but with no solution.
It is a problem about the Master and backup hosts flooding each other with
packets after a switchover.
The biggest problem I see is that keepalived seems an abandoned project...

Post by Rimbalza Rimbalza
I read about the problem in several posts but have not found no

resolutive

Post by Rimbalza Rimbalza
answer.
The problem is 100% CPU usage, both by irq and keepalived process.

Just a wild guess: has this system been up since the previous leap
second was added?
date -s "`date`"
might help. We've noticed a lot of different things eating CPU due to
the leap second problem. Java and MySQL were mentioned a lot at the
time, but other things also may suffer.
Paul
------------------------------------------------------------------------------
1. A cloud service to automate IT design, transition and operations
2. Dashboards that offer high-level views of enterprise services
3. A single system of record for all IT processes
http://p.sf.net/sfu/servicenow-d2d-j
_______________________________________________
Keepalived-devel mailing list
https://lists.sourceforge.net/lists/listinfo/keepalived-devel

Paul Robert Marino

2013-06-04 12:59:28 UTC

Rimbalza Rimbalza

2013-06-04 13:15:04 UTC

(damn, I forgot to cc the list. Re-re-posting. Re-sorry).
Glad to hear it not dead, I just started using it! Just noting that a
showstopper bug is reported 7 years ago and today it is here preventing the
usage of keepalived in a production environment. If it is due to a miscfg,
I'm very happy, here is one from "machine A".
Machine B has the two vrrp with MASTER/BACKUP exchanged and the same for
priorities.
To be precise this file is generated from a script, so BNAME1 and 2 are
examples of a generic MASTER and generic BACKUP.
The checking scripts just check the existence of the socket. Replacing them
with "exit 0" or removing them all have no impact on keepalived demon going
unstable.
Bye

# autocfg
# Created on Tue, 04 Jun 2013 15:02:57 +0200
# My role is: MASTER

global_defs {
notification_email {
***@email
}
notification_email_from ***@email
smtp_server 172.30.3.75
smtp_connect_timeout 5
}

# Description: Desc bil
vrrp_script chk_BNAME1 {
script "/lb/chkha.sh /lb/pids/ha-c1.sock"
interval 5
weight 75
}

vrrp_instance BNAME1 {
interface eth0
state MASTER
virtual_router_id 133
priority 150
vrrp_unicast_bind 172.30.111.10
vrrp_unicast_peer 172.30.111.11
smtp_alert
virtual_ipaddress {
172.30.111.12/16 dev eth0
}
authentication {
auth_type PASS
auth_pass supersecret
}
track_script {
chk_BNAME1
}
}

# Description: Desc bil 2
vrrp_script chk_BNAME2 {
script "/lb/chkha.sh /lb/pids/ha-c2.sock"
interval 5
weight 75
}

vrrp_instance BNAME2 {
interface eth0
state BACKUP
virtual_router_id 134
priority 100
vrrp_unicast_bind 172.30.111.10
vrrp_unicast_peer 172.30.111.11
smtp_alert
virtual_ipaddress {
172.30.111.13/16 dev eth0
}
authentication {
auth_type PASS
auth_pass supersecret
}
track_script {
chk_BNAME2
}
}

1)
Keepalived is not an abandon project not by a long shot. There were
several versions released last year and many people still write patches for
it its just a very slow release cycle in comparison to some other projects.
2) without seeing your config we can't help you. It sounds like you have
seriously misconfigured some thing. Most likely the priorities but it could
be other thing in any case it sounds like they are getting into a perpetual
election cycle they can't resolve.
-- Sent from my HP Pre3
------------------------------
"This" is really several couples of servers I'm working with.
They range from machines online for days to machines online from seconds,
since I rebooted them several times to study the 100% behavior.
In the lists the problem seems known from ages, but never solved. Just
google for keepalived 100% cpu.
http://marc.info/?l=keepalived-devel&m=114040573028656
The link to the "remediation" is flawed, I contacted the author (very kind
in responding) but with no solution.
It is a problem about the Master and backup hosts flooding each other with
packets after a switchover.
The biggest problem I see is that keepalived seems an abandoned project...

Post by Rimbalza Rimbalza
I read about the problem in several posts but have not found no

resolutive

Post by Rimbalza Rimbalza
answer.
The problem is 100% CPU usage, both by irq and keepalived process.

Paul Slootman

2013-06-04 14:11:30 UTC

Post by Rimbalza Rimbalza
The checking scripts just check the existence of the socket. Replacing them
with "exit 0" or removing them all have no impact on keepalived demon going
unstable.

Rimbalza Rimbalza

2013-06-04 14:17:28 UTC

They are. I have echos that append to a file and also they do their work,
commanding a failover if the socket is not present.

Post by Paul Slootman

Post by Rimbalza Rimbalza
The checking scripts just check the existence of the socket. Replacing

them

Post by Rimbalza Rimbalza
with "exit 0" or removing them all have no impact on keepalived demon

going

Post by Rimbalza Rimbalza
unstable.

Please check whether the scripts are actually executed;
I posted a question about vrrp scripts a couple of weeks ago,
apparently the config is not being read correctly and instead of the
script I defined, an empty shell command is called (which is replaced
with "exit 0" by the libc system() function).
Paul
------------------------------------------------------------------------------
1. A cloud service to automate IT design, transition and operations
2. Dashboards that offer high-level views of enterprise services
3. A single system of record for all IT processes
http://p.sf.net/sfu/servicenow-d2d-j
_______________________________________________
Keepalived-devel mailing list
https://lists.sourceforge.net/lists/listinfo/keepalived-devel

Ryan O'Hara

2013-06-04 20:15:01 UTC

Perhaps it is not abandoned, but it is not actively maintained. Saying
that is just has a "very slow release cycle" is being kind. People do
write patches, including me, but they aren't being incorporated into
the project. I've been collecting patches in my github trees for some
time now.

Ryan

Pasi Kärkkäinen

2013-06-05 09:26:01 UTC

What are the urls for your github trees btw?

It'd be nice to get a new official keepalived release out,
I've also sent a patch (add missing smtp to-header) that I'd like to get included upstream.

Thanks,

-- Pasi

Ryan O'Hara

2013-06-05 15:34:15 UTC

What are the urls for your github trees btw?

Bug fixes can be found in my devel branch:
https://github.com/rohara/keepalived/tree/devel

I believe all commits in my devel branch are part of this pull request:
https://github.com/acassen/keepalived/pull/15

Ryan

Rimbalza Rimbalza

2013-06-05 15:49:34 UTC

Did not try the unicast patches but I moved the same binaries and
configuration on two physical servers, and the 100% CPU problem is here
again...
So I'll exclude some virtualization stack fault in managing L2
communications.

Paul Robert Marino

2013-06-05 15:58:32 UTC

I'm curious what version are you using and on what distribution and version
of Linux
additionally is this a binary you compiled your self or one provided by the
distribution.

Post by Rimbalza Rimbalza
Did not try the unicast patches but I moved the same binaries and
configuration on two physical servers, and the 100% CPU problem is here
again...
So I'll exclude some virtualization stack fault in managing L2
communications.
------------------------------------------------------------------------------
1. A cloud service to automate IT design, transition and operations
2. Dashboards that offer high-level views of enterprise services
3. A single system of record for all IT processes
http://p.sf.net/sfu/servicenow-d2d-j
_______________________________________________
Keepalived-devel mailing list
https://lists.sourceforge.net/lists/listinfo/keepalived-devel

Rimbalza Rimbalza

2013-06-05 16:00:12 UTC

It's in one of my first posts.
Compiled 1.2.7 from source on Ubuntu 12.04.2 x64 (Linux xyz
3.5.0-32-generic #53~precise1-Ubuntu SMP Wed May 29 20:33:37 UTC 2013
x86_64 x86_64 x86_64 GNU/Linux)

Post by Paul Robert Marino
I'm curious what version are you using and on what distribution and
version of Linux
additionally is this a binary you compiled your self or one provided by
the distribution.

Rimbalza Rimbalza

2013-06-05 16:08:21 UTC

Yes, I'm using ONLY that version from that very same file. No patches yet.

I'm curious have you tried the stock version with no patches to see if you
get the same problem?
http://www.keepalived.org/software/keepalived-1.2.7.tar.gz

Post by Rimbalza Rimbalza
It's in one of my first posts.
Compiled 1.2.7 from source on Ubuntu 12.04.2 x64 (Linux xyz
3.5.0-32-generic #53~precise1-Ubuntu SMP Wed May 29 20:33:37 UTC 2013
x86_64 x86_64 x86_64 GNU/Linux)

Rimbalza Rimbalza

2013-06-05 16:26:36 UTC

Graeme Fowler

2013-06-05 16:34:57 UTC

Post by unknown
The stock binary behaves the same.
And also just returns to normality reloading the configuration.
1.2.2 stock is the same as 1.2.7 from official sources.

Just a thought…

Do you have SELinux enabled, in enforcing mode?

# getenforce

to switch off

# setenforce 0

If you're trying to do certain things that SELinux disapproves of, all sorts of fun and games can occur. And for the record, in somewhere approaching 8 or 9 years of using keepalived I have *never* had a process runaway to 100% CPU and wedge there. Not that it helps you much, but I can assure you that this isn't a widespread problem.

Graeme

Rimbalza Rimbalza

2013-06-05 16:59:57 UTC

SELinux disabled and no check scripts by me, same story.
When I restart one host, the other goes 100%.
Whet the other comes up, they both go 100%.
Reloading cfg restores the 0% status.
Tried again 1.2.2 stock binary and 1.2.7 from sources.
I'm starting to get mad...

Post by unknown
The stock binary behaves the same.
And also just returns to normality reloading the configuration.
1.2.2 stock is the same as 1.2.7 from official sources.

Just a thought
Do you have SELinux enabled, in enforcing mode?
# getenforce
to switch off
# setenforce 0
If you're trying to do certain things that SELinux disapproves of, all
sorts of fun and games can occur. And for the record, in somewhere
approaching 8 or 9 years of using keepalived I have *never* had a process
runaway to 100% CPU and wedge there. Not that it helps you much, but I can
assure you that this isn't a widespread problem.
Graeme
------------------------------------------------------------------------------
1. A cloud service to automate IT design, transition and operations
2. Dashboards that offer high-level views of enterprise services
3. A single system of record for all IT processes
http://p.sf.net/sfu/servicenow-d2d-j
_______________________________________________
Keepalived-devel mailing list
https://lists.sourceforge.net/lists/listinfo/keepalived-devel

Rimbalza Rimbalza

2013-06-05 15:58:21 UTC

git clone https://github.com/rohara/keepalived/tree/devel
Cloning into 'devel'...
fatal: https://github.com/rohara/keepalived/tree/devel/info/refs not found:
did you run git update-server-info on the server?

1)
Keepalived is not an abandon project not by a long shot. There

were

several versions released last year and many people still write

patches

for it its just a very slow release cycle in comparison to some

other

projects.

What are the urls for your github trees btw?

https://github.com/rohara/keepalived/tree/devel
https://github.com/acassen/keepalived/pull/15
Ryan
------------------------------------------------------------------------------
1. A cloud service to automate IT design, transition and operations
2. Dashboards that offer high-level views of enterprise services
3. A single system of record for all IT processes
http://p.sf.net/sfu/servicenow-d2d-j
_______________________________________________
Keepalived-devel mailing list
https://lists.sourceforge.net/lists/listinfo/keepalived-devel

Ryan O'Hara

2013-06-05 16:13:27 UTC

Post by Rimbalza Rimbalza
git clone https://github.com/rohara/keepalived/tree/devel
Cloning into 'devel'...
did you run git update-server-info on the server?

git clone https://github.com/rohara/keepalived.git

Then checkout 'devel' for 1.2.7 + bug fixes.
Or checkout 'vrrp_unicast' for my revised/updated unicast patches.

Ryan

1)
Keepalived is not an abandon project not by a long shot. There

were

several versions released last year and many people still write

patches

for it its just a very slow release cycle in comparison to some

other

projects.

What are the urls for your github trees btw?

Rimbalza Rimbalza

2013-06-05 16:15:59 UTC

Tried devel.
make[2]: Leaving directory `/tmp/keepalived/keepalived/libipvs-2.6'
Building ../bin/keepalived
../lib/vty.o: In function `vty_auth':
/tmp/keepalived/lib/vty.c:265: undefined reference to `crypt'
../lib/command.o: In function `zencrypt':
/tmp/keepalived/lib/command.c:436: undefined reference to `crypt'
collect2: ld returned 1 exit status
make[1]: *** [all] Error 1
make[1]: Leaving directory `/tmp/keepalived/keepalived'
make: *** [all] Error 2

Post by Rimbalza Rimbalza
git clone https://github.com/rohara/keepalived/tree/devel
Cloning into 'devel'...
fatal: https://github.com/rohara/keepalived/tree/devel/info/refs not
did you run git update-server-info on the server?

git clone https://github.com/rohara/keepalived.git
Then checkout 'devel' for 1.2.7 + bug fixes.
Or checkout 'vrrp_unicast' for my revised/updated unicast patches.
Ryan

1)
Keepalived is not an abandon project not by a long shot. There

were

several versions released last year and many people still

write

patches

for it its just a very slow release cycle in comparison to

some

other

projects.

Perhaps it is not abandoned, but it is not actively maintained.

Saying

Post by Ryan O'Hara
that is just has a "very slow release cycle" is being kind. People

Post by Ryan O'Hara
write patches, including me, but they aren't being incorporated

into

Post by Ryan O'Hara
the project. I've been collecting patches in my github trees for

some

Post by Ryan O'Hara
time now.

What are the urls for your github trees btw?

https://github.com/rohara/keepalived/tree/devel
https://github.com/acassen/keepalived/pull/15
Ryan

------------------------------------------------------------------------------

1. A cloud service to automate IT design, transition and operations
2. Dashboards that offer high-level views of enterprise services
3. A single system of record for all IT processes
http://p.sf.net/sfu/servicenow-d2d-j
_______________________________________________
Keepalived-devel mailing list
https://lists.sourceforge.net/lists/listinfo/keepalived-devel

Ryan O'Hara

2013-06-05 18:10:43 UTC

Post by Rimbalza Rimbalza
Tried devel.
make[2]: Leaving directory `/tmp/keepalived/keepalived/libipvs-2.6'
Building ../bin/keepalived
/tmp/keepalived/lib/vty.c:265: undefined reference to `crypt'
/tmp/keepalived/lib/command.c:436: undefined reference to `crypt'
collect2: ld returned 1 exit status
make[1]: *** [all] Error 1
make[1]: Leaving directory `/tmp/keepalived/keepalived'
make: *** [all] Error 2

This is problem in the master branch. My devel branch follows the head
so that pull requests are easier. There is some new code the master
branch that created this problem, specifically because -lcrypt is
missing from the LDFLAGS. This has been reported on the mailing list
at least once, and I believe a patch was submitted.

Ryan

git clone https://github.com/rohara/keepalived.git
Then checkout 'devel' for 1.2.7 + bug fixes.
Or checkout 'vrrp_unicast' for my revised/updated unicast patches.
Ryan

1)
Keepalived is not an abandon project not by a long shot. There

were

several versions released last year and many people still

write

patches

for it its just a very slow release cycle in comparison to

some

other

projects.

Perhaps it is not abandoned, but it is not actively maintained.

Saying

Post by Ryan O'Hara
that is just has a "very slow release cycle" is being kind. People

Post by Ryan O'Hara
write patches, including me, but they aren't being incorporated

into

Post by Ryan O'Hara
the project. I've been collecting patches in my github trees for

some

Post by Ryan O'Hara
time now.

What are the urls for your github trees btw?

https://github.com/rohara/keepalived/tree/devel
https://github.com/acassen/keepalived/pull/15
Ryan

------------------------------------------------------------------------------

Rimbalza Rimbalza

2013-06-06 10:20:00 UTC

Compiled your 'devel' version.
Same problem.

git clone https://github.com/rohara/keepalived.git
Then checkout 'devel' for 1.2.7 + bug fixes.
Or checkout 'vrrp_unicast' for my revised/updated unicast patches.
Ryan

On Tue, Jun 04, 2013 at 08:59:28AM -0400, Paul Robert Marino

1)
Keepalived is not an abandon project not by a long shot.

There

were

several versions released last year and many people still

write

patches

for it its just a very slow release cycle in comparison to

some

other

projects.

Perhaps it is not abandoned, but it is not actively maintained.

Saying

that is just has a "very slow release cycle" is being kind.

People

Post by Ryan O'Hara
do

write patches, including me, but they aren't being incorporated

into

the project. I've been collecting patches in my github trees

for

Post by Ryan O'Hara
some

time now.

What are the urls for your github trees btw?

https://github.com/rohara/keepalived/tree/devel
I believe all commits in my devel branch are part of this pull
https://github.com/acassen/keepalived/pull/15
Ryan

------------------------------------------------------------------------------

Paul Robert Marino

2013-06-04 16:52:12 UTC

forgot to do a reply all

---------- Forwarded message ----------
From: Paul Robert Marino <***@gmail.com>
Date: Tue, Jun 4, 2013 at 12:51 PM
Subject: Re: [Keepalived-devel] 100% CPU usage
To: Marcello Gorlani <***@gorlani.com>

no this is not what I'm saying

for example if your master has a priority of 150 the backup in normal
operation should have a priority of 149 if you have multiple backups they
should decrement by 1 for each additional backup.

Note: a difference of 50 in priorities between 2 instances constitutes a
fault regardless of what the advertised state is.

also that howto is ancient, out of date and should not be used any more. I
wish they would take it off the site or update but they never
have.following that doc and any other howto based on it based on it causes
problems.

The real up to date doc is here
https://github.com/acassen/keepalived/blob/master/doc/keepalived.conf.SYNOPSIS

I've also proven repeatedly that hard setting the initial state to "MASTER"
is problematic for an number of reasons. you are better off setting all
instances to "BACKUP" and allowing the priorities to determine the master.
also if you set one to initial state of "MASTER" it invalidates the
nopreempt and preempt_delay options.

for more information on this please see
rfc5798
section 6
http://tools.ietf.org/html/rfc5798#section-6

here is an example of how I do it and it works very well both in production
and every lab bench test scenario I've put it through.

on the primary

"
vrrp_script ROUTERCHECK {

script "/usr/bin/fpingvrrpcheck.sh 192.169.100.4 192.169.101.4"
interval 5
weight -51
}

vrrp_instance MY_VRRP_INSTANCE {
state BACKUP
interface bond2
dont_track_primary

lvs_sync_daemon_interface bond2
virtual_router_id 51
priority 101
garp_master_delay 0
advert_int 1
authentication {
auth_type PASS
auth_pass 6cN1qQ~bi
}

virtual_ipaddress {
192.169.100.1/29 dev bond0.100 label bond0.100:1
192.169.101.1/29 dev bond1.101 label bond1.101:1

192.169.102.1/27 dev bond1.102 label bond1.102:1
}
track_interface {
bond0
bond1
}
track_script {
ROUTERCHECK
}

nopreempt
notify_master "/usr/bin/keepalived-conntrack-manager.sh primary"
notify_backup "/usr/bin/keepalived-conntrack-manager.sh backup"
notify_fault "/usr/bin/keepalived-conntrack-manager.sh fault"

smtp_alert # Send email notification during state transit
}

"
On the backup I change only one option.
"
priority 100
"

Note I don't like preempt but if you want that behavior you should set a
preempt_delay or you can risk flapping conditions creating chaos

note the /usr/bin/keepalived-conntrack-manager.sh is not the standard one

Also bond2 is a directly connected pair of crossover cables with no switch
between which is why i set "dont_track_primary" because it will go down if
the other box is powered down and in those cases it does not constitute a
fault.

http://www.keepalived.org/LVS-NAT-Keepalived-HOWTO.html
Allows the same prio on different vrrp addresses, so I don't definitely
understand what you meant.
Thanks

Thank you for the reply but I don't understand "same".
I have 150 prio for Master and 100 for backup.
Is a problem that every different Master has the same 150 prio?
My cisco routers all have the same 100 prio for each different master
VIP.
You say I should have 150 for first master and 100 for its backup, 151
for second master and so on?
Thanks

Same priorities is your problem its a violation of the protocol spec for
VRRP
-- Sent from my HP Pre3
------------------------------
Glad to hear it not dead, I just started using it! Just noting that a
showstopper bug is reported 7 years ago and today it is here preventing the
usage of keepalived in a production environment. If it is due to a miscfg,
I'm very happy, here is one from "machine A".
Machine B has the two vrrp with MASTER/BACKUP exchanged and the same for
priorities.
To be precise this file is generated from a script, so BNAME1 and 2 are
examples of a generic MASTER and generic BACKUP.
The checking scripts just check the existence of the socket. Replacing
them with "exit 0" or removing them all have no impact on keepalived demon
going unstable.
Bye
# autocfg
# Created on Tue, 04 Jun 2013 15:02:57 +0200
# My role is: MASTER
global_defs {
notification_email {
}
smtp_server 172.30.3.75
smtp_connect_timeout 5
}
# Description: Desc bil
vrrp_script chk_BNAME1 {
script "/lb/chkha.sh /lb/pids/ha-c1.sock"
interval 5
weight 75
}
vrrp_instance BNAME1 {
interface eth0
state MASTER
virtual_router_id 133
priority 150
vrrp_unicast_bind 172.30.111.10
vrrp_unicast_peer 172.30.111.11
smtp_alert
virtual_ipaddress {
172.30.111.12/16 dev eth0
}
authentication {
auth_type PASS
auth_pass supersecret
}
track_script {
chk_BNAME1
}
}
# Description: Desc bil 2
vrrp_script chk_BNAME2 {
script "/lb/chkha.sh /lb/pids/ha-c2.sock"
interval 5
weight 75
}
vrrp_instance BNAME2 {
interface eth0
state BACKUP
virtual_router_id 134
priority 100
vrrp_unicast_bind 172.30.111.10
vrrp_unicast_peer 172.30.111.11
smtp_alert
virtual_ipaddress {
172.30.111.13/16 dev eth0
}
authentication {
auth_type PASS
auth_pass supersecret
}
track_script {
chk_BNAME2
}
}

1)
Keepalived is not an abandon project not by a long shot. There were
several versions released last year and many people still write patches for
it its just a very slow release cycle in comparison to some other projects.
2) without seeing your config we can't help you. It sounds like you
have seriously misconfigured some thing. Most likely the priorities but it
could be other thing in any case it sounds like they are getting into a
perpetual election cycle they can't resolve.
-- Sent from my HP Pre3
------------------------------
"This" is really several couples of servers I'm working with.
They range from machines online for days to machines online from
seconds, since I rebooted them several times to study the 100% behavior.
In the lists the problem seems known from ages, but never solved. Just
google for keepalived 100% cpu.
http://marc.info/?l=keepalived-devel&m=114040573028656
The link to the "remediation" is flawed, I contacted the author (very
kind in responding) but with no solution.
It is a problem about the Master and backup hosts flooding each other
with packets after a switchover.
The biggest problem I see is that keepalived seems an abandoned
project...

Post by Rimbalza Rimbalza
I read about the problem in several posts but have not found no

resolutive

Post by Rimbalza Rimbalza
answer.
The problem is 100% CPU usage, both by irq and keepalived process.

Rimbalza Rimbalza

2013-06-04 17:06:25 UTC

Thanks a lot!
Tomorrow I'm going to re-create my lab setup with the new configurations.
To make it short:
- priority "distance" limited to 1 or so (only have a master/backup)
- BACKUP on start
- preempt_delay: i suspect this have something to do with my crazy
ping-pong behavior. I was missing this.

I'm only trying to figure out the correct setting for weight in the check
script. You used a negative one, I was using positive.
Reading the doc you pointed out leaves me with doubts. In a two host
scenario, with hosts exposing VIPs on the same interface as the host IP
(i.e. eth0), should negative or positive weights be used?

Hope this will solve my troubles.
Thanks again.

Post by Paul Robert Marino
forgot to do a reply all
---------- Forwarded message ----------
Date: Tue, Jun 4, 2013 at 12:51 PM
Subject: Re: [Keepalived-devel] 100% CPU usage
no this is not what I'm saying
for example if your master has a priority of 150 the backup in normal
operation should have a priority of 149 if you have multiple backups they
should decrement by 1 for each additional backup.
Note: a difference of 50 in priorities between 2 instances constitutes a
fault regardless of what the advertised state is.
also that howto is ancient, out of date and should not be used any more. I
wish they would take it off the site or update but they never
have.following that doc and any other howto based on it based on it causes
problems.
The real up to date doc is here
https://github.com/acassen/keepalived/blob/master/doc/keepalived.conf.SYNOPSIS
I've also proven repeatedly that hard setting the initial state to
"MASTER" is problematic for an number of reasons. you are better off
setting all instances to "BACKUP" and allowing the priorities to determine
the master.
also if you set one to initial state of "MASTER" it invalidates the
nopreempt and preempt_delay options.
for more information on this please see
rfc5798
section 6
http://tools.ietf.org/html/rfc5798#section-6
here is an example of how I do it and it works very well both in
production and every lab bench test scenario I've put it through.
on the primary
"
vrrp_script ROUTERCHECK {
script "/usr/bin/fpingvrrpcheck.sh 192.169.100.4 192.169.101.4"
interval 5
weight -51
}
vrrp_instance MY_VRRP_INSTANCE {
state BACKUP
interface bond2
dont_track_primary
lvs_sync_daemon_interface bond2
virtual_router_id 51
priority 101
garp_master_delay 0
advert_int 1
authentication {
auth_type PASS
auth_pass 6cN1qQ~bi
}
virtual_ipaddress {
192.169.100.1/29 dev bond0.100 label bond0.100:1
192.169.101.1/29 dev bond1.101 label bond1.101:1
192.169.102.1/27 dev bond1.102 label bond1.102:1
}
track_interface {
bond0
bond1
}
track_script {
ROUTERCHECK
}
nopreempt
notify_master "/usr/bin/keepalived-conntrack-manager.sh primary"
notify_backup "/usr/bin/keepalived-conntrack-manager.sh backup"
notify_fault "/usr/bin/keepalived-conntrack-manager.sh fault"
smtp_alert # Send email notification during state transit
}
"
On the backup I change only one option.
"
priority 100
"
Note I don't like preempt but if you want that behavior you should set a
preempt_delay or you can risk flapping conditions creating chaos
note the /usr/bin/keepalived-conntrack-manager.sh is not the standard one
Also bond2 is a directly connected pair of crossover cables with no switch
between which is why i set "dont_track_primary" because it will go down if
the other box is powered down and in those cases it does not constitute a
fault.

http://www.keepalived.org/LVS-NAT-Keepalived-HOWTO.html
Allows the same prio on different vrrp addresses, so I don't definitely
understand what you meant.
Thanks

Same priorities is your problem its a violation of the protocol spec
for VRRP
-- Sent from my HP Pre3
------------------------------
Glad to hear it not dead, I just started using it! Just noting that a
showstopper bug is reported 7 years ago and today it is here preventing the
usage of keepalived in a production environment. If it is due to a miscfg,
I'm very happy, here is one from "machine A".
Machine B has the two vrrp with MASTER/BACKUP exchanged and the same
for priorities.
To be precise this file is generated from a script, so BNAME1 and 2
are examples of a generic MASTER and generic BACKUP.
The checking scripts just check the existence of the socket. Replacing
them with "exit 0" or removing them all have no impact on keepalived demon
going unstable.
Bye
# autocfg
# Created on Tue, 04 Jun 2013 15:02:57 +0200
# My role is: MASTER
global_defs {
notification_email {
}
smtp_server 172.30.3.75
smtp_connect_timeout 5
}
# Description: Desc bil
vrrp_script chk_BNAME1 {
script "/lb/chkha.sh /lb/pids/ha-c1.sock"
interval 5
weight 75
}
vrrp_instance BNAME1 {
interface eth0
state MASTER
virtual_router_id 133
priority 150
vrrp_unicast_bind 172.30.111.10
vrrp_unicast_peer 172.30.111.11
smtp_alert
virtual_ipaddress {
172.30.111.12/16 dev eth0
}
authentication {
auth_type PASS
auth_pass supersecret
}
track_script {
chk_BNAME1
}
}
# Description: Desc bil 2
vrrp_script chk_BNAME2 {
script "/lb/chkha.sh /lb/pids/ha-c2.sock"
interval 5
weight 75
}
vrrp_instance BNAME2 {
interface eth0
state BACKUP
virtual_router_id 134
priority 100
vrrp_unicast_bind 172.30.111.10
vrrp_unicast_peer 172.30.111.11
smtp_alert
virtual_ipaddress {
172.30.111.13/16 dev eth0
}
authentication {
auth_type PASS
auth_pass supersecret
}
track_script {
chk_BNAME2
}
}

1)
Keepalived is not an abandon project not by a long shot. There were
several versions released last year and many people still write patches for
it its just a very slow release cycle in comparison to some other projects.
2) without seeing your config we can't help you. It sounds like you
have seriously misconfigured some thing. Most likely the priorities but it
could be other thing in any case it sounds like they are getting into a
perpetual election cycle they can't resolve.
-- Sent from my HP Pre3
------------------------------
"This" is really several couples of servers I'm working with.
They range from machines online for days to machines online from
seconds, since I rebooted them several times to study the 100% behavior.
In the lists the problem seems known from ages, but never solved. Just
google for keepalived 100% cpu.
http://marc.info/?l=keepalived-devel&m=114040573028656
The link to the "remediation" is flawed, I contacted the author (very
kind in responding) but with no solution.
It is a problem about the Master and backup hosts flooding each other
with packets after a switchover.
The biggest problem I see is that keepalived seems an abandoned
project...

Post by Rimbalza Rimbalza
I read about the problem in several posts but have not found no

resolutive

Post by Rimbalza Rimbalza
answer.
The problem is 100% CPU usage, both by irq and keepalived process.

------------------------------------------------------------------------------
1. A cloud service to automate IT design, transition and operations
2. Dashboards that offer high-level views of enterprise services
3. A single system of record for all IT processes
http://p.sf.net/sfu/servicenow-d2d-j
_______________________________________________
Keepalived-devel mailing list
https://lists.sourceforge.net/lists/listinfo/keepalived-devel

Paul Robert Marino

2013-06-04 17:26:22 UTC

Todd Fleisher

2013-06-04 17:57:32 UTC

Paul,
Can you elaborate on this point? I have multiple environments where I set 1 node to MASTER on start to avoid forcing an election during a configuration reload, but I have never experienced this pingpong behavior as a result.

-T

The pingpong behavior is actually a result of hard setting master on start that's why I say that old how-to is a problem

Paul Robert Marino

2013-06-04 18:56:16 UTC

Todd

First of all having an election during a config reload isn't really a
serious problem just use a SIG HUP instead of restarting the process. It
should reload the config but preserve the current state although I may be
wrong. That said elections are so quick the only thing that would cause is
a slight bit of latency during a configuration reload.

By the way recent versions of keepalived send out a gratuitous priority 0
advertisement on shutdown which tells the other node to take over and
ignore the hold down timer so it still transitions but it happens so
quickly you don't notice it. Doing that as you said more likely than not
causes packet drops and or errors to occur, and will certainly damage your
connection tracking table if you are using the old conntrackd
primary-backup.sh script which many people still are because its included
as an example along with a broken keepalived configuration in the
conntrack-tools tar file. That old how-to and the examples for conntrack
tools date from around the same time, and both programs and the Linux
kernel have come a long way since.
The priority 0 advertisement behavior is per rfc5798 section 6.1 sub
heading Priority.
http://tools.ietf.org/html/rfc5798#section-6.1

That said the problem appears mostly when you have multiple VRRP instances
in a sync group.
here is what happens
node A = primary
node B = backup

1) VRRP instance 1 faults on node A

2) VRRP instance 1 on node B becomes MASTER

2) VRRP instance 2 switches to MASTER on node B because instance 1 and 2
are in a VRRP sync group

3) VRRP instance 2 hits its next check cycle on node A checks out fine.

4) Since node A it is set to MASTER the nopreempt and preempt_delay options
are ignored, and VRRP instance 2 immediately transitions to MASTER state on
node A because it should be MASTER.

4) Due to the fact that they are in a sync group VRRP instance 1 on node A
becomes MASTER.

5) Repeat from item 1 in the list when VRRP instance 1 does its next check.

This is a race condition that happens with vrrp sync groups all the time
and its essentially a race condition caused by setting the initial state of
MASTER.

Post by Todd Fleisher
Paul,
Can you elaborate on this point? I have multiple environments where I set
1 node to MASTER on start to avoid forcing an election during a
configuration reload, but I have never experienced this pingpong behavior
as a result.
-T
The pingpong behavior is actually a result of hard setting master on start
that's why I say that old how-to is a problem

Rimbalza Rimbalza

2013-06-05 09:45:00 UTC

Hi Paul,
same big problem after implementing changes. Prio are 110 (master) and 105
(backup).
Put int preempt_delay. All members BACKUP on start.
The attached cfg is 2 hosts each one with an active and a backup VIPs.
Starting manually often results in no problem (not always).
Upon rebooting one of the two servers, keepalived goes 100% on each
machines. After some time (minutes) at 100%, VIPs stop responding.
Reloading the cfg of a machine makes the CPU go normal, but only on that
machine. Sad again...
Here is my cfg:

global_defs {
notification_email {
***@email
}
notification_email_from ***@email
smtp_server 172.30.3.75
smtp_connect_timeout 5
}

# Description: IP DUMMY DMZ
vrrp_script chk_DMZ-DUMMY {
script "/lb/chkha.sh /lb/pids/ha-test1.sock"
interval 5
weight 75
}

vrrp_instance DMZ-DUMMY {
interface eth0
state BACKUP
#notify_backup /lb/notify_backup.sh
virtual_router_id 90
priority 110
#vrrp_unicast_bind 172.30.3.25
#vrrp_unicast_peer 172.30.3.26
smtp_alert
preempt_delay 2
virtual_ipaddress {
172.30.3.27/16 dev eth0
}
authentication {
auth_type PASS
auth_pass dont-tell
}
track_script {
chk_DMZ-DUMMY
}
}

# Description: IP DUMMY28 DMZ
vrrp_script chk_DMZ-DUMMY28 {
script "/lb/chkha.sh /lb/pids/ha-test1.sock"
interval 5
weight 75
}

vrrp_instance DMZ-DUMMY28 {
interface eth0
state BACKUP
#notify_backup /lb/notify_backup.sh
virtual_router_id 91
priority 105
#vrrp_unicast_bind 172.30.3.25
#vrrp_unicast_peer 172.30.3.26
smtp_alert
preempt_delay 2
virtual_ipaddress {
172.30.3.28/16 dev eth0
}
authentication {
auth_type PASS
auth_pass dont-tell
}
track_script {
chk_DMZ-DUMMY28
}
}

# Description: SMTP
vrrp_script chk_SMTPOUT {
script "/lb/chkha.sh /lb/pids/ha-smtpout.sock"
interval 5
weight 75
}

vrrp_instance SMTPOUT {
interface eth0
state BACKUP
#notify_backup /lb/notify_backup.sh
virtual_router_id 92
priority 110
#vrrp_unicast_bind 172.30.3.25
#vrrp_unicast_peer 172.30.3.26
smtp_alert
preempt_delay 2
virtual_ipaddress {
172.30.3.75/16 dev eth0
}
authentication {
auth_type PASS
auth_pass dont-tell
}
track_script {
chk_SMTPOUT
}
}

# Description: Corp site
vrrp_script chk_CORP {
script "/lb/chkha.sh /lb/pids/ha-CORP.sock"
interval 5
weight 75
}

vrrp_instance CORP {
interface eth0
state BACKUP
#notify_backup /lb/notify_backup.sh
virtual_router_id 93
priority 105
#vrrp_unicast_bind 172.30.3.25
#vrrp_unicast_peer 172.30.3.26
smtp_alert
preempt_delay 2
virtual_ipaddress {
172.30.3.91/16 dev eth0
}
authentication {
auth_type PASS
auth_pass dont-tell
}
track_script {
chk_CORP
}
}

The pingpong behavior is actually a result of hard setting master on start
that's why I say that old how-to is a problem. But the preempt delay solves
an other set of conditions that can cause the same problem again though it
is ignored if you hard set a node to master which confuses people.
Also note if you have more than one backup node you want to modify the
weight on the check script accordingly.
-- Sent from my HP Pre3
------------------------------
Thanks a lot!
Tomorrow I'm going to re-create my lab setup with the new configurations.
- priority "distance" limited to 1 or so (only have a master/backup)
- BACKUP on start
- preempt_delay: i suspect this have something to do with my crazy
ping-pong behavior. I was missing this.
I'm only trying to figure out the correct setting for weight in the check
script. You used a negative one, I was using positive.
Reading the doc you pointed out leaves me with doubts. In a two host
scenario, with hosts exposing VIPs on the same interface as the host IP
(i.e. eth0), should negative or positive weights be used?
Hope this will solve my troubles.
Thanks again.

Post by Paul Robert Marino
forgot to do a reply all
---------- Forwarded message ----------
Date: Tue, Jun 4, 2013 at 12:51 PM
Subject: Re: [Keepalived-devel] 100% CPU usage
no this is not what I'm saying
for example if your master has a priority of 150 the backup in normal
operation should have a priority of 149 if you have multiple backups they
should decrement by 1 for each additional backup.
Note: a difference of 50 in priorities between 2 instances constitutes a
fault regardless of what the advertised state is.
also that howto is ancient, out of date and should not be used any more.
I wish they would take it off the site or update but they never
have.following that doc and any other howto based on it based on it causes
problems.
The real up to date doc is here
https://github.com/acassen/keepalived/blob/master/doc/keepalived.conf.SYNOPSIS
I've also proven repeatedly that hard setting the initial state to
"MASTER" is problematic for an number of reasons. you are better off
setting all instances to "BACKUP" and allowing the priorities to determine
the master.
also if you set one to initial state of "MASTER" it invalidates the
nopreempt and preempt_delay options.
for more information on this please see
rfc5798
section 6
http://tools.ietf.org/html/rfc5798#section-6
here is an example of how I do it and it works very well both in
production and every lab bench test scenario I've put it through.
on the primary
"
vrrp_script ROUTERCHECK {
script "/usr/bin/fpingvrrpcheck.sh 192.169.100.4 192.169.101.4"
interval 5
weight -51
}
vrrp_instance MY_VRRP_INSTANCE {
state BACKUP
interface bond2
dont_track_primary
lvs_sync_daemon_interface bond2
virtual_router_id 51
priority 101
garp_master_delay 0
advert_int 1
authentication {
auth_type PASS
auth_pass 6cN1qQ~bi
}
virtual_ipaddress {
192.169.100.1/29 dev bond0.100 label bond0.100:1
192.169.101.1/29 dev bond1.101 label bond1.101:1
192.169.102.1/27 dev bond1.102 label bond1.102:1
}
track_interface {
bond0
bond1
}
track_script {
ROUTERCHECK
}
nopreempt
notify_master "/usr/bin/keepalived-conntrack-manager.sh primary"
notify_backup "/usr/bin/keepalived-conntrack-manager.sh backup"
notify_fault "/usr/bin/keepalived-conntrack-manager.sh fault"
smtp_alert # Send email notification during state transit
}
"
On the backup I change only one option.
"
priority 100
"
Note I don't like preempt but if you want that behavior you should set a
preempt_delay or you can risk flapping conditions creating chaos
note the /usr/bin/keepalived-conntrack-manager.sh is not the standard one
Also bond2 is a directly connected pair of crossover cables with no
switch between which is why i set "dont_track_primary" because it will go
down if the other box is powered down and in those cases it does not
constitute a fault.

http://www.keepalived.org/LVS-NAT-Keepalived-HOWTO.html
Allows the same prio on different vrrp addresses, so I don't definitely
understand what you meant.
Thanks

Same priorities is your problem its a violation of the protocol spec
for VRRP
-- Sent from my HP Pre3
------------------------------
Glad to hear it not dead, I just started using it! Just noting that a
showstopper bug is reported 7 years ago and today it is here preventing the
usage of keepalived in a production environment. If it is due to a miscfg,
I'm very happy, here is one from "machine A".
Machine B has the two vrrp with MASTER/BACKUP exchanged and the same
for priorities.
To be precise this file is generated from a script, so BNAME1 and 2
are examples of a generic MASTER and generic BACKUP.
The checking scripts just check the existence of the socket. Replacing
them with "exit 0" or removing them all have no impact on keepalived demon
going unstable.
Bye
# autocfg
# Created on Tue, 04 Jun 2013 15:02:57 +0200
# My role is: MASTER
global_defs {
notification_email {
}
smtp_server 172.30.3.75
smtp_connect_timeout 5
}
# Description: Desc bil
vrrp_script chk_BNAME1 {
script "/lb/chkha.sh /lb/pids/ha-c1.sock"
interval 5
weight 75
}
vrrp_instance BNAME1 {
interface eth0
state MASTER
virtual_router_id 133
priority 150
vrrp_unicast_bind 172.30.111.10
vrrp_unicast_peer 172.30.111.11
smtp_alert
virtual_ipaddress {
172.30.111.12/16 dev eth0
}
authentication {
auth_type PASS
auth_pass supersecret
}
track_script {
chk_BNAME1
}
}
# Description: Desc bil 2
vrrp_script chk_BNAME2 {
script "/lb/chkha.sh /lb/pids/ha-c2.sock"
interval 5
weight 75
}
vrrp_instance BNAME2 {
interface eth0
state BACKUP
virtual_router_id 134
priority 100
vrrp_unicast_bind 172.30.111.10
vrrp_unicast_peer 172.30.111.11
smtp_alert
virtual_ipaddress {
172.30.111.13/16 dev eth0
}
authentication {
auth_type PASS
auth_pass supersecret
}
track_script {
chk_BNAME2
}
}
On Tue, Jun 4, 2013 at 2:59 PM, Paul Robert Marino <

1)
Keepalived is not an abandon project not by a long shot. There were
several versions released last year and many people still write patches for
it its just a very slow release cycle in comparison to some other projects.
2) without seeing your config we can't help you. It sounds like you
have seriously misconfigured some thing. Most likely the priorities but it
could be other thing in any case it sounds like they are getting into a
perpetual election cycle they can't resolve.
-- Sent from my HP Pre3
------------------------------
"This" is really several couples of servers I'm working with.
They range from machines online for days to machines online from
seconds, since I rebooted them several times to study the 100% behavior.
In the lists the problem seems known from ages, but never solved.
Just google for keepalived 100% cpu.
http://marc.info/?l=keepalived-devel&m=114040573028656
The link to the "remediation" is flawed, I contacted the author (very
kind in responding) but with no solution.
It is a problem about the Master and backup hosts flooding each other
with packets after a switchover.
The biggest problem I see is that keepalived seems an abandoned
project...

Post by Rimbalza Rimbalza
I read about the problem in several posts but have not found no

resolutive

Post by Rimbalza Rimbalza
answer.
The problem is 100% CPU usage, both by irq and keepalived process.

Rimbalza Rimbalza

2013-06-05 11:15:48 UTC

Another test was starting the daemon with --vrrp.
No change. 100% CPU.

Rimbalza Rimbalza

2013-06-05 11:23:07 UTC

I suspect all my messages were kept due to excessive size. Reposting the
relevant changes also tested:

The same with negative weight -51.
The same with nopreempt option or delay set to 7.
Now I'm noting that is the same also if I put the VIPs not in a "crossed"
fashion, but all on the same host as master.
strace shows a loop of read-select, dunno if the correct ones

Post by Rimbalza Rimbalza
Another test was starting the daemon with --vrrp.
No change. 100% CPU.

Rimbalza Rimbalza

2013-06-05 12:51:10 UTC

Another test with no success. Moved one of the 2 VMs in another physically
different VMWare cluster (L2 connected of course), same behavior.

Post by Rimbalza Rimbalza
I suspect all my messages were kept due to excessive size. Reposting the
The same with negative weight -51.
The same with nopreempt option or delay set to 7.
Now I'm noting that is the same also if I put the VIPs not in a "crossed"
fashion, but all on the same host as master.
strace shows a loop of read-select, dunno if the correct ones

Post by Rimbalza Rimbalza
Another test was starting the daemon with --vrrp.
No change. 100% CPU.

Paul Slootman

2013-06-05 13:20:55 UTC

Post by Rimbalza Rimbalza
Another test with no success. Moved one of the 2 VMs in another physically
different VMWare cluster (L2 connected of course), same behavior.

That's a good point; what I once saw was that sometimes the VRRP packets
issued from inside a VM could be seen twice by tcpdump: once with the
MAC address bound to the VM's (virtual) interface, and again with the
MAC address bound to the host's interface. So now I have an iptables
rule:

iptables -I INPUT -s 192.168.0.57 -m mac --mac-source ! 00:21:f6:00:00:38 -p vrrp -j DROP

192.168.0.57 is the VM's IP address, 00:21:f6:00:00:38 is the VM's MAC
address. That combination is the only one I want to receive...

In a month's time:

Chain INPUT (policy ACCEPT 129M packets, 59G bytes)
pkts bytes target prot opt in out source destination
73999 2960K DROP 112 -- * * 192.168.0.57 0.0.0.0/0 MAC ! 00:21:F6:00:00:38

Without the rule it confused the incoming VRRP packets as from
another candidate and gave up the IP address, and then took it back,
rinse, repeat.

Paul

Rimbalza Rimbalza

2013-06-05 13:43:46 UTC

Mumbling... Did someone tried to use Willy (or others') patch for unicast
made for 1.1.9 in current 1.2.7?
In a two host environment this can overcome the problem.

Post by Rimbalza Rimbalza
Another test with no success. Moved one of the 2 VMs in another

physically

Post by Rimbalza Rimbalza
different VMWare cluster (L2 connected of course), same behavior.

[...]

Thomas Heil

2013-06-05 13:58:46 UTC

Hi,

Post by Rimbalza Rimbalza
Mumbling... Did someone tried to use Willy (or others') patch for
unicast made for 1.1.9 in current 1.2.7?
In a two host environment this can overcome the problem.

You can use the patch from Ryan O'Hara on top of 1.2.7. (patch is
attached), which is very good.
Then in every vrrp instance use something like that
unicast {
<ip>
}

The <ip> should be one that belongs to the interface you are using for
that instance.

Post by Rimbalza Rimbalza
Another test with no success. Moved one of the 2 VMs in another

physically

Post by Rimbalza Rimbalza
different VMWare cluster (L2 connected of course), same behavior.

That's a good point; what I once saw was that sometimes the VRRP packets
issued from inside a VM could be seen twice by tcpdump: once with the
MAC address bound to the VM's (virtual) interface, and again with the
MAC address bound to the host's interface. So now I have an iptables
[...]
------------------------------------------------------------------------------
1. A cloud service to automate IT design, transition and operations
2. Dashboards that offer high-level views of enterprise services
3. A single system of record for all IT processes
http://p.sf.net/sfu/servicenow-d2d-j
_______________________________________________
Keepalived-devel mailing list
https://lists.sourceforge.net/lists/listinfo/keepalived-devel

cheers
thomas

Ryan O'Hara

2013-06-05 15:28:14 UTC

Post by Thomas Heil
Hi,

Post by Rimbalza Rimbalza
Mumbling... Did someone tried to use Willy (or others') patch for
unicast made for 1.1.9 in current 1.2.7?
In a two host environment this can overcome the problem.

You can use the patch from Ryan O'Hara on top of 1.2.7. (patch is
attached), which is very good.
Then in every vrrp instance use something like that
unicast {
<ip>
}
The <ip> should be one that belongs to the interface you are using for
that instance.

One of the things I added to Willy's original patch when I ported it
to 1.2.7 is the ability to have multiple peers. I believe I tested
with as many as 6 nodes. Another difference is that I did not
forward port the 'unicast_bind' option that appears in Willy's
patch. It wasn't being used and I'm not entirely sure it is
needed.

If you do used my patch, I'd love some feedback. I've not yet issued a
pull request for this patch.

Ryan

Post by Thomas Heil

Post by Rimbalza Rimbalza
Another test with no success. Moved one of the 2 VMs in another

physically

Post by Rimbalza Rimbalza
different VMWare cluster (L2 connected of course), same behavior.

That's a good point; what I once saw was that sometimes the VRRP packets
issued from inside a VM could be seen twice by tcpdump: once with the
MAC address bound to the VM's (virtual) interface, and again with the
MAC address bound to the host's interface. So now I have an iptables
[...]
------------------------------------------------------------------------------
1. A cloud service to automate IT design, transition and operations
2. Dashboards that offer high-level views of enterprise services
3. A single system of record for all IT processes
http://p.sf.net/sfu/servicenow-d2d-j
_______________________________________________
Keepalived-devel mailing list
https://lists.sourceforge.net/lists/listinfo/keepalived-devel

cheers
thomas

Paul Robert Marino

2013-06-05 12:58:54 UTC

Rimbalza Rimbalza

2013-06-05 13:10:36 UTC

Just tested with advert_int 1 and 5.
100 CPU% immediately after one machine goes for reboot.

I Just took a quick glance and something struck me as odd in your config.
None of the instances have an advert_int set that'd supposed to be a
required field
If some how it defaulted to 0 that would cause it to eat a CPU core.
Try setting "advert_int 1" or higher
Also if your if one node has a priority of 105 and the other 110 you're
weight on the scripts should be -55
If you use nopreempt.
-- Sent from my HP Pre3
------------------------------
I suspect all my messages were kept due to excessive size. Reposting the
The same with negative weight -51.
The same with nopreempt option or delay set to 7.
Now I'm noting that is the same also if I put the VIPs not in a "crossed"
fashion, but all on the same host as master.
strace shows a loop of read-select, dunno if the correct ones

Post by Rimbalza Rimbalza
Another test was starting the daemon with --vrrp.
No change. 100% CPU.

unknown

1970-01-01 00:00:00 UTC

--047d7bacb8ea6276f104de6aa9af
Content-Type: text/plain; charset=ISO-8859-1

The stock binary behaves the same.
And also just returns to normality reloading the configuration.
1.2.2 stock is the same as 1.2.7 from official sources.
:( :(

what about the stock version that comes with Ubuntu I know its a little
older version in percise 1.2.2 but if its know to work then that can tell
you if its a binary or configuration problem.

Post by Rimbalza Rimbalza
Yes, I'm using ONLY that version from that very same file. No patches yet.

I'm curious have you tried the stock version with no patches to see if
you get the same problem?
http://www.keepalived.org/software/keepalived-1.2.7.tar.gz

--047d7bacb8ea6276f104de6aa9af
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable <div dir="ltr">The stock binary behaves the same.<br><div style>And also just returns to normality reloading the configuration.</div><div style>1.2.2 stock is the same as 1.2.7 from official sources.</div><div style>:( :(</div> </div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Jun 5, 2013 at 6:14 PM, Paul Robert Marino <span dir="ltr"><<a href="mailto:***@gmail.com" target="_blank">***@gmail.com</a>></span> wrote:<br> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">what about the stock version that comes with Ubuntu I know its a little older version in percise 1.2.2 but if its know to work then that can tell you if its a binary or configuration problem.<br> <br><br></div><div class="HOEnZb"><div class="h5"> <div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Jun 5, 2013 at 12:08 PM, Rimbalza Rimbalza <span dir="ltr"><<a href="mailto:***@gmail.com" target="_blank">***@gmail.com</a>></span> wrote:<br> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Yes, I'm using ONLY that version from that very same file. No patches yet.</div><div> <div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Jun 5, 2013 at 6:07 PM, Paul Robert Marino <span dir="ltr"><<a href="mailto:***@gmail.com" target="_blank">***@gmail.com</a>></span> wrote:<br> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">I'm curious have you tried the stock version with no patches to see if you get the same problem?<br> <br><a href="http://www.keepalived.org/software/keepalived-1.2.7.tar.gz" target="_blank">http://www.keepalived.org/software/keepalived-1.2.7.tar.gz</a><br> </div><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Jun 5, 2013 at 12:00 PM, Rimbalza Rimbalza <span dir="ltr"><<a href="mailto:***@gmail.com" target="_blank">***@gmail.com</a>></span> wrote:<br> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">It's in one of my first posts.<div>Compiled 1.2.7 from source on Ubuntu 12.04.2 x64 (Linux xyz 3.5.0-32-generic #53~precise1-Ubuntu SMP Wed May 29 20:33:37 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux)</div> <div><br></div><div><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote"><div>On Wed, Jun 5, 2013 at 5:58 PM, Paul Robert Marino <span dir="ltr"><<a href="mailto:***@gmail.com" target="_blank">***@gmail.com</a>></span> wrote:<br> </div><div><div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><br></div>I'm curious what version are you using and on what distribution and version of Linux<br> </div>additionally is this a binary you compiled your self or one provided by the distribution.<br> </div><div class="gmail_extra"><br><br><div class="gmail_quote"><div>On Wed, Jun 5, 2013 at 11:49 AM, Rimbalza Rimbalza <span dir="ltr"><<a href="mailto:***@gmail.com" target="_blank">***@gmail.com</a>></span> wrote:<br>

</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div dir="ltr">Did not try the unicast patches but I moved the same binaries and configuration on two physical servers, and the 100% CPU problem is here again...<div>

So I'll exclude some virtualization stack fault in managing L2 communications.</div>
<div><div><br></div></div></div>
<br></div><div>------------------------------------------------------------------------------<br>
How ServiceNow helps IT people transform IT departments:<br>
1. A cloud service to automate IT design, transition and operations<br>
2. Dashboards that offer high-level views of enterprise services<br>
3. A single system of record for all IT processes<br>
<a href="http://p.sf.net/sfu/servicenow-d2d-j" target="_blank">http://p.sf.net/sfu/servicenow-d2d-j</a><br>_______________________________________________<br>
Keepalived-devel mailing list<br>
<a href="mailto:Keepalived-***@lists.sourceforge.net" target="_blank">Keepalived-***@lists.sourceforge.net</a><br>
<a href="https://lists.sourceforge.net/lists/listinfo/keepalived-devel" target="_blank">https://lists.sourceforge.net/lists/listinfo/keepalived-devel</a><br>
<br></div></blockquote></div><br></div>
</blockquote></div></div></div><br></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>

--047d7bacb8ea6276f104de6aa9af--

Paul Robert Marino

2013-06-05 16:46:42 UTC

Rimbalza Rimbalza

2013-06-05 16:50:52 UTC

I started from scratch dozens (literally) of times, also used the cfg
sample you sent some mails ago.
The thing I learned is that is seems to get problematic adding the 3rd vrrp
VIP.
My chk scripts are the first thing I removed when testing, being the only
non-keepalived thing into the equation.
Just now I'm experimenting the SELinux thing...

Start from scratch on your config and start simple and add to it as you go.
There is something wrong some where in the configuration.
It could be any thing even your check scripts could be causing it who
knows.
But I can tell you I've seen far more complex config then yours with a lot
more in them that didn't have your issue.
The only thing like it I've seen was when I did that hack to implement
subsecond intervals and I tested at 0.01 second intervals.
-- Sent from my HP Pre3
------------------------------
The stock binary behaves the same.
And also just returns to normality reloading the configuration.
1.2.2 stock is the same as 1.2.7 from official sources.
:( :(

what about the stock version that comes with Ubuntu I know its a little
older version in percise 1.2.2 but if its know to work then that can tell
you if its a binary or configuration problem.

Post by Rimbalza Rimbalza
Yes, I'm using ONLY that version from that very same file. No patches
yet.

I'm curious have you tried the stock version with no patches to see if
you get the same problem?
http://www.keepalived.org/software/keepalived-1.2.7.tar.gz

Rimbalza Rimbalza

2013-06-05 17:07:55 UTC

Also the stock binary has the problem and a x86 Ubunt 12.04.2 (not amd64) I
tested last week.
I MUST find a solution to this problem...
Tomorrow I will try to install a couple of Debian servers and start all
over again.
This problem was found in the lists 7 years ago, so I doubt it is a problem
with the current release of Ubuntu.

I have boxes in production which have as many as 6 vrrp instances on RHEL
so I doubt its that. this makes me wonder if there is a bug in one of the
dependencies in your version of Ubuntu

Post by Rimbalza Rimbalza
I started from scratch dozens (literally) of times, also used the cfg
sample you sent some mails ago.
The thing I learned is that is seems to get problematic adding the 3rd
vrrp VIP.
My chk scripts are the first thing I removed when testing, being the only
non-keepalived thing into the equation.
Just now I'm experimenting the SELinux thing...

Start from scratch on your config and start simple and add to it as you go.
There is something wrong some where in the configuration.
It could be any thing even your check scripts could be causing it who
knows.
But I can tell you I've seen far more complex config then yours with a
lot more in them that didn't have your issue.
The only thing like it I've seen was when I did that hack to implement
subsecond intervals and I tested at 0.01 second intervals.
-- Sent from my HP Pre3
------------------------------
The stock binary behaves the same.
And also just returns to normality reloading the configuration.
1.2.2 stock is the same as 1.2.7 from official sources.
:( :(

what about the stock version that comes with Ubuntu I know its a little
older version in percise 1.2.2 but if its know to work then that can tell
you if its a binary or configuration problem.

Post by Rimbalza Rimbalza
Yes, I'm using ONLY that version from that very same file. No patches
yet.
On Wed, Jun 5, 2013 at 6:07 PM, Paul Robert Marino <

I'm curious have you tried the stock version with no patches to see
if you get the same problem?
http://www.keepalived.org/software/keepalived-1.2.7.tar.gz
On Wed, Jun 5, 2013 at 12:00 PM, Rimbalza Rimbalza <

Rimbalza Rimbalza

2013-06-06 06:43:55 UTC

Rimbalza Rimbalza

2013-06-06 07:17:00 UTC

Another report. Traffic just reached 2 mbit and all is stuck again.
I suspect the traffic is not related.

Post by unknown
Early morning report.
I rebuilt from scratch forcing -O0 on gcc (gcc version 4.6.3
(Ubuntu/Linaro 4.6.3-1ubuntu5)).
Same servers (hw), it seems the problem is not showing.
The only problem with the current setup is that there is a little load on
the servers. Also yesterday I noted (to verify today) that removing a
specific VIP positively impacts keepalived. This VIP is bound to a site
generating a steady load of 4mbit. I know it is not much (previously
running FreeBSD 8 on the same hw I balanced >80Mbit with less than 30% cpu
load on a single core).
Now I'm waiting for some load to build up.
In the meantime I'm building a couple of CentOS, that should be similar yo
your RHEL.
I'll be back as soon as I tested the new findings.
Bye

try it on fedora or any other distro. Debian and Ubuntu are mostly the
same thing except for the desktop enhancements and the proprietary software
added.
also if you're paying Connonical for support you might want to get them
involved.

Post by Rimbalza Rimbalza
Also the stock binary has the problem and a x86 Ubunt 12.04.2 (not
amd64) I tested last week.
I MUST find a solution to this problem...
Tomorrow I will try to install a couple of Debian servers and start all
over again.
This problem was found in the lists 7 years ago, so I doubt it is a
problem with the current release of Ubuntu.

I have boxes in production which have as many as 6 vrrp instances on
RHEL so I doubt its that. this makes me wonder if there is a bug in one of
the dependencies in your version of Ubuntu

Post by Rimbalza Rimbalza
I started from scratch dozens (literally) of times, also used the cfg
sample you sent some mails ago.
The thing I learned is that is seems to get problematic adding the 3rd
vrrp VIP.
My chk scripts are the first thing I removed when testing, being the
only non-keepalived thing into the equation.
Just now I'm experimenting the SELinux thing...
On Wed, Jun 5, 2013 at 6:46 PM, Paul Robert Marino <

Start from scratch on your config and start simple and add to it as you go.
There is something wrong some where in the configuration.
It could be any thing even your check scripts could be causing it who
knows.
But I can tell you I've seen far more complex config then yours with
a lot more in them that didn't have your issue.
The only thing like it I've seen was when I did that hack to
implement subsecond intervals and I tested at 0.01 second intervals.
-- Sent from my HP Pre3
------------------------------
The stock binary behaves the same.
And also just returns to normality reloading the configuration.
1.2.2 stock is the same as 1.2.7 from official sources.
:( :(
On Wed, Jun 5, 2013 at 6:14 PM, Paul Robert Marino <

what about the stock version that comes with Ubuntu I know its a
little older version in percise 1.2.2 but if its know to work then that can
tell you if its a binary or configuration problem.
On Wed, Jun 5, 2013 at 12:08 PM, Rimbalza Rimbalza <

Post by Rimbalza Rimbalza
Yes, I'm using ONLY that version from that very same file. No
patches yet.
On Wed, Jun 5, 2013 at 6:07 PM, Paul Robert Marino <

I'm curious have you tried the stock version with no patches to
see if you get the same problem?
http://www.keepalived.org/software/keepalived-1.2.7.tar.gz
On Wed, Jun 5, 2013 at 12:00 PM, Rimbalza Rimbalza <

Post by Paul Robert Marino
I'm curious what version are you using and on what distribution
and version of Linux
additionally is this a binary you compiled your self or one
provided by the distribution.
On Wed, Jun 5, 2013 at 11:49 AM, Rimbalza Rimbalza <

Post by Rimbalza Rimbalza
Did not try the unicast patches but I moved the same binaries
and configuration on two physical servers, and the 100% CPU problem is here
again...
So I'll exclude some virtualization stack fault in managing L2
communications.
------------------------------------------------------------------------------
1. A cloud service to automate IT design, transition and operations
2. Dashboards that offer high-level views of enterprise services
3. A single system of record for all IT processes
http://p.sf.net/sfu/servicenow-d2d-j
_______________________________________________
Keepalived-devel mailing list
https://lists.sourceforge.net/lists/listinfo/keepalived-devel

Rimbalza Rimbalza

2013-06-06 07:56:37 UTC

Here is what I found deeper digging.
Using a single VIP definition in keepalived.conf, there are no problems.
Adding a second one generates the problem.
Findings: when the process is stuck 100% CPU, on the wire advertisements
are still regular, a couple every second (coherent with the cfg file), one
for each VRRP address. The only thing to note, for what its worth, is
Wireshark saying
Priority: 101 (Non-default backup priority)
The packets contain the correct VIP, checksun is ok and there's the auth
string.

Attached is my current cfg. If you remove one of the vrrp_instance
definitions, it works ok, so the single cfg are ok by themselves.
The only difference for backup host is prio 100 instead of 101.

global_defs {
notification_email {
***@email
}
notification_email_from ***@email
smtp_server 172.30.3.75
smtp_connect_timeout 5
}

# Description: SMTP
vrrp_script chk_SMTPOUT {
script "/lb/chkha.sh /lb/pids/ha-smtpout.sock"
interval 5
weight -51
}

vrrp_instance SMTPOUT {
interface eth0
state BACKUP
virtual_router_id 90
garp_master_delay 0
priority 101
smtp_alert
advert_int 1
preempt_delay 5
virtual_ipaddress {
172.30.3.75/16 dev eth0
}
authentication {
auth_type PASS
auth_pass xxx
}
track_script {
chk_SMTPOUT
}
track_interface {
eth0
}
}

# Description: corp
vrrp_script chk_CORP {
script "/lb/chkha.sh /lb/pids/ha-CORP.sock"
interval 5
weight -51
}

vrrp_instance CORP {
interface eth0
state BACKUP
virtual_router_id 91
garp_master_delay 0
priority 101
smtp_alert
advert_int 1
preempt_delay 5
virtual_ipaddress {
172.30.3.91/16 dev eth0
}
authentication {
auth_type PASS
auth_pass xxx
}
track_script {
chk_CORP
}
track_interface {
eth0
}
}

Rimbalza Rimbalza

2013-06-06 08:42:24 UTC

Here again.
I refactored the configuration to use only a vrrp router with 2 addresses.
This is very sub-optimal, but just to try...
Same result. I can sniff a single advertisement with the 2 IPs, once a
second as configured, but rebooting one server makes the other 100% CPU. On
restart the two have 100% usage. During "madness" moments, vrrps adverts on
the wire are regular at 1 sec.
I don't know what to try now...

Post by Rimbalza Rimbalza
Here is what I found deeper digging.
Using a single VIP definition in keepalived.conf, there are no problems.
Adding a second one generates the problem.
Findings: when the process is stuck 100% CPU, on the wire advertisements
are still regular, a couple every second (coherent with the cfg file), one
for each VRRP address. The only thing to note, for what its worth, is
Wireshark saying
Priority: 101 (Non-default backup priority)
The packets contain the correct VIP, checksun is ok and there's the auth
string.
Attached is my current cfg. If you remove one of the vrrp_instance
definitions, it works ok, so the single cfg are ok by themselves.
The only difference for backup host is prio 100 instead of 101.
global_defs {
notification_email {
}
smtp_server 172.30.3.75
smtp_connect_timeout 5
}
# Description: SMTP
vrrp_script chk_SMTPOUT {
script "/lb/chkha.sh /lb/pids/ha-smtpout.sock"
interval 5
weight -51
}
vrrp_instance SMTPOUT {
interface eth0
state BACKUP
virtual_router_id 90
garp_master_delay 0
priority 101
smtp_alert
advert_int 1
preempt_delay 5
virtual_ipaddress {
172.30.3.75/16 dev eth0
}
authentication {
auth_type PASS
auth_pass xxx
}
track_script {
chk_SMTPOUT
}
track_interface {
eth0
}
}
# Description: corp
vrrp_script chk_CORP {
script "/lb/chkha.sh /lb/pids/ha-CORP.sock"
interval 5
weight -51
}
vrrp_instance CORP {
interface eth0
state BACKUP
virtual_router_id 91
garp_master_delay 0
priority 101
smtp_alert
advert_int 1
preempt_delay 5
virtual_ipaddress {
172.30.3.91/16 dev eth0
}
authentication {
auth_type PASS
auth_pass xxx
}
track_script {
chk_CORP
}
track_interface {
eth0
}
}

Rimbalza Rimbalza

2013-06-06 09:46:29 UTC

Ok, I think I exhausted all of the possible tests now.
Installed 2 CentOS systems (Linux centos1 2.6.32-358.6.2.el6.i686 #1 SMP
Thu May 16 18:12:13 UTC 2013 i686 i686 i386 GNU/Linux). Used gcc version
4.4.7 20120313 (Red Hat 4.4.7-3) (GCC). Using keepalived from packages, v.
1.2.7.
Moved the cfg files I posted above to the new machines. Different OS,
kernel, libs and compiler.
Very same problem. Again.
At this stage I can say for sure there is a problem in keepalived. If the
problem lays in the configuration, please please please someone give me a
hint? It is a plain stupid one, works for single vrrp, but makes keepalived
explode when using two VIPs, being them in a single vrrp instance or two
separated ones.
Don't really know what to do now...

Post by Rimbalza Rimbalza
Here again.
I refactored the configuration to use only a vrrp router with 2 addresses.
This is very sub-optimal, but just to try...
Same result. I can sniff a single advertisement with the 2 IPs, once a
second as configured, but rebooting one server makes the other 100% CPU. On
restart the two have 100% usage. During "madness" moments, vrrps adverts on
the wire are regular at 1 sec.
I don't know what to try now...

Graeme Fowler

2013-06-06 10:23:56 UTC

Post by Rimbalza Rimbalza
Don't really know what to do now...

Humour me.

Try it with no authentication.

Graeme

Rimbalza Rimbalza

2013-06-06 10:43:29 UTC

I'm lacking humor, but tried to remove auth of course.
Same 100%. Perf top is
Samples: 64K of event 'cycles', Event count (approx.): 29111165416
11.04% keepalived [.] 0x0000000000003900
7.27% [kernel] [k] __ticket_spin_lock
6.69% [kernel] [k] copy_user_generic_string
5.42% [vdso] [.] 0x00007fffb77ff9d6
4.41% [kernel] [k] __pollwait
4.31% [kernel] [k] do_select
3.94% [kernel] [k] _raw_spin_unlock_irqrestore
3.63% [kernel] [k] system_call
3.28% [kernel] [k] _raw_spin_lock_irqsave
2.83% [kernel] [k] fput
2.67% [kernel] [k] memset
2.38% [kernel] [k] native_read_tsc
2.31% [kernel] [k] fget_light
1.93% [kernel] [k] datagram_poll
1.71% [kernel] [k] core_sys_select
1.66% [kernel] [k] add_wait_queue
1.63% [kernel] [k] remove_wait_queue
1.55% libc-2.15.so [.] __select
1.53% libc-2.15.so [.] read

Post by Graeme Fowler

Post by Rimbalza Rimbalza
Don't really know what to do now...

Humour me.
Try it with no authentication.
Graeme
------------------------------------------------------------------------------
1. A cloud service to automate IT design, transition and operations
2. Dashboards that offer high-level views of enterprise services
3. A single system of record for all IT processes
http://p.sf.net/sfu/servicenow-d2d-j
_______________________________________________
Keepalived-devel mailing list
https://lists.sourceforge.net/lists/listinfo/keepalived-devel

Graeme Fowler

2013-06-06 11:00:20 UTC

Post by Rimbalza Rimbalza
I'm lacking humor, but tried to remove auth of course.
Same 100%.

Hrm. Was worth a punt.

I don't know what the output from perf means, either - I'm an strace man myself.

On the system which isn't currently running keepailved, run:

strace -o /tmp/keepalived.strace -s2048 -fFvtt <keepalived command line>

where the command line isn't the init script, but what the init script ends up running for you.

Force it to go to 100%. Leave it for 30 seconds or so, then CTRL-C. This will generate a *lot* of data in the output file in /tmp but you should be able to run a careful eye over it and see where the process starts to spin, and what it's spinning on.

I do have an idle suggestion, though… is there any chance you're running some sort of network bridge here (I've not read back through the thread) which means you're seeing packets looping? If there's any way you could create a network storm which ends up amplifying or looping the announcements then that could easily explain the behaviour.

Graeme

Rimbalza Rimbalza

2013-06-06 11:12:12 UTC

Hope it is the same, I run
strace -o /tmp/keepalived.strace -s2048 -fFvtt -p 1530
since keepalived was already running and at 100%

The console shows
Process 1530 attached - interrupt to quit
Process 2010 attached
Process 2011 attached
Process 2012 attached
Process 2013 attached (waiting for parent)
Process 2011 suspended
Process 2013 resumed (parent 2010 ready)
Process 2010 suspended
Process 2014 attached
Process 2015 attached
Process 2016 attached (waiting for parent)
Process 2016 resumed (parent 2014 ready)
Process 2017 attached (waiting for parent)
Process 2017 resumed (parent 2015 ready)
Process 2016 detached
Process 2017 detached
Process 2019 attached
Process 2020 attached (waiting for parent)
Process 2020 resumed (parent 2015 ready)
Process 2020 detached
Process 2015 detached
Process 2011 resumed
...

The log file contains these 3 repeating lines forever.
1530 13:08:46.986966 read(14, "", 512) = 0
1530 13:08:46.986993 read(13, "", 512) = 0
1530 13:08:46.987021 select(1024, [4 6 11 13 14], [], [], {0, 62960}) = 2
(in [13 14], left {0, 62958})
1530 13:08:46.987090 read(14, "", 512) = 0
1530 13:08:46.987117 read(13, "", 512) = 0
1530 13:08:46.987145 select(1024, [4 6 11 13 14], [], [], {0, 62836}) = 2
(in [13 14], left {0, 62834})

Post by Graeme Fowler

Post by Rimbalza Rimbalza
I'm lacking humor, but tried to remove auth of course.
Same 100%.

Hrm. Was worth a punt.
I don't know what the output from perf means, either - I'm an strace man myself.
strace -o /tmp/keepalived.strace -s2048 -fFvtt <keepalived command line>
where the command line isn't the init script, but what the init script
ends up running for you.
Force it to go to 100%. Leave it for 30 seconds or so, then CTRL-C. This
will generate a *lot* of data in the output file in /tmp but you should be
able to run a careful eye over it and see where the process starts to spin,
and what it's spinning on.
I do have an idle suggestion, though is there any chance you're running
some sort of network bridge here (I've not read back through the thread)
which means you're seeing packets looping? If there's any way you could
create a network storm which ends up amplifying or looping the
announcements then that could easily explain the behaviour.
Graeme
------------------------------------------------------------------------------
1. A cloud service to automate IT design, transition and operations
2. Dashboards that offer high-level views of enterprise services
3. A single system of record for all IT processes
http://p.sf.net/sfu/servicenow-d2d-j
_______________________________________________
Keepalived-devel mailing list
https://lists.sourceforge.net/lists/listinfo/keepalived-devel

Graeme Fowler

2013-06-06 11:14:58 UTC

Post by Rimbalza Rimbalza
Hope it is the same, I run
strace -o /tmp/keepalived.strace -s2048 -fFvtt -p 1530
since keepalived was already running and at 100%

Ah.

Post by Rimbalza Rimbalza
The log file contains these 3 repeating lines forever.
1530 13:08:46.986966 read(14, "", 512) = 0
1530 13:08:46.986993 read(13, "", 512) = 0
1530 13:08:46.987021 select(1024, [4 6 11 13 14], [], [], {0, 62960}) = 2 (in [13 14], left {0, 62958})
1530 13:08:46.987090 read(14, "", 512) = 0
1530 13:08:46.987117 read(13, "", 512) = 0
1530 13:08:46.987145 select(1024, [4 6 11 13 14], [], [], {0, 62836}) = 2 (in [13 14], left {0, 62834})

The reason I suggested running it from startup is that you need to know what those two file descriptors 13 and 14 are.

Graeme

Rimbalza Rimbalza

2013-06-06 11:45:55 UTC

Ok, I'm bzipping my 300Mb file obtained with your metod. If I upload it
somewhere, could someone look into it?

Post by Rimbalza Rimbalza
Hope it is the same, I run
strace -o /tmp/keepalived.strace -s2048 -fFvtt -p 1530
since keepalived was already running and at 100%

Ah.

2 (in [13 14], left {0, 62958})

Post by Rimbalza Rimbalza
1530 13:08:46.987090 read(14, "", 512) = 0
1530 13:08:46.987117 read(13, "", 512) = 0
1530 13:08:46.987145 select(1024, [4 6 11 13 14], [], [], {0, 62836}) =

2 (in [13 14], left {0, 62834})
The reason I suggested running it from startup is that you need to know
what those two file descriptors 13 and 14 are.
Graeme
------------------------------------------------------------------------------
1. A cloud service to automate IT design, transition and operations
2. Dashboards that offer high-level views of enterprise services
3. A single system of record for all IT processes
http://p.sf.net/sfu/servicenow-d2d-j
_______________________________________________
Keepalived-devel mailing list
https://lists.sourceforge.net/lists/listinfo/keepalived-devel

Rimbalza Rimbalza

2013-06-06 11:58:25 UTC

Ok, it is a 20Mb file now.
http://www.morosa.it/keepalived.strace.bz2

I suppose it is the way strace works, but I didn't see 100% on keepalived,
but 80% strace, 20% keepalived.

Post by Rimbalza Rimbalza
Ok, I'm bzipping my 300Mb file obtained with your metod. If I upload it
somewhere, could someone look into it?

Post by Rimbalza Rimbalza
Hope it is the same, I run
strace -o /tmp/keepalived.strace -s2048 -fFvtt -p 1530
since keepalived was already running and at 100%

Ah.

= 2 (in [13 14], left {0, 62958})

Post by Rimbalza Rimbalza
1530 13:08:46.987090 read(14, "", 512) = 0
1530 13:08:46.987117 read(13, "", 512) = 0
1530 13:08:46.987145 select(1024, [4 6 11 13 14], [], [], {0, 62836})

= 2 (in [13 14], left {0, 62834})
The reason I suggested running it from startup is that you need to know
what those two file descriptors 13 and 14 are.
Graeme
------------------------------------------------------------------------------
1. A cloud service to automate IT design, transition and operations
2. Dashboards that offer high-level views of enterprise services
3. A single system of record for all IT processes
http://p.sf.net/sfu/servicenow-d2d-j
_______________________________________________
Keepalived-devel mailing list
https://lists.sourceforge.net/lists/listinfo/keepalived-devel

Rimbalza Rimbalza

2013-06-06 12:03:31 UTC

Not a big expert, but something weird seems happening at
07:18:59.506186
and again after 07:19:04.508077. It seems a kill/respawn/fail loop. Or
maybe I need to study some more :)

Post by Rimbalza Rimbalza
Ok, it is a 20Mb file now.
http://www.morosa.it/keepalived.strace.bz2
I suppose it is the way strace works, but I didn't see 100% on keepalived,
but 80% strace, 20% keepalived.

Post by Rimbalza Rimbalza
Ok, I'm bzipping my 300Mb file obtained with your metod. If I upload it
somewhere, could someone look into it?

Post by Rimbalza Rimbalza
Hope it is the same, I run
strace -o /tmp/keepalived.strace -s2048 -fFvtt -p 1530
since keepalived was already running and at 100%

Ah.

= 2 (in [13 14], left {0, 62958})

Post by Rimbalza Rimbalza
1530 13:08:46.987090 read(14, "", 512) = 0
1530 13:08:46.987117 read(13, "", 512) = 0
1530 13:08:46.987145 select(1024, [4 6 11 13 14], [], [], {0, 62836})

Graeme Fowler

2013-06-06 13:55:25 UTC

Good news, and bad news…

It seems that your mail server is the culprit. I don't fully understand why - a couple of messages get out, then a bunch of connectons get stuck:

kevin:development graeme$ egrep "htons$25$|sendto.+(MAIL FROM|RCPT TO|HELO|EHLO|DATA|QUIT|Subject|owning)" keepalived.strace

7364 07:18:59.504875 connect(13, {sa_family=AF_INET, sin_port=htons(25), sin_addr=inet_addr("172.30.3.75")}, 128) = -1 EINPROGRESS (Operation now in progress)
7364 07:18:59.505146 connect(14, {sa_family=AF_INET, sin_port=htons(25), sin_addr=inet_addr("172.30.3.75")}, 128) = -1 EINPROGRESS (Operation now in progress)
7364 07:18:59.552312 sendto(13, "HELO tmpha2\r\n", 13, 0, NULL, 0 <unfinished ...>
7364 07:18:59.555856 sendto(14, "HELO tmpha2\r\n", 13, 0, NULL, 0 <unfinished ...>
7364 07:18:59.556904 sendto(13, "MAIL FROM:<***@vittoriaassicurazioni.it>\r\n", 48, 0, NULL, 0 <unfinished ...>
7364 07:18:59.558642 sendto(14, "MAIL FROM:<***@vittoriaassicurazioni.it>\r\n", 48, 0, NULL, 0 <unfinished ...>
7364 07:18:59.558954 sendto(13, "RCPT TO:<***@vittoriaassicurazioni.it>\r\n", 43, 0, NULL, 0 <unfinished ...>
7364 07:18:59.560898 sendto(14, "RCPT TO:<***@vittoriaassicurazioni.it>\r\n", 43, 0, NULL, 0 <unfinished ...>
7364 07:18:59.561709 sendto(13, "DATA\r\n", 6, 0, NULL, 0 <unfinished ...>
7364 07:18:59.563130 sendto(14, "DATA\r\n", 6, 0, NULL, 0 <unfinished ...>
7364 07:18:59.570996 sendto(13, "Date: Thu, 06 Jun 2013 11:18:59 +0000\r\nFrom: ***@vittoriaassicurazioni.it\r\nSubject: [tmpha2] VRRP Instance SMTPOUT - Entering BACKUP state\r\nX-Mailer: Keepalived\r\n\r\n", 170, 0, NULL, 0 <unfinished ...>
7364 07:18:59.571184 sendto(13, "=> VRRP Instance is nolonger owning VRRP VIPs <=\r\n", 50, 0, NULL, 0) = 50
7364 07:18:59.571505 sendto(14, "Date: Thu, 06 Jun 2013 11:18:59 +0000\r\nFrom: ***@vittoriaassicurazioni.it\r\nSubject: [tmpha2] VRRP Instance ISTITUZIONALE - Entering BACKUP state\r\nX-Mailer: Keepalived\r\n\r\n", 176, 0, NULL, 0 <unfinished ...>
7364 07:18:59.571571 sendto(14, "=> VRRP Instance is nolonger owning VRRP VIPs <=\r\n", 50, 0, NULL, 0 <unfinished ...>
7364 07:18:59.611025 sendto(13, "QUIT\r\n", 6, 0, NULL, 0) = 6
7364 07:18:59.611450 sendto(14, "QUIT\r\n", 6, 0, NULL, 0) = 6

OK, so all good that far, one message per instance gets sent. But then this happens:

7364 07:19:48.767849 connect(13, {sa_family=AF_INET, sin_port=htons(25), sin_addr=inet_addr("172.30.3.75")}, 128) = -1 EINPROGRESS (Operation now in progress)
7364 07:19:48.769976 connect(14, {sa_family=AF_INET, sin_port=htons(25), sin_addr=inet_addr("172.30.3.75")}, 128) = -1 EINPROGRESS (Operation now in progress)
7364 07:19:53.770282 connect(15, {sa_family=AF_INET, sin_port=htons(25), sin_addr=inet_addr("172.30.3.75")}, 128) = -1 EINPROGRESS (Operation now in progress)
7364 07:19:53.772589 connect(16, {sa_family=AF_INET, sin_port=htons(25), sin_addr=inet_addr("172.30.3.75")}, 128) = -1 EINPROGRESS (Operation now in progress)
7364 07:20:43.469544 connect(17, {sa_family=AF_INET, sin_port=htons(25), sin_addr=inet_addr("172.30.3.75")}, 128) = -1 EINPROGRESS (Operation now in progress)
7364 07:20:43.472064 connect(18, {sa_family=AF_INET, sin_port=htons(25), sin_addr=inet_addr("172.30.3.75")}, 128) = -1 EINPROGRESS (Operation now in progress)
7364 07:21:23.485573 connect(19, {sa_family=AF_INET, sin_port=htons(25), sin_addr=inet_addr("172.30.3.75")}, 128) = -1 EINPROGRESS (Operation now in progress)
7364 07:21:23.488321 connect(20, {sa_family=AF_INET, sin_port=htons(25), sin_addr=inet_addr("172.30.3.75")}, 128) = -1 EINPROGRESS (Operation now in progress)

…and this is where you see the endless CPU utilisation. The process spins round and round trying to read from the open SMTP connections.

Then we see another process send another two messages successfully - this is a new child process as the previus one (7364) terminated.

8026 07:22:22.326167 connect(13, {sa_family=AF_INET, sin_port=htons(25), sin_addr=inet_addr("172.30.3.75")}, 128) = -1 EINPROGRESS (Operation now in progress)
8026 07:22:22.326424 connect(14, {sa_family=AF_INET, sin_port=htons(25), sin_addr=inet_addr("172.30.3.75")}, 128) = -1 EINPROGRESS (Operation now in progress)
8026 07:22:22.374481 sendto(13, "HELO tmpha2\r\n", 13, 0, NULL, 0 <unfinished ...>
8026 07:22:22.376086 sendto(14, "HELO tmpha2\r\n", 13, 0, NULL, 0 <unfinished ...>
8026 07:22:22.377182 sendto(13, "MAIL FROM:<***@vittoriaassicurazioni.it>\r\n", 48, 0, NULL, 0 <unfinished ...>
8026 07:22:22.377498 sendto(14, "MAIL FROM:<***@vittoriaassicurazioni.it>\r\n", 48, 0, NULL, 0 <unfinished ...>
8026 07:22:22.378816 sendto(13, "RCPT TO:<***@vittoriaassicurazioni.it>\r\n", 43, 0, NULL, 0 <unfinished ...>
8026 07:22:22.379362 sendto(14, "RCPT TO:<***@vittoriaassicurazioni.it>\r\n", 43, 0, NULL, 0 <unfinished ...>
8026 07:22:22.380825 sendto(13, "DATA\r\n", 6, 0, NULL, 0 <unfinished ...>
8026 07:22:22.381210 sendto(14, "DATA\r\n", 6, 0, NULL, 0 <unfinished ...>
8026 07:22:22.383241 sendto(13, "Date: Thu, 06 Jun 2013 11:22:22 +0000\r\nFrom: ***@vittoriaassicurazioni.it\r\nSubject: [tmpha2] VRRP Instance SMTPOUT - Entering BACKUP state\r\nX-Mailer: Keepalived\r\n\r\n", 170, 0, NULL, 0 <unfinished ...>
8026 07:22:22.383413 sendto(13, "=> VRRP Instance is nolonger owning VRRP VIPs <=\r\n", 50, 0, NULL, 0 <unfinished ...>
8026 07:22:22.384515 sendto(14, "Date: Thu, 06 Jun 2013 11:22:22 +0000\r\nFrom: ***@vittoriaassicurazioni.it\r\nSubject: [tmpha2] VRRP Instance ISTITUZIONALE - Entering BACKUP state\r\nX-Mailer: Keepalived\r\n\r\n", 176, 0, NULL, 0 <unfinished ...>
8026 07:22:22.385274 sendto(14, "=> VRRP Instance is nolonger owning VRRP VIPs <=\r\n", 50, 0, NULL, 0 <unfinished ...>
8026 07:22:22.422562 sendto(13, "QUIT\r\n", 6, 0, NULL, 0) = 6
8026 07:22:22.430301 sendto(14, "QUIT\r\n", 6, 0, NULL, 0) = 6

There's an awful lot of stuff in your system logs from this instance which might be helpful. But the first thing is for you to work out why the (remote?) SMTP server doesn't respond in a nice timely fashion.

It's quite possible that you're being caught out here by the system's rlimits too… could be that there are too many open files.

Graeme

Rimbalza Rimbalza

2013-06-06 14:17:38 UTC

Thanks for pointing that out. The SMTP server is reached via one of the
VIPs. So it is normal, in my thoughts, that it is not reachable while
transitioning. That's why I raised to 10 seconds the timeout for SMTP,
following the logic that the first/second attempt to send will be useless,
but the following will succeed. In fact from an "external" point of view,
the SMTP service on the VIP is interrupted only for a little lapse.
It seems that there is no wait/check/resend cycle on keepalived, and it
keeps hammering the CPU, without noticing the VIP and its associated SMTP
port aree available.
The reason I have this config is that my mail relay listens on an alternate
(not 25) port. I digged into the keepalived docs and easily found the way
to configure the server, but how can I use a non standard port on it?
This will make me do some other tests since I have (by now) other 4 couples
of server (4 hw and 4 vm) that have the very same behavior and no "local
SMTP VIP".
Thanks again.

Paul Slootman

2013-06-06 14:49:21 UTC

Post by Rimbalza Rimbalza
The reason I have this config is that my mail relay listens on an alternate
(not 25) port. I digged into the keepalived docs and easily found the way
to configure the server, but how can I use a non standard port on it?

Often the way is to specify the server as ipadress:port

Otherwise it might be a good idea to configure a local MTA that handles
the remote delivery.

Paul

Rimbalza Rimbalza

2013-06-06 14:55:46 UTC

Yes, there are several alternative solutions, but I wanted to avoid loading
potentially critical sw such an MTA.
Using the :port notation was my first test, and resulted in (from logs)
SMTP connection ERROR to [::]:25.
In the docs there is clearly only the IP address to put here.
In any case this is a workable problem. The CPU hogging isn't...

Post by Rimbalza Rimbalza
The reason I have this config is that my mail relay listens on an

alternate

Post by Rimbalza Rimbalza
(not 25) port. I digged into the keepalived docs and easily found the way
to configure the server, but how can I use a non standard port on it?

Often the way is to specify the server as ipadress:port
Otherwise it might be a good idea to configure a local MTA that handles
the remote delivery.
Paul
------------------------------------------------------------------------------
1. A cloud service to automate IT design, transition and operations
2. Dashboards that offer high-level views of enterprise services
3. A single system of record for all IT processes
http://p.sf.net/sfu/servicenow-d2d-j
_______________________________________________
Keepalived-devel mailing list
https://lists.sourceforge.net/lists/listinfo/keepalived-devel

Paul Slootman

2013-06-06 17:15:33 UTC

Post by Rimbalza Rimbalza
Yes, there are several alternative solutions, but I wanted to avoid loading
potentially critical sw such an MTA.
Using the :port notation was my first test, and resulted in (from logs)
SMTP connection ERROR to [::]:25.

Hmm :(

Post by Rimbalza Rimbalza
In the docs there is clearly only the IP address to put here.
In any case this is a workable problem. The CPU hogging isn't...

So have you tried it without configuring any email stuff?

Rimbalza Rimbalza

2013-06-07 07:23:24 UTC

So have you tried it without configuring any email stuff?

John Sullivan

2013-06-06 20:11:15 UTC

Not convinced the keepalived smtp engine is right here.

Check out keepalived/core/smtp.c, smtp_read_thread().

It does the read, checks for -1 - mostly error conditions except
for EAGAIN which, on a non-blocking socket, simply means no data
is available *yet* but the socket is still good.

Otherwise it assumes it got some data, appends it to the buffer,
parses for the status code etc., requeues the select for read.

At no point does it check for a zero return from read(), which
means the socket has been closed by the other end (or otherwise
failed). On a closed socket (where all input has already been
drained), read() will *always* return immediately with 0, and
select() will always indicate readability, hence the spin.

John

--
Dead stars still burn

Pasi Kärkkäinen

2013-06-10 08:59:51 UTC

Post by John Sullivan
Not convinced the keepalived smtp engine is right here.

Check out keepalived/core/smtp.c, smtp_read_thread().
It does the read, checks for -1 - mostly error conditions except
for EAGAIN which, on a non-blocking socket, simply means no data
is available *yet* but the socket is still good.
Otherwise it assumes it got some data, appends it to the buffer,
parses for the status code etc., requeues the select for read.
At no point does it check for a zero return from read(), which
means the socket has been closed by the other end (or otherwise
failed). On a closed socket (where all input has already been
drained), read() will *always* return immediately with 0, and
select() will always indicate readability, hence the spin.

Wanna send a patch? :)

-- Pasi

Ryan O'Hara

2013-06-14 16:48:19 UTC

Post by John Sullivan
Not convinced the keepalived smtp engine is right here.

Check out keepalived/core/smtp.c, smtp_read_thread().
It does the read, checks for -1 - mostly error conditions except
for EAGAIN which, on a non-blocking socket, simply means no data
is available *yet* but the socket is still good.
Otherwise it assumes it got some data, appends it to the buffer,
parses for the status code etc., requeues the select for read.
At no point does it check for a zero return from read(), which
means the socket has been closed by the other end (or otherwise
failed). On a closed socket (where all input has already been
drained), read() will *always* return immediately with 0, and
select() will always indicate readability, hence the spin.

Wanna send a patch? :)

I've attached a quick-and-dirty patch. Note that I have not completely
tested this patch because I am unable to easily recreate the
problem. Comments welcome.

Ryan

Rimbalza Rimbalza

2013-06-06 11:16:09 UTC

About the "bridging" issue. I have the two hw connected via a single
adapter (each one) to a common switch.
Also, if you read my previous comments, I sniffed the net and found that
even when 100% busy, vrrp packets are sent out 1 each second, as configured.

Post by Graeme Fowler

Post by Rimbalza Rimbalza
I'm lacking humor, but tried to remove auth of course.
Same 100%.

Bengt Gördén

2013-06-06 09:40:42 UTC

Post by Rimbalza Rimbalza
Here again.
I refactored the configuration to use only a vrrp router with 2
addresses. This is very sub-optimal, but just to try...
Same result. I can sniff a single advertisement with the 2 IPs, once a
second as configured, but rebooting one server makes the other 100%
CPU. On restart the two have 100% usage. During "madness" moments,
vrrps adverts on the wire are regular at 1 sec.
I don't know what to try now...

Rimbalza Rimbalza

2013-06-06 10:37:40 UTC

Not sure what these mean, but here you are:
stat

Performance counter stats for 'keepalived':

0.861522 task-clock # 0.760 CPUs utilized
0 context-switches # 0.000 K/sec
0 CPU-migrations # 0.000 K/sec
367 page-faults # 0.426 M/sec
2,608,186 cycles # 3.027 GHz
<not supported> stalled-cycles-frontend
<not supported> stalled-cycles-backend
1,493,716 instructions # 0.57 insns per cycle
289,235 branches # 335.726 M/sec
9,961 branch-misses # 3.44% of all branches
[ 2.78%]

0.001133157 seconds time elapsed

And Top
Samples: 24K of event 'cycles', Event count (approx.): 14855008749
10.94% keepalived [.] 0x000000000001cb16
7.04% [kernel] [k] copy_user_generic_string
6.81% [kernel] [k] __ticket_spin_lock
5.58% [vdso] [.] 0x00007fffb77ff60c
4.51% [kernel] [k] __pollwait
4.17% [kernel] [k] do_select
4.12% [kernel] [k] _raw_spin_unlock_irqrestore
3.86% [kernel] [k] system_call
3.31% [kernel] [k] _raw_spin_lock_irqsave
2.86% [kernel] [k] fput
2.57% [kernel] [k] memset
2.41% [kernel] [k] native_read_tsc
2.32% [kernel] [k] fget_light
2.06% [kernel] [k] datagram_poll
1.80% [kernel] [k] core_sys_select
1.71% [kernel] [k] add_wait_queue
1.67% libc-2.15.so [.] read
1.66% [kernel] [k] remove_wait_queue
1.63% [kernel] [k] ktime_get_ts
1.59% libc-2.15.so [.] __select
1.38% [kernel] [k] select_estimate_accuracy.part.5

And good old top

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
928 root 20 0 25564 1348 908 R 100 0.0 3:42.04 keepalived

Post by Rimbalza Rimbalza
Here again.
I refactored the configuration to use only a vrrp router with 2
addresses. This is very sub-optimal, but just to try...
Same result. I can sniff a single advertisement with the 2 IPs, once a
second as configured, but rebooting one server makes the other 100%
CPU. On restart the two have 100% usage. During "madness" moments,
vrrps adverts on the wire are regular at 1 sec.
I don't know what to try now...

There was a suggestion to use perf to investigate the problem and to see
where it spends its time. Did you do that?
If not.
sudo apt-get install linux-tools-common
sudo apt-get install linux-tools
perf stat keepalived
perf top -p `pidof -s keepalived`
If you've got more than one keepalived process checkout the others with
"pidof kreepalived".
perf top -p <pid>
https://perf.wiki.kernel.org/index.php/Tutorial
/bengan
------------------------------------------------------------------------------
1. A cloud service to automate IT design, transition and operations
2. Dashboards that offer high-level views of enterprise services
3. A single system of record for all IT processes
http://p.sf.net/sfu/servicenow-d2d-j
_______________________________________________
Keepalived-devel mailing list
https://lists.sourceforge.net/lists/listinfo/keepalived-devel

unknown

1970-01-01 00:00:00 UTC