"access-list logging rate-limited or missed <X> packets"

[John Caruso]

We're frequently seeing this message from two separate Internet-facing
Cisco routers which send their syslog output to a central logging server.
Both routers are running 12.3(1a). The routers both have plenty of CPU
and RAM available, have no "logging rate-limit" specified, and are
generating these messages even when the logging buffer is nearly empty.

The volume of these messages is running well above the volume of actual,
useful messages from these routers...as an example, out of 104388 syslog
messages one of the routers generated last week, 59794 of them were these
"%SEC-6-IPACCESSLOGRL: access-list logging rate-limited or missed <X>
packets" messages.

At Cisco's request we've tried upping the logging buffer size and setting
"logging rate-limit 10000" (even though the default is supposedly that
there's no limit); neither action helped.

Can anyone say what might be going on here? How do we get our routers to
stop dropping useful log information on the ground? I can't think of any
valid reason for a router with this much free CPU and RAM to refuse to
log so many messages.

- John

[Aaron Woody]

John,

It sounds like to me you are logging in two places. 1 - Syslog
server, 2 - buffer. Is it your intention to log in both locations? If
not, stop the logging to the buffer and just log to syslog server.

Aaron

Post by John Caruso
We're frequently seeing this message from two separate Internet-facing
Cisco routers which send their syslog output to a central logging server.
Both routers are running 12.3(1a). The routers both have plenty of CPU
and RAM available, have no "logging rate-limit" specified, and are
generating these messages even when the logging buffer is nearly empty.
The volume of these messages is running well above the volume of actual,
useful messages from these routers...as an example, out of 104388 syslog
messages one of the routers generated last week, 59794 of them were these
"%SEC-6-IPACCESSLOGRL: access-list logging rate-limited or missed <X>
packets" messages.
At Cisco's request we've tried upping the logging buffer size and setting
"logging rate-limit 10000" (even though the default is supposedly that
there's no limit); neither action helped.
Can anyone say what might be going on here? How do we get our routers to
stop dropping useful log information on the ground? I can't think of any
valid reason for a router with this much free CPU and RAM to refuse to
log so many messages.
- John

[John Caruso]

Thanks for the response (I'm surprised that I didn't hear more responses,
since I'm sure we're not the only site that's run into this issue).

Post by Aaron Woody
It sounds like to me you are logging in two places. 1 - Syslog
server, 2 - buffer. Is it your intention to log in both locations? If
not, stop the logging to the buffer and just log to syslog server.

I can do that (the buffer logging isn't really necessary, though it's nice
as a fallback), but it doesn't do anything about the actual problem. I'm
guessing that you were thinking the rate limit on buffer logging was causing
rate-limiting messages in syslog, but that's not the case.

The only workaround I've found so far is to use "ip access-list log-update
threshold 1", which forces a flush of the access-list logging buffers after
every single violation. However, Cisco strongly recommends against using
this, and I certainly don't want to use it since it greatly increases the
number of messages logged. In fact I'm not sure why this even does fix the
problem, since Cisco claims that routers will never log two messages within
1 second of each other (an assertion that's contradicted by our own syslog
logfiles, but still, that's what they say).

If anyone has any other suggestions they'd be appreciated.

- John

[Juraj Ljubesic]

On Mon, 27 Oct 2003 22:29:20 GMT, John Caruso
<***@myprivacy.ca> wrote:

.....

Post by John Caruso
The only workaround I've found so far is to use "ip access-list log-update
threshold 1", which forces a flush of the access-list logging buffers after
every single violation. However, Cisco strongly recommends against using
this, and I certainly don't want to use it since it greatly increases the
number of messages logged. In fact I'm not sure why this even does fix the
problem, since Cisco claims that routers will never log two messages within
1 second of each other (an assertion that's contradicted by our own syslog
logfiles, but still, that's what they say).

I have similar problem with 2691.
In this circumstances I really don't understand what is purpouse of
command "logging rate-limit X"
where is X "<1-10000> Messages per second".

Jura

Post by John Caruso
If anyone has any other suggestions they'd be appreciated.
- John

[John Caruso]

Post by Juraj Ljubesic

Post by John Caruso
The only workaround I've found so far is to use "ip access-list log-update
threshold 1", which forces a flush of the access-list logging buffers after
every single violation. However, Cisco strongly recommends against using
this, and I certainly don't want to use it since it greatly increases the
number of messages logged. In fact I'm not sure why this even does fix the
problem, since Cisco claims that routers will never log two messages within
1 second of each other (an assertion that's contradicted by our own syslog
logfiles, but still, that's what they say).

I have similar problem with 2691.
In this circumstances I really don't understand what is purpouse of
command "logging rate-limit X"
where is X "<1-10000> Messages per second".

My understanding thus far in going through this with Cisco support is that
"logging rate-lmiit X" is global to ALL logging messages, but the builtin
1 message per second rate limit is specific to access list logging (and
there's no limit by default on the number of logging messages other than
ACL logging messages). So you can set "logging rate-limit" all you want,
but it won't affect the operation of ACL logging.

That's why you have to use "ip access-list log-update threshold" instead
to affect ACL logging (there's also an undocumented "ip access-list log
interval" command that can be used to change the default 5-minute interval
for generation of duplicate log messages).

This is all rather cloudy. The Cisco support guy is still trying to get
a straight answer on all this from the developers--especially in light of
the fact that we've seen 1) multiple ACL logging messages within a second
of each other, and 2) "rate-limiting" logging messages more than a second
after a valid logging message. Neither of those should be possible if the
1-second limit is really in place, and if it's the source of our problems.

- John

[Juraj Ljubesic]

Thanks a lot!

Now I lose about 1% of logg records instead of 50-70%.

Thanks again.

Jura