snmpd error on subcontainer ‘ia_addr’ insert (-1)

Found that the log files on two separate Centos 6.3 boxes where filling up with the error

error on subcontainer 'ia_addr' insert (-1)

This was being written with each connect. It seems that this is a harmless debugging message and the fix appears to be to change the OPTIONS setting in “/etc/sysconfig/snmpd” from

-LSd

Which means log to syslog under the daemon facility, to

-LS6d

Which means log only LOG_INFO level messages and above to the syslog using the “daemon” facility.

This is annoying since I was using -Lf, but adding the “6” there for the log level causes net-snmpd to create the file as a socket instead of a regular file. It seems that there is no method to filter the debugging notices when logging to a file.

aide.conf syntax errors contain junk line information

The latest servers that we added to run our e-Classifieds (r) platform use CentOS 6. I definitely like the boxes but Prelink and AIDE have been a pain.

After trying to stop AIDE checking some folders I started getting this report:

271:syntax error: <junk>
271:Error while reading configuration: <junk>

It turns out that this really was a syntax error. It really was on line 271 because rather daftly I had been thinking in terms of RegExps and added the line as

!^/usr/local/....

The ^ generates the syntax error. But apparently AIDE has a free/malloc/pointer bug in this error message as it prints random junk after the error message. Initally made me think that the error that was being reported was that the aide.conf file contained those characters and I couldn’t see then and therefore oooh filesystem/disk corruption… I was relieved to find that it was a bug in AIDE’s error message.

Why does IPTables not log when it is started/stopped?

I have to say that it is very strange that iptables doesn’t log when it is started, stopped, restarted or even when a rule is added. Given how big of a part in the security scene iptables plays, you’d think that by default it would send a notice to syslog, probably at daemon.notice level, when it is stopped and started. After all this is one fairly important tool. Not the only (hopefully) but usually a pretty important one and in many cases it really is the only thing between a server and the bad guys (yes, I know, that it shouldn’t be the only thing..).

I mention this because I just came back from lunch to find my SSH connection wouldn’t respond to input. This has happened occasionally, but as far as I’ve been able to tell (by piecing together information afterwards), it has invariably been because iptables was restarted.

Now me, I just add the new rule to the firewall and if it works and I want to keep it, add it to the config file separately. I don’t believe in restarting iptables because if you f*ck up the new rule and restart the firewall you’re not only screwed then, but if you reboot the server too. For a remote connection that’s bad news. At least if you screw up the rule you know that you did the moment that you add it to the firewall, so if it comes to remotely rebooting the box you know it will come back up without a problem. And yes, I did do that once. It was a case of

“I’ll just update the firewall rules before I leave this morni… oh er.. oh, what did I type? Sh*t!”.

But my colleague believes that you should add it to the config file and restart because that way you’re only entering it once. Which is nice, and does indeed have its merits because there’s less chance of a typo, getting the order wrong, etc., but it also means you need to stop and restart iptables. Aside from the risk of locking yourself out of the server to me that’s a royal pain. You lose any temporary rules, and you worst of all you lose the connection tracking. Which means my nice and quiet SSH session which was working before lunch and has my open Vim session in it … is now not working because its state is not new or established or related, and thus not a valid connection state as far as iptables is concerned and thus the packets are blocked, and the connection gets dropped. *grr*

More irritatingly though is that I can’t *know* that that’s what happened because iptables doesn’t log when it stopped or started. Which is pretty crazy really if you think about the fact that every daemon under the sun logs that kind of information. It never occurred to me before, but its true, at least on my box.

Incidentally, if you’re looking to fiddle with the firewall remotely, a few good tricks are;

  1. Add a rule to allow your SSH connection through as a first rule. E.g.iptables -I INPUT 1 -p tcp --dport 22 -s <your ip> -j ACCEPT
  2. Use a sleep; undo system. This works for both the command line and restarting iptables. For example;
    1. Add a rule on the command line, knowing the rule number you’ll give it, and then set it to undo a moment later;iptables -I INPUT 23 <what you want to happen>; \
      sleep 10; \
      iptables -D INPUT 23
    2. Restart iptables, then stop it again if it doesn’t work;/etc/init.d/iptables restart; sleep 10; /etc/init.d/iptables stop

The second tip uses the a chain of commands which do whatever might go wrong, sleep and then undo it. The reason for this is that it will execute a command and then move on to the next command in the list, so if you submit the commands like this, where you must include the semi-colons between the commands, (don’t forget the \’s in the first case, or just ignore them and don’t hit return) , then the shell will execute them one after another like this … do “fiddle with firewall command”, sleep, and do “undo fiddle command”. In the even that you don’t do anything, the firewall will be restarted and then ten seconds later, stopped. This is good because in the event that you f*ck up the firewall rules and can’t type, 10s later you’ll be able to type again because the firewall has been taken down again.

But the best part of this trick is that in the event that you can type you do … you wait a second or so, and then type Ctrl-C to interupt the currently running command. This will be the sleep command, so you’ll interrupt that and interrupt the list of commands to execute so your last command, the “undo” command, is never run. So if you can’t get into the box it unsets its firewall. If you can get in then you kill the sleep command before it finishes thus the “undo” is never executed.

As a tip, if you’re not sure when to hit Ctrl-C, add a command echo “Sleeping” into the list just before the sleep command. If you don’t see that then the firewall was messed up. If you do see it then your changes worked, and you can kill the rest of the commands with Ctrl-C.

Colin.