SourceXR

C/C++ Cross-Reference Tool

Application Monitoring with SNMP

When operating an online application, it is critical to know what is going on.

Externally, it is easy to get feedback and monitoring in terms of consumed CPU, memory, bandwidth, file descriptors and so on. But how to get information on the data handled by the application itself? As soon as the application handles data, we may ask for some metrics, like the following:

  • how many sessions are created?
  • what is the lifetime of a session?
  • how many requests on behalf of a session are performed?
  • if the application caches data, how are they used: access rate, hit and miss ratios?
  • etc.

And these are only a very small part of metrics that we would like to know! What about monitoring, marketing, business metrics that are key indicators for a company?

We will only focus here on a simple mechanism to collect these data, leaving reporting to more specialized tools.

Logs and logfiles are a convenient way of reporting what is going on, but they require a post processing to be viewed in a more user-friendly way, and they are not realtime.

Here, we'll describe several possible ways to perform application monitoring using SNMP. SNMP is a de facto standard for network monitoring but can be used for anything else.

Its wide use means that it is implemented in almost all network management solutions. Therefore, using SNMP for your applications means you will immediately be able to see and monitor the data without wondering how they can be collected for reporting, alerts, etc.

Among other things, SNMP supports as well asynchronous reporting of events called traps. Traps are single UDP messages sent to a snmptrapd daemon whose purpose is to handle the trap or hands over it to any other program. Traps should be seen as a notification mechanism. This is the realtime property of SNMP reporting.

There are several ways of integrating SNMP monitoring to your application. One is external, that is, the monitored data is already provided in some way by your application; the other one is internal, in that case, the SNMP API is used to access the data.

We use here the SNMP implementation from Net-SNMP.

External Monitoring

snmpd is a process running on the server whose purpose is to reply to SNMP request about metrics on the server. Many requests are built-in such as CPU, memory, disks, process monitoring (number of process instance running for a given process). You can easily add your own (check the proc entry in your snmpd.conf file).

exec and extend Directives

What is interesting is that you can add your own program to be run when querying snmpd for a predefined OID (OID are object identifiers, corresponding to the data your are requesting, for example current free memory).

If your application has already a way of communicating the requested data to the outside world, the exec or the somewhat more flexible extend directive can be used.

  • Add the program used to collect the data,
  • Have snmpd reload its configuration,
  • Add the OID to your favorite network management application, and data start to be collected, analyzed, archived, etc:
    • for exec, the OID starts at .1.3.6.1.4.1.2021.8, and the execution will give you the exit status and the command output;
    • for extend, an additional parameter is used to customize the OID, for example if CPU is used as parameter the OID will be .1.3.6.1.4.1.8072.1.3.2.4.1.2.3.67.80.85 (the last three characters corresponds to the ASCII codes of the characters of the parameter).

pass_persist Directive

snmpd supports many more ways to get data. In the previous example, exec and extend will fork a new process and snmpd will wait for the commands to return.

If this is too costly (for example if the metrics are retrieved very frequently), or if collected data change very frequently (as snmpd caches the returned data) the pass_persist directive can be used.

The first time snmpd receives a request for the metric, it will fork and then reuse the process.

The protocol between snmpd and the process is simple. snmpd forks the process and communicates with it using its stdin and stdout channels.

Initialization

SNMPd writes 'PING\n' on the process stdin and it should reply 'PONG\n' on its stdout channel.

Requests

For get and getnext SNMP requests, snmpd writes two lines on stdin: the command (get or getnext) and the requested OID. The process should reply three lines: the OID, the type of the value and the value.

This directive supports as well to set a specific OID value if needed.

Internal Monitoring & Traps

If your application is able to speak SNMP directly, you do not need to go through an external program to get the data. Simply use the proxy directive of snmpd configuration file.

When snmpd receives a request for the predefined OID, it will forward the request to your application, using the supplied parameters (possibly remotely and changing the OID on the fly if needed).

SNMP Traps

SNMP traps are asynchonous events sent to the trap manager which handles it possibly raising alerts, escalating, etc.

SNMP traps can be sent through the snmptrap command line utility, supplying the right command line parameters in accordance with the configuration of the snmptrapd daemon that handles it.

They can be sent as well through the Net-SNMP API. The sample code below sets up a SNMP session and sends a sample trap message to the snmptrapd daemon located on localhost. Compile it with the following command line:

g++ -lnetsnmphelpers -lnetsnmp -lnetsnmpagent sourcefile

(The development headers of net-snmp are needed, on Debian, the package is named libsnmp-dev).

#include <net-snmp/net-snmp-config.h>
#include <net-snmp/net-snmp-includes.h>
#include <net-snmp/agent/net-snmp-agent-includes.h>
#include <string>

int main() {

    struct snmp_session session, *ss;
    init_traps ();
    // session init
    snmp_sess_init (&session);
    session.peername = const_cast<char *> ("localhost:162");
    session.version = SNMP_VERSION_2c;

    // set the SNMPv1 community name used for authentication
    const char name[] = "demopublic";
    const u_char *comm = reinterpret_cast<const u_char *> (name);
    session.community = const_cast<u_char *>(comm);
    session.community_len = sizeof (name) / sizeof (name[0]) - 1;

    // open the session
    ss = snmp_open (&session);
    if (!ss) {
        snmp_perror ("ack");
        snmp_log (LOG_ERR, "open_session failed!\n");
        return 1;
    }

    add_trap_session (ss, SNMP_MSG_TRAP2, 0, SNMP_VERSION_2c);

    oid objid_snmptrap[] = { 1, 3, 6, 1, 6, 3, 1, 1, 4, 1, 0 };
    size_t objid_snmptrap_len = OID_LENGTH (objid_snmptrap);

    netsnmp_variable_list *notification_vars = NULL;

    // define here the custom OIDs for your application events
    oid server_failed_oid[] = { 1, 3, 6, 1, 4, 1, 1, 2, 3, 4, 5, 6 };
    size_t server_failed_oid_len = OID_LENGTH (server_failed_oid);

    oid notification_oid[] = { 1, 3, 6, 1, 4, 1, 1, 2, 3, 0, 1 };
    size_t notification_oid_len = OID_LENGTH (notification_oid);

    snmp_varlist_add_variable (&notification_vars,
                               objid_snmptrap, objid_snmptrap_len,
                               ASN_OBJECT_ID,
                               reinterpret_cast<const u_char *> (notification_oid),
                               notification_oid_len * sizeof (oid));

    std::string err ("Server failed!");
    snmp_varlist_add_variable (&notification_vars,
                               server_failed_oid, server_failed_oid_len,
                               ASN_OCTET_STR,
                               reinterpret_cast<const u_char *> (err.data ()),
                               err.size ());

    send_v2trap (notification_vars);
    snmp_free_varbind (notification_vars);

    remove_trap_session (ss);
    snmp_close (ss);
    return 0;

}

Do not forget to setup the snmptrapd daemon (not running by default) and configure the trap handling.

Conclusion

It is quite easy to configure snmpd and start getting application metrics, either directly or using the application's own facility for metrics reporting.

Traps can be sent easily as well using the Net-SNMP provided command line utility or using the above code snippet if you want to implement the trap sending directly within your application.

Comments !