The syslog-ng Java destinations

The syslog-ng application can read messages from the sources. It processes them with filters, rewrite rules, parsers and finally sends messages to their destinations. It has Java destinations so that will give an opportunity to easily work around with Java code based applications.

In this blog post, we will discuss about how we can write a simple Java destination to write into a file from my Google summer of code task experience with syslog-ng.  Before you begin, you need to setup an environment variable LD_LIBRARY_PATH as follows the path to libjvm.so in your JDK folder. Since syslog-ng destinations are written using Gradle builder, we’ll also use Gradle for building purposes. If you haven’t installed Gradle or any IDE, please follow my another blogpost about setting up dev environment in Ubuntu.

Starting Gradle Java project

After creating simple gradle project, you need to add few dependencies to Java destination in your build.gradle

apply plugin: 'java'

sourceCompatibility = 1.7

task fatJar(type: Jar) {
    manifest {
        attributes 'Implementation-Title': 'Java destination',
                'Implementation-Version': version,
                'Main-Class': 'FileDestination'
    }
    baseName = project.name + '-all'
    from { configurations.compile.collect { it.isDirectory() ? it : zipTree(it) } }
    with jar
}

repositories {
    flatDir {
        dirs '/usr/local/lib/syslog-ng/java-modules/'
    }
}

dependencies {
    testCompile group: 'junit', name: 'junit', version: '4.11'
    compile name: 'syslog-ng-core'
    compile name: 'syslog-ng-common'

}

 

We need to have syslog-ng-core and syslong-ng-common dependencies to create syslog-ng Java destinations. These jars will be available in syslong-ng installed directory. For example,

/usr/local/lib/syslog-ng/java-modules/

Check your jar path and add it to the build.gradle as I showed.

We can have a class file or jar file for the logpath of syslog-ng configuration file, but since here we have dependencies in two jars, its better to have a fatJar with all dependencies.

Sample Java class for destination

import org.syslog_ng.InternalMessageSender;
import org.syslog_ng.TextLogDestination;
import org.syslog_ng.options.*;

import java.io.BufferedWriter;
import java.io.File;
import java.io.FileWriter;
import java.io.IOException;

public class FileDestination extends TextLogDestination {

    private String fullPath;
    private BufferedWriter bufferedWriter = null;

    public FileDestination (long arg0){
        super(arg0);
    }

    @Override
    protected boolean send(String s) {
        try {
            this.bufferedWriter.write(s);
            this.bufferedWriter.newLine();
            this.bufferedWriter.flush();
            return true;
        } catch (IOException e) {
            InternalMessageSender.error(e.getMessage());
        }
        return false;
    }

    @Override
    protected boolean open() {
        File file = new File(fullPath);
        if(!file.exists()){
            try {
                file.createNewFile();
            } catch (IOException e) {
                InternalMessageSender.error(e.getMessage());
            }
        }
        if(file.exists()){
            try {
                FileWriter fileWriter = new FileWriter(file.getAbsoluteFile());
                bufferedWriter = new BufferedWriter(fileWriter);
                return true;
            } catch (IOException e) {
                InternalMessageSender.error(e.getMessage());
                return false;
            }
        }
        return false;
    }

    @Override
    protected void close() {
        try {
            this.bufferedWriter.close();
        } catch (IOException e) {
            InternalMessageSender.error(e.getMessage());
        }
    }

    @Override
    protected boolean isOpened() {
        if (bufferedWriter != null) {
            return true;
        }
        return false;
    }

    @Override
    protected String getNameByUniqOptions() {
        return "FileDestination("+fullPath+")";
    }

    @Override
    protected boolean init() {
        String filename;
        String filepath;
        Options requiredOptions = new Options();
        requiredOptions.put (new RequiredOptionDecorator(new StringOption(this,"filename")));
        requiredOptions.put (new RequiredOptionDecorator(new StringOption(this,"filepath")));
        try {
            requiredOptions.validate();
        } catch (InvalidOptionException e) {
            InternalMessageSender.error("Some options are missing");
            return false;
        }

        filename = getOption("filename");
        filepath = getOption("filepath");
        fullPath = filepath + filename;
        return true;
    }

    @Override
    protected void deinit() {

    }
}

Lets see, how this works.

My main Java class that you might have noticed in build.gradle is FileDestination. If you need to create a syslog-ng Java destination from this, you need to extend your main class from TextLogDestination or StructuredLogDestination abstract classes. This will override the methods from above abstract classes.

Use case of init() method

This is the first method that will be invoked in the pipeline. Here you should check whether all your required options are set in the configuration file and do other initiation works you need.

Use case of open() method

Here you should open the connection to the destination. For example, in my case my destination is file, so I’m opening a connection to file using bufferedWriter.

Use case of isOpened() method

This method should check whether the open() method correctly opened the connection to the destination, otherwise application will break.

Use case of send() method

This method will receive the message from logWriter and it will send the String messages to the opened connection.

Use case of getNameByUniqueOptions() method

Here we need to setup a unique name for the destination because this will be shown in the statistics and also syslog-ng will assign disk buffer to the destination using this name.

Building and running the application

Once you’re done with your coding now you can build the gradle project using the following command in the gradle project home.

gradle fatJar

It will build a jar file with all your dependencies in $gradle_project_home/build/lib

Now you need to create a configuration file for your Java destination.

Sample configuration

@version: 3.7

source sample_text {
	file("/home/vithulan/syslog-ng/sample-logs/kern.log"
	follow_freq(1));	
};

destination d_java_file {
	java(
	 class_name("FileDestination")
	 class_path("/home/vithulan/IdeaProjects/syslogng-java-destination/build/libs/javadestination-all-1.0-SNAPSHOT.jar")
	 option("name","fileDest")
	 option("filename","test.txt")
	 option("filepath","/home/vithulan/syslog-ng/tests/")
	);
};

log {
	source(sample_text);
	destination(d_java_file);
        # in case of file source with remote destination in most cases we should use flow-control
        # you can read more about flow-control in the documentation
	flags(flow-control);
};

You might want to change the class_path, options, file as per your paths for files.

Running syslog-ng with this configuration.

Go to the folder where you’ve installed syslog-ng and run the following command. (Assuming you’ve saved the above configuration as sample.conf)

syslog-ng -Fe -f $path_to_your_configuration_file/sample.conf

This will open the connection to the file you’ve mentioned in the configuration file and will send the messages to the destination that you have indicated in options.

You can clone the actual project at github – https://github.com/VIthulan/syslog-ng-java-sample

I hope this helps you to understand the basics of syslog-ng Java destinations. If you have any doubts or problems or feedbacks please feel free to leave a comment below.

Thanks 🙂
Happy Coding!! 😀