Logging to RudderStack

An optional guide to sending logs to RudderStack

We recommend setting up Fluentd on all the machines running Backend to send logs to our S3 bucket. This will help us in debugging in case of any potential issues you might face.

Installing Fluentd

Please follow the clear and detailed installation instructions to set up Fluentd

Once you have successfully installed Fluentd, follow the steps below:

The following commands are run on an example installation on EC2 machine running Ubuntu

$ sudo apt-get install ruby
$ sudo apt-get install ruby-dev
$ sudo gem install fluentd --no-ri --no-rdoc
#Verify fluentd is installed:
$ fluentd --setup ./fluent
$ fluentd -c ./fluent/fluent.conf -vv &
$ echo '{"json":"message"}' | fluent-cat debug.test

Also, install few plugins to facilitate reading journalctl logs and sending them to S3

$ sudo gem install fluent-plugin-systemd -v 1.0.1 --no-document
$ sudo gem install uuidtools --no-document
$ sudo gem install fluent-plugin-record-modifier --no-document
$ sudo gem install fluent-plugin-s3 -v 1.0.0 --no-document

Configuration

Create a configuration file with the following contents:

Run fluentd with created configuration in background
$ sudo fluentd -c <configuration_file_path> &
Stop fluentd
$ sudo pkill -f fluentd

Variables Please contact us to get the values for the following variables and replace them in the below config

  • APP_NAME

  • AWS_USER_KEY

  • AWS_USER_SECRET

  • RUDDER_S3_BUCKET_NAME

Also make sure to replace the <STORAGE_PATH> with your preferred location (eg. /home/ubuntu)

<source>
@type systemd
tag <APP_NAME>-rudder-server
path /run/log/journal
matches [{ "_SYSTEMD_UNIT": "rudder.service" }]
read_from_head true
<storage>
@type local
path <STORAGE_PATH>/fluentd-journald-rudder-server-cursor.json
</storage>
<entry>
field_map {"MESSAGE": "log", "_PID": ["process", "pid"], "_CMDLINE": "process", "_COMM": "cmd"}
fields_strip_underscores true
fields_lowercase true
</entry>
</source>
<source>
@type systemd
tag <APP_NAME>-rudder-transformer
path /run/log/journal
matches [{ "_SYSTEMD_UNIT": "rudder-transformer.service" }]
read_from_head true
<storage>
@type local
path <STORAGE_PATH>/fluentd-journald-rudder-transformer-cursor.json
</storage>
<entry>
field_map {"MESSAGE": "log", "_PID": ["process", "pid"], "_CMDLINE": "process", "_COMM": "cmd"}
fields_strip_underscores true
fields_lowercase true
</entry>
</source>
<filter <APP_NAME>-rudder-server>
@type record_modifier
<record>
tag ${tag}
formatted_time ${Time.at(time).to_s}
</record>
</filter>
<filter <APP_NAME>-rudder-transformer>
@type record_modifier
<record>
tag ${tag}
formatted_time ${Time.at(time).to_s}
</record>
</filter>
<match <APP_NAME>-rudder-server>
@type s3
aws_key_id <AWS_USER_KEY>
aws_sec_key <AWS_USER_SECRET>
s3_bucket <RUDDER_S3_BUCKET_NAME>
s3_region us-east-1
acl bucket-owner-full-control
auto_create_bucket false
check_bucket false
check_object false
check_apikey_on_start false
path logs/${tag}/%Y/%m/%d/
s3_object_key_format %{path}%{time_slice}_%{uuid_flush}_%{index}.%{file_extension}
# if you want to use ${tag} or %Y/%m/%d/ like syntax in path / s3_object_key_format,
# need to specify tag for ${tag} and time for %Y/%m/%d in <buffer> argument.
<buffer tag,time>
@type file
path /var/log/fluent/s3
timekey 1h # 1 hour partition
timekey_wait 10m
timekey_use_utc true # use utc
</buffer>
<format>
@type json
</format>
</match>
<match <APP_NAME>-rudder-transformer>
@type s3
aws_key_id <AWS_USER_KEY>
aws_sec_key <AWS_USER_SECRET>
s3_bucket <RUDDER_S3_BUCKET_NAME>
s3_region us-east-1
acl bucket-owner-full-control
auto_create_bucket false
check_bucket false
check_object false
check_apikey_on_start false
path logs/${tag}/%Y/%m/%d/
s3_object_key_format %{path}%{time_slice}_%{uuid_flush}_%{index}.%{file_extension}
# if you want to use ${tag} or %Y/%m/%d/ like syntax in path / s3_object_key_format,
# need to specify tag for ${tag} and time for %Y/%m/%d in <buffer> argument.
<buffer tag,time>
@type file
path /var/log/fluent/s3
timekey 1h # 1 hour partition
timekey_wait 10m
timekey_use_utc true # use utc
</buffer>
<format>
@type json
</format>
</match>