Fluentd - Configuration for Hipaa Bucket : Each Match

 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

 This was the first -- Fluentd configuration with Folder creates 

dd/mm/yyyy 

  messages

  secure

  kern

 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

 

<source>
  @type syslog
  @id input_syslog
  port 4889
  bind 0.0.0.0
  <transport tcp>
  </transport>
  <parse>
    @type syslog
    with_priority false
    message_format auto
  </parse>
  emit_unmatched_lines true
  severity_key message_severity
  source_address_key source_address
  tag oci.0.8x8.vo.dev.os.syslog.*
</source>

<source>
  @type tail
#  @id input_tail_audit
  path /var/log/audit/audit.log
  pos_file /var/lib/fluent_oci_outplugin/pos/audit.pos
  pos_file_compaction_interval 24h
  enable_watch_timer true
  enable_stat_watcher true
  follow_inodes true
  <parse>
#    @type json
    @type none
  </parse>
  tag oci.0.8x8.vo.dev.os.audit.*
</source>

<source>
  @type tail
  @id input_tail_go-audit
  path /var/log/go-audit/go-audit.log
  pos_file /var/lib/fluent_oci_outplugin/pos/go-audit.pos
  pos_file_compaction_interval 24h
  enable_watch_timer true
  enable_stat_watcher true
  follow_inodes true
  <parse>
    @type json
  </parse>
  tag oci.0.8x8.vo.dev.os.go-audit.*
</source>

<source>
  @type tail
  path /var/log/secure
  pos_file /var/lib/fluent_oci_outplugin/pos/secure.pos
  pos_file_compaction_interval 24h
  enable_watch_timer true
  enable_stat_watcher true
  follow_inodes true
  <parse>
    @type none
  </parse>
  tag oci.0.8x8.vo.dev.os.secure.*
</source>

<source>
  @type tail
  path /var/log/messages
  pos_file /var/lib/fluent_oci_outplugin/pos/messages.pos
  pos_file_compaction_interval 24h
  enable_watch_timer true
  enable_stat_watcher true
  follow_inodes true
  <parse>
    @type none
  </parse>
  tag oci.0.8x8.vo.dev.os.messages.*
</source>

<source>
  @type tail
  path /var/log/kern.log
  pos_file /var/lib/fluent_oci_outplugin/pos/kern.pos
  pos_file_compaction_interval 24h
  enable_watch_timer true
  enable_stat_watcher true
  follow_inodes true
  <parse>
    @type none
  </parse>
  tag oci.0.8x8.vo.dev.os.kern.*
</source>

<filter tag oci.0.8x8.vo.dev.os.go-audit.**>
  @type go_audit_parser
  @id go-audit.parser
</filter>

<filter oci.0.8x8.vo.dev.os.**>
    @type record_transformer
    enable_ruby false
    <record>
        hostname "#{Socket.gethostname}"
        host "compute73-cc"
        fqdn "compute73-cc.8x8hosts.pilot"
        osDistribution "CentOS"
        osMajorVersion "7"
        logSource "OSAuditSyslog"
        businessUnit "vo"
        environment "development"
        securityCompliance "SOC2"
        tag ${tag}
    </record>
</filter>

<match oci.0.8x8.vo.dev.os.syslog.**>
        @type copy
        <store>
                @type s3
                aws_key_id e577cbec64dad7f892683dbde024fc0a077f4b5d
                aws_sec_key fyGrYtUUdUuK9uIY9/4aH/09/bjFnu4GgW9wjiCdvwA=
                s3_bucket idxixi5bmstw
                s3_region us-sanjose-1
                s3_endpoint https://idxixi5bmstw.compat.objectstorage.us-sanjose-1.oraclecloud.com
                path HIPAA_Logs_Bucket/${environment}/${fqdn}/%F/syslogs/
                s3_object_key_format %{path}%{time_slice}_%{index}.%{file_extension}
                check_apikey_on_start false
                ssl_verify_peer false
                check_bucket false
                <buffer tag,time,oci_la_log_source_name,hostIpAddress,logName,environment,fqdn,businessUnit>
                     @type file
                     tag ${tag}
                     path /var/log/fluent_oci_outplugin/buffer/3/syslogs/
                     timekey 120 # (default 3600) 1 hour partition
                     timekey_wait 10s
                     timekey_use_utc true # use utc
                     flush_thread_count 10
                </buffer>
                <format>
                   @type json
                </format>
                    time_slice_format %Y%m%d%H%M
        </store>
</match>

<match oci.0.8x8.vo.dev.os.audit.**>
        @type copy
        <store>
                @type s3
                aws_key_id e577cbec64dad7f892683dbde024fc0a077f4b5d
                aws_sec_key fyGrYtUUdUuK9uIY9/4aH/09/bjFnu4GgW9wjiCdvwA=
                s3_bucket idxixi5bmstw
                s3_region us-sanjose-1
                s3_endpoint https://idxixi5bmstw.compat.objectstorage.us-sanjose-1.oraclecloud.com
                path HIPAA_Logs_Bucket/${environment}/${fqdn}/%F/audit/
                s3_object_key_format %{path}%{time_slice}_%{index}.%{file_extension}
                check_apikey_on_start false
                ssl_verify_peer false
                check_bucket false
                <buffer tag,time,oci_la_log_source_name,hostIpAddress,logName,environment,fqdn,businessUnit>
                     @type file
                     tag ${tag}
                     path /var/log/fluent_oci_outplugin/buffer/2/audit/
                     timekey 120 # (default 3600) 1 hour partition
                     timekey_wait 10s
                     timekey_use_utc true # use utc
                     flush_thread_count 10
                </buffer>
 #               <format>
 #                  @type json
 #              </format>
                    time_slice_format %Y%m%d%H%M
        </store>
</match>

<match oci.0.8x8.vo.dev.os.go-audit.**>
        @type copy
        <store>
                @type s3
                aws_key_id e577cbec64dad7f892683dbde024fc0a077f4b5d
                aws_sec_key fyGrYtUUdUuK9uIY9/4aH/09/bjFnu4GgW9wjiCdvwA=
                s3_bucket idxixi5bmstw
                s3_region us-sanjose-1
                s3_endpoint https://idxixi5bmstw.compat.objectstorage.us-sanjose-1.oraclecloud.com
                path HIPAA_Logs_Bucket/${environment}/${fqdn}/%F/go_audit/
                s3_object_key_format %{path}%{time_slice}_%{index}.%{file_extension}
                check_apikey_on_start false
                ssl_verify_peer false
                check_bucket false
                <buffer tag,time,oci_la_log_source_name,hostIpAddress,logName,environment,fqdn,businessUnit>
                     @type file
                     tag ${tag}
                     path /var/log/fluent_oci_outplugin/buffer/0/go_audit/
                     timekey 120 # (default 3600) 1 hour partition
                     timekey_wait 10s
                     timekey_use_utc true # use utc
                     flush_thread_count 10
                </buffer>
                <format>
                   @type json
                </format>
                    time_slice_format %Y%m%d%H%M
        </store>
</match>

<match oci.0.8x8.vo.dev.os.messages.**>
        @type copy
        <store>
                @type s3
                aws_key_id e577cbec64dad7f892683dbde024fc0a077f4b5d
                aws_sec_key fyGrYtUUdUuK9uIY9/4aH/09/bjFnu4GgW9wjiCdvwA=
                s3_bucket idxixi5bmstw
                s3_region us-sanjose-1
                s3_endpoint https://idxixi5bmstw.compat.objectstorage.us-sanjose-1.oraclecloud.com
                path HIPAA_Logs_Bucket/${environment}/${fqdn}/%F/messages/
                s3_object_key_format %{path}%{time_slice}_%{index}.%{file_extension}
                check_apikey_on_start false
                ssl_verify_peer false
                check_bucket false
                <buffer tag,time,oci_la_log_source_name,hostIpAddress,logName,environment,fqdn,businessUnit>
                     @type file
                     tag ${tag}
                     path /var/log/fluent_oci_outplugin/buffer/0/messages/
                     timekey 120 # (default 3600) 1 hour partition
                     timekey_wait 10s
                     timekey_use_utc true # use utc
                     flush_thread_count 10
                </buffer>
  #              <format>
  #                  @type json
  #              </format>
                    time_slice_format %Y%m%d%H%M
        </store>
</match>

<match oci.0.8x8.vo.dev.os.secure.**>
        @type copy
        <store>
                @type s3
                aws_key_id e577cbec64dad7f892683dbde024fc0a077f4b5d
                aws_sec_key fyGrYtUUdUuK9uIY9/4aH/09/bjFnu4GgW9wjiCdvwA=
                s3_bucket idxixi5bmstw
                s3_region us-sanjose-1
                s3_endpoint https://idxixi5bmstw.compat.objectstorage.us-sanjose-1.oraclecloud.com
                path HIPAA_Logs_Bucket/${environment}/${fqdn}/%F/secure/
                s3_object_key_format %{path}%{time_slice}_%{index}.%{file_extension}
                check_apikey_on_start false
                ssl_verify_peer false
                check_bucket false
                <buffer tag,time,oci_la_log_source_name,hostIpAddress,logName,environment,fqdn,businessUnit>
                     @type file
                     tag ${tag}
                     path /var/log/fluent_oci_outplugin/buffer/1/secure/
                     timekey 120 # (default 3600) 1 hour partition
                     timekey_wait 10s
                     timekey_use_utc true # use utc
                     flush_thread_count 10
                </buffer>
 #               <format>
 #                  @type json
 #               </format>
                    time_slice_format %Y%m%d%H%M
        </store>
</match>
<match oci.0.8x8.vo.dev.os.kern.**>
        @type copy
        <store>
                @type s3
                aws_key_id e577cbec64dad7f892683dbde024fc0a077f4b5d
                aws_sec_key fyGrYtUUdUuK9uIY9/4aH/09/bjFnu4GgW9wjiCdvwA=
                s3_bucket idxixi5bmstw
                s3_region us-sanjose-1
                s3_endpoint https://idxixi5bmstw.compat.objectstorage.us-sanjose-1.oraclecloud.com
                path HIPAA_Logs_Bucket/${environment}/${fqdn}/%F/kern/
                s3_object_key_format %{path}%{time_slice}_%{index}.%{file_extension}
                check_apikey_on_start false
                ssl_verify_peer false
                check_bucket false
                <buffer tag,time,oci_la_log_source_name,hostIpAddress,logName,environment,fqdn,businessUnit>
                     @type file
                     tag ${tag}
                     path /var/log/fluent_oci_outplugin/buffer/1/kern/
                     timekey 120 # (default 3600) 1 hour partition
                     timekey_wait 10s
                     timekey_use_utc true # use utc
                     flush_thread_count 10
                </buffer>
 #               <format>
 #                  @type json
 #               </format>
                    time_slice_format %Y%m%d%H%M
        </store>
</match>
 

 

Comments

Post a Comment

Popular posts from this blog

FluentD - Parse

Image
 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++  FluentD : Parse +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++  Some of the Fluentd plugins support the <parse> section to specify how to parse the raw data. Parse Section Overview : The parse section can be under <source> , <match> or <filter> section. It is enabled for the plugins that support parser plugin features. <source>   @type tail   # ...   <parse>     # ...   </parse> </source> Parse Plugin Type : The @type parameter of <parse> section specifies the type of the parser plugin. Fluentd core bundles some useful  <parse>   @ type apache2 </parse> Third-party plugins may also be installed and configured.                      
Read more

FluentD : Tags : Prefix : Suffix

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++   +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ You can also access to a certain portion of a tag using the following notations: tag_parts[N] refers to the Nth part of the tag. tag_prefix[N] refers to the [0..N] part of the tag. tag_suffix[N] refers to the [N..] part of the tag. All indices are zero-based. For example, if you have an incoming event tagged debug.my.app, then tag_parts[1] will represent my. Also in this case, tag_prefix[N] and tag_suffix[N] will work as follows: tag_prefix[0] = debug          tag_suffix[0] = debug.my.app tag_prefix[1] = debug.my       tag_suffix[1] = my.app tag_prefix[2] = debug.my.app   tag_suffix[2] = app
Read more

Fluentd :

 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++     +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++  Interlude Routing : The source submits events to the Fluentd routing engine. An event consists of three entities: tag, time and record. The tag is a string separated by dots (e.g. myapp.access), and is used as the directions for Fluentd internal routing engine. The time field is specified by input plugins, and it must be in the Unix time format. The record is a JSON object
Read more