Technical preview

Ingest and parse plaintext logs, including existing logs, from any programming language or framework without modifying your application or its configuration.

Plaintext logs require some additional setup that structured logs do not require:

• To search, filter, and aggregate effectively, you need to parse plaintext logs using an ingest pipeline to extract structured fields. Parsing is based on log format, so you might have to maintain different settings for different applications.
• To correlate plaintext logs, you need to inject IDs into log messages and parse them using an ingest pipeline.

To ingest, parse, and correlate plaintext logs:

1. Ingest plaintext logs with Filebeat or Elastic Agent and parse them before indexing with an ingest pipeline.
2. Correlate plaintext logs with an APM agent.
3. View logs in Logs Explorer

Ingest logs

Send application logs to your project using one of the following shipping tools:

• Filebeat: A lightweight data shipper that sends log data to your project.
• Elastic Agent: A single agent for logs, metrics, security data, and threat prevention. With Fleet, you can centrally manage Elastic Agent policies and lifecycles directly from your project.

Ingest logs with Filebeat

Follow these steps to ingest application logs with Filebeat.

Step 1: Install Filebeat

Install Filebeat on the server you want to monitor by running the commands that align with your system:

curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-8.13.4-darwin-x86_64.tar.gz
tar xzvf filebeat-8.13.4-darwin-x86_64.tar.gz

Step 2: Connect to your project

Connect to your project using an API key to set up Filebeat. Set the following information in the filebeat.yml file:

output.elasticsearch:
  hosts: ["your-projects-elasticsearch-endpoint"]
  api_key: "id:api_key"

1. Set the hosts to your project's Elasticsearch endpoint. Locate your project's endpoint by clicking the help icon () and selecting Endpoints. Add the Elasticsearch endpoint to your configuration.

2. From Developer tools, run the following command to create an API key that grants manage permissions for the cluster and the filebeat-* indices using:

   POST /_security/api_key
   {
     "name": "your_api_key",
     "role_descriptors": {
       "filebeat_writer": {
         "cluster": ["manage"],
         "index": [
           {
             "names": ["filebeat-*"],
             "privileges": ["manage", "create_doc"]
           }
         ]
       }
     }
   }

   Refer to Grant access using API keys for more information.

Step 3: Configure Filebeat

Add the following configuration to the filebeat.yaml file to start collecting log data.

filebeat.inputs:
- type: filestream 
  enabled: true
  paths: /path/to/logs.log 

You can add additional settings to the filebeat.yml file to meet the needs of your specific set up. For example, the following settings would add a parser to manage messages that span multiple lines and add service fields:

  parsers:
  - multiline:
      type: pattern
      pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
      negate: true
      match: after
  fields_under_root: true
  fields:
    service.name: your_service_name
    service.environment: your_service_environment
    event.dataset: your_event_dataset

Step 4: Set up and start Filebeat

From the Filebeat installation directory, set the index template by running the command that aligns with your system:

./filebeat setup --index-management

from the Filebeat installation directory, start filebeat by running the command that aligns with your system:

sudo chown root filebeat.yml
sudo ./filebeat -e

Step 5: Parse logs with an ingest pipeline

Use an ingest pipeline to parse the contents of your logs into structured, Elastic Common Schema (ECS)-compatible fields.

Create an ingest pipeline with a dissect processor to extract structured ECS fields from your log messages. In your project, go to Developer Tools and use a command similar to the following example:

PUT _ingest/pipeline/filebeat*
{
  "description": "Extracts the timestamp log level and host ip",
  "processors": [
    {
      "dissect": {
        "field": "message",
        "pattern": "%{@timestamp} %{log.level} %{host.ip} %{message}"
      }
    }
  ]
}

Refer to Extract structured fields for more on using ingest pipelines to parse your log data.

After creating your pipeline, specify the pipeline for filebeat in the filebeat.yml file:

output.elasticsearch:
  hosts: ["your-projects-elasticsearch-endpoint"]
  api_key: "id:api_key"
  pipeline: "your-pipeline"

Ingest logs with Elastic Agent

Follow these steps to ingest and centrally manage your logs using Elastic Agent and Fleet.

Step 1: Add the custom logs integration to your project

To add the custom logs integration to your project:

1. In your Observability project, go to Project Settings → Integrations.
2. Type custom in the search bar and select Custom Logs.
3. Click Add Custom Logs.
4. Click Install Elastic Agent at the bottom of the page, and follow the instructions for your system to install the Elastic Agent.
5. After installing the Elastic Agent, configure the integration from the Add Custom Logs integration page.
6. Give your integration a meaningful name and description.
7. Add the Log file path. For example, /var/log/your-logs.log.
8. An agent policy is created that defines the data your Elastic Agent collects. If you've previously installed an Elastic Agent on the host you're collecting logs from, you can select the Existing hosts tab and use an existing agent policy.
9. Click Save and continue.

You can add additional settings to the integration under Custom log file by clicking Advanced options and adding YAML configurations to the Custom configurations. For example, the following settings would add a parser to manage messages that span multiple lines and add service fields. Service fields are used for Log correlation.

  parsers:
  - multiline:
      type: pattern
      pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
      negate: true
      match: after
  fields_under_root: true
  fields:
    service.name: your_service_name
    service.version: your_service_version
    service.environment: your_service_environment

Step 2: Add an ingest pipeline to your integration

To aggregate or search for information in plaintext logs, use an ingest pipeline with your integration to parse the contents of your logs into structured, Elastic Common Schema (ECS)-compatible fields.

1. From the custom logs integration, select Integration policies tab.

2. Select the integration policy you created in the previous section.

3. Click Change defaults → Advanced options.

4. Under Ingest pipelines, click Add custom pipeline.

5. Create an ingest pipeline with a dissect processor to extract structured fields from your log messages.

   Click Import processors and add a similar JSON to the following example:

   {
     "description": "Extracts the timestamp log level and host ip",
     "processors": [
       {
         "dissect": {
           "field": "message",
           "pattern": "%{@timestamp} %{log.level} %{host.ip} %{message}"
         }
       }
     ]
   }

6. Click Create pipeline.

7. Save and deploy your integration.

Correlate logs

Correlate your application logs with trace events to:

• view the context of a log and the parameters provided by a user
• view all logs belonging to a particular trace
• easily move between logs and traces when debugging application issues

Log correlation works on two levels:

• at service level: annotation with service.name, service.version, and service.environment allow you to link logs with APM services
• at trace level: annotation with trace.id and transaction.id allow you to link logs with traces

Learn about correlating plaintext logs in the agent-specific ingestion guides:

• Go
• Java
• .NET
• Node.js
• Python
• Ruby

View logs

To view logs ingested by Filebeat, go to Discover. Create a data view based on the filebeat-* index pattern. Refer to Create a data view for more information.

To view logs ingested by Elastic Agent, go to Discover and select the Logs Explorer tab. Refer to the Filter and aggregate logs documentation for more on viewing and filtering your log data.