When using ForgeRock ID Cloud (ID Cloud), all the logs (application, debug and audit) are stored in GCP. An API is available for customers to get the log data over REST. It is documented here https://backstage.forgerock.com/docs/idcloud/latest/tenant-audit-logs.html.
The log events are sent in the REST response. While calling the REST API directly is great for a quick round of troubleshooting, if one wants to do some cross ref queries based on
transactionId etc., importing the log events in a tool like Elasticsearch will be of great help.
There already are tools available which make life a bit easier when dealing with ID Cloud logs - like this one - https://github.com/vscheuber/fidc-debug-tools (
The advantage of sending logs to ELK is that they will be available in a central location for everyone to view locally. Also, the advanced query and visualization features can greatly help with monitoring and troubleshooting.
When using ELK to ingest and analyze ID Cloud logs, there are two main components involved.
- The ELK stack which stores, indexes and lets one query and visualize log data
- Something that can make REST API calls to ID Cloud and can be configured to connect to different ID Cloud tenants or different ELK stack. filebeat is used for this.
Here is what the architecture looks like:
How to use
2. Follow the steps from https://elk-docker.readthedocs.io/#prerequisites
1. First, export environment variables to point to a ID Cloud tenant. Best practice is to create separate
env files specific to each environment. An example
.env.customername file could be:
export FIDC_ORIGIN="https://<tenant url>" export FIDC_API_KEY_ID="67xxxxxxxxxxxxxxxxxxxxxxxxxxx221" export FIDC_API_KEY_SECRET="acxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxd6" export FIDC_LOG_SOURCE="idm-sync,am-access,am-authentication,idm-core,am-core" export FIDC_LOG_START_TIME="2021-03-10T00:00:00Z" # optional: only needed when pulling historical logs (not tailing) export FIDC_PULL_INTERVAL="10s" # optional: default 10s (this is only used when FIDC_LOG_START_TIME is used) export FIDC_LOG_REQUEST_TIMEOUT="1m" # optional: default 1m
Or, download env.tenantname
If FIDC_LOG_START_TIME is set, filebeat will not "tail" the logs, instead it will start pulling logs from the specified instance in past. If you need to tail as well as pull historical logs, you can start a separate filebeat container without the FIDC_LOG_START_TIME variable set. How to do that is not covered in this doc.
LOG_SOURCE can be a comma separated list of any of the following:
am-access am-activity am-authentication am-config am-core am-everything ctsstore ctsstore-access ctsstore-config-audit ctsstore-upgrade idm-access idm-activity idm-authentication idm-config idm-core idm-everything idm-sync userstore userstore-access userstore-config-audit userstore-ldif-importer userstore-upgrade
2. Source the environment file
$ source .env.tenantname
3. To run the ELK container, use the following code to create a
docker-compose.yml on your machine.
version: '3.7' services: elk: image: sandeepc0/elk-fidc container_name: elk-forgerock-idcloud environment: - ES_HEAP_SIZE="4G" - LOGSTASH_START="0" ports: - "5601:5601" - "9200:9200" - "5044:5044" filebeat: image: sandeepc0/filebeat-fidc container_name: filebeat-forgerock-idcloud environment: - TOGGLE_FOR_RESTART="no" - FIDC_ORIGIN - FIDC_API_KEY_ID - FIDC_API_KEY_SECRET - FIDC_LOG_SOURCE - FIDC_LOG_START_TIME # optional: only needed when pulling historical logs (not tailing) - FIDC_PULL_INTERVAL # optional: default 10s (this is only used when FIDC_LOG_START_TIME is used) - FIDC_LOG_REQUEST_TIMEOUT # optional: default 1m
Or, download docker-compose.yml
4. Then, to start the containers, from the directory where the above downloaded
docker-compose.yml is, run the following to start the stack.
$ docker-compose up -d # you can also run "docker-compose -p <project name> up -d" to manage as a docker-compose project
5. Wait for a a minute or two for the stack to start up. Optionally, you can tail the container logs to check when the services have started successfully.
6. After that, go to http://localhost:5601/
7. Click the sandwich menu icon and select "Discover".
The reason for using a Canvas vs a Dashboard is because the data (transactionId) in the Canvas is clickable.
Clicking on a
transactionId in one of the tables in the Canvas will take you to the Discover app filtered for that
Log event fields
In the logs events, JSON formatted logs are available in two fields:
json_payload: contains events which are logged in JSON format by ForgeRock
text_payload: contains events which are logged as plain text.
This information can be used to further query data and create saved queries and visualizations etc in Kibana.
The ELK stack
This is implemented as a docker container consisting of Elasticsearch, Logstash and Kibana. The image is available at: https://hub.docker.com/repository/docker/sandeepc0/elk-fidc (this is based on https://hub.docker.com/r/sebp/elk/).
The ELK stack (specially Elasticsearch) can require substantial system resources, specially if large number of records (which log data tends to have) are involved. Please follow the prerequisite steps to prepare your environment.
"Filebeat is a lightweight shipper for forwarding and centralizing log data"
Filebeat has an httpjson input which is used here instead of
httpjson input supports authentication, paged results (with page cookie) and also honors rate limit headers from ID Cloud API.
To use filebeat with ID Cloud, we use the following basic configuration. This is reproduced here just for explanation, you do not need to create this as the docker container takes care of this.
filebeat.inputs: - type: httpjson config_version: 2 request.url: https://<tenant host>/monitoring/logs/tail request.transforms: - set: target: header.Authorization value: 'Basic <base64 encoded(api_key:api_secret)>' - set: target: url.params.source value: 'am-authentication' - set: target: url.params._pagedResultsCookie value: '[[.last_response.body.pagedResultsCookie]]' request.rate_limit: limit: '[[.last_response.header.Get "x-ratelimit-limit"]]' remaining: '[[.last_response.header.Get "x-ratelimit-remaining"]]' reset: '[[.last_response.header.Get "x-ratelimit-reset"]]' response.split: target: body.result type: array output.logstash: hosts: ["localhost:5044"] # output.console: # codec.json: # pretty: true
httpjson input and sends Basic HTTP authentication header with the request. The above is configured to send the output to Logstash running on localhost (in docker), at port 5044. Optionally, it can also output to console (commented out).
Docker hub links:
- ELK image: https://hub.docker.com/repository/docker/sandeepc0/elk-fidc (https://github.com/sandman0/elk-docker)
- Filebeat image: https://hub.docker.com/repository/docker/sandeepc0/filebeat-fidc (https://github.com/sandman0/filebeat-docker)
- Added capability to pull logs from the past (not tailing)
- Added configurable API request timeout (default 1m)
- Added configurable interval between calls (does not apply to tail mode)
- 16-Mar-2021 - Added ingest pipeline to extract GeoIP and User Agent data from (applicable) log entries
- 15-Mar-2021 - Disabled logstash and automatic creation of index pattern and Canvas in Kibana
- 02-Mar-2021 - Using filebeat to pull logs from ForgeRock ID Cloud
- 22-Feb-2021 - Moved from OSS versions to "Basic" licensed ones. Also filtered out IDM "ping" entries.
- 18-Dec-2020 - Sync'ed the tail script with upstream (https://github.com/vscheuber/fidc-debug-tools), fixing the issue where the
pagedResultsCookiewas getting reset in case of an error response from the log API.