This document describes the various configuration parameters for the config.toml
file. You can fine-tune them to suit your application's needs.
No. | Parameter name | Type | Description | Default |
1 |
|
| Number of parallel threads used in server. Should be set to number of cores. | 12 |
2 |
|
|
The amount of time events are stored in gateway database after they have been processed | 0 |
3 |
|
|
The amount of time events are stored in router database after they have been processed | 0 |
4 |
|
| A boolean that enables or disables the processor module. Will be set to false when running in degraded mode | true |
5 |
|
| A boolean that enables or disables the router module. Will be set to false when running in degraded mode | true |
6 |
|
| A boolean that enables or disables stats. | true |
No. | Parameter name | Type | Description | Default |
1 |
|
| The port on which the server runs. | 8080 |
2 |
|
| As requests come in to the gateway and are batched, | 64 |
3 |
|
| For creation of a job in jobsDB, | "GW" |
4 |
|
| Batch size used in Gateway. Requests are batched up to size | 32 |
5 |
|
|
In case the request batches do not meet the | 20 |
6 |
|
| One example where an error message ("Request size exceeds max limit") is thrown for a particular request is when its size in KB crosses | 100000 |
7 |
|
| Enable or disable deduplication of events. We use | false |
8 |
|
|
Events with the same message_id with in this time frame are considered duplicate and are dropped | 86400 |
9 |
|
| Rate limit the number of requests accepted by the gateway. Used for running hosted service | false |
No. | Parameter name | Type | Description | Default |
1 |
|
| A bool to enable or disable event schema upload. | false |
2 |
|
| Maximum size of the live events batch sent to config-backend. | 32 |
3 |
|
| Maximum size of the live events queue in the memory. | 1024 |
4 |
|
| Maximum number of attempts by the server to upload the request batches, in case of errors. | 3 |
5 |
|
|
In case the request batches do not meet the | 2 |
6 |
|
|
In case of error while uploading the request batches, the server instead of retrying continuously for | 100 |
No. | Parameter name | Type | Description | Default |
1 |
|
| If ( | 0.8 |
2 |
|
| If ( migrate the table of jobs that have been processed. | 5 |
3 |
|
| The maximum size of a table. If the maximum size is reached, migrate all the jobs in that table. | 100000 |
4 |
|
| The maximum number of tables that can be migrated together. | 10 |
5 |
|
| Maximum size of tables in MB. Tables are migrated if they cross this limit. | 300 |
6 |
|
|
| 30 |
7 |
|
| The maximum number of tables probed to find if they can be migrated. | 10 |
8 |
|
|
| 5 |
9 |
|
|
| 2 |
10 |
|
| Enables/Disables backup. This is set to false in degraded mode. | true |
No. | Parameter name | Type | Description | Default |
1 |
|
| The size of a batch of jobs to get from database. A list of retry list, processing list or unprocessed list or executed list. | 10000 |
2 |
|
| The minimum size needed to update the status of a batch of jobs. | 1000 |
3 |
|
|
| 10 |
4 |
|
| The total number of workers that are started to send events to destinations. | 64 |
5 |
|
| The number of jobs a channel in each worker can contain. So, each worker gets 1000 jobs. | 1000 |
6 |
|
| ​ | 3 |
7 |
|
|
| 60 |
8 |
|
|
| 0 |
9 |
|
|
| 5 |
11 |
|
| Run internal tests. | false |
12 |
|
| The maximum number of times a job can fail before marking it as aborted. | 8 |
13 |
|
| Maintain the user events order | true |
14 |
|
| Minimum retry window in case of errors like 5XX, 429 | 180 |
15 |
|
| Minimum time in seconds before the next retry in case of errors like 5XX, 429 | 10 |
16 |
|
| Maximum time in seconds between errors in case of errors like 5XX, 429 | 300 |
No. | Parameter name | Type | Description | Default |
1 |
|
| Timeout in seconds while running main loop. | 30 |
2 |
|
| The number of workers to batch jobs before deletion. | 8 |
3 |
|
| Number of events picked up from batch router's jobsdb in each query | 100000 |
4 |
|
| The Frequency with which batch router dumps events to storage destinations. | 30 |
5 |
|
| The maximum number of times a job can fail before marking it as aborted. | 128 |
You can ignore these settings if you don't have any warehouses (like redshift, bigquery, etc.) enabled in your config.
​
No. | Parameter name | Type | Description | Default |
1 | stagingFilesTable |
| Table name of staging files. | wh_staging_files |
2 | loadFilesTable |
| Table name of load files. | wh_load_files |
3 | uploadsTable |
| Table name of uploads. | wh_uploads |
4 | schemasTable |
| Table name of schemas. | wh_schemas |
5 | uploadFreqInS |
| Frequency of upload in seconds | 1800 |
6 | noOfWorkers |
| Number of concurrent writes to the warehouse | 8 |
7 | mainLoopSleepInS |
|
Time to wait between multiple warehouse writes | 600 |
8 | stagingFilesBatchSize |
| Batch size of staging files | 240 |
No. | Variable name | Type | Description | Default |
1 |
|
|
In case the length of user jobs process queue is zero or the sum of unprocessed and retry list is empty, sleep for 10 milliseconds is called. | 10 |
2 |
|
| Max processor loop sleep time in milli seconds | 5000 |
2 |
|
| The total number of events to get as a batch from the database. | 10000 |
3 |
|
| Batch size of events added to request queue before sending to transformation server. | 10 |
4 |
|
| Batch size of events added to request queue before sending to custom transformation server. Only used when user transformation functions are connected to a destination. | 200 |
5 |
|
| The minimum number of events needed to be process further. | 20 |
6 |
|
|
| 10 |
7 |
|
| The maximum channel size for request and response queue in transformer. | 2048 |
8 |
|
| If | false |
9 |
|
| The number of go transform workers. | 8 |
10 |
|
| The maximum number of times a transformer can retry hitting the API in case of an error | 30 |
11 |
|
|
| 100 |
No. | Variable name | Type | Description | Default |
1 |
|
|
| 5 |
2 |
|
| Reads backend workspace config from a JSON file instead of fetching form api. | false |
3 |
|
| Path of the JSON file which contains the backend workspace config | ./workspaceConfig.json |
No. | Variable name | Type | Description | Default |
1 |
|
| Max number of events to be allowed in a time interval | 1000 |
2 |
|
|
Rolling time interval used to limit number of events allowed | 60 |
3 |
|
| Number of buckets | 12 |
No. | Parameter name | Type | Description | Default |
1 |
|
| Boolean to send server diagnostics report to us. Disabling this will disable sending all diagnostics information. | true |
2 |
|
| Time interval to send gateway requests report in seconds | 60 |
3 |
|
| Time interval to send router requests report in seconds | 60 |
4 |
|
| Time interval to send batch router requests report in seconds | 600 |
5 |
|
| Boolean to send server start event | true |
6 |
|
| Boolean to send workspace config received event | true |
7 |
|
| Boolean to send server successfully started event | true |
8 |
|
| Boolean to send workspace config details | true |
9 |
|
| Boolean to send gateway request metrics | true |
10 |
|
| Boolean to send router request metrics | true |
11 |
|
| Boolean to send batch router request metrics | true |
12 |
|
| Boolean to send destination failures metrics | true |