Configuration Parameters

Various types of configuration parameters explained with their types, descriptions, and default values

This document describes the various configuration parameters for the config.toml file. You can fine-tune them to suit your application's needs.
Global Parameters
No.
Parameter name
Type
Description
Default
1
maxProcess
int
Number of parallel threads used in server. Should be set to number of cores.
12
2
gwDBRetentionInHr
int64
Time in hours
The amount of time events are stored in gateway database after they have been processed
0
3
routerDBRetention
int64
Time in seconds
The amount of time events are stored in router database after they have been processed
0
4
enableProcessor
bool
A boolean that enables or disables the processor module. Will be set to false when running in degraded mode
true
5
enableRouter
bool
A boolean that enables or disables the router module. Will be set to false when running in degraded mode
true
6
enableStats
bool
A boolean that enables or disables stats. 
true
[Gateway]
No.
Parameter name
Type
Description
Default
1
webPort
int
The port on which the server runs.
8080
2
maxDBWriterProcess
int
As requests come in to the gateway and are batched, maxDBWriterProcess writers are run to send these batches to db and config-backend
64
3
CustomVal
string
For creation of a job in jobsDB, CustomVal  i.e.GW will be assigned for the CustomVal column in jobsDB
"GW"
4
maxBatchSize
int
Batch size used in Gateway. Requests are batched up to size maxBatchSize before writing to DB.
32
5
batchTimeoutInMS
int64
Time in MilliSeconds
In case the request batches do not meet the maxBatchSize, every 20 milliseconds these request batches are are sent to db and config-backend.
20
6
maxReqSizeInKB
int
One example where an error message ("Request size exceeds max limit") is thrown for a particular request is when its size in KB crosses maxReqSizeInKB.
100000
7
enableDedup
bool
Enable or disable deduplication of events. We use message_id to de-dup. Duplicate events are dropped at the gateway. 
false
8
dedupWindowInS
int
Time in Seconds
Events with the same message_id with in this time frame are considered duplicate and are dropped
86400
9
enableRateLimit
bool
Rate limit the number of requests accepted by the gateway. Used for running hosted service
false
[SourceDebugger]
No.
Parameter name
Type
Description
Default
1
disableEventUploads
bool
A bool to enable or disable event schema upload.
false
2
maxBatchSize
int
Maximum size of the live events batch sent to config-backend.
32
3
maxESQueueSize
int
Maximum size of the live events queue in the memory.
1024
4
maxRetry
int
Maximum number of attempts by the server to upload the request batches, in case of errors.
3
5
batchTimeoutInS
int64
Time in seconds
In case the request batches do not meet the maxBatchSize, every 2 seconds these request batches are uploaded to config-backend.
2
6
retrySleepInMS
int64
Time in milliseconds
In case of error while uploading the request batches, the server instead of retrying continuously for maxRetry times will sleep for 100 milliseconds before retrying.
100
[JobsDB]
No.
Parameter name
Type
Description
Default
1
jobDoneMigrateThres
float64
If (deletedJobsCount / totalJobsCount) > jobDoneMigrateThres, migrate the table of the jobs that have been processed.
0.8
2
jobStatusMigrateThres
float64
If (statusCount / totalCount) > jobStatusMigrateThres, 
migrate the table of jobs that have been processed.
5
3
maxDSSize
int
The maximum size of a table. If the maximum size is reached, migrate all the jobs in that table.
100000
4
maxMigrateOnce
int
The maximum number of tables that can be migrated together.
10
5
maxTableSizeInMB
int
Maximum size of tables in MB. Tables are migrated if they cross this limit.
300
6
migrateDSLoopSleepDurationInS
int64
Time in seconds 
Time to sleep before migrating datasets.
30
7
maxMigrateDSProbe
int64
The maximum number of tables probed to find if they can be migrated.
10
8
addNewDSLoopSleepDurationInS
int64
Time in seconds 
Time to sleep before adding a dataset.
5
9
backupCheckSleepDurationIns
int64
Time in seconds 
Time interval to sleep before checking if a dataset needs backing up or not.
2
10
enableBackup
bool
Enables/Disables backup. This is set to false in degraded mode. 
true
[Router]
No.
Parameter name
Type
Description
Default
1
jobQueryBatchSize
int
The size of a batch of jobs to get from database. A list of retry list, processing list or unprocessed list or executed list.
10000
2
updateStatusBatchSize
int
The minimum size needed to update the status of a batch of jobs.
1000
3
readSleepInMS
int64
Time in milliseconds 
The time to wait for before the next iteration of the loop. 
In case the length of unprocessed and retry list is zero, sleep for 10  milliseconds. Else add it to another list. 
10
4
noOfWorkers
int
The total number of workers that are started to send events to destinations. 
64
5
noOfJobsPerChannel
int 
The number of jobs a channel in each worker can contain. So, each worker gets 1000 jobs.
1000
6
ser
int
​
3
7
maxSleepInS
int64
Time in seconds 
The time to sleep when response status code is not 200. This is to give time before the next retry.
60
8
minSleepInS
int64
Time in seconds 
Time to sleep when response status code is 200.
0
9
maxStatusUpdateWaitInS
int64
Time in seconds 
Time to sleep before ending one round of stat collection.
5
10
randomWorkerAssign
bool
Randomly assign workers that send events to destinations.
false
11
useTestSink
bool
Run internal tests.
false
12
maxFailedCountForJob
int
The maximum number of times a job can fail before marking it as aborted.
8
13
keepOrderOnFailure
bool
Keep the order of jobs on failure.
true
14
retryTimeWindowInMins
int
Minimum retry window in case of errors like 5XX, 429
180
15
minRetryBackoffInS
int
Minimum time in seconds before the next retry in case of errors like 5XX, 429
10
16
maxRetryBackoffInS
int
Maximum time in seconds between errors in case of errors like 5XX, 429
300
[BatchRouter]
No.
Parameter name
Type
Description
Default
1
mainLoopSleepInS
int64
Timeout in seconds while running main loop.
30
2
noOfWorkers
int
The number of workers to batch jobs before deletion.
8
3
jobQueryBatchSize
int
Number of events picked up from batch router's jobsdb in each query 
100000
4
uploadFreqInS
int
The Frequency with which batch router dumps events to storage destinations.
30
5
maxFailedCountForJob
int
The maximum number of times a job can fail before marking it as aborted.
128
[Warehouse]
You can ignore these settings if you don't have any warehouses (like redshift, bigquery, etc.) enabled in your config.
​
No.
Parameter name
Type
Description
Default
1
stagingFilesTable
string
Table name of staging files.
wh_staging_files
2
loadFilesTable
string
Table name of load files.
wh_load_files
3
uploadsTable
string
Table name of uploads.
wh_uploads
4
schemasTable
string
Table name of schemas.
wh_schemas
5
uploadFreqInS
int
Frequency of upload in seconds
1800
6
noOfWorkers
int
Number of concurrent writes to the warehouse
8
7
mainLoopSleepInS
int
Time in Seconds 
Time to wait between multiple warehouse writes
600
8
stagingFilesBatchSize
int
Batch size of staging files
240
[Processor]
No.
Variable name
Type
Description
Default
1
loopSleepInMS
int64
Time in MilliSeconds
In case the length of user jobs process queue is zero or the sum of unprocessed and retry list is empty, sleep for 10 milliseconds is called.
10
2
maxLoopSleepInMS
int64
Max processor loop sleep time in milli seconds
5000
2
dbReadBatchSize
int
The total number of events to get as a batch from the database.
10000
3
transformBatchSize
int
Batch size of events added to request  queue before sending to transformation server.
10
4
userTransformBatchSize
int
Batch size of events added to request queue before sending to custom transformation server. Only used when user transformation functions are connected to a destination.
200
5
sessionThresholdEvents
int
The minimum number of events needed to be process further.
20
6
sessionThresholdInS
int64
Time in seconds 
Minimum time needed before a new session is created.
10
7
maxChanSize
int
The maximum channel size for request and response queue in transformer.
2048
8
processSessions
bool
 If processSessions is marked true, the status of the job in db is updated.
false
9
numTransformWorker
int 
The number of go transform workers.
8
10
maxRetry
int
The maximum number of times a transformer can retry hitting the API in case of an error
30
11
retrySleepInMS
int64
Time in milliseconds
Sleep time in case of an error by transformer while hitting the API, before trying to hit the API again
100
[BackendConfig]
No.
Variable name
Type
Description
Default
1
pollIntervalInS
int64
Time in seconds
Frequency of updating data from config-backend . 
5
2
configFromFile
bool
Reads backend workspace config from a JSON file instead of fetching form api.
false
3
configJSONPath
string
Path of the JSON file which contains the backend workspace config
./workspaceConfig.json
[RateLimiting]
No.
Variable name
Type
Description
Default
1
eventLimit
int64
Max number of events to be allowed in a time interval 
1000
2
rateLimitWindowInMins
int32
Time in minutes 
Rolling time interval used to limit number of events allowed
60
3
noOfBucketsInWindow
int32
Number of buckets rateLimitWindowInMins is broken down into 
12
[Diagnosis]
No.
Parameter name
Type
Description
Default
1
enableDiagnosis
bool
Boolean to send server diagnostics report to us. Disabling this will disable sending all diagnostics information.
true
2
gatewayTimePeriodInS
int32
Time interval to send gateway requests report in seconds
60
3
routerTimePeriodInS
int32
Time interval to send router requests report in seconds
60
4
batchRouterTimePeriodInS
int32
Time interval to send batch router requests report in seconds
600
5
enableServerStartMetric
bool
Boolean to send server start event
true
6
enableConfigIdentifyMetric
bool
Boolean to send workspace config received event
true
7
enableServerStartedMetric
bool
Boolean to send server successfully started event
true
8
enableConfigProcessedMetric
bool
Boolean to send workspace config details
true
9
enableGatewayMetric
bool
Boolean to send gateway request metrics
true
10
enableRouterMetric
bool
Boolean to send router request metrics
true
11
enableBatchRouterMetric
bool
Boolean to send batch router request metrics
true
12
enableDestinationFailuresMetric
bool
Boolean to send destination failures metrics
true

Monitoring and Metrics

Troubleshooting Guide

Last updated 4 weeks ago

No.	Parameter name	Type	Description	Default
1	`maxProcess`	`int`	Number of parallel threads used in server. Should be set to number of cores.	12
2	`gwDBRetentionInHr`	`int64`	`Time in hours` The amount of time events are stored in gateway database after they have been processed	0
3	`routerDBRetention`	`int64`	`Time in seconds` The amount of time events are stored in router database after they have been processed	0
4	`enableProcessor`	`bool`	A boolean that enables or disables the processor module. Will be set to false when running in degraded mode	true
5	`enableRouter`	`bool`	A boolean that enables or disables the router module. Will be set to false when running in degraded mode	true
6	`enableStats`	`bool`	A boolean that enables or disables stats.	true

No.	Parameter name	Type	Description	Default
1	`webPort`	`int`	The port on which the server runs.	8080
2	`maxDBWriterProcess`	`int`	As requests come in to the gateway and are batched, `maxDBWriterProcess` writers are run to send these batches to db and config-backend	64
3	`CustomVal`	`string`	For creation of a job in jobsDB, `CustomVal` i.e.`GW` will be assigned for the `CustomVal` column in jobsDB	"GW"
4	`maxBatchSize`	`int`	Batch size used in Gateway. Requests are batched up to size `maxBatchSize` before writing to DB.	32
5	`batchTimeoutInMS`	`int64`	`Time in MilliSeconds` In case the request batches do not meet the `maxBatchSize`, every 20 milliseconds these request batches are are sent to db and config-backend.	20
6	`maxReqSizeInKB`	`int`	One example where an error message ("Request size exceeds max limit") is thrown for a particular request is when its size in KB crosses `maxReqSizeInKB`.	100000
7	`enableDedup`	`bool`	Enable or disable deduplication of events. We use `message_id` to de-dup. Duplicate events are dropped at the gateway.	false
8	`dedupWindowInS`	`int`	`Time in Seconds` Events with the same message_id with in this time frame are considered duplicate and are dropped	86400
9	`enableRateLimit`	`bool`	Rate limit the number of requests accepted by the gateway. Used for running hosted service	false

No.	Parameter name	Type	Description	Default
1	`disableEventUploads`	`bool`	A bool to enable or disable event schema upload.	false
2	`maxBatchSize`	`int`	Maximum size of the live events batch sent to config-backend.	32
3	`maxESQueueSize`	`int`	Maximum size of the live events queue in the memory.	1024
4	`maxRetry`	`int`	Maximum number of attempts by the server to upload the request batches, in case of errors.	3
5	`batchTimeoutInS`	`int64`	`Time in seconds` In case the request batches do not meet the `maxBatchSize`, every 2 seconds these request batches are uploaded to config-backend.	2
6	`retrySleepInMS`	`int64`	`Time in milliseconds` In case of error while uploading the request batches, the server instead of retrying continuously for `maxRetry` times will sleep for 100 milliseconds before retrying.	100

No.	Parameter name	Type	Description	Default
1	`jobDoneMigrateThres`	`float64`	If (`deletedJobsCount` / `totalJobsCount`) > `jobDoneMigrateThres`, migrate the table of the jobs that have been processed.	0.8
2	`jobStatusMigrateThres`	`float64`	If (`statusCount` / `totalCount`) > `jobStatusMigrateThres`, migrate the table of jobs that have been processed.	5
3	`maxDSSize`	`int`	The maximum size of a table. If the maximum size is reached, migrate all the jobs in that table.	100000
4	`maxMigrateOnce`	`int`	The maximum number of tables that can be migrated together.	10
5	`maxTableSizeInMB`	`int`	Maximum size of tables in MB. Tables are migrated if they cross this limit.	300
6	`migrateDSLoopSleepDurationInS`	`int64`	`Time in seconds` Time to sleep before migrating datasets.	30
7	`maxMigrateDSProbe`	`int64`	The maximum number of tables probed to find if they can be migrated.	10
8	`addNewDSLoopSleepDurationInS`	`int64`	`Time in seconds` Time to sleep before adding a dataset.	5
9	`backupCheckSleepDurationIns`	`int64`	`Time in seconds` Time interval to sleep before checking if a dataset needs backing up or not.	2
10	`enableBackup`	`bool`	Enables/Disables backup. This is set to false in degraded mode.	true

No.	Parameter name	Type	Description	Default
1	`jobQueryBatchSize`	`int`	The size of a batch of jobs to get from database. A list of retry list, processing list or unprocessed list or executed list.	10000
2	`updateStatusBatchSize`	`int`	The minimum size needed to update the status of a batch of jobs.	1000
3	`readSleepInMS`	`int64`	`Time in milliseconds` The time to wait for before the next iteration of the loop. In case the length of unprocessed and retry list is zero, sleep for 10 milliseconds. Else add it to another list.	10
4	`noOfWorkers`	`int`	The total number of workers that are started to send events to destinations.	64
5	`noOfJobsPerChannel`	`int`	The number of jobs a channel in each worker can contain. So, each worker gets 1000 jobs.	1000
6	`ser`	`int`		3
7	`maxSleepInS`	`int64`	`Time in seconds` The time to sleep when response status code is not 200. This is to give time before the next retry.	60
8	`minSleepInS`	`int64`	`Time in seconds` Time to sleep when response status code is 200.	0
9	`maxStatusUpdateWaitInS`	`int64`	`Time in seconds` Time to sleep before ending one round of stat collection.	5
10	`randomWorkerAssign`	`bool`	Randomly assign workers that send events to destinations.	false
11	`useTestSink`	`bool`	Run internal tests.	false
12	`maxFailedCountForJob`	`int`	The maximum number of times a job can fail before marking it as aborted.	8
13	`keepOrderOnFailure`	`bool`	Keep the order of jobs on failure.	true
14	`retryTimeWindowInMins`	`int`	Minimum retry window in case of errors like 5XX, 429	180
15	`minRetryBackoffInS`	`int`	Minimum time in seconds before the next retry in case of errors like 5XX, 429	10
16	`maxRetryBackoffInS`	`int`	Maximum time in seconds between errors in case of errors like 5XX, 429	300

No.	Variable name	Type	Description	Default
1	`loopSleepInMS`	`int64`	`Time in MilliSeconds` In case the length of user jobs process queue is zero or the sum of unprocessed and retry list is empty, sleep for 10 milliseconds is called.	10
2	`maxLoopSleepInMS`	`int64`	Max processor loop sleep time in milli seconds	5000
2	`dbReadBatchSize`	`int`	The total number of events to get as a batch from the database.	10000
3	`transformBatchSize`	`int`	Batch size of events added to request queue before sending to transformation server.	10
4	`userTransformBatchSize`	`int`	Batch size of events added to request queue before sending to custom transformation server. Only used when user transformation functions are connected to a destination.	200
5	`sessionThresholdEvents`	`int`	The minimum number of events needed to be process further.	20
6	`sessionThresholdInS`	`int64`	`Time in seconds` Minimum time needed before a new session is created.	10
7	`maxChanSize`	`int`	The maximum channel size for request and response queue in transformer.	2048
8	`processSessions`	`bool`	If `processSessions` is marked true, the status of the job in db is updated.	false
9	`numTransformWorker`	`int`	The number of go transform workers.	8
10	`maxRetry`	`int`	The maximum number of times a transformer can retry hitting the API in case of an error	30
11	`retrySleepInMS`	`int64`	`Time in milliseconds` Sleep time in case of an error by transformer while hitting the API, before trying to hit the API again	100