Configuration Parameters

Various types of configuration parameters explained with their types, descriptions, and default values

This document describes the various configuration parameters for the config.toml file. You can fine-tune them to suit your application's needs.

Global Parameters

No.

Parameter name

Type

Description

Default

1

maxProcess

int

Number of parallel threads used in server. Should be set to number of cores.

12

2

gwDBRetentionInHr

int64

Time in hours

The amount of time events are stored in gateway database after they have been processed

0

3

routerDBRetention

int64

Time in seconds

The amount of time events are stored in router database after they have been processed

0

4

enableProcessor

bool

A boolean that enables or disables the processor module. Will be set to false when running in degraded mode

true

5

enableRouter

bool

A boolean that enables or disables the router module. Will be set to false when running in degraded mode

true

6

enableStats

bool

A boolean that enables or disables stats.

true

[Gateway]

No.

Parameter name

Type

Description

Default

1

webPort

int

The port on which the server runs.

8080

2

maxDBWriterProcess

int

As requests come in to the gateway and are batched, maxDBWriterProcess writers are run to send these batches to db and config-backend

64

3

CustomVal

string

For creation of a job in jobsDB, CustomVal i.e.GW will be assigned for the CustomVal column in jobsDB

"GW"

4

maxBatchSize

int

Batch size used in Gateway. Requests are batched up to size maxBatchSize before writing to DB.

32

5

batchTimeoutInMS

int64

Time in MilliSeconds

In case the request batches do not meet the maxBatchSize, every 20 milliseconds these request batches are are sent to db and config-backend.

20

6

maxReqSizeInKB

int

One example where an error message ("Request size exceeds max limit") is thrown for a particular request is when its size in KB crosses maxReqSizeInKB.

100000

7

enableDedup

bool

Enable or disable deduplication of events. We use message_id to de-dup. Duplicate events are dropped at the gateway.

false

8

dedupWindowInS

int

Time in Seconds

Events with the same message_id with in this time frame are considered duplicate and are dropped

86400

9

enableRateLimit

bool

Rate limit the number of requests accepted by the gateway. Used for running hosted service

false

[SourceDebugger]

No.

Parameter name

Type

Description

Default

1

disableEventUploads

bool

A bool to enable or disable event schema upload.

false

2

maxBatchSize

int

Maximum size of the live events batch sent to config-backend.

32

3

maxESQueueSize

int

Maximum size of the live events queue in the memory.

1024

4

maxRetry

int

Maximum number of attempts by the server to upload the request batches, in case of errors.

3

5

batchTimeoutInS

int64

Time in seconds

In case the request batches do not meet the maxBatchSize, every 2 seconds these request batches are uploaded to config-backend.

2

6

retrySleepInMS

int64

Time in milliseconds

In case of error while uploading the request batches, the server instead of retrying continuously for maxRetry times will sleep for 100 milliseconds before retrying.

100

[JobsDB]

No.

Parameter name

Type

Description

Default

1

jobDoneMigrateThres

float64

If (deletedJobsCount / totalJobsCount) > jobDoneMigrateThres, migrate the table of the jobs that have been processed.

0.8

2

jobStatusMigrateThres

float64

If (statusCount / totalCount) > jobStatusMigrateThres,

migrate the table of jobs that have been processed.

5

3

maxDSSize

int

The maximum size of a table. If the maximum size is reached, migrate all the jobs in that table.

100000

4

maxMigrateOnce

int

The maximum number of tables that can be migrated together.

10

5

maxTableSizeInMB

int

Maximum size of tables in MB. Tables are migrated if they cross this limit.

300

6

mainCheckSleepDurationInS

int64

Time in seconds Time to sleep before adding or migrating a dataset.

2

7

backupCheckSleepDurationIns

int64

Time in seconds Time interval to sleep before checking if a dataset needs backing up or not.

2

8

enableBackup

bool

Enables/Disables backup. This is set to false in degraded mode.

true

[Router]

No.

Parameter name

Type

Description

Default

1

jobQueryBatchSize

int

The size of a batch of jobs to get from database. A list of retry list, processing list or unprocessed list or executed list.

10000

2

updateStatusBatchSize

int

The minimum size needed to update the status of a batch of jobs.

1000

3

readSleepInMS

int64

Time in milliseconds The time to wait for before the next iteration of the loop. In case the length of unprocessed and retry list is zero, sleep for 10 milliseconds. Else add it to another list.

10

4

noOfWorkers

int

The total number of workers that are started to send events to destinations.

64

5

noOfJobsPerChannel

int

The number of jobs a channel in each worker can contain. So, each worker gets 1000 jobs.

1000

6

ser

int

3

7

maxSleepInS

int64

Time in seconds The time to sleep when response status code is not 200. This is to give time before the next retry.

60

8

minSleepInS

int64

Time in seconds Time to sleep when response status code is 200.

0

9

maxStatusUpdateWaitInS

int64

Time in seconds Time to sleep before ending one round of stat collection.

5

10

randomWorkerAssign

bool

Randomly assign workers that send events to destinations.

false

11

useTestSink

bool

Run internal tests.

false

12

maxFailedCountForJob

int

The maximum number of times a job can fail before marking it as aborted.

8

13

keepOrderOnFailure

bool

Keep the order of jobs on failure.

true

[BatchRouter]

No.

Parameter name

Type

Description

Default

1

mainLoopSleepInS

int64

Timeout in seconds while running main loop.

30

2

noOfWorkers

int

The number of workers to batch jobs before deletion.

8

3

jobQueryBatchSize

int

Number of events picked up from batch router's jobsdb in each query

100000

4

uploadFreqInS

int

The Frequency with which batch router dumps events to storage destinations.

30

5

maxFailedCountForJob

int

The maximum number of times a job can fail before marking it as aborted.

128

[Warehouse]

You can ignore these settings if you don't have any warehouses (like redshift, bigquery, etc.) enabled in your config.

No.

Parameter name

Type

Description

Default

1

stagingFilesTable

string

Table name of staging files.

wh_staging_files

2

loadFilesTable

string

Table name of load files.

wh_load_files

3

uploadsTable

string

Table name of uploads.

wh_uploads

4

schemasTable

string

Table name of schemas.

wh_schemas

5

uploadFreqInS

int

Frequency of upload in seconds

1800

6

noOfWorkers

int

Number of concurrent writes to the warehouse

8

7

mainLoopSleepInS

int

Time in Seconds

Time to wait between multiple warehouse writes

600

8

stagingFilesBatchSize

int

Batch size of staging files

240

[Processor]

No.

Variable name

Type

Description

Default

1

loopSleepInMS

int64

Time in MilliSeconds

In case the length of user jobs process queue is zero or the sum of unprocessed and retry list is empty, sleep for 10 milliseconds is called.

10

2

maxLoopSleepInMS

int64

Max processor loop sleep time in milli seconds

5000

2

dbReadBatchSize

int

The total number of events to get as a batch from the database.

10000

3

transformBatchSize

int

Batch size of events added to request queue before sending to transformation server.

10

4

userTransformBatchSize

int

Batch size of events added to request queue before sending to custom transformation server. Only used when user transformation functions are connected to a destination.

200

5

sessionThresholdEvents

int

The minimum number of events needed to be process further.

20

6

sessionThresholdInS

int64

Time in seconds Minimum time needed before a new session is created.

10

7

maxChanSize

int

The maximum channel size for request and response queue in transformer.

2048

8

processSessions

bool

If processSessions is marked true, the status of the job in db is updated.

false

9

numTransformWorker

int

The number of go transform workers.

8

10

maxRetry

int

The maximum number of times a transformer can retry hitting the API in case of an error

30

11

retrySleepInMS

int64

Time in milliseconds Sleep time in case of an error by transformer while hitting the API, before trying to hit the API again

100

[BackendConfig]

No.

Variable name

Type

Description

Default

1

pollIntervalInS

int64

Time in seconds Frequency of updating data from config-backend .

5

2

configFromFile

bool

Reads backend workspace config from a JSON file instead of fetching form api.

false

3

configJSONPath

string

Path of the JSON file which contains the backend workspace config

./workspaceConfig.json

[RateLimiting]

No.

Variable name

Type

Description

Default

1

eventLimit

int64

Max number of events to be allowed in a time interval

1000

2

rateLimitWindowInMins

int32

Time in minutes

Rolling time interval used to limit number of events allowed

60

3

noOfBucketsInWindow

int32

Number of buckets rateLimitWindowInMins is broken down into

12

[Diagnosis]

No.

Parameter name

Type

Description

Default

1

enableDiagnosis

bool

Boolean to send server diagnostics report to us. Disabling this will disable sending all diagnostics information.

true

2

gatewayTimePeriodInS

int32

Time interval to send gateway requests report in seconds

60

3

routerTimePeriodInS

int32

Time interval to send router requests report in seconds

60

4

batchRouterTimePeriodInS

int32

Time interval to send batch router requests report in seconds

600

5

enableServerStartMetric

bool

Boolean to send server start event

true

6

enableConfigIdentifyMetric

bool

Boolean to send workspace config received event

true

7

enableServerStartedMetric

bool

Boolean to send server successfully started event

true

8

enableConfigProcessedMetric

bool

Boolean to send workspace config details

true

9

enableGatewayMetric

bool

Boolean to send gateway request metrics

true

10

enableRouterMetric

bool

Boolean to send router request metrics

true

11

enableBatchRouterMetric

bool

Boolean to send batch router request metrics

true