Agent configuration¶

One JSON file: /usr/local/etc/tis-agent.json on FreeBSD, /etc/tis-agent/config.json on Linux (-config names another). Keep it owned by root with mode 600 – it holds the token. Unknown keys are errors, so a typo doesn't silently change nothing; tis-agent check validates the file, and start/reload of the service do too.

Durations are strings such as "30s", "5m" or "1h30m". Commands are lists – the program, then its arguments: ["rndc", "reload"]; they run without a shell (for pipes or redirections use ["sh", "-c", "…"]).

A complete example for a FreeBSD firewall with BIND:

{
  "platform": { "url": "https://api.scrany.net", "token": "<token>" },
  "reporter": "fw-01",
  "targets": {
    "rpz":       { "type": "bind", "path": "/usr/local/etc/namedb/tis.rpz" },
    "blocklist": { "type": "ipfw-table", "table": "tis-blocklist", "rule": 1000 }
  },
  "jobs": [
    { "name": "rpz", "every": "5m",
      "pull": { "list": "customer-x", "format": "rpz", "type": "fqdn", "target": "rpz" },
      "healthcheck": "https://hc-ping.com/<uuid>" },
    { "name": "blocklist", "every": "10m",
      "pull": { "list": "customer-x", "type": "ipv4", "target": "blocklist" } },
    { "name": "bruteforce", "every": "15m",
      "push": { "source": "fw-01-bruteforce", "command": ["pfctl", "-t", "bruteforce", "-T", "show"], "tag": "ssh" } }
  ],
  "hits": {
    "logs": [ { "format": "rpz", "path": "/var/log/named/rpz.log", "zones": ["tis.rpz"] } ],
    "divert": { "port": 8888, "table": "tis-blocklist" }
  }
}

Top level¶

Key	Default	Description
`platform`		How to reach the platform (below)
`reporter`	hostname	This machine's name in the hit statistics
`state_dir`	`/var/db/tis-agent` (FreeBSD), `/var/lib/tis-agent`	ETags, loaded tables, job results, log positions
`log_timestamps`	`true`	Date and time in log lines (`-log-timestamps` overrides it; the services turn it off as syslog and the journal add their own)
`targets`		Named local destinations of lists (below)
`jobs`		What to do and how often (below)
`hits`		Hit counting; omitted = off (below)
`checkin`	`{ "enabled": true }`	Reporting the state to the platform (below)
`managed`	`false`	Pull jobs from the platform (below)
`fail2ban`		Send fail2ban's bans (below)
`healthchecks`		Healthcheck URLs of central jobs, by job name (below)

platform¶

Key	Default	Description
`url`	`https://api.scrany.net`	Platform URL, without `/api/v1` (or `TIS_URL`)
`token`	`TIS_TOKEN`	API token, without `Bearer`
`insecure`	`false`	Skip TLS certificate verification
`ca_file`		Extra CA certificates (PEM) to verify the platform with
`timeout`	`5m`	Limit of a whole request
`connect_timeout`	`15s`	Limit of connecting
`retries`	`3`	Attempts of a request (network errors, HTTP 408/429/5xx)
`retry_delay`	`5s`	Wait before a retry, times the attempt number

An empty url or token in the file leaves the default or the environment's.

A 429 (e.g. the export limit per token and minute) or 503 with Retry-After is retried after the time asked for, up to a minute; a longer one ends the attempts, and the next scheduled run follows with the backoff.

targets¶

A target is a named local destination; only this file defines them – the platform never says what is written where or which command runs.

`type`	Keys	Defaults
`file`	`path`, `check`, `reload`, `revert`	–
`bind`	as `file`	format `zone`: `named-checkconf {file}`, `rndc reconfig`; format `rpz`: `named-checkzone -q rpz.invalid {file}`, `rndc reload`
`unbound`	as `file`	`unbound-checkconf {file}`, `unbound-control reload`
`dnsmasq`	as `file`	`dnsmasq --test --conf-file={file}`, `service dnsmasq restart`
`ipfw-table`	`table`, `create_options`, `rule`, `legacy`, `reload`	`create_options` `["type", "addr"]`
`pf-table`	`table`, `reload`	–
`ipset`	`table`, `create_options`, `reload`	`create_options` `hash:net family inet` (or `inet6`) with room for the list
`nftables-set`	`family`, `table`, `set`, `reload`	`family` `inet`

Commands given replace the defaults; a target belongs to one job.

Files¶

path is absolute and its directory must exist. A deployment:

compares the download with the file – when they differ only in the generation time and the SOA serial, nothing happens;
refuses an empty download over a list that isn't empty, unless the job has allow_empty;
writes the new list next to the file and runs check on it – {file} stands for the new file and is appended when no argument has it; a failed check leaves everything as it was (exit code 3);
moves it in place atomically, keeping the file's mode and owner (0644 for a new file);
runs reload ({file} = the deployed file); when it fails, the previous content is put back – an empty file when there was none – and revert runs, or reload once more (exit code 2).

Firewall tables¶

Only IP addresses and networks of the download are loaded (an export of type all also carries domains), deduplicated; a table is loaded only when its list changed or it doesn't exist, and created when missing.

ipfw-table (FreeBSD): fills <table>_TEMP and swaps it in. rule adds deny log logamount 0 ip from table(<table>) to me with that number when it is missing; legacy supports ipfw without create … missing / or-flush.
pf-table: pfctl -t <table> -T replace in one transaction.
ipset (Linux): ipset restore of a temporary set, then swap; the job's type must be ipv4 or ipv6. Names have at most 26 characters (31 with _TEMP).
nftables-set (Linux): the set set in the table table of the family (inet, ip, ip6, bridge, netdev) is updated in one nft -f transaction, so rules using it see the old or the new list and a failure keeps the old one: a filled set gets only the changes (removed elements deleted, new ones added – the counters of the others go on), otherwise – or when that fails – it is flushed and filled. Networks inside others are left out, as interval sets refuse overlapping elements. A missing set is created with its table as type ipv4_addr; flags interval (or ipv6_addr; with counter when its hits are read); an existing one – declared in nftables.conf together with the rules using it – keeps its definition. When a reload of the ruleset (flush ruleset) emptied the set, the agent notices it and loads the list again. The job's type must be ipv4 or ipv6; for both, use two sets and two jobs.
```
# /etc/nftables.conf
table inet filter {
    set tis_v4 { type ipv4_addr; flags interval; }
    set tis_v6 { type ipv6_addr; flags interval; }
    chain input {
        type filter hook input priority 0;
        ip saddr @tis_v4 drop
        ip6 saddr @tis_v6 drop
    }
}
```
```
"targets": {
  "tis_v4": { "type": "nftables-set", "table": "filter", "set": "tis_v4" },
  "tis_v6": { "type": "nftables-set", "table": "filter", "set": "tis_v6" }
},
"jobs": [
  { "name": "tis_v4", "every": "10m", "pull": { "list": "customer-x", "type": "ipv4", "target": "tis_v4" } },
  { "name": "tis_v6", "every": "10m", "pull": { "list": "customer-x", "type": "ipv6", "target": "tis_v6" } }
]
```

reload runs after a successful load, e.g. to save the set.

jobs¶

Key	Description
`name`	Unique; letters, digits, `_`, `.`, `-` – used by `tis-agent run <name>` and in the state
`every`	Interval, at least `30s`; omitted = only by `tis-agent run`
`pull`	Download an export and deploy it (below)
`push`	Send local content to a source (below)
`healthcheck`	Ping URL (below)

A job has either pull or push.

pull¶

Key	Description
`list`	A list (group) by ID, UUID or name – `/export/group/…`
`sources`	Sources by ID, UUID or name – `/export/private/…`; with neither, the public export
`format`	Export format; default by target: `zone` for `bind`, `unbound`, `dnsmasq`, otherwise `plain`
`type`	`ipv4`, `ipv6`, `fqdn`, `url`, `json` or `all` (default)
`tag`	Only indicators with these tags (comma-separated)
`whitelist`	Values never exported – IPs, CIDRs, domain globs (list)
`country`, `exclude_country`	GeoIP country codes for IP indicators (lists)
`rpz_ip`	With format `rpz`: also `response`, `client` or `nsip` IP triggers
`target`	The target to deploy to
`allow_empty`	Deploy an empty download over a list that isn't empty

The agent sends the ETag of the last download (If-None-Match) as long as the deployed file or table is still exactly what it put there, so an unchanged list costs a 304; a file edited by hand or a flushed table is downloaded and deployed again. tis-agent run -force always downloads.

Combinations that can't work are refused by check: firewall tables need format plain and address types (ipset and nftables sets ipv4 or ipv6), bind needs zone or rpz, unbound and dnsmasq their own formats.

push¶

Key	Description
`source`	The source by ID, UUID or name – `POST /import/…`
`file`	Content from a file (absolute path), or
`url`	from an http(s) URL – downloaded without the token, or
`command`	from a command's standard output (5-minute limit; its error output goes to the log)
`tag`	Tags added to the imported indicators (comma-separated)
`refresh`	Send unchanged content again after this long (default `24h`)

The platform parses the content with the source's format and options – usually an import-only source (without a URL). Content is sent when it changed since the last successful push, and unchanged content again after refresh: the platform prunes indicators by when they were last observed (retention), and each push renews that. A push only adds and renews indicators – values removed locally leave the source with its retention. Empty content is not sent; more than 10 MB is an error. tis-agent run -force always sends.

Healthchecks¶

healthcheck is a Healthchecks.io-style ping URL: before each run GET <url>/start, after it POST <url>/<exit code> (the codes of tis-agent run) with the end of the run's output (100 kB) – the lines it logged, the output of a failed check or reload command, and the error.

Give each job its own check, with a period matching every.
Pings go without the platform token and are retried (10 s per attempt); a failed ping is logged and changes nothing else. The log doesn't show the URL – it holds the check's key.
Dry runs don't ping, and neither does a run that finds the job busy. A run stopped by a shutdown pings only the start.
A query in the URL (e.g. ?create=1 of slug URLs) stays at its end.

hits¶

Key	Default	Description
`interval`	`5m`	How often hits are reported (and the logs read)
`divert`		Count packets ipfw sends to a divert socket (below)
`logs`		Count the blocks logs record (below)
`nftables`		Read the element counters of nftables sets (below)
`nflog`		Count packets nftables rules log to a netlink group (below)
`healthcheck`		Ping URL for the passes over logs and sets (exit code 0 or 1)

Hits are reported to POST /api/v1/hits (API) as reporter, per hour and listed value, in chunks of at most 10 000 values. The token needs the hits rule.

divert¶

Key	Default	Description
`port`	`8888`	Divert port the ipfw rule sends to
`table`	`tis-blocklist`	ipfw table the packets are matched in
`buf_size`	`65535`	Packet buffer, bytes
`reload_interval`	`5m`	How often the table is re-read

Each packet counts for the table entry its source address falls into (longest prefix), or its destination when the source isn't listed; IPv4 and IPv6. A failed report (network error, timeout, 408/429/5xx) is kept for the next one, for at most 24 hours; on stop the remaining counts are sent. A failing divert socket stops only the divert counting.

divert hands the packet to the agent, which doesn't re-inject it, so the rule works as a deny even when the agent isn't running:

ipfw add 4990 divert 8888 ip from 'table(tis-blocklist)' to me

tee sends a copy and lets the original continue, so a separate rule blocks it:

ipfw add 4990 tee 8888 ip from 'table(tis-blocklist)' to me
ipfw add 4991 deny ip from 'table(tis-blocklist)' to me

nftables sets¶

Since tis-agent 0.4.0. On Linux, the counters of the elements of the agent's nftables sets count what the lists blocked – no logging needed:

"hits": { "nftables": ["tis_v4", "tis_v6"] }

The entries are names of nftables-set targets. Every interval the agent reads their elements' counters (nft -j list set) and reports what they grew by since the last read, for the element – the listed address or network, so the counts match the indicators. A set the agent creates gets counter; a set declared in nftables.conf needs it there (otherwise the read fails, saying so):

set tis_v4 { type ipv4_addr; flags interval; counter; }

The first read only takes the counters. The counters read are kept in state_dir and move on once the report was accepted: after a failed report the increments are counted again, also after a restart.
Hits go to the hour of the read.
Updates of the list keep the counters of the elements that stay; what an element counted between the last read and a full reload of the set (flush ruleset, a failed update) is lost.

For the time of every packet, use NFLOG.

nflog¶

Since tis-agent 0.7.0. On Linux, the counterpart of the divert: an nftables rule copies the packets it blocks to a netlink log group, and the agent counts each one for the element of the listed sets its source – or else destination – address falls in (longest prefix).

"hits": { "nflog": { "group": 100, "targets": ["tis_v4", "tis_v6"] } }

ip saddr @tis_v4 log group 100 drop
ip6 saddr @tis_v6 log group 100 drop

Key	Default	Description
`group`		The NFLOG group of the rules (0–65535)
`targets`		`nftables-set` targets whose elements the packets are matched against
`reload_interval`	`5m`	How often the sets' elements are read again

Compared with the element counters: every packet counts in its own hour, nothing is lost when a set is reloaded, and the sets need no counter. Both can run side by side for different sets.
The agent binds the group itself (it runs as root); ulogd or another logger must not use the same group (the agent says so when the group is taken).
The kernel copies only the first 128 bytes of each packet – the IP header is enough. When the agent can't keep up, the kernel drops log copies, never the blocked packets; the agent warns once a minute.
The counts are reported every interval and kept for 24 hours while the platform is unreachable, like the divert's. Report hits now from the platform doesn't apply – they go every interval anyway.

logs¶

"hits": {
  "logs": [
    { "format": "rpz",  "path": "/var/log/named/rpz.log", "zones": ["tis-blocklist.rpz"] },
    { "format": "ipfw", "path": "/var/log/security", "rules": [4990], "table": "tis-blocklist" }
  ]
}

Key	Description
`format`	`rpz` – BIND's RPZ rewrites, or `ipfw` – packets of ipfw logging rules
`path`	The log file
`zones`	rpz: the RPZ zones of the TIS lists
`rules`	ipfw: the numbers of the logging rules
`table`	ipfw: a packet counts for the entry of this table its source (or else destination) matches
`side`	ipfw without `table`: count the `src` or `dst` address
`from_start`	Count what the log already holds when it is read the first time (default: start at its end)

rpz – each rewrite counts for its trigger, the listed entry, so the counts match the indicators: *.evil.example → evil.example, IP triggers (rpz-ip, rpz-client-ip, rpz-nsip) → addresses and networks, rpz-nsdname → the name. PASSTHRU (whitelisted) rewrites don't count. BIND logs the rewrites with the rpz category, e.g.

logging {
    channel rpz_log { file "/var/log/named/rpz.log" versions 3 size 20m; print-time yes;
                      print-category yes; print-severity yes; severity info; };
    category rpz { rpz_log; };
};

ipfw – only the lines of the given rules count; "last message repeated N times" counts N more. The rule needs log (ipfw add 4990 deny log ip from 'table(tis-blocklist)' to any), and the kernel messages go to /var/log/security with net.inet.ip.fw.verbose=1 (mind net.inet.ip.fw.verbose_limit). The table is matched in the agent, like the divert packets.

Every interval the agent reads what each log got since the last time, from the positions kept in state_dir:

Hits go to the hour of the line's timestamp – BIND's print-time, BSD syslog or RFC 5424 – so a backlog read after an outage lands in the right hours.
The positions move on once the report was accepted: after a failed report the same lines are counted again, also after a restart. A report accepted only in part keeps the rest for the next one.
A line still being written waits for the next pass.
When a log was rotated, the rest of the old file is counted if it was renamed to <path>.0 or <path>.1; otherwise the new file is counted from its start.

tis-agent hits -dry-run prints what the logs would report, per hour and value, without sending or saving anything; tis-agent hits makes one pass.

checkin¶

Since v0.5.0, with platform v1.41.0+. The daemon reports its state to the platform right after it starts, then every minute (the platform may ask for another interval) and shortly after a scheduled job ran, so the device shows up on the Devices page: the agent's version, the host (hostname, OS, release, architecture, uptime), a SHA-256 of the configuration file, the targets (names and types only – no paths or commands), and per job its list or source, schedule, last and next run, result, error, failed runs in a row and entries.

Key	Default	Description
`enabled`	`true`	`false` turns the check-in off.

A device is the token and reporter, as in the hit statistics – give each machine its own sub-token, or at least its own reporter.
The token needs agent/* – in the guest and user defaults; add it to a token with its own ACL.
A failed check-in changes nothing else: the jobs run on, the failure is logged when it starts or changes, and the next attempt waits longer (like a job's backoff). An older platform answers 404, which is logged once and tried again hourly; 401/403 hints at a revoked or expired token or the missing right.
Without a token there is no check-in either.

managed¶

Since v0.6.0, with platform v1.42.0+. With "managed": true the pull jobs come from the platform – the template of the customer's token and the device's own jobs, edited on the Devices page – and replace the local pull jobs. Targets, push jobs and hit counting stay local.

"managed": true,
"healthchecks": { "fw": "https://hc-ping.com/<uuid>" }

A central job sets only its name, every and what to pull into which of this device's targets. The platform never sets a command, a file or a URL: it can change what is deployed – as it always could – but not run anything on the device or read its files.
Central jobs arrive with the check-in and go through the same checks as this file (known target, a format it takes, every at least 30 s, no other fields), with the target presets following their formats. Valid ones are saved to central.saved in the state directory and the jobs restart with them; invalid ones are refused as a whole – the running jobs stay and the reason shows on the Devices page.
At start the saved central jobs run. Before the first ones arrive, or when the platform has none for the device, the local pull jobs run.
healthchecks gives central jobs their healthcheck URLs, by job name.
The platform can ask to pull now (a job or all, as tis-agent run) and to report hits now (the logs and nftables sets; divert hits are reported every interval anyway); the outcome goes back with the next check-in, which follows right away.
tis-agent status marks central jobs with *, tis-agent check says which central configuration runs, tis-agent run <job> runs central jobs too.

fail2ban¶

Since tis-agent 0.8.0. The agent reads fail2ban's log and sends the IPs it bans to the platform – as indicators into an import-only source, so the lists can block them on other machines too, and as hits for the IPs the TIS lists already have. fail2ban needs no change: it logs every ban with its jail.

"fail2ban": { "source": "fw-01-fail2ban", "hits": true, "jails": ["sshd", "postfix-sasl"] }

Key	Default	Description
`log`	`/var/log/fail2ban.log`	fail2ban's `logtarget` (a syslog file works too)
`jails`	all	Only these jails
`source`		Import-only source the banned IPs go to
`tag`	`fail2ban`	Tags of the imported IPs, with `jail:<jail>`
`hits`	`false`	Also report the bans as hits (listed IPs only)
`interval`	`1m`	How often the log is read and the bans sent
`from_start`	`false`	Read what the log already holds at the first start

Lines [<jail>] Ban <ip> count, with the time of the line; Restore Ban (bans fail2ban loads again after a restart), Unban and Found don't. The log is read like the hit logs: from its end at the first start, from where it stopped afterwards, also across rotation.
Indicators: the new IPs of every interval go to POST /api/v1/import/<source> per jail, tagged fail2ban,jail:sshd. Use an import-only source with format plain and a retention that suits you – unbans aren't sent, repeat offenders keep their indicator fresh. The token needs import/<source>.
Hits (platform v1.43.0+): every ban counts in its hour, but only for the IPs the TIS lists have when it is reported (listedOnly) – fail2ban sees far more than the lists. Hits are sent before the import, so a ban doesn't count just because it was imported; an IP imported by an earlier ban does. The token needs hits.
What couldn't be sent waits in the state directory (24 hours at most) and goes with the next interval; imports wait while hits do.
tis-agent hits -dry-run lists the new bans without sending them, tis-agent check shows the setting; Report hits now from the platform sends them too.