# Taking DFE for a first look

> **This applies to you if** you want to see DFE actually work - on your laptop, a
> demo box, or a proof of concept - without committing to anything yet.

The goal here is narrow and worth stating: get one event in one end and see it
come out the other. Everything else can wait.

## What you need

- Docker and Docker Compose v2
- Python 3
- Network access to `ghcr.io/hyperi-io`, where the images are public and pull with no login

## Getting it running

```bash
make init                  # create .env and env/, generate secrets
make stack VERSION=X.Y.Z   # pin every image from the DFE stack manifest
make ci                    # pull and start
```

`make stack` is not optional. The stack refuses to start on unpinned images
rather than quietly pulling `latest`, so a fresh checkout without it fails with a
message telling you exactly which pin is missing. That is deliberate: on
something you might later deploy, "which image am I running?" should never have a
vague answer.

`make ci` finishes by running the power-on self test, which injects a few marked
events and proves they reached ClickHouse. If it prints `PASS`, the pipeline
works - you have already seen the thing you came to see.

## Seeing your own data go through

Send an event to dfe-receiver, which every profile runs:

```bash
curl -X POST http://localhost:8080/ingest \
  -H 'Content-Type: application/json' \
  -d '{"_source":"main","message":"hello dfe"}'
```

`make post` reads the resolved profile and skips on one with no receiver, so if you are not sure which you have, run that instead.

Then look for it. `make init` gave ClickHouse's `default` user a generated
password, so the read needs it -- and a password on a command line is read back
out of `ps` and your shell history, so this reads it from `.env` and sends it as
a header:

```bash
python3 - <<'PY'
import urllib.request
from pathlib import Path

password = dict(
    line.split("=", 1)
    for line in Path(".env").read_text().splitlines()
    if "=" in line and not line.startswith("#")
)["CLICKHOUSE_PASSWORD"]
query = "SELECT * FROM dfe.main ORDER BY _timestamp_load DESC LIMIT 5 FORMAT Vertical"
print(
    urllib.request.urlopen(
        urllib.request.Request(
            "http://localhost:8123/",
            data=query.encode(),
            headers={"X-ClickHouse-User": "default", "X-ClickHouse-Key": password},
        )
    )
    .read()
    .decode()
)
PY
```

`_source` is what routes the event - the loader writes it to
`<database>.<_source>`, so `"_source":"main"` lands in `dfe.main`.

It will not appear instantly. The pipeline is asynchronous, and via Kafka
genuinely so; a couple of seconds is normal.

## Raw syslog, parsed per vendor

`make demo` streams Cisco ASA and FortiGate firewall logs into the running stack as raw syslog. dfe-transform-elastic works out which device sent each line, parses it, and lands it in that vendor's own table. Palo Alto and Cisco IOS come back once [dfe-receiver#212](https://github.com/hyperi-io/dfe-receiver/issues/212) and [dfe-transform-elastic#133](https://github.com/hyperi-io/dfe-transform-elastic/issues/133) land.

It needs a profile that runs dfe-transform-elastic, so set `DFE_PROFILE=single-kafka` or `DFE_PROFILE=single-grpc` in `.env` before `make ci`. Then:

```bash
make demo                      # 20 lines a second over TCP, until you stop it
make demo DEMO_RATE=100        # replaces a running stream with a faster one
make demo-stop                 # stops the stream; `make down` stops it too
```

It creates the Sources in [`resources/demo/demo.yaml`](https://github.com/hyperi-io/dfe-docker/blob/406c01cb997337c517042dc1ea04cda78e45e77f/resources/demo/demo.yaml) through the engine: a `demo-syslog` intake, and the tables `dfe.cisco-asa` and `dfe.fortinet`. It turns on dfe-receiver's syslog listener, UDP and TCP on port 514, and recreates what that changes. The FortiGate lines are dfe-transform-elastic's own test fixtures, and the ASA lines come from [`resources/demo/samples/`](https://github.com/hyperi-io/dfe-docker/tree/406c01cb997337c517042dc1ea04cda78e45e77f/resources/demo/samples). It returns as soon as every vendor table has gained rows, and the stream keeps running:

```text
demo -   dfe.cisco-asa: +N rows
demo -   dfe.fortinet: +N rows
demo -   dfe.demo-syslog: +N rows
demo -   dfe.main: +0 rows
demo - PASS every vendor table gained rows, ready to query in the console or HyperDX. The stream runs until `make demo-stop`
```

`dfe.demo-syslog` holds the lines no detector claimed. The replay script comes from the dfe-transform-elastic release the stack pins, so the demo needs a release with agentless detection and stops before changing anything on one without it. A `make dev` stack takes `make demo DEV=1`.

## What you are looking at

| Where | What |
|-------|------|
| http://localhost:3000 | DFE UI |
| http://localhost:8081 | Kafka UI (topics, consumer groups, lag) |
| http://localhost:8123 | ClickHouse HTTP |
| http://localhost:8003 | dfe-engine API |

The console asks for a login; Kafbat, HyperDX and the ClickHouse port do not
authenticate you as a person. `make creds` prints the console login again.

The ingest ports bind all interfaces so anything can push to them. Everything
else -- every web UI included -- binds loopback by default, so the four links
above work on this box and nowhere else. `DFE_BIND_SCOPE=all` publishes the UIs
on every interface, and needs `DFE_EXTERNAL_ORIGIN` set to the address browsers
use -- `make` stops and says so otherwise. Read the auth section of
[operating.md](/v/2.2.1/dfe-docker/operating.md) before you widen anything.

## Stopping

```bash
make down    # stop the containers, keep the data
make clean   # stop everything and DELETE the volumes, including ClickHouse
```

`make clean` really does delete the warehouse. On a laptop that is usually what
you want; anywhere else, read it twice.

## What this is and is not

It **is** the same image, from the same registry, that a Kubernetes deployment
would run. There is no separate "demo build" - what you are evaluating is the
real thing, packaged differently.

It is **not** secured, though it is not open either. `make init` mints two
logins -- `admin` for the console and the engine API, and `breakglass` for
recovery -- and generates ClickHouse's password with them. `make creds` prints
the admin one on a terminal and names the `.env` key holding the other;
`access-summary.md`, written beside `.env` and readable only by you, carries
both. Kafbat and HyperDX answer to anyone who can reach them on these profiles,
the opt-in `auth` profile puts an OIDC sign-in in front of both, and nothing here
serves TLS until you turn [console TLS](/v/2.2.1/dfe-docker/configuration.md#console-tls-opt-in) on.
That is fine on your machine and on a throwaway demo box. It is not
fine on anything reachable by people you have not met.

If the proof of concept goes well and the box becomes something real - even
something small - read [operating.md](/v/2.2.1/dfe-docker/operating.md) before it does. Small is
still production, and the defaults here are tuned for a laptop.

## Where next

| You want to | Read |
|-------------|------|
| Understand how the pieces fit | [architecture.md](/v/2.2.1/dfe-docker/architecture.md) |
| Run this where others depend on it | [operating.md](/v/2.2.1/dfe-docker/operating.md) |
| Stand up a real deployment, or upgrade one | [deploying.md](/v/2.2.1/dfe-docker/deploying.md) |
| Know what the stack reports about itself | [observability.md](/v/2.2.1/dfe-docker/observability.md) |
| Look up a variable or a port | [configuration.md](/v/2.2.1/dfe-docker/configuration.md) |
| Work on a DFE component | [developing.md](https://github.com/hyperi-io/dfe-docker/blob/406c01cb997337c517042dc1ea04cda78e45e77f/docs/developing.md) |
| Work out why something is broken | [troubleshooting.md](/v/2.2.1/dfe-docker/troubleshooting.md) |