What is an agent ?
QALITA agents are workers for the platform and sources. They connect them together and compute analysis, then send them to the platform.
Deploy an Agent
Agents must be installed the closest possible to the source as they will interact with them. take into account network configuration (proxy/firewall) and bandwidth for databases, or IO disk usage for files and no-SQL sources.
Hardware Requirements
Agent characteristics depend heavily on the typology and volume of sources for which they perform analysis.
| Usage | Memory | CPU | GPU |
|---|---|---|---|
| Minimal | 50 MiB | 0.2 | NVIDIA RTX 5060 Ti (for Studio) |
| Depends on source volume and analysis frequency | . | . |
NVIDIA GPU are recommended when using agent in UI mode and using Studio
Modalities
| Method | Description | Documentation |
|---|---|---|
| Desktop | Use your Terminal to deploy a local agent on your workstation. | CLI > Quick Start |
| Docker | Deploy an agent using Docker. | CLI > Docker |
| Kubernetes | Deploy an agent with the qalita helm chart and enable the agent flag. | See the documentation for available configuration options. |
You will then see your agent in the platform's agent page.
(Advanced) Kubernetes Architecture details
Same namespace
- Deployment with agent in the same namespace
agent.enabled=true - Deployment of data sources in the same namespace
Different namespace
- Deployment of an agent in another namespace
- Deployment of data sources in the same namespace
Different namespace + remote source
- Deployment of an agent in another namespace
- To connect to a source in any other environment (VM, localhost, etc.)
Fully Remote
- Deployment of an agent in any other environment (VM, localhost, etc.)
- To connect to a source located in any other environment (VM, localhost, etc.)
Agent Operating Modes
The agent has 2 operating modes:
Job
Job mode can be useful when developing your own packs. To run a pack in job mode, you will need:
- A source registered on the platform
- A pack registered on the platform
You will need to get their ID with:
qalita source list
and
qalita pack list
Finally, you can run a job locally with your agent using:
qalita agent -m job run -s <source_id> -p <pack_id>
This will run your pack with your source using your local agent.
This Helps you debug your packs when developping them allowing fast itteration.
Worker
Worker mode allows you to run an agent as a worker for the platform. It allows you to run tasks and routines in the background.
This agent will remain online waiting for tasks to execute.
Worker mode agents will only run tasks on sources that they have in their configuration ~/.qalita/sources-conf.yaml. If the source is not present with an id in the agent's configuration, it will not be able to run tasks on that source.
Make sure your agent in worker mode has the configuration of the sources on which you want to run tasks. And that there is indeed a source id. meaning the source has been referenced to the platform.
Routines (scheduled tasks)
Create Routines
Routines are scheduled tasks that run automatically at a given schedule by any compatible worker agents. You can create routines from a pack's detail page.
| From a Pack Page | From a Source Page |
|---|---|
|
|
The routine creation panel allows you to select the pack to use, the frequency of routine execution, the start date of the routine. Once the pack is selected, the default configuration is loaded. You can then override the default pack configuration specifically for this routine. You can expand the pack's documentation drawer to understand better what config is most suited for your needs.
Quick Run
The Quick Run Feature allows you to create a routine in just one click.
When clicking on it on a source page, it will select and create a routine using the first available Pack compatible with your source, and specific for that data quality dimension.
You will still be able to edit or remove the routine in the agent page.
Track Routine Executions
You can track routine executions on the platform's agent page:
- View the status of tasks
- Check the logs of a task
- Enable or disable a routine
- Trigger immediately a routine will generate a task to be executed immediately.
- Modify the configuration of a routine
- Delete a routine
