Finetune Models

Framework Agnostic

Weaver is a framework-agnostic training API. You can use the SDK directly, integrate NexRL, or adapt another RL framework. NexRL is the reference implementation most deeply integrated with the Nex-AGI ecosystem, so this page uses it as the main path.

What is NexRL?

NexRL is a distributed post-training framework for large language models. It organizes data, rollout, rewards, trajectory pools, and training algorithms. Weaver acts as the training-service backend that performs remote training, weight saves, and sampler weight sync.

In Weaver mode, a typical pipeline is:

DataLoader -> RolloutWorker -> TrajectoryPool -> Trainer -> Weaver Training API
                    |                                      |
                    v                                      v
                 Rewards                         Updated / Exported Weights

Core Components

Component	Role
DataLoader	Provides prompts, questions, environment states, or task examples.
RolloutWorker	Calls a model or agent to generate trajectories and compute rewards.
TrajectoryPool	Collects, groups, and batches trajectories.
Trainer	Implements GRPO, PPO, SFT, or custom algorithms and calls Weaver APIs.
WeightSyncController	Synchronizes updated weights between training and inference/rollout services.

Most Weaver users customize two pieces:

• RolloutWorker: how tasks become trajectories, answers, and rewards.
• Trainer: how trajectories become advantages and Weaver loss inputs.

Install NexRL

git clone git@github.com:nex-agi/NexRL.git
cd NexRL

# Full install with training dependencies
pip install -e ".[core]"

# Or lightweight CLI/config installation
pip install -e .

Configure Weaver:

export WEAVER_API_KEY=<your-api-key>

Run a Weaver Recipe

NexRL organizes training jobs as recipes. A recipe usually contains:

• A training YAML file.
• Optional environment setup scripts.
• Data paths, rollout worker, trainer, service backend, and logging configuration.

Pig Latin example:

nexrl -m training-service \
  -c recipe/pig_latin/weaver.yaml \
  --run-nexrl \
  --tag pig-latin-v1

Script entry:

python scripts/run.py \
  --mode training-service \
  --train-config recipe/pig_latin/weaver.yaml \
  --run-nexrl \
  --tag pig-latin-v1

Common options:

Option	Description
-m / --mode	Run mode. Weaver recipes usually use training-service.
-c / --train-config	Recipe YAML path.
--run-nexrl	Start training automatically after resources are ready.
--tag	Experiment tag for distinguishing runs.

Recipe Examples

SFT: Pig Latin

Location: NexRL/recipe/pig_latin/weaver.yaml

This recipe demonstrates supervised fine-tuning through NexRL and Weaver:

rollout_worker:
  type: "pig_latin"
  need_llm_inference: false

trainer:
  type: "remote_api_cross_entropy"

service:
  train_service:
    backend: weaver
    config:
      loss_fn: "cross_entropy"
      learning_rate: 1e-4

Highlights:

• The rollout worker uses ground-truth labels and does not need model inference.
• The trainer converts examples into target_tokens and weights for cross_entropy.
• Weaver performs remote LoRA or full fine-tuning.

Full Fine-Tuning: Math

Location: NexRL/recipe/math/weaver_full_ft.yaml

This recipe demonstrates Weaver full fine-tuning, useful when LoRA capacity is not enough or longer training requires deeper adaptation.

Typical configuration:

service:
  weaver_service:
    training_mode: "full_ft"

  train_service:
    backend: weaver

Agent Training: NexAU

Locations:

• NexRL/recipe/nexau_deepsearch/weaver.yaml
• NexRL/recipe/nexau_news/weaver.yaml

These recipes show how to send multi-turn NexAU tool-use trajectories through NexRL and update the model with Weaver.

Custom RolloutWorker

RolloutWorker turns input tasks into trainable trajectories:

from nexrl.rollout_worker import BaseRolloutWorker
from nexrl.nexrl_types import Trajectory


class MyTaskWorker(BaseRolloutWorker):
    def rollout(self, task: dict) -> str | None:
        prompt = task["prompt"]
        answer = task.get("answer", "")

        completion = self._inference_client.completion(prompt)
        prompt_tokens = completion["prompt_tokens"]
        response_tokens = completion["response_tokens"]
        response = completion["response"]

        reward = 1.0 if self._extract_answer(response) == answer else 0.0

        trajectory = Trajectory(
            tokens=prompt_tokens + response_tokens,
            loss_mask=[0] * len(prompt_tokens) + [1] * len(response_tokens),
            reward=reward,
            extra_fields={
                "response": response,
                "answer": answer,
            },
        )
        return self._put_trajectory(trajectory)

Reference it in a recipe:

rollout_worker:
  type: "custom"
  custom_rollout_worker_module_path: "recipe/my_task/rollout_worker.py"
  custom_rollout_worker_class_name: "MyTaskWorker"

Custom Trainer

Trainer converts trajectories into Weaver Datum objects and chooses the loss. You can inherit a NexRL remote API trainer and implement advantage computation in _prepare_trajectories():

from nexrl.trainer import RemoteApiTrainer


class MyAlgorithmTrainer(RemoteApiTrainer):
    def _prepare_trajectories(self, trajectories, metrics):
        for traj in trajectories:
            traj.advantage = traj.reward
        return trajectories

Recipe reference:

trainer:
  type: "custom"
  custom_trainer_module_path: "recipe/my_task/trainer.py"
  custom_trainer_class_name: "MyAlgorithmTrainer"

Weaver Integration Notes

• SFT usually uses remote_api_cross_entropy and loss_fn: "cross_entropy".
• RL usually uses importance_sampling, ppo_clip, grpo, or OPD-related losses.
• For Weaver full fine-tuning, set training_mode: "full_ft" in the Weaver service config.
• For frequent rollout weight sync, use sampler exports and TTL instead of durable checkpoints.
• Long experiments should periodically call save_state(checkpoint_type="weight_and_optimizer") so optimizer state can be restored.

Next Steps

• Build & Finetune Agents: define tool-using NexAU agents.
• Loss Functions: understand Weaver loss input contracts.
• Saving and Loading: plan checkpoint and sampler export lifetimes.