Consulting

Find out why your LLM feature fails, and what to fix first.

I am a Principal AI Engineer. I read your architecture, your code and your evaluation results, then deliver a prioritized plan for fixing retrieval quality, latency and cost. When you want those fixes built rather than described, I do that too.

The technical write-ups on this site are the portfolio. They document the implementation choices, the approaches that failed, and what I learned. If the problems in them look like yours, we should talk.

Every engagement begins with a free 30-minute call. Then a written, fixed-price proposal. Nothing starts, and nothing is billed, until the scope and the number are agreed in writing.

You might be here because

  • Your RAG prototype impressed the demo audience and disappointed the first real users, and nobody can say precisely why.
  • Prompt changes have stopped improving your evaluation results, the training run keeps dying, and the GPU bill keeps growing.
  • Your team needs an independent review before committing to an LLM architecture or a vendor it will be living with for two years.

How I can help

Three ways to start, and one way to continue. Each is scoped and quoted in writing after the free call. If your problem does not fit any of these, describe it anyway.

LLM architecture review

Fixed scope · about two weeks

I read your architecture, your code and your evaluation results, then tell you what is actually wrong and what to fix first. One existing LLM workflow per review.

  • Written findings, with the evidence behind each one
  • A prioritized plan, each item with acceptance criteria
  • A walkthrough call, and a second one after your team has read it

Fit: Best when a prototype works in a demo and nobody can say precisely why it disappoints real users.

Retrieval and search implementation

Scoped project · six to eight weeks

Building the fixes rather than describing them, on one existing retrieval pipeline: hybrid search over BM25 and dense vectors, reranking, and query routing where it earns its place.

  • Working code in your repositories, on your cloud accounts
  • An evaluation harness your team owns and can run without me
  • Handover documentation and a recorded walkthrough

Fit: Best when answers are inconsistent, retrieval quality is guesswork, or a specialized domain needs more than vanilla RAG.

Fine-tuning feasibility sprint

Scoped project · four to six weeks

One model, one task, one bounded experiment. Supervised fine-tuning or DPO on open-weight models, measured against a real baseline, so the decision to invest in training is made on numbers instead of hope.

  • A baseline your team agrees is fair
  • One bounded experiment, with the training and serving cost measured
  • A recommendation: train, do not train, or change the problem

Larger multi-node training programmes are scoped and quoted separately.

Fit: Best when prompting has stopped moving your evaluation numbers and nobody wants to commit six figures of GPU time to find out why.

AI technical advisory

Monthly retainer · eight hours a month · follows an engagement

A standing working session plus asynchronous review of designs, pull requests and vendor decisions. Eight hours a month in total, meetings included, so the boundary is clear on both sides.

  • A scheduled working session every other week
  • Written review of designs, pull requests and vendor shortlists
  • A month-by-month agreement, cancellable with two weeks notice

This is a continuation, not a starting point. I take it on after a review or an implementation project, because advising well on a system I have not read is guesswork, and guesswork is not worth paying a retainer for.

Fit: Best when a review or a build has finished and your team wants the same judgement available while they execute the plan.

What a review actually contains

The most common question is what arrives at the end. This is the table of contents, and it is the same every time. Nothing in it is a slide.

  1. 1What the system does today, written plainly enough to circulate internally
  2. 2Where it fails, with the queries, traces and numbers that show it failing
  3. 3Retrieval quality: what is being retrieved, what should be, and how far apart those are
  4. 4Evaluation: whether your harness can detect a regression, and what it currently misses
  5. 5Cost and latency per request, broken down by the component responsible
  6. 6The failure modes that will appear at ten times your current traffic
  7. 7A prioritized plan: each change, the reason for its rank, and how you will know it worked

Every finding cites the evidence it came from, so your team can disagree with a conclusion without having to take my word for the measurement. The plan is ordered by expected effect per unit of effort, and I say which items I would skip.

Selected work

Each of these links to a technical write-up on this site. They document the implementation choices, the approaches that failed, and what I learned.

DeepSeek-V3 multi-node training in BF16

Multi-node BF16 training runs of the 671B-parameter DeepSeek-V3 mixture-of-experts model across three and four A100 nodes: DeepSpeed and Ray Train orchestration, NCCL networking, driver and fabric-manager stability, checkpoint hygiene and Prometheus/Grafana monitoring. Written up as an eight-part series plus a technical overview of the model.

DeepSpeedRayNCCLObservability
→

Qwen 2.5-72B two-stage fine-tuning

Supervised fine-tuning with ZeRO-3 CPU offload and LoRA, followed by Direct Preference Optimization on preference data, for a content-generation workload.

SFTDPOLoRA
→

Agentic RAG and hybrid search architecture for medical search

How I approach query routing to specialized agents over a lexical-plus-semantic retrieval stack with neural reranking and diversity optimization, plus the data pipelines that feed it. The architecture I would start from for a domain-specific search product, written up in full.

RAGHybrid searchHealthcare
→

Marketing mix modeling platform

Architecture and lessons from building Bayesian MMM as a product rather than a PDF: asynchronous GPU training, budget optimization, and a UI that turns posteriors into recommendations non-technical stakeholders can act on.

BayesianPlatformMarketing
→

LLaMA 3.1 8B distributed fine-tuning

Sixteen-GPU fine-tuning with DeepSpeed ZeRO-3 offload and LoRA, with stable loss curves and efficient memory use.

DistributedDeepSpeed
→

Image model fine-tuning (SDXL, Flux)

Fine-tuning and serving diffusion models for image generation workloads, including the technical details most write-ups skip.

DiffusionSDXLFlux
→

How an engagement works

  1. 1

    A 30-minute call

    You describe the problem; I ask the questions that decide whether I can help. No charge, no pitch deck. If I am the wrong person, I say so on this call.

  2. 2

    A written proposal

    Scope, deliverables, acceptance criteria, timeline and a fixed price, in writing, before any work starts. It also lists what I need from your team and what is explicitly out of scope. If a smaller engagement would get you there, I propose that instead.

  3. 3

    Delivery against milestones

    Reviews end with written findings and a walkthrough. Implementation projects put code in your repositories, with documentation and a recorded weekly demo, so progress is visible without a standing meeting.

About me

I am Thomas Kalnik, a Principal AI Engineer. I have built AI systems in healthcare, digital marketing and the creator economy. Before that I worked in data engineering and data science at Criteo, 1upHealth, Dynamo Software and Wellington Management. That background matters: most production AI problems turn out to be data and infrastructure problems wearing a model costume.

I hold a Master's in Computer Science from the University of Illinois Urbana-Champaign, an MBA and MS in Finance from Northeastern University, and a Bachelor's from SUNY Geneseo. The finance training shows up in how I scope work: I care about the cost of a system and the business outcome it is supposed to move, not just whether the model is clever.

More on the about page, and in the technical posts.

Questions I usually get

What does it cost?
It depends on which engagement fits and how much of it you need, so I quote after the call rather than before it. A review costs less than an implementation project, and the advisory retainer is only available once one of those has finished. Put a budget range on the form and I will tell you on the call whether the scope and the number meet. Whatever the figure, it is fixed in writing before any work starts and does not move once signed.
What kind of companies do you work with?
Mostly product companies with an engineering team that is shipping, or trying to ship, an LLM-based feature: healthcare, marketing technology and the creator economy are the industries I know best, but the engineering problems travel well.
Will you work inside our existing codebase and infrastructure?
That is the default. Work happens in your repositories and on your cloud accounts, so nothing is locked in a consultant-owned environment. Cloud and GPU spend runs on your accounts against a limit we agree in advance, and it is never billed through me.
Who owns the work?
You do. All code, models and documentation produced during an engagement belong to you. I am happy to sign your NDA before the first call.
How many clients do you take on?
One delivery engagement at a time, plus a small number of advisory retainers. If I am not available when you reach out, I will tell you when I will be, or point you to someone who fits.

Start a conversation

Tell me what you are building and where it is stuck. I reply personally within two business days, usually with a few questions and a suggested next step.

Not sure which engagement fits? Describe the problem and leave the budget blank. I will suggest the shape on the call.

No newsletter, no follow-up sequence. Just a reply from me.