Fine tuning AI to return structured JSON business process models

Question

I use Open AI to generate a business process graph (structured JSON nodes and edges) based on a defined set inputs (also JSON). While it can generate a result, it is a little underwhelming and inconsistent.

I have a lot of examples of business process graphs and so wanted to improve my results by fine tuning a model using that data. These existing graphs can get quite complicated and large, so the resulting JSON data can be quite long.

I generated a jsonl file that has lines like this:

{"messages":[{"role":"system","content":"system prompt here..."},{"role":"user", "content":{"industry":"Banking", "function":"Create customer account", "description":"Customer request a new account online", ...}, {"role":"assistant":"content":{"nodes":[...]}, "edges":[...]}]}

It includes a system prompt, user input as json data and assistant output as json data. This data failed when Open AI tried to load it on every line.

Details - Samples of lines per error type: contains invalid schema: Line numbers --> 1, 2, 3, 4, 5, 6, 7, 8

I tried using openai tools fine_tunes.prepare_data -f ai-tuning-output.jsonl and got

ERROR in necessary_column validator: prompt column/key is missing. Please make sure you name your columns/keys appropriately, then retry

I tried using prompt and completion pairs (again with JSON data), but got errors because the lines were too long (which they are because the business process models can be quite large).

It could be that my use case does not fit into how these models are setup and used. Was just wondering if anybody had experience or suggestions as to how I might tune my model to improve the results.

Answer 1

Hey Antony, It sounds like you're trying to get models to consistently generate structured JSON with fine-tuning. I've been there, and I think I have some insights that might help.

Fine-Tuning might not be the best solution, it's primarily designed to help the model learn a specific linguistic style or "tone of voice" for your project. Trying to force a strict JSON output structure, especially with very long interactions, might not be its strong suit.

Instead of relying solely on fine-tuning for structure, I'd suggest focusing on a robust system prompt combined with tool definitions. This approach is often more effective for enforcing a precise JSON output:

• Craft a Detailed System Prompt:
  • The exact output schema for your JSON (e.g., specifying fields, data types, and nesting).
  • Specific instructions and rules the AI agent needs to follow when generating the JSON. Think of it as giving the AI a strict blueprint.
• Leverage SDK Tools (if applicable): If you're using an SDK, make full use of its tooling capabilities. You can often pass the JSON schema of your desired tool directly to the SDK's tool definition. Additionally, reinforce this by including the JSON schema of the tool within your system prompt. This dual approach helps the model understand precisely what structured output it needs to produce. This one helped me a lot.

If you still want to explore fine-tuning, here are some points to consider, though I wouldn't make it your main strategy for JSON structure:

• Chunking and Weighting: You could try breaking down long interactions into smaller chunks. You might also assign a "weight" of 1 to assistant responses that you deem perfectly aligned with your desired tone, and 0 to examples where the model deviates. However, avoid trying to fine-tune with entire, lengthy conversation threads, as this tends to be problematic for long JSON outputs. Grab a few of 'wrong' outputs aswell.