How to Generate Narrative From Structured Data

Generating a narrative from structured data using Large Language Models (LLMs) involves converting tabular or structured data (e.g., from a spreadsheet or database) into coherent, natural language text. Here's a step-by-step guide to achieve this:

1. Preprocessing the Data

a. Understand the Data

• Review the structure, types of fields, and the relationships between them.
• Identify the key points or insights to be communicated.

b. Prepare the Input

• Convert the structured data into a text format (e.g., JSON, CSV-to-text).
• Normalize or aggregate the data if needed (e.g., totals, averages, percentages).

c. Summarize Key Elements

• Extract the most relevant rows, columns, or metrics to include in the narrative.
• Use filters to emphasize significant insights (e.g., "top-performing categories").

2. Formulate the Prompt

a. Structure the Prompt

• Create a descriptive instruction for the LLM that explains what you want. For example: ``` Here is sales data for Q3 2024. Generate a narrative summarizing the key insights:

  • Highlight top-performing categories.
  • Identify underperforming areas.
  • Provide year-over-year comparisons where available.

  Data: [Structured data or JSON here] ```

b. Use Examples

• Provide an example of the desired output format if possible. This improves specificity.

c. Manage Context

• Limit the prompt size by focusing only on essential data points to fit within the LLM’s token limits.

3. Use an LLM with a Specialized API or Framework

• Use LLMs like OpenAI’s GPT-4, Google’s Bard, or other fine-tuned LLMs.
• Tools like LangChain, LLamaIndex, or OpenAI Function Calling can help integrate LLMs with structured data.

a. Direct Text Query

• Pass the structured data as a JSON object directly to the model.

• Example: json { "sales_data": { "region": "North America", "total_sales": "$1.5M", "growth_rate": "10%", "top_products": ["Product A", "Product B"] } }

  Prompt: Generate a narrative summarizing the sales data.

b. Pre-Formatted Queries

• Use libraries to convert data to natural language queries automatically.
• Example: pip install pandasai for querying pandas DataFrames using natural language.

4. Enhance the Output

a. Add Context or Domain Knowledge

• Augment the narrative with context, benchmarks, or industry standards.

b. Use Templates for Consistency

• Create templates with placeholders for the model to populate. Example: ``` Sales Summary for {time_period}:
  • Total Sales: {total_sales}
  • Growth Rate: {growth_rate}
  • Top Performers: {top_performers} ```

c. Iterate and Refine

• Fine-tune the prompt or post-process the output for accuracy and tone.

5. Automate and Scale

• Use pipelines for automated narrative generation:
• Fetch structured data from APIs or databases.
• Preprocess and format the data.
• Generate narratives using an LLM API.
• Save and distribute the output (e.g., reports, dashboards).

6. Fine-Tuning (Optional)

• Fine-tune an LLM on a dataset of structured data and example narratives to improve relevance and coherence for your domain.

Example Workflow

Input Data:

json { "quarter": "Q3 2024", "total_sales": 1500000, "growth_rate": 0.1, "top_products": ["Product A", "Product B"], "underperforming_regions": ["South America"] }

Prompt:

Create a business report based on the following data: Quarter: Q3 2024 Total Sales: $1.5M Growth Rate: 10% Top Products: Product A, Product B Underperforming Regions: South America

Output:

"In Q3 2024, the company achieved total sales of $1.5M, marking a 10% growth compared to the previous quarter. The top-performing products were Product A and Product B, which significantly contributed to the revenue. However, sales in South America underperformed, indicating a need for targeted strategies in this region."

By iterating on these steps, you can generate effective narratives tailored to your needs.