How to Feed Large, Time-Varying Context to Server-Side LLMs for Summarization and Managing Token Limits When Sending Evolving Context to LLMs

Question

How to Feed Large, Time-Varying Context to Server-Side LLMs for Summarization and Managing Token Limits When Sending Evolving Context to LLMs

Ankita Inaniya 0 Microsoft Employee

We have a use case where data from one of our APIs evolves over a two-day period. This data is shown to customers on the UI, and we want to pass the same data to a Copilot model so users can ask questions or request summaries based on that context.

However, the dataset is quite large, and since we’re using a client-server architecture with the LLM hosted on the server, we’re hitting token size limitations when trying to send the full context to the model.

One approach we discussed is having the client break the data into chunks and send them incrementally during chat initialization—essentially streaming the context to the server-side LLM until the full payload is delivered.

We’d like to understand: Is this a recommended approach for handling large context inputs, or are there better alternatives for managing token limits on the client side?

SriLakshmi C 5,050 Reputation points Microsoft External Staff Moderator

2025-05-08T08:50:30.4933333+00:00

Hi @Ankita Inaniya,

Following up to see if the below answer was helpful. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Thank you!
SriLakshmi C 5,050 Reputation points Microsoft External Staff Moderator

2025-05-09T10:13:20.98+00:00

Hi @Ankita Inaniya,

Just checking in to see if you have got a chance to see my response to your question in resolving the issue.

If you are still facing any further issues, please don't hesitate to reach out to us. We are happy to assist you.

Looking forward to your response and appreciate your time on this.

If you feel that your quires have been resolved, please accept the answer by clicking the "Upvote" and "Accept Answer" on the post.

Thank you!

1 answer

Your answer

SriLakshmi C 5,050 Reputation points Microsoft External Staff Moderator

2025-05-08T08:50:30.4933333+00:00

Hi @Ankita Inaniya,

Following up to see if the below answer was helpful. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Thank you!
SriLakshmi C 5,050 Reputation points Microsoft External Staff Moderator

2025-05-09T10:13:20.98+00:00

Hi @Ankita Inaniya,

Just checking in to see if you have got a chance to see my response to your question in resolving the issue.

If you are still facing any further issues, please don't hesitate to reach out to us. We are happy to assist you.

Looking forward to your response and appreciate your time on this.

If you feel that your quires have been resolved, please accept the answer by clicking the "Upvote" and "Accept Answer" on the post.

Thank you!

Answer 1

SriLakshmi C 5,050 Microsoft External Staff Moderator

Hello @Ankita Inaniya,

To effectively manage large and evolving datasets when interacting with server-side language models (LLMs), especially in the context of summarization and question-answering, streaming the context data in chunks from the client to the server is a practical and scalable solution. This method helps navigate token limitations such as the 4096-token limit in models like gpt-35-turbo by breaking down the input into smaller, manageable units instead of sending one large payload.

When implementing this approach, it's important to ensure that each chunk contains coherent and contextually meaningful information to maintain the relevancy of responses. Additionally, maintaining the conversational state is crucial; this can be achieved through a session manager that tracks user interactions and updates context as new data is streamed. Incorporating dynamic truncation techniques can also help prioritize and retain the most relevant information, especially as the conversation evolves and token constraints become tighter.

Alternatively, for extremely large datasets, a summarization step before sending context to the LLM can be beneficial, reducing input size while preserving key information. Regardless of the approach, careful memory and performance monitoring is recommended, as increased context size may impact latency and resource utilization.

Chunk-based context streaming offers a flexible and efficient method to manage large inputs within token constraints, while supporting real-time updates and sustained conversational relevance in client-server LLM applications.

I Hope this helps. Do let me know if you have any further queries.

If this answers your query, please do click Accept Answer and Yes for was this answer helpful.

Thank you!

Ankita Inaniya 0 Reputation points Microsoft Employee

2025-05-12T06:23:58.9566667+00:00

hi @SriLakshmi C thanks for the input! I’ll review your response in detail and discuss it with the internal team. At first glance, this chunk-based streaming approach looks promising, and it’s helpful that you’ve highlighted key areas to focus on during implementation.

One question I do have: since we’ll primarily be dealing with JSON data, my understanding is that recursive chunking tends to work best in this context. Do you agree with that, or would you recommend a different strategy?
SriLakshmi C 5,050 Reputation points Microsoft External Staff Moderator

2025-05-12T08:57:13.3433333+00:00

Hi Ankita Inaniya,

You're absolutely right recursive chunking is often a highly effective strategy when working with JSON data, especially in use cases involving large and evolving structures.

This approach allows you to break down the data hierarchically, preserving nested relationships and semantic meaning within the structure. This is particularly important for large language models (LLMs), as it ensures each chunk maintains logical integrity and can be independently processed or summarized without losing contextual relevance.

Recursive chunking also works well with incremental summarization and embedding pipelines, enabling chunk-wise processing and later recombination for coherent conversation flow or broader summarization. That said, the optimal strategy should be informed by the nature of your JSON data if it’s deeply nested or contains highly relevant fields scattered throughout, recursive chunking combined with relevance-based filtering can help prioritize what’s most important. For flatter data structures, fixed-size or semantic chunking might be more efficient.

Additionally, combining this with vector embeddings and hybrid retrieval can enhance performance by storing chunked data semantically and retrieving only the most relevant portions at query time reducing token load while preserving conversational richness. Ultimately, choosing the best approach depends on your data structure and the specific goals of your application.

Please refer this RAG chunking phase and Chunking approaches.

I Hope this helps. Do let me know if you have any further queries.

If this answers your query, please do click Accept Answer and Yes for was this answer helpful.

Thank you!
SriLakshmi C 5,050 Reputation points Microsoft External Staff Moderator

2025-05-13T09:53:23.55+00:00

Hi Ankita Inaniya,

Following up to see if the above suggestion was helpful. If this answers your query, please do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Thank you!

Share via

How to Feed Large, Time-Varying Context to Server-Side LLMs for Summarization and Managing Token Limits When Sending Evolving Context to LLMs

1 answer

Your answer