SynapseML OpenAIChatCompletion lacks image support

Hiob Gebisso 121 Reputation points
2024-07-02T07:26:20.9166667+00:00

OpenAIs GPT4-o model allows for an "image_url" as argument type in its Chat Completions API, but the corresponding SynapseML function seems to lack this option according to the documentation

OPENAI chat completions API:

response = client.chat.completions.create(
    model=MODEL,
    messages=[
        {"role": "system", "content": "You are a helpful assistant that responds in Markdown. Help me with my math homework!"},
        {"role": "user", "content": [
            {"type": "text", "text": "What's the area of the triangle?"},
            {"type": "image_url", "image_url": {
                "url": "https://upload.wikimedia.org/wikipedia/commons/e/e2/The_Algebra_of_Mohammed_Ben_Musa_-_page_82b.png"}
            }
        ]}
    ],
    temperature=0.0,
)

SynapseML function from the documentation:

def setMessagesCol(self, value):
        """
        Args:
            messagesCol: The column messages to generate chat completions for, in the chat format. This column should have type Array(Struct(role: String, content: String)).
        """
        self._set(messagesCol=value)
        return self

Simply providing the image url inside the content results in erroneous behaviour of the model, where it makes up the answer without actually processing the image. And a base64 encoded image string exceeds the token limit.

System Information:

  • Language version: python 3.10, scala 2.12.15
  • Spark Version: 3.3
  • Spark Platform : Synapse
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,671 questions
Azure OpenAI Service
Azure OpenAI Service
An Azure service that provides access to OpenAI’s GPT-3 models with enterprise capabilities.
2,613 questions
{count} votes

Accepted answer
  1. Bhargava-MSFT 28,951 Reputation points Microsoft Employee
    2024-07-02T16:08:49.04+00:00

    Hello Hiob Gebisso,

    Welcome to the Microsft Q&A forum.

    The issue seems to be that SynapseML does not natively support the image_url argument that is available in OpenAI's API.

    I would suggest submitting a feature request using the URL below. This is open for the user community to upvote and comment on. It allows product teams to effectively prioritize your request against our existing feature backlog and provides insight into the potential impact of implementing the suggested feature.

    https://feedback.azure.com/d365community/forum/9b9ba8e4-0825-ec11-b6e6-000d3a4f07b8

    Please let me know if you have any further questions.

    1 person found this answer helpful.
    0 comments No comments

1 additional answer

Sort by: Most helpful
  1. Deleted

    This answer has been deleted due to a violation of our Code of Conduct. The answer was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.


    Comments have been turned off. Learn more