SynapseML OpenAIChatCompletion lacks image support

Question

OpenAIs GPT4-o model allows for an "image_url" as argument type in its Chat Completions API, but the corresponding SynapseML function seems to lack this option according to the documentation

OPENAI chat completions API:

response = client.chat.completions.create(
    model=MODEL,
    messages=[
        {"role": "system", "content": "You are a helpful assistant that responds in Markdown. Help me with my math homework!"},
        {"role": "user", "content": [
            {"type": "text", "text": "What's the area of the triangle?"},
            {"type": "image_url", "image_url": {
                "url": "https://upload.wikimedia.org/wikipedia/commons/e/e2/The_Algebra_of_Mohammed_Ben_Musa_-_page_82b.png"}
            }
        ]}
    ],
    temperature=0.0,
)

SynapseML function from the documentation:

def setMessagesCol(self, value):
        """
        Args:
            messagesCol: The column messages to generate chat completions for, in the chat format. This column should have type Array(Struct(role: String, content: String)).
        """
        self._set(messagesCol=value)
        return self

Simply providing the image url inside the content results in erroneous behaviour of the model, where it makes up the answer without actually processing the image. And a base64 encoded image string exceeds the token limit.

System Information:

• Language version: python 3.10, scala 2.12.15
• Spark Version: 3.3
• Spark Platform : Synapse

Answer 1

Hello Hiob Gebisso,

Welcome to the Microsft Q&A forum.

The issue seems to be that SynapseML does not natively support the image_url argument that is available in OpenAI's API.

I would suggest submitting a feature request using the URL below. This is open for the user community to upvote and comment on. It allows product teams to effectively prioritize your request against our existing feature backlog and provides insight into the potential impact of implementing the suggested feature.

https://feedback.azure.com/d365community/forum/9b9ba8e4-0825-ec11-b6e6-000d3a4f07b8

Please let me know if you have any further questions.