OpenAIs GPT4-o model allows for an "image_url" as argument type in its Chat Completions API, but the corresponding SynapseML function seems to lack this option according to the documentation
OPENAI chat completions API:
response = client.chat.completions.create(
model=MODEL,
messages=[
{"role": "system", "content": "You are a helpful assistant that responds in Markdown. Help me with my math homework!"},
{"role": "user", "content": [
{"type": "text", "text": "What's the area of the triangle?"},
{"type": "image_url", "image_url": {
"url": "https://upload.wikimedia.org/wikipedia/commons/e/e2/The_Algebra_of_Mohammed_Ben_Musa_-_page_82b.png"}
}
]}
],
temperature=0.0,
)
SynapseML function from the documentation:
def setMessagesCol(self, value):
"""
Args:
messagesCol: The column messages to generate chat completions for, in the chat format. This column should have type Array(Struct(role: String, content: String)).
"""
self._set(messagesCol=value)
return self
Simply providing the image url inside the content results in erroneous behaviour of the model, where it makes up the answer without actually processing the image. And a base64 encoded image string exceeds the token limit.
System Information:
- Language version: python 3.10, scala 2.12.15
- Spark Version: 3.3
- Spark Platform : Synapse