How to convert a TensorFlow Data and BatchDataset into Azure DataTable ?

Question

How to convert a TensorFlow Data and BatchDataset into Azure DataTable ?

Ghada 31

I am training a model using Azure PCA-based Anomaly Detection module and streaming the data for model training and evaluation using Kafka. The train and test dataset are in Azure DataTable format. How do I convert the tf BatchDataset into an Azure DataTable?

romungi-MSFT 48,906 Reputation points Microsoft Employee Moderator

2023-04-12T11:31:13.5666667+00:00

@Ghada With respect to Azure Machine learning are you using the Azure ML workspace where you want to import or register your data as a dataset? I do not have much experience with Azure Data tables format, could you add more details about the format?

With respect to using TF data you could use tensorflow datasets package and convert the same to a dataframe or numpy array and then try to import it or register them as a dataset on your Azure ML workspace and then consume the dataset in your experiment.
romungi-MSFT 48,906 Reputation points Microsoft Employee Moderator

2023-04-17T06:37:56.4433333+00:00

@Ghada Did you get a chance to check the above response to add more details?

Ghada 31

Thank you for your comments. The Azure DataTable class I am using is part of the azureml-designer-classic-modules 0.0.177 available from https://pypi.org/project/azureml-designer-classic-modules/

I am using Apache Kafka and Tensorflow IO to stream data for model training. I converted the dataframe to a list format to feed to the Kafka server. I am training an Azure studio model on an Azure ML studio platform that uses a DataTable format. Here are some code snippets to add more details:

#import the DataTable class
from azureml.studio.common.datatable.data_table import DataTable

#Convert each test and train dataframe to list form to feed to kafka
x_train = list(filter(None, x_train_df.to_csv(index=False).split("\n")[1:]))
y_train = list(filter(None, y_train_df.to_csv(index=False).split("\n")[1:]))

x_test = list(filter(None, x_test_df.to_csv(index=False).split("\n")[1:]))
y_test = list(filter(None, y_test_df.to_csv(index=False).split("\n")[1:]))
len(x_train), len(y_train), len(x_test), len(y_test)

#createan online train dataset 
online_train_ds = tfio.experimental.streaming.KafkaGroupIODataset(
    topics=["train_ds"],
    group_id="cgonline",
    servers="localhost:9092",
    stream_timeout=10000, # in milliseconds, to block indefinitely, set it to -1.
    configuration=[
        "session.timeout.ms=7000",
        "max.poll.interval.ms=8000",
        "auto.offset.reset=earliest"
    ],
)

def decode_kafka_online_item(raw_message, raw_key):
  message = tf.io.decode_csv(raw_message, [[0.0] for i in range(NUM_COLUMNS)])
  key = tf.strings.to_number(raw_key)
  return (message, key)

BATCH_SIZE = 32
online_train_ds = online_train_ds.shuffle(buffer_size=32)
online_train_ds = online_train_ds.map(decode_kafka_online_item)
online_train_ds = online_train_ds.batch(BATCH_SIZE)

When I print the online_train_ds, I get the following results:

Your answer

romungi-MSFT 48,906 Reputation points Microsoft Employee Moderator

2023-04-12T11:31:13.5666667+00:00

@Ghada With respect to Azure Machine learning are you using the Azure ML workspace where you want to import or register your data as a dataset? I do not have much experience with Azure Data tables format, could you add more details about the format?

With respect to using TF data you could use tensorflow datasets package and convert the same to a dataframe or numpy array and then try to import it or register them as a dataset on your Azure ML workspace and then consume the dataset in your experiment.
romungi-MSFT 48,906 Reputation points Microsoft Employee Moderator

2023-04-17T06:37:56.4433333+00:00

@Ghada Did you get a chance to check the above response to add more details?

Share via

How to convert a TensorFlow Data and BatchDataset into Azure DataTable ?

Your answer