TextCatalog.TokenizeIntoWords Método

Definición

Espacio de nombres:

Microsoft.ML

Ensamblado:

Microsoft.ML.Transforms.dll

Paquete:

Microsoft.ML v3.0.1

Cree un WordTokenizingEstimatorobjeto , que tokeniza el texto de entrada mediante separators como separadores.

public static Microsoft.ML.Transforms.Text.WordTokenizingEstimator TokenizeIntoWords (this Microsoft.ML.TransformsCatalog.TextTransforms catalog, string outputColumnName, string inputColumnName = default, char[] separators = default);

static member TokenizeIntoWords : Microsoft.ML.TransformsCatalog.TextTransforms * string * string * char[] -> Microsoft.ML.Transforms.Text.WordTokenizingEstimator

<Extension()>
Public Function TokenizeIntoWords (catalog As TransformsCatalog.TextTransforms, outputColumnName As String, Optional inputColumnName As String = Nothing, Optional separators As Char() = Nothing) As WordTokenizingEstimator

Parámetros

catalog

TransformsCatalog.TextTransforms

Catálogo de transformación relacionado con texto.

outputColumnName

String

Nombre de la columna resultante de la transformación de inputColumnName. El tipo de datos de esta columna será un vector de tamaño variable de texto.

inputColumnName

String

Nombre de la columna que se va a transformar. Si se establece en null, el valor de outputColumnName se usará como origen. Este estimador funciona en escalar de texto y vector de tipo de datos de texto.

separators

Char[]

Separadores que se van a usar (usa el carácter de espacio de forma predeterminada).

Devoluciones

WordTokenizingEstimator

Ejemplos

using System;
using System.Collections.Generic;
using Microsoft.ML;

namespace Samples.Dynamic
{
    public static class TokenizeIntoWords
    {
        public static void Example()
        {
            // Create a new ML context, for ML.NET operations. It can be used for
            // exception tracking and logging, as well as the source of randomness.
            var mlContext = new MLContext();

            // Create an empty list as the dataset. The 'TokenizeIntoWords' does
            // not require training data as the estimator
            // ('WordTokenizingEstimator') created by 'TokenizeIntoWords' API is not
            // a trainable estimator. The empty list is only needed to pass input
            // schema to the pipeline.
            var emptySamples = new List<TextData>();

            // Convert sample list to an empty IDataView.
            var emptyDataView = mlContext.Data.LoadFromEnumerable(emptySamples);

            // A pipeline for converting text into vector of words.
            // The following call to 'TokenizeIntoWords' tokenizes text/string into
            // words using space as a separator. Space is also a default value for
            // the 'separators' argument if it is not specified.
            var textPipeline = mlContext.Transforms.Text.TokenizeIntoWords("Words",
                "Text", separators: new[] { ' ' });

            // Fit to data.
            var textTransformer = textPipeline.Fit(emptyDataView);

            // Create the prediction engine to get the word vector from the input
            // text /string.
            var predictionEngine = mlContext.Model.CreatePredictionEngine<TextData,
                TransformedTextData>(textTransformer);

            // Call the prediction API to convert the text into words.
            var data = new TextData()
            {
                Text = "ML.NET's TokenizeIntoWords API " +
                "splits text/string into words using the list of characters " +
                "provided as separators."
            };

            var prediction = predictionEngine.Predict(data);

            // Print the length of the word vector.
            Console.WriteLine($"Number of words: {prediction.Words.Length}");

            // Print the word vector.
            Console.WriteLine($"\nWords: {string.Join(",", prediction.Words)}");

            //  Expected output:
            //   Number of words: 15
            //   Words: ML.NET's,TokenizeIntoWords,API,splits,text/string,into,words,using,the,list,of,characters,provided,as,separators.
        }

        private class TextData
        {
            public string Text { get; set; }
        }

        private class TransformedTextData : TextData
        {
            public string[] Words { get; set; }
        }
    }
}