Microsoft.ML.Core Intended to be applied to types and members with internal scope to indicate that friend access of this internal item is OK from another assembly. This restriction applies only to assemblies that declare the assembly level attribute. Note that this attribute is not transferable: an internal member with this attribute does not somehow make a containing internal type accessible. Conversely, neither does marking an internal type make any unmarked internal members accessible. This is an assembly level attribute to signal that friend accesses on this assembly should be checked for usage of . If this attribute is missing, normal access rules for friends should apply. Allows control of command line parsing. Attach this attribute to instance fields of types used as the destination of command line argument parsing. Allows control of command line parsing. Specifies the error checking to be done on the argument. The error checking to be done on the argument. The short name(s) of the argument. Set to null means use the default short name if it does not conflict with any other parameter name. Set to String.Empty for no short name. More than one short name can be separated by commas or spaces. This property should not be set for DefaultArgumentAttributes. The help text for the argument. Allows the GUI or other tools to inspect the intended purpose of the argument and pick a correct custom control. Used to control parsing of command line arguments. Indicates that this field is required. An error will be displayed if it is not present when parsing arguments. Only valid in conjunction with Multiple. Duplicate values will result in an error. Indicates that the argument may be specified more than once. Only valid if the argument is a collection The default type for non-collection arguments. The argument is not required, but an error will be reported if it is specified more than once. For non-collection arguments, when the argument is specified more than once no error is reported and the value of the argument is the last value which occurs in the argument list. The default type for collection arguments. The argument is permitted to occur multiple times, but duplicate values will cause an error to be reported. Consume the characters of the next token and append them to the string builder. Returns true if it quoted. A delegate used in error reporting. An IComponentFactory that is used in the command line. This allows components to be created by name, signature type, and a settings string. Parser for command line arguments. The parser specification is inferred from the instance fields of the object specified as the destination of the parse. Valid argument types are: int, uint, string, bool, enums Also argument types of Array of the above types are also valid. Error checking options can be controlled by adding a ArgumentAttribute to the instance fields of the destination object. At most one field may be marked with the DefaultArgumentAttribute indicating that arguments without a '-' or '/' prefix will be parsed as that argument. If not specified then the parser will infer default options for parsing each instance field. The default long name of the argument is the field name. The default short name is the first character of the long name. Long names and explicitly specified short names must be unique. Default short names will be used provided that the default short name does not conflict with a long name or an explicitly specified short name. Arguments which are array types are collection arguments. Collection arguments can be specified multiple times. Parses a command line. This assumes that the .exe name has been stripped off. Errors are output on Console.Error. Use ArgumentAttributes to control parsing behavior. The host environment The command line The object to receive the options true if no errors were detected Parses a command line. This assumes that the .exe name has been stripped off. Use ArgumentAttributes to control parsing behavior. The host environment The command line The object to receive the options The type of 'destination' The destination for parse errors. true if no errors were detected Parses a command line. This assumes that the exe name has been stripped off. Use ArgumentAttributes to control parsing behavior. The host environment The command line The object to receive the options The destination for parse errors. true if no errors were detected Check whether a certain type is numeric. Returns a Usage string for command line argument parsing. Use ArgumentAttributes to control parsing behavior. The host environment. The type of the arguments to display usage for. The default values. Whether to show the @file item. The number of columns to format the output to. Printable string containing a user friendly description of command line arguments. Parses an argument list into an object true if successful Parses an argument list. The arguments to parse. The destination of the parsed arguments. true if no parse errors were encountered. GetSettingsCore handles the top-level case. This handles the nested custom record case. It deals with custom "unparse" functionality, as well as quoting. It also appends to a StringBuilder instead of returning a string. A user friendly usage string describing the command line argument syntax. Creates an ICommandLineComponentFactory given the factory type, signature type, and a command line string. This class exposes those parts of this wrapped appropriate for public consumption. A factory class for creating IComponentFactory instances. Indicates that this argument is the default argument. '/' or '-' prefix only the argument value is specified. The ShortName property should not be set for DefaultArgumentAttribute instances. The LongName property is used for usage text only and does not affect the usage of the argument. Indicates that this argument is the default argument. Specifies the error checking to be done on the argument. On an enum value - specifies the display name. On an enum value - indicates that the value should not be shown in help or UI. This is used to specify a column mapping of a data transform. This is meant to be a large text (like a c# code block, for example). This is used to specify a column mapping of a data transform. Make sure the given assemblies are loaded and that their loadable classes have been catalogued. Given an assembly path, load the assembly and register it with the ComponentCatalog. Checks whether references the assembly containing LoadableClassAttributeBase, and therefore can contain components. This catalogs instantiable components (aka, loadable classes). Components are registered via a descendant of , identifying the names and signature types under which the component type should be registered. Signatures are delegate types that return void and specify that parameter types for component instantiation. Each component may also specify an "arguments object" that should be provided at instantiation time. Provides information on an instantiable component, aka, loadable class. Used for dictionary lookup based on signature and name. Count of component construction arguments, NOT including the arguments object (if there is one). This matches the number of arguments for the signature type delegate(s). The type that contains the construction method, whether static Instance property, static Create method, or constructor. Summary of the component. UserName may be null or empty, indicating that it should be hidden in UI. Whether this is a "hidden" component, that generally shouldn't be displayed to users. All load names. The first is the default. The static property that returns an instance of this loadable class. This creation method does not support an arguments class. Only one of Ctor, Create and InstanceGetter can be non-null. The constructor to create an instance of this loadable class. This creation method supports an arguments class. Only one of Ctor, Create and InstanceGetter can be non-null. The static method that creates an instance of this loadable class. This creation method supports an arguments class. Only one of Ctor, Create and InstanceGetter can be non-null. A name of an embedded resource containing documentation for this loadable class. This is non-null only in the event that we have verified the assembly of actually contains this resource. The type that contains the arguments to the component. Create an instance, given the arguments object and arguments to the signature delegate. The args should be non-null iff ArgType is non-null. The length of the extra array should match the number of parameters for the signature delegate. When that number is zero, extra may be null. Create an instance, given the arguments object and arguments to the signature delegate. The args should be non-null iff ArgType is non-null. The length of the extra array should match the number of parameters for the signature delegate. When that number is zero, extra may be null. Create an instance with default arguments. If is not null, returns a new default instance of . Otherwise, returns null. A description of a single entry point. A description of a single component. The 'component' is a non-standalone building block that is used to parametrize entry points or other ML.NET components. For example, 'Loss function', or 'similarity calculator' could be components. Registers all the components in the specified assembly by looking for loadable classes and adding them to the catalog. The assembly to register. true to throw an exception if there are errors with registering the components; false to skip any errors. Return an array containing information for all instantiable components. If provided, the given set of assemblies is loaded first. Return an array containing information for instantiable components with the given signature and base type. If provided, the given set of assemblies is loaded first. Return an array containing all the known signature types. If provided, the given set of assemblies is loaded first. Returns a string name for a given signature type. Get all registered entry points. Akin to , except if the regular (case sensitive) comparison fails, it will attempt to back off to a case-insensitive comparison. Returns all valid component kinds. Returns all components of the specified kind. Returns all components that implement the specified interface. The valid names for the components and entry points must consist of letters, digits, underscores and dots, and begin with a letter or digit. Create an instance of the indicated component with the given extra parameters. Try to create an instance of the indicated component and settings with the given extra parameters. If there is no such component in the catalog, returns false. Any other error results in an exception. Parses arguments using CmdParser. If there's a problem, it throws an InvalidOperationException, with a message giving usage. The host environment The argument object The settings string The name is used for error reporting only Gets a value indicating whether can contain extensions. All ML.NET product assemblies won't contain extensions. This is a token interface that all component factories must implement. An interface for creating a component with no extra parameters (other than an ). An interface for creating a component when we take one extra parameter (and an ). An interface for creating a component when we take two extra parameters (and an ). An interface for creating a component when we take three extra parameters (and an ). A utility class for creating instances. Creates a component factory with no extra parameters (other than an ) that simply wraps a delegate which creates the component. Creates a component factory when we take one extra parameter (and an ) that simply wraps a delegate which creates the component. Creates a component factory when we take two extra parameters (and an ) that simply wraps a delegate which creates the component. Creates a component factory when we take three extra parameters (and an ) that simply wraps a delegate which creates the component. A class for creating a component with no extra parameters (other than an ) that simply wraps a delegate which creates the component. A class for creating a component when we take one extra parameter (and an ) that simply wraps a delegate which creates the component. A class for creating a component when we take one extra parameter (and an ) that simply wraps a delegate which creates the component. A class for creating a component when we take three extra parameters (and an ) that simply wraps a delegate which creates the component. A channel provider can create new channels and generic information pipes. Start a standard message channel. Start a generic information pipe. Utility class for IHostEnvironment Return a file handle for an input "file". Create an output "file" and return a handle to it. The host environment interface creates hosts for components. Note that the methods of this interface should be called from the main thread for the environment. To get an environment to service another thread, call Fork and pass the return result to that thread. Create a host with the given registration name. The catalog of loadable components () that are available in this host. Signal to stop execution in all the hosts. Flag which indicates host execution has been stopped. The seed property that, if assigned, makes components requiring randomness behave deterministically. The location for the temp files created by ML.NET Allow falling back to run on CPU if couldn't run on GPU. GPU device ID to run execution on, to run on CPU. A host is coupled to a component and provides random number generation and concurrency guidance. Note that the random number generation, like the host environment methods, should be accessed only from the main thread for the component. The random number generator issued to this component. Note that random number generators are NOT thread safe. A generic information pipe. Note that pipes are disposable. Generally, Done should be called before disposing to signal a normal shut-down of the pipe, as opposed to an aborted completion. The caller relinquishes ownership of the object. The kinds of standard channel messages. Note: These values should never be changed. We can add new kinds, but don't change these values. Other code bases, including native code for other projects depends on these values. A flag that can be attached to a message or exception to indicate that it has a certain class of sensitive data. By default, messages should be specified as being of unknown sensitivity, which is to say, every sensitivity flag is turned on, corresponding to . Messages that are totally safe should be marked as . However, if, say, one prints out data from a file (for example, this might be done when expressing parse errors), it should be flagged in that case with . For non-sensitive data. For messages that may contain user-data from data files. For messages that contain information like column names from datasets. Note that, despite being part of the schema, annotations should be treated as user data, since it is often derived from user data. Note also that types, despite being part of the schema, are not considered "sensitive" as such, in the same way that column names might be. The default value, unknown, is treated as if everything is sensitive. An alias for , so it is functionally the same, except semantically it communicates the idea that we want all bits set. A channel message. Line endings may not be normalized. A standard communication channel. General utility extension methods for objects in the "host" universe, i.e., , , and that do not belong in more specific areas, for example, or component creation. Convenience variant of setting . Convenience variant of setting . Convenience variant of setting . Convenience variant of setting . Convenience variant of setting . Convenience variant of setting . Convenience variant of setting . Convenience variant of setting . This is a factory interface for . Both and implement this interface, to allow for nested progress reporters. REVIEW: make implement this, instead of the environment? Create a progress channel for a computation named . A common interface for progress reporting. It is expected that the progress channel interface is used from only one thread. Supported workflow: 1) Create the channel via . 2) Call as many times as desired (including 0). Each call to supersedes the previous one. 3) Report checkpoints (0 or more) by calling . 4) Repeat steps 2-3 as often as necessary. 5) Dispose the channel. Set up the reporting structure: - Set the 'header' of the progress reports, defining which progress units and metrics are going to be reported. - Provide a thread-safe delegate to be invoked whenever anyone needs to know the progress. It is acceptable to call multiple times (or none), regardless of whether the calculation is running or not. Because of synchronization, the computation should not deny calls to the 'old' delegates even after a new one is provided. The header object. The delegate to provide actual progress. The parameter of the delegate will correspond to the provided . Submit a 'checkpoint' entry. These entries are guaranteed to be delivered to the progress listener, if it is interested. Typically, this would contain some intermediate metrics, that are only calculated at certain moments ('checkpoints') of the computation. For example, SDCA may report a checkpoint every time it computes the loss, or LBFGS may report a checkpoint every iteration. The only parameter, , is interpreted in the following fashion: * First MetricNames.Length items, if present, are metrics. * Subsequent ProgressNames.Length items, if present, are progress units. * Subsequent ProgressNames.Length items, if present, are progress limits. * If any more values remain, an exception is thrown. The metrics, progress units and progress limits. This is the 'header' of the progress report. These are the names of the progress 'units', from the least granular to the most granular. For example, neural network might have {'epoch', 'example'} and FastTree might have {'tree', 'split', 'feature'}. Will never be null, but can be empty. These are the names of the reported metrics. For example, this could be the 'loss', 'weight updates/sec' etc. Will never be null, but can be empty. Initialize the header. This will take ownership of the arrays. Both arrays can be null, even simultaneously. This 'empty' header indicated that the calculation doesn't report any units of progress, but the tracker can still track start, stop and elapsed time. Of course, if there's any progress or metrics to report, it is always better to report them. The metrics that the calculation reports. These are completely independent, and there is no contract on whether the metric values should increase or not. As naming convention, can have multiple words with spaces, and should be title-cased. The names of the progress units, listed from least granular to most granular. The idea is that the progress should be lexicographically increasing (like [0,0], [0,10], [1,0], [1,15], [2,5] etc.). As naming convention, should be lower-cased and typically plural (for example, iterations, clusters, examples). A constructor for no metrics, just progress units. As naming convention, should be lower-cased and typically plural (for example, iterations, clusters, examples). A metric/progress holder item. Set the progress value for the index to , and the limit value for the progress becomes 'unknown'. Set the progress value for the index to , and the limit value to . If is a NAN, it is set to null instead. Sets the metric with index to . The progress reporting classes used by descendants. The progress channel for . This is coupled with a that aggregates all events and returns them on demand. The pair of (header, fill action) is updated atomically. Normally this should be readonly field, but we want to null it in Dispose to prevent memory leaking. Initialize a for the process identified by . The exception context. The tracker to couple with. The computation name. Pull the current progress by invoking the fill delegate, if any. This is a 'derived' or 'subordinate' progress channel. The subordinates' Start/Stop events and checkpoints will not be propagated. When the status is requested, all of the subordinate channels are also invoked, and the resulting metrics are then returned in the order of their 'subordinate level'. If there's more than one channel with the same level, the order is not defined. The pair of (header, fill action) is updated atomically. Pull the current progress by invoking the fill delegate, if any. This class listens to the progress reporting channels, caches all checkpoints and start/stop events and, on demand, requests current progress on all active calculations. The public methods of this class should only be called from one thread. Log of pending events. For each calculation, its properties. This list is protected by , and it's updated every time a new calculation starts. The entries are cleaned up when the start and stop events are reported (that is, after the first pull request after the calculation's 'Stop' event). This is a 'process index' that gets incremented whenever a new calculation is started. The set of used process names. This class is an 'event log' for one calculation. Every time a calculation is 'started', it gets its own log, so if there are multiple 'start' calls, there will be multiple logs. Auto-assigned index to serve as a unique ID. Name is auto-modified from the calculation name provided by the pipe. A log of pending checkpoint entries. Whether the calculation has finished. Get progress reports from all current calculations. For every calculation the following events will be returned: * A start event. * Each checkpoint. * If the calculation is finished, the stop event. Each of the above events will be returned exactly once. If, for one calculation, there's no events in the above categories, the tracker will request ('pull') the current progress and return this as an event. An array-backed implementation of . The header (names of metrics and units). The contents of the header should be treated as read-only. The calculation itself doesn't even need to access the header, since it will know it anyway. Whether the progress entry is a 'checkpoint' (that is, it's being pushed by the component). The actual progress (amount of completed units), in the units that are contained in the header. Parallel to the header's . Null value indicates 'not applicable now'. The computation should not modify these arrays directly, and instead rely on , and . The lim values of each progress unit. Parallel to the header's . Null value indicates unbounded or unknown. The reported metrics. Parallel to the header's . Null value indicates unknown. Set the progress value for the index to , and the limit value for the progress becomes 'unknown'. Set the progress value for the index to , and the limit value to . Sets the metric with index to . Creates the progress entry corresponding to a given header. An event about calculation progress. It could be either start/stop of the calculation, or a progress entry. Instances of this class are used to set up a bundle of named delegates. These delegates are registered through and its overloads. Once all registrations are done, is called and a message of type is sent through the input channel provider. The intended use case is that any information surfaced through these delegates will be published in some fashion, with the target scenario being that the library will publish some sort of restful API. Returns either this object, or null if there are no listeners on this server channel. This can be used in conjunction with the ?. operator to have more performant though more robust calls to and . Starts a new server channel. The channel provider, on which to send the notification that a server is being constructed A semi-unique identifier for this "bundle" that is being constructed The constructed server channel, or null if there was no listeners for server channels registered on Finalizes all registrations of delegates, and pipes the bundle of objects in a up through the pipe to be consumed by any listeners. Entry point factory for creating instances. Classes that want to publish the bundles from server channels in some fashion should implement this interface. The intended simple use case is that this will be some form of in-process web server, and then when disposed, they should stop themselves. Note that the primary communication with the server from the client code's perspective is not through method calls on this interface, but rather communication through an that the server will listen to throughout its lifetime. This should return the base address where the server is. If this server is not actually serving content at any URL, this property should be null. Creates what might be considered a good "default" server factory, if possible, or null if no good default was possible. A null value could be returned, for example, if a user opted to remove all implementations of and the associated for security reasons. When a is created, the creation method will send an implementation is a notification sent through an , to indicate that a may be pending soon. Listeners that want to receive the bundle to expose it, for example, a web service, should register this interest by passing in an action to be called. If no listener registers interest, the server channel that sent the notification will act differently by, say, acting as a no-op w.r.t. client calls to it. Any publisher of the named delegates will call this method, upon receiving an instance of this object through the pipe. This method serves two purposes: firstly it detects whether anyone is even interested in publishing anything at all, so that we can just ignore any input delegates in the case where no one is listening (which, we must expect, is the majority of scenarios). The second is that it provides an action to call, once all publishing is complete, and has been called by the client code. The callback to perform when all named delegates have been registered, and is called. The final bundle of published named delegates that a listener can serve. This contains a name to delegate mappings. The delegates contained herein are gauranteed to be some variety of , , , etc. This should be a more-or-less unique identifier for the type of API this bundle is producing. Its intended use is that it will form part of the URL for the RESTful API, so to the extent that it contains multiple tokens they must be slash delimited. Convenience method for that looks more idiomatic to typical channel creation methods on . The channel provider. This is an identifier of the "type" of bundle that is being published, and should form a path with forward-slash '/' delimiters. The newly created server channel, or null if there was no listener for server channels on . The console environment. As its name suggests, should be limited to those applications that deliberately want console functionality. Query all progress and: * If there's any checkpoint/start/stop event, print all of them. * If there's none, print a dot. * If there's dots, print the current status for all running calculations. If we printed any dots so far, finish the line. This call is expected to be protected by _lock. Print a progress dot. Returns whether it is 'time' to print more info. This call is expected to be protected by _lock. Create an ML.NET for local execution, with console feedback. Random seed. Set to null for a non-deterministic environment. Set to true for fully verbose logging. Allowed message sensitivity. Text writer to print normal messages to. Text writer to print error messages to. Optional TextWriter to write messages if the host is a test environment. Pull running calculations for their progress and output all messages to the console. If no messages are available, print a dot. If a specified number of dots are printed, print an ad-hoc status of all running calculations. Redirects the channel output through the specified writers. This method is not thread-safe. Base class for channel providers. This is a common base class for. The ParentFullName, ShortName, and FullName may be null or empty. Data keys that are attached to the exception thrown via the exception context. The channel depth, NOT host env depth. ExceptionContext description. Override this method to change the way full names are constructed. Message source (a channel) that generated the message being dispatched. A basic host environment suited for many environments. This also supports modifying the concurrency factor, provides the ability to subscribe to pipes via the AddListener/RemoveListener methods, and exposes the to query progress. Base class for hosts. Classes derived from may choose to provide their own host class that derives from this class. This encapsulates the random number generator and name information. This method registers and returns the host for the calling component. The generated host is also added to and encapsulated by . It becomes necessary to remove these hosts when they are reclaimed by the Garbage Collector. Base class for implementing . Deriving classes can optionally override the Done() and the DisposeCore() methods. If no overrides are needed, the sealed class may be used. A base class for implementations. A message is dispatched as a . Deriving classes can optionally override the Done() and the DisposeCore() methods. An optional implementation of . Base class for . The master host environment has a map from to . Strongly typed dispatcher class. This field is actually used as a , which holds the listener actions for all listeners that are currently subscribed. The action itself is an immutable object, so every time any listener subscribes or unsubscribes, the field is replaced with a modified version of the delegate. The field can be null, if no listener is currently subscribed. The dispatch delegate invokes the current dispatching action (wchch calls all current listeners). The main constructor. This constructor is for forking. This method registers and returns the host for the calling component. The generated host is also added to and encapsulated by . It becomes necessary to remove these hosts when they are reclaimed by the Garbage Collector. This method is called when a channel is created and when a listener is registered. This method is not invoked on every message. Line endings in message may not be normalized, this method provides normalized printing. The text writer to write to. The message, which if it contains newlines will be normalized. If false, then two newlines will be printed at the end, making messages be bracketed by blank lines. If true then only the single newline at the end of a message is printed. A prefix that will be written to every line, except the first line. If contains no newlines then this prefix will not be written at all. This prefix is not written to the newline written if is false. Interface for "processing" exceptions before they are thrown. This can be used to add context to the exception, wrap the exception in another one, totally replace the exception, etc. It is not legal to return null from Process (unless null was passed in, which really shouldn't happen). A string describing the context itself. Does standard processing of an exception (typically called after construction but before it is thrown). Mark the exception by setting in the exception to 1. Indicates whether the exception was "marked" the Contracts code. Exceptions whose message communicates potentially sensitive information should be marked using this method, before they are thrown. Note that if the exception already had this flag set, the message will be flagged with the bitwise or of the existing flag, alongside the passed in sensitivity. This is a convenience method to get the sensitivity of an exception, as encoded with . If there is no key, then the message is assumed to be of unknown sensitivity, i.e., it is assumed that it might contain literally anything. The exception to query The value encoded at the , if it is a value. If neither of these conditions hold then is returned. This is an internal convenience implementation of an exception context to make marking exceptions with a specific sensitivity flag a bit less onerous. The alternative to a scheme like this, where messages are marked through use of , would be that every check and exception method in this file would need some "peer" where sensitivity was set. Since there are so many, we have this method instead. I'm not sure if there will be performance implications. There shouldn't be, since checks rarely happen in tight loops. We will run this instances first. This can be null. Exceptions will be marked with this. If happens to mark it with a sensitivity flag, then the result will not only be this value, but the bitwise or of this with the existing value. A convenience context for marking exceptions from checks and excepts with . A convenience context for marking exceptions from checks and excepts with . A convenience context for marking exceptions from checks and excepts with . A convenience context for marking exceptions from checks and excepts with . A convenience context for marking exceptions from checks and excepts with . A convenience context for marking exceptions from checks and excepts with . Sets the assert handler to the given function, returning the previous handler. Default exception type (currently InvalidOperationException) For signaling bad user input. For signaling bad function parameters. For signaling null function parameters. For signaling null or empty function parameters (strings, arrays, collections, etc). For signaling null, empty or white-space function parameters (strings, arrays, collections, etc). For signaling errors in decoding information, whether while reading from a file, parsing user input, etc. For signaling IO failures. For signaling functionality that is not YET implemented. For signaling functionality that is not implemented by design. For signaling schema validation issues. CheckUserArg / ExceptUserArg should be used when the validation of user-provided arguments failed. Typically, this is shortly after the arguments are parsed using CmdParser. Check state of the host and throw exception if host marked to stop all execution. This documents that the parameter can legally be null. This documents that the parameter can legally be null. This documents that the parameter can legally be null. The base attribute type for all attributes used for extensibility purposes. Common signature type with no extra parameters. Assembly attribute used to specify that a class is loadable by a machine learning host environment, such as TLC The class type that is loadable The argument type that the constructor takes (may be null) The signature of the constructor of this class (in addition to the arguments parameter) The name to use when presenting a list to users The names that can be used to load the class, for example, from a command line Assembly attribute used to specify that a class is loadable by a machine learning host environment, such as TLC The class type that is loadable The class type that contains the construction method The argument type that the constructor takes (may be null) The signature of the constructor of this class (in addition to the arguments parameter) The name to use when presenting a list to users The names that can be used to load the class, for example, from a command line Assembly attribute used to specify that a class is loadable by a machine learning host environment, such as TLC The description summary of the class type The class type that is loadable The argument type that the constructor takes (may be null) The signature of the constructor of this class (in addition to the arguments parameter) The name to use when presenting a list to users The names that can be used to load the class, for example, from a command line Assembly attribute used to specify that a class is loadable by a machine learning host environment, such as TLC The description summary of the class type The class type that is loadable The class type that contains the construction method The argument type that the constructor takes (may be null) The signature of the constructor of this class (in addition to the arguments parameter) The name to use when presenting a list to users The names that can be used to load the class, for example, from a command line The type that is created/loaded. The type that contains the construction method, whether static Instance property, static Create method, or constructor. Of course, a constructor is only permissible if this type derives from InstanceType. This defaults to the same as InstanceType. The command line arguments object type. This should be null if there isn't one. This indicates the extra parameter types. It must be a delegate type. The return type should be void. The parameter types of the SigType delegate should NOT include the ArgType. Note that CtorTypes includes the ArgType (if there is one), and the parameter types of the SigType. The description summary of the class type. UserName may be null or empty indicating that it should be hidden in UI. This should indicate a path within the doc/public directory next to the TLC solution, where the documentation lies. This value will be used as part of a URL, so, the path separator should be phrased as '/' forward slashes rather than backslashes. Add slot names annotation. The to which to add the slot names. The size of the slot names vector. The getter delegate for the slot names. Add key values annotation. The value type of key values. The to which to add the key values. The size of key values vector. The value type of key values. Its raw type must match . The getter delegate for the key values. Utilities for implementing and using the annotation API of . This class lists the canonical annotation kinds Annotation kind for names associated with slots/positions in a vector-valued column. The associated annotation type is typically fixed-sized vector of Text. Annotation kind for values associated with the key indices when the column type's item type is a key type. The associated annotation type is typically fixed-sized vector of a primitive type. The primitive type is frequently Text, but can be anything. Annotation kind for sets of score columns. The value is typically a with raw type U4. Annotation kind that indicates the prediction kind as a string. For example, "BinaryClassification". The value is typically a ReadOnlyMemory<char>. Annotation kind that indicates the value kind of the score column as a string. For example, "Score", "PredictedLabel", "Probability". The value is typically a ReadOnlyMemory. Annotation kind that indicates if a column is normalized. The value is typically a Bool. Annotation kind that indicates if a column is visible to the users. The value is typically a Bool. Not to be confused with IsHidden() that determines if a column is masked. Annotation kind for the label values used in training to be used for the predicted label. The value is typically a fixed-sized vector of Text. Annotation kind that indicates the ranges within a column that are categorical features. The value is a vector type of ints with dimension of two. The first dimension represents the number of categorical features and second dimension represents the range and is of size two. The range has start and end index(both inclusive) of categorical slots within that column. This class holds all pre-defined string values that can be found in canonical annotations Helper delegate for marshaling from generic land to specific types. Used by the Marshal method below. Returns a standard exception for responding to an invalid call to GetAnnotation. Returns a standard exception for responding to an invalid call to GetAnnotation. Helper to marshal a call to GetAnnotation{TValue} to a specific type. Returns a vector type with item type text and the given size. The size must be positive. This is a standard type for annotation consisting of multiple text values, eg SlotNames. Returns a vector type with item type int and the given size. The range count must be a positive integer. This is a standard type for annotation consisting of multiple int values that represent categorical slot ranges with in a column. The type of the ScoreColumnSetId annotation. Returns a key-value pair useful when implementing GetAnnotationTypes(col). Returns a key-value pair useful when implementing GetAnnotationTypes(col). This assumes that the values of the key type are Text. Given a type and annotation kind string, returns a key-value pair. This is useful when implementing GetAnnotationTypes(col). Prepends a params array to an enumerable. Useful when implementing GetAnnotationTypes. Returns the max value for the specified annotation kind. The annotation type should be a with raw type U4. colMax will be set to the first column that has the max value for the specified annotation. If no column has the specified annotation, colMax is set to -1 and the method returns zero. The filter function is called for each column, passing in the schema and the column index, and returns true if the column should be considered, false if the column should be skipped. Returns the set of column ids which match the value of specified annotation kind. The annotation type should be a with raw type U4. Returns the set of column ids which match the value of specified annotation kind. The annotation type should be of type text. Returns true if the specified column: * has a SlotNames annotation * annotation type is VBuffer<ReadOnlyMemory<char>> of length . Returns whether a column has the annotation indicated by the schema shape. The schema shape column to query True if and only if the column has the annotation of a scalar type, which we assume, if set, should be true. Returns whether a column has the annotation indicated by the schema shape. The schema shape column to query True if and only if the column is a definite sized vector type, has the annotation of definite sized vectors of text. Tries to get the annotation kind of the specified type for a column. The raw type of the annotation, should match the PrimitiveType type The schema The type of the annotation The annotation kind The column The value to return, if successful True if the annotation of the right type exists, false otherwise The categoricalFeatures is a vector of the indices of categorical features slots. This vector should always have an even number of elements, and the elements should be parsed in groups of two consecutive numbers. So if its value is the range of numbers: 0,2,3,4,8,9 look at it as [0,2],[3,4],[8,9]. The way to interpret that is: feature with indices 0, 1, and 2 are one categorical Features with indices 3 and 4 are another categorical. Features 5 and 6 don't appear there, so they are not categoricals. Produces sequence of columns that are generated by trainer estimators. whether we should also append 'IsNormalized' (typically for probability column) Produces annotations for the score column generated by trainer estimators for multiclass classification. If input LabelColumn is not available it produces slotnames annotation by default. Label column. Returns a value getter delegate to fetch the value of column with the given columnIndex, from the row. This throws if the column is not active in this row, or if the type differs from this column's type. is the column's content type. is the output column whose getter should be returned. Returns whether the given column is active in this row. Presents a as a an . The annotations to wrap. A row that wraps an input annotations. Extension methods related to the ColumnType class. Whether this type is a standard scalar type completely determined by its (not a or , etc). Zero return means it's not a key type. Sometimes it is necessary to cast the Count to an int. This performs overflow check. Zero return means it's not a key type. For non-vector types, this returns the column type itself (i.e., return ). For vector types, this returns the type of the items stored as values in vector. Zero return means either it's not a vector or the size is unknown. For non-vectors, this returns one. For unknown size vectors, it returns zero. For known sized vectors, it returns size. Whether this is a vector type with known size. Returns false for non-vector types. Equivalent to > 0. Gets the equivalent for the 's RawType. This can return default() if the RawType doesn't have a corresponding . Equivalent to calling Equals(ColumnType) for non-vector types. For vector type, returns true if current and other vector types have the same size and item type. Specifies a simple data type. or [text](xref:Microsoft.ML.Data.TextDataViewType) | Empty or `null` string (both result in empty `System.ReadOnlyMemory` | | | [Key](xref:Microsoft.ML.Data.KeyDataViewType) type (supported by the unsigned integer types in `DataKind`) | Not defined | Always `false` | | All other types | Default value of the corresponding system type as defined by .NET standard. In C#, default value expression `default(T)` provides that value. | Equality test with the default value | The table below shows the missing value definition for each of the data types. | Type | Missing Value | IsMissing Indicator | | -- | -- | -- | | or [text](xref:Microsoft.ML.Data.TextDataViewType) | Not defined | Always `false` | | [Key](xref:Microsoft.ML.Data.KeyDataViewType) type (supported by the unsigned integer types in `DataKind`) | `0` | Equality test with `0` | | | | | | | | | | All other types | Not defined | Always `false` | ]]> 1-byte integer, type of . 1-byte unsigned integer, type of . 2-byte integer, type of . 2-byte unsigned integer, type of . 4-byte integer, type of . 4-byte unsigned integer, type of . 8-byte integer, type of . 8-byte unsigned integer, type of . 4-byte floating-point number, type of . 8-byte floating-point number, type of . string, type of , where T is . Also compatible with . boolean variable type, type of . type of . type of . type of . Data type specifier used in command line. is the underlying version of used for command line and entry point BC. Extension methods related to the DataKind enum. Maps a DataKind to a value suitable for indexing into an array of size KindCount. Maps from an index into an array of size KindCount to the corresponding DataKind This function converts to . Because is a subset of , the conversion is straightforward. This function converts to . Because is a subset of , we should check if can be found in . For integer DataKinds, this returns the maximum legal value. For un-supported kinds, it returns zero. For integer Types, this returns the maximum legal value. For un-supported Types, it returns zero. For integer DataKinds, this returns the minimum legal value. For un-supported kinds, it returns one. Maps a DataKind to the associated .Net representation type. Try to map a System.Type to a corresponding DataKind value. Get the canonical string for a DataKind. Note that using DataKind.ToString() is not stable and is also slow, so use this instead. A file handle. Returns whether CreateWriteStream is expected to succeed. Typically, once CreateWriteStream has been called once, this will forever more return false. Returns whether OpenReadStream is expected to succeed. Create a writable stream for this file handle. Open a readable stream for this file handle. A simple disk-based file handle. This interface maps an input to an output . Typically, the output contains both the input columns and new columns added by the implementing class, although some implementations may return a subset of the input columns. This interface is similar to , except it does not have any input role mappings, so to rebind, the same input column names must be used. Implementations of this interface are typically created over defined input . Mappers are defined as accepting inputs with this very specific schema. Gets an instance of which describes the columns' names and types in the output generated by this mapper. Given a set of columns, return the input columns that are needed to generate those output columns. Get an with the indicated active columns, based on the input . Getting values on inactive columns of the returned row will throw. The of should be the same object as . Implementors of this method should throw if that is not the case. Conversely, the returned value must have the same schema as . This method creates a live connection between the input and the output . In particular, when the getters of the output are invoked, they invoke the getters of the input row and base the output values on the current values of the input . The output values are re-computed when requested through the getters. Also, the returned will dispose when it is disposed. A mapper that can be bound to a (which encapsulates a and has mappings from column kinds to columns of that schema). Binding an to a produces an , which is an interface that has methods to return the names and indices of the input columns needed by the mapper to compute its output. The is an extention to this interface, that can also produce an output given an input . The produced generally contains only the output columns of the mapper, and not the input columns (but there is nothing preventing an from mapping input columns directly to outputs). This interface is implemented by wrappers of IValueMapper based predictors, which are predictors that take a single features column. New predictors can implement directly. Implementing includes implementing a corresponding (or ) and a corresponding ISchema for the output schema of the . In case the interface is implemented, the SimpleRow class can be used in the method. This interface is used to map a schema from input columns to output columns. The should keep track of the input columns that are needed for the mapping. The that was passed to the in the binding process. Gets schema of this mapper's output. A property to get back the that produced this . This method returns the binding information: which input columns are used and in what roles. This interface extends . Input schema accepted. Given a set of columns, from the newly generated ones, return the input columns that are needed to generate those output columns. Get an with the indicated active columns, based on the input . Getting values on inactive columns of the returned row will throw. The of should be the same object as . Implementors of this method should throw if that is not the case. Conversely, the returned value must have the same schema as . This method creates a live connection between the input and the output . In particular, when the getters of the output are invoked, they invoke the getters of the input row and base the output values on the current values of the input . The output values are re-computed when requested through the getters. Also, the returned will dispose when it is disposed. Delegate type to map/convert a value. Delegate type to map/convert among three values, for example, one input with two outputs, or two inputs with one output. Interface for mapping a single input value (of an indicated ColumnType) to an output value (of an indicated ColumnType). This interface is commonly implemented by predictors. Note that the input and output ColumnTypes determine the proper type arguments for GetMapper, but typically contain additional information like vector lengths. Get a delegate used for mapping from input to output values. Note that the delegate should only be used on a single thread - it should NOT be assumed to be safe for concurrency. Interface for mapping a single input value (of an indicated ColumnType) to an output value plus distribution value (of indicated ColumnTypes). This interface is commonly implemented by predictors. Note that the input, output, and distribution ColumnTypes determine the proper type arguments for GetMapper, but typically contain additional information like vector lengths. Get a delegate used for mapping from input to output values. Note that the delegate should only be used on a single thread - it should NOT be assumed to be safe for concurrency. Extension methods related to the class. Sometimes it is necessary to cast the Count to an int. This performs overflow check. Base class for a cursor has an input cursor, but still needs to do work on . Gets the input cursor. Returns the root cursor of the input. It should be used to perform operations, but with the distinction, as compared to , that this is not a simple passthrough, but rather very implementation specific. For example, a common usage of this class is on filter cursor implementations, where how that input cursor is consumed is very implementation specific. That is why this is , not . Base class for creating a cursor of rows that filters out some input rows. Return whether the current input row should be returned by this cursor. A base class for a that has an input cursor, but still needs to do work on . Note that the default assumes that each input column is exposed as an output column with the same column index. Gets row's schema. Returns whether the given column is active in this row. Returns a value getter delegate to fetch the value of column with the given columnIndex, from the row. This throws if the column is not active in this row, or if the type differs from this column's type. is the column's content type. is the output column whose getter should be returned. Compare equality with the given system string value. Splits on the left-most occurrence of separator and produces the left and right of values. If does not contain the separator character, this returns false and sets to this instance and to the default of value. Splits on the left-most occurrence of an element of separators character array and produces the left and right of values. If does not contain any of the characters in separators, this return false and initializes to this instance and to the default of value. Returns a of with leading and trailing spaces trimmed. Note that this will remove only spaces, not any form of whitespace. Returns a of with leading and trailing whitespace trimmed. Returns a of with trailing whitespace trimmed. Encapsulates a plus column role mapping information. The purpose of role mappings is to provide information on what the intended usage is for. That is: while a given data view may have a column named "Features", by itself that is insufficient: the trainer must be fed a role mapping that says that the role mapping for features is filled by that "Features" column. This allows things like columns not named "Features" to actually fill that role (as opposed to insisting on a hard coding, or having every trainer have to be individually configured). Also, by being a one-to-many mapping, it is a way for learners that can consume multiple features columns to consume that information. This class has convenience fields for several common column roles (for example, , ), but can hold an arbitrary set of column infos. The convenience fields are non-null if and only if there is a unique column with the corresponding role. When there are no such columns or more than one such column, the field is null. The , , and methods provide some cardinality information. Note that all columns assigned roles are guaranteed to be non-hidden in this schema. Note that instances of this class are, like instances of , immutable. It is often the case that one wishes to bundle the actual data with the role mappings, not just the schema. For that case, please use the class. Note that there is no need for components consuming a or to make use of every defined mapping. Consuming components are also expected to ignore any they do not handle. They may very well however complain if a mapping they wanted to see is not present, or the column(s) mapped from the role are not of the form they require. Instances of this are the keys of a . This class also holds some important commonly used pre-defined instances available (for example, , ) that should be used when possible for consistency reasons. However, practitioners should not be afraid to declare custom roles if approppriate for their task. Role for features. Commonly used as the independent variables given to trainers, and scorers. Role for labels. Commonly used as the dependent variables given to trainers, and evaluators. Role for group ID. Commonly used in ranking applications, for defining query boundaries, or sequence classification, for defining the boundaries of an utterance. Role for sample weights. Commonly used to point to a number to make trainers give more weight to a particular example. Role for sample names. Useful for informational and tracking purposes when scoring, but typically without affecting results. Role for feature contributions. Useful for specific diagnostic functionality. The string value for the role. Guaranteed to be non-empty. Constructor for the column role. The value for the role. Must be non-empty. Convenience method for creating a mapping pair from a role to a column name for giving to constructors of and . The column name to map to. Can be null, in which case when used to construct a role mapping structure this pair will be ignored A key-value pair with this instance as the key and as the value The source . The column, when there is exactly one (null otherwise). The column, when there is exactly one (null otherwise). The column, when there is exactly one (null otherwise). The column, when there is exactly one (null otherwise). The column, when there is exactly one (null otherwise). Returns whether there are any columns with the given column role. Returns whether there is exactly one column of the given role. Returns whether there are two or more columns of the given role. If there are columns of the given role, this returns the infos as a readonly list. Otherwise, it returns null. An enumerable over all role-column associations within this object. An enumerable over all role-column associations within this object. An enumerable over all role-column associations for the given role. This is a helper function for implementing the method. Returns the corresponding to if there is exactly one such mapping, and otherwise throws an exception. The role to look up The column corresponding to that role, assuming there was only one column mapped to that Constructor given a schema, and mapping pairs of roles to columns in the schema. This skips null or empty column-names. It will also skip column-names that are not found in the schema if is true. The schema over which roles are defined Whether to consider the column names specified "optional" or not. If false then any non-empty values for the column names that does not appear in will result in an exception being thrown, but if true such values will be ignored The column role to column name mappings Constructor given a schema, and mapping pairs of roles to columns in the schema. This skips null or empty column names. It will also skip column-names that are not found in the schema if is true. The schema over which roles are defined The column role to column name mappings Whether to consider the column names specified "optional" or not. If false then any non-empty values for the column names that does not appear in will result in an exception being thrown, but if true such values will be ignored Convenience constructor for role-mappings over the commonly used roles. Note that if any column name specified is null or whitespace, it is ignored. The schema over which roles are defined The column name that will be mapped to the role The column name that will be mapped to the role The column name that will be mapped to the role The column name that will be mapped to the role The column name that will be mapped to the role Any additional desired custom column role mappings Whether to consider the column names specified "optional" or not. If false then any non-empty values for the column names that does not appear in will result in an exception being thrown, but if true such values will be ignored Encapsulates an plus a corresponding . Note that the schema of of is guaranteed to equal the the of . The data. The role mapped schema. Note that 's is guaranteed to be the same as 's . Constructor given a data view, and mapping pairs of roles to columns in the data view's schema. This skips null or empty column-names. It will also skip column-names that are not found in the schema if is true. The data over which roles are defined Whether to consider the column names specified "optional" or not. If false then any non-empty values for the column names that does not appear in 's schema will result in an exception being thrown, but if true such values will be ignored The column role to column name mappings Constructor given a data view, and mapping pairs of roles to columns in the data view's schema. This skips null or empty column-names. It will also skip column-names that are not found in the schema if is true. The schema over which roles are defined The column role to column name mappings Whether to consider the column names specified "optional" or not. If false then any non-empty values for the column names that does not appear in 's schema will result in an exception being thrown, but if true such values will be ignored Convenience constructor for role-mappings over the commonly used roles. Note that if any column name specified is null or whitespace, it is ignored. The data over which roles are defined The column name that will be mapped to the role The column name that will be mapped to the role The column name that will be mapped to the role The column name that will be mapped to the role The column name that will be mapped to the role Any additional desired custom column role mappings Whether to consider the column names specified "optional" or not. If false then any non-empty values for the column names that does not appear in 's schema will result in an exception being thrown, but if true such values will be ignored Base class for creating a cursor with default tracking of . All calls to will be seen by subclasses of this cursor. For a cursor that has an input cursor and does not need notification on , use instead. Zero-based position of the cursor. Convenience property for checking whether the current state of the cursor is one where data can be fetched. Creates an instance of the class Channel provider Core implementation of , called if no prior call to this method has returned . Legacy method to get the column index. DO NOT USE: use instead. Base class for creating a cursor on top of another cursor that does not add or remove rows. It forces one-to-one correspondence between items in the input cursor and this cursor. It delegates all functionality except Dispose() to the root cursor. Dispose is virtual with the default implementation delegating to the input cursor. The synchronized cursor base, as it merely passes through requests for all "positional" calls (including , , , and so forth), offers an opportunity for optimization for "wrapping" cursors (which are themselves often implementors) to get this root cursor. But, this can only be done by exposing this root cursor, as we do here. Internal code should be quite careful in using this as the potential for misuse is quite high. Convenience property for checking whether the cursor is in a good state where values can be retrieved, that is, whenever is non-negative. Convenient base class for implementors that wrap a single as their input. The , , and are taken from this . The wrapped input row. This override of the dispose method by default only calls 's method, but subclasses can enable additional functionality via the functionality. Called from with in the case where that method has never been called before, and right after has been disposed. The default implementation does nothing. Whether this was called through the dispose path, as opposed to the finalizer path. The signature for commands. A set of 'requirements' to the incoming schema, as well as a set of 'promises' of the outgoing schema. This is more relaxed than the proper , since it's only a subset of the columns, and also since it doesn't specify exact 's for vectors and keys. The column name. The type of the column: scalar, fixed vector or variable vector. The 'raw' type of column item: must be a primitive type or a structured type. The flag whether the column is actually a key. If yes, is representing the underlying primitive type. The annotations that are present for this column. Returns whether is a valid input, if this object represents a requirement. Namely, it returns true iff: - The , , , fields match. - The columns of of is a superset of our columns. - Each such annotation column is itself compatible with the input annotation column. Return if this structure is not identical to the default value of . If true, it means this structure is initialized properly and therefore considered as valid. Given a , extract the type parameters that describe this type as a 's column type. The actual column type to process. The vector kind of . The item type of . Whether (or its item type) is a key. Create a schema shape out of the fully defined schema. Returns if there is a column with a specified and if so stores it in . The 'data loader' takes a certain kind of input and turns it into an . The type of input the loader takes. Produce the data view from the specified input. Note that 's are lazy, so no actual loading happens here, just schema validation. The output schema of the loader. Sometimes we need to 'fit' an . A DataLoader estimator is the object that does it. Train and return a data loader. The 'promise' of the output schema. It will be used for schema propagation. The transformer is a component that transforms data. It also supports 'schema propagation' to answer the question of 'how will the data with this schema look, after you transform it?'. Schema propagation for transformers. Returns the output schema of the data, if the input schema is like the one provided. Take the data in, make transformations, output the data. Note that 's are lazy, so no actual transformations happen here, just schema validation. Whether a call to should succeed, on an appropriate schema. Constructs a row-to-row mapper based on an input schema. If is false, then an exception should be thrown. If the input schema is in any way unsuitable for constructing the mapper, an exception should likewise be thrown. The input schema for which we should get the mapper. The row to row mapper. The estimator (in Spark terminology) is an 'untrained transformer'. It needs to 'fit' on the data to manufacture a transformer. It also provides the 'schema propagation' like transformers do, but over instead of . Train and return a transformer. Schema propagation for estimators. Returns the output schema shape of the estimator, if the input schema shape is like the one provided. This spells 'ML MODEL' with zero replacing space (assuming little endian). Initialize the header and writer for writing. The value of fpMin and header should be passed to the other utility methods here. The current writer position should be the end of the model blob. Records the model size, writes the string table, completes and writes the header, and writes the tail. The current writer position should be where the tail belongs. Writes the header and tail. Typically this isn't called directly unless you are doing custom string table serialization. In that case you should have called EndModelCore before writing the string table information. The current writer position should be the end of the model blob. Records the size of the model blob. Typically this isn't called directly unless you are doing custom string table serialization. Sets the version information the header. Record the given loader sig in the header. If sig is null, clears the loader sig. Record the given alternate loader sig in the header. If sig is null, clears the alternate loader sig. Low level method for copying bytes from a header structure into a byte array. Read the model header, strings, etc from reader. Also validates the header (throws if bad). Leaves the reader position at the beginning of the model blob. Finish reading. Checks that the current reader position is the end of the model blob. Seeks to the end of the entire model file (after the tail). Performs standard version validation. Low level method for copying bytes from a byte array to a header structure. Checks the basic validity of the header, assuming the stream is at least the given size. Returns false (and the out exception) on failure. Checks the validity of the header, reads the string table, etc. Extract and return the loader sig from the header, trimming trailing zeros. Extract and return the alternate loader sig from the header, trimming trailing zeros. This is used to simplify version checking boiler-plate code. It is an optional utility type. Construct version info with a string value for modelSignature. The string must be 8 characters all less than 0x100. Spaces are mapped to zero. This assumes little-endian. This is a convenience context object for loading models from a repository, for implementors of ICanSaveModel. It is not mandated but designed to reduce the amount of boiler plate code. It can also be used when loading from a single stream, for implementors of ICanSaveInBinaryFormat. When in repository mode, this is the repository we're reading from. It is null when in single-stream mode. When in repository mode, this is the directory we're reading from. Null means the root of the repository. It is always null in single-stream mode. The main stream reader. The strings loaded from the main stream's string table. The name of the assembly that the loader lives in. This may be null or empty if one was never written to the model, or is an older model version. The main stream's model header. The min file position of the main stream. Exception context provided by Repository (can be null). Returns whether this context is in repository mode (true) or single-stream mode (false). Create a ModelLoadContext supporting loading from a repository, for implementors of ICanSaveModel. Create a ModelLoadContext supporting loading from a single-stream, for implementors of ICanSaveInBinaryFormat. Performs version checks. Reads an integer from the load context's reader, and returns the associated string, or null (encoded as -1). Reads an integer from the load context's reader, and returns the associated string. Reads an integer from the load context's reader, and returns the associated string. Throws if the string is empty or null. Commit the load operation. This completes reading of the main stream. When in repository mode, it disposes the Reader (but not the repository). When in repository mode, this disposes the Reader (but no the repository). Returns the new assembly name to maintain backward compatibility. Return whether this context contains a directory and stream for a sub-model with the indicated name. This does not attempt to load the sub-model. Load an optional object from the repository directory. Returns false iff no stream was found for the object, iff result is set to null. Throws if loading fails for any other reason. Load an object from the repository directory. Load a sub model from the given sub directory if it exists. This requires InRepository to be true. Returns false iff no stream was found for the object, iff result is set to null. Throws if loading fails for any other reason. Load a sub model from the given sub directory. This requires InRepository to be true. Try to load from the given repository entry using the default loader(s) specified in the header. Returns false iff the default loader(s) could not be bound to a compatible loadable class. Load from the given repository entry using the default loader(s) specified in the header. Try to load from the given stream (non-Repository). Returns false iff the default loader(s) could not be bound to a compatible loadable class. Load from the given stream (non-Repository) using the default loader(s) specified in the header. Try to load from the given reader (non-Repository). Returns false iff the default loader(s) could not be bound to a compatible loadable class. Load from the given reader (non-Repository) using the default loader(s) specified in the header. Tries to load. Returns false iff the default loader(s) could not be bound to a compatible loadable class. Try to load a sub model from the given sub directory. This requires InRepository to be true. Try to load a binary stream from the current directory. This requires InRepository to be true. Try to load a text stream from the current directory. This requires InRepository to be true. Signature for a repository based model loader. This is the dual of . Convenience context object for saving models to a repository, for implementors of . This class reduces the amount of boiler plate code needed to implement . It can also be used when saving to a single stream, for implementors of . When in repository mode, this is the repository we're writing to. It is null when in single-stream mode. When in repository mode, this is the directory we're reading from. Null means the root of the repository. It is always null in single-stream mode. The main stream writer. The strings that will be saved in the main stream's string table. The main stream's model header. The min file position of the main stream. The wrapped entry. Exception context provided by Repository (can be null). The assembly name where the loader resides. Returns whether this context is in repository mode (true) or single-stream mode (false). Create a supporting saving to a repository, for implementors of . Create a supporting saving to a single-stream, for implementors of . Set the version information in the main stream's header. This should be called before is called. Puts a string into the context pool, and writes the integer code of the string ID to the write stream. If str is null, this writes -1 and doesn't add it to the pool. Puts a string into the context pool, and writes the integer code of the string ID to the write stream. Checks that str is not null. Puts a string into the context pool, and writes the integer code of the string ID to the write stream. Commit the save operation. This completes writing of the main stream. When in repository mode, it disposes (but not ). When in repository mode, this disposes the Writer (but not the repository). Save a sub model to the given sub directory. This requires to be . Save the object by calling TrySaveModel then falling back to .net serialization. Save to a single-stream by invoking the given action. Save to the given sub directory by invoking the given action. This requires to be . For saving a model into a repository. Classes implementing should do an explicit implementation of . Classes inheriting from a base class should overwrite the function invoked by in that base class, if there is one. For saving to a single stream. Note that this interface is mostly deprecated in favor of saving more comprehensive and composable "model" objects, via . Abstraction around a or other hierarchical storage. The relative path of this entry. /// The stream for this entry. This is either a memory stream or a file stream in the temporary directory. In either case, it is seekable and NOT the actual archive stream. Exception context. Force all open entries to be disposed. Remove the entry from _open. Note that under normal access patterns, entries are LIFO, so we search from the end of _open. The entry is being disposed. Note that overrides should always call RemoveEntry, in addition to whatever they need to do with the corresponding stream. When considering entries inside one of our model archives, we want to ensure that we use a consistent directory separator. Zip archives are stored as flat lists of entries. When we load those entries into our look-up dictionary, we normalize them to always use backward slashes. When building paths to our local file system, we want to force both forward and backward slashes to the system directory separator character. We do this for cases where we either used Windows-specific path building logic, or concatenated filesystem paths with zip archive entries on Linux. Constructs both the relative path to the entry and the absolute path of a corresponding temporary file. If createDir is true, makes sure the directory exists within the temp directory. Commit the writing of the repository. This signals successful completion of the write. This is a signature for classes that are 'holders' of entry points and components. A simplified assembly attribute for marking EntryPoint modules. Performs checks on an EntryPoint input class equivalent to the checks that are done when parsing a JSON EntryPoint graph. Call this method from EntryPoint methods to ensure that range and required checks are performed in a consistent manner when EntryPoints are created directly from code. Searches for the given column name in the schema. This method applies a common policy that throws an exception if the column is not found and the column name was explicitly specified. If the column is not found and the column name was not explicitly specified, it returns null. Converts EntryPoint Optional{T} types into nullable types, with the implicit value being converted to the null value. This class defines attributes to annotate module inputs, outputs, entry points etc. when defining the module interface. An attribute used to annotate the component. The load name of the component. Must be unique within its kind. UI friendly name. Can contain spaces and other forbidden for Name symbols. Alternative names of the component. Each alias must also be unique in the component's kind. Comma-separated . Description of the component. This should indicate a name of an embedded resource that contains detailed documents for the component, for example, markdown document with the .md extension. The embedded resource is assumed to be in the same assembly as the class on which this attribute is ascribed. An attribute used to annotate the signature interface. Effectively, this is a way to associate the signature interface with a user-friendly name. An attribute used to annotate the kind of entry points. Typically it is used on the input classes. An attribute used to annotate the outputs of the module. Official name of the output. If it is not specified, the field name is used. The description of the output. The rank order of the output. Because .NET reflection returns members in an unspecified order, this is the only way to ensure consistency. An attribute to indicate that a field is optional in an EntryPoint module. A node can be run without optional input fields. An attribute used to annotate the valid range of a numeric input. The target type of this range attribute, as determined by the type of the set range bound values. An inclusive lower bound of the value. An inclusive upper bound of the value. An exclusive lower bound of the value. An exclusive upper bound of the value. An attribute used to indicate suggested sweep ranges for parameter sweeping. An attribute used to indicate suggested sweep ranges for discrete parameter sweeping. The value is the index of the option chosen. Use Options[Value] to get the corresponding option using the index. An attribute used to indicate suggested sweep ranges for float parameter sweeping. An attribute used to indicate suggested sweep ranges for long parameter sweeping. An attribute to mark an entry point of a module. The entry point name. The entry point description. UI friendly name. Can contain spaces and other forbidden for Name symbols. Short name of the Entry Point The list of data types that are supported as inputs or outputs. Not used. Integer, including long. Unsigned integer, including ulong. Floating point, including double. A char. A string. A boolean value. A dataset, represented by an . A file handle, represented by an . A transform model, represented by an . A predictor model, represented by an . An enum: one value of a specified list. An array (0 or more values of the same type, accessible by index). A dictionary (0 or more values of the same type, identified by a unique string key). The underlying C# representation is A component of a specified kind. The component is identified by the "load name" (unique per kind) and, optionally, a set of parameters, unique to each component. Example: "BinaryClassifierEvaluator{threshold=0.5}". The C# representation is . The untyped base class for 'maybe'. Whether the value was set 'explicitly', or 'implicitly'. This is a 'maybe' class that is able to differentiate the cases when the value is set 'explicitly', or 'implicitly'. The idea is that if the default value is specified by the user, in some cases it needs to be treated differently than if it's auto-filled. An example is the weight column: the default behavior is to use 'Weight' column if it's present. But if the user explicitly sets the weight column to be 'Weight', we need to actually enforce the presence of the column. The type of the value Create the 'implicit' value. The implicit conversion into . The implicit conversion from . This will assume that the parameter is set 'explicitly'. Base type for standard predictor model port type. Save the model to the given stream. Extract only the transform portion of the predictor model. Extract the predictor object out of the predictor model. Apply the predictor model to the transform model and return the resulting predictor model. For a given input data, return role mapped data and the predictor object. The scoring entry point will hopefully know how to construct a scorer out of them. Returns a string array containing the label names of the label column type predictor was trained on. If the training label is a key with text key value annotation, it should return this annotation. The order of the labels should be consistent with the key values. Otherwise, it returns null. The column type of the label the predictor was trained on. Returns the that was used in training. Interface for standard transform model port type. The input schema that this transform model was originally instantiated on. Note that the schema may have columns that aren't needed by this transform model. If an exists with this schema, then applying this transform model to it shouldn't fail because of column type issues. The output schema that this transform model was originally instantiated on. The schema resulting from may differ from this, similarly to how may differ from the schema of dataviews we apply this transform model to. Apply the transform(s) in the model to the given input data. Apply the transform(s) in the model to the given transform model. Save the model to the given stream. Returns the transform model as an that can output a row given a row with the same schema as . The transform model as an . If not all transforms in the pipeline are then it returns . A telemetry message. Message with one long text and bunch of small properties (limit on value is ~1020 chars) Message with exception Message with metric value and it properites Type of prediction task. Note that this is a legacy structure and usage of this should generally be discouraged in future projects. Its presence suggests that there are privileged and supported tasks, and anything outside of this is unsupported. This runs rather contrary to the idea of this being an expandable framework, and it is inappropriately limiting. For legacy pipelines based on and it is still useful, but for things based on the idiom, it is inappropriate. Weakly typed version of IPredictor. Return the type of prediction task. A predictor the produces values of the indicated type. REVIEW: Determine whether this is just a temporary shim or long term solution. A predictor that produces values and distributions of the indicated types. Note that from a public API perspective this is bad. Loadable class signatures for trainers. Typically each trainer should register with both SignatureTrainer and SignatureXxxTrainer where Xxx is the prediction kind. The base interface for a trainers. Implementors should not implement this interface directly, but rather implement the more specific . Auxiliary information about the trainer in terms of its capabilities and requirements. Return the type of prediction task for the produced predictor. Trains a predictor. A context containing at least the training data The trained predictor Strongly typed generic interface for a trainer. A trainer object takes training data and produces a predictor. Type of predictor produced Trains a predictor. A context containing at least the training data The trained predictor Convenience train extension for the case where one has only a training set with no auxiliary information. Equivalent to calling on a constructed with . The trainer The training data. The trained predictor Convenience train extension for the case where one has only a training set with no auxiliary information. Equivalent to calling on a constructed with . The trainer The training data. The trained predictor Predictor that has ensemble tree structures and returns collection of trees. Returns the number of trees in the ensemble. Returns the collection of trees. Collection of trees Type of tree used in ensemble of tree based predictors Returns the array of right(Greater than) child nodes of every interior nodes Returns the array of left(Leser than or equal to) nodes of every interior nodes returns the number of interior nodes. Returns the number of leaf nodes. Returns node structure for the given node Node id Flag to denote whether the node is leaf or not Feature names collection Node structure Type of tree used in ensemble of tree based predictors Type of features container (instance) on which to make predictions Returns the leaf node for the given instance. Type of features container (instance) on which to make predictions node id Type of features container (instance) on which to make predictions Type to represent the structure of node Returns Key value pairs representing the properties of the node. Keys to represent the properties of node. Name of the the interior node. It is Feature name if it is fasttree. Type is string for default trees. Split gain of the interior node. Type is double for default trees. Threshold value of the interior node. Type is string for default trees. It is expected that the string has exactly two space separated components. i. The first one should be the operator ii. The second one should be the actual threshold For ex., for a split like f1 <= 10, expected Threshold is "<= 10" For a split like color not-in { blue, green }, expected Threshold is "not-in { blue, green }" Gain value (specific to fasttree) of the interior node. Type is double for default trees. Previous leaf value(specific to fasttree) of the interior node. Type is double for default trees. Leaf value of the leaf node. Type is double for default trees. Extra items that will be displayed in the tool-tip. Type is string for default trees. Holds information relevant to trainers. Instances of this class are meant to be constructed and passed into or . This holds at least a training set, as well as optionally a predictor. The training set. Cannot be null. The validation set. Can be null. Note that passing a non-null validation set into a trainer that does not support validation sets should not be considered an error condition. It should simply be ignored in that case. The test set, whose uses are very similar to validation set but it should not directly and indirectly affect the training process. One major difference between validation set and test test is that validation can affect the training process by, for example, early stopping. Note that early stopping is a technique which terminates the training process once the scores computed on validation set starts getting worse. The initial predictor, for incremental training. Note that if a implementor does not support incremental training, then it can ignore it similarly to how one would ignore . However, if the trainer does support incremental training and there is something wrong with a non-null value of this, then the trainer ought to throw an exception. Constructor, given a training set and optional other arguments. Will set to this value. This must be specified Will set to this value if specified Will set to this value if specified Will set to this value if specified Characteristics of a trainer. Exposed via the Info property of each trainer. Whether the trainer needs to see data in normalized form. Only non-parametric trainers will tend to return here. Whether the trainer needs calibration to produce probabilities. As a general rule only trainers that produce binary classifier predictors that also do not have a natural probabilistic interpretation should have a value here. Whether this trainer could benefit from a cached view of the data. Trainers that have few passes over the data, or that need to build their own custom data structure over the data, will have a false here. Whether the trainer supports validation set via . Not implementing this interface and returning false from this property is an indication the trainer does not support that. Whether the trainer can use test set via . Not implementing this interface and returning false from this property is an indication the trainer does not support that. Whether the trainer can support incremental trainers via . Not implementing this interface and returning true from this property is an indication the trainer does not support that. Initializes with the given parameters. The parameters have default values for the most typical values for most classical trainers. The value for the property The value for the property The value for the property The value for the property The value for the property The value for the property Akin to FindIndexSorted, except stores the found index in the output index parameter, and returns whether that index is a valid index pointing to a value equal to the input parameter value. Assumes input is sorted and finds value using BinarySearch. If value is not found, returns the logical index of 'value' in the sorted list i.e index of the first element greater than value. In case of duplicates it returns the index of the first one. It guarantees that items before the returned index are < value, while those at and after the returned index are >= value. Assumes input is sorted and finds value using BinarySearch. If value is not found, returns the logical index of 'value' in the sorted list i.e index of the first element greater than value. In case of duplicates it returns the index of the first one. It guarantees that items before the returned index are < value, while those at and after the returned index are >= value. An array-like data structure that supports storing more than many entries, up to 0x7FEFFFFF00000L. The entries are indexed by 64-bit integers, and a single entry can be accessed by the indexer if no modifications to the entries is desired, or the method. Efficient looping can be accomplished by calling the method. This data structure employs the "length and capacity" pattern. The logical length can be retrieved from the property, which can possibly be strictly less than the total capacity. The type of entries. Gets the logical length of the big array. Gets or sets the entry at . This indexer is not efficient for looping. If looping access to entries is desired, use the method instead. Note that unlike a normal array, the value returned from this indexer getter cannot be modified (for example, by ++ operator or passing into a method as a ref parameter). To modify an entry, use the method instead. Initializes a new instance of the class with a specified size. Applies a method at a given . Implements a more efficient way to loop over index range in [min, lim) and apply the specified method delegate. Fills the entries with index in [min, lim) with the given value. Resizes the array so that its logical length equals . This method is more efficient than initialize another array and copy the entries because it preserves existing blocks. The actual capacity of the array may become larger than . If equals , then no operation is done. If is less than , the array shrinks in size so that both its length and its capacity equal . If is larger than , the array capacity grows to the smallest integral multiple of that is larger than , unless is less than , in which case the capacity grows to double its current capacity or , which ever is larger, but up to . Trims the capacity to logical length. Appends the elements of to the end. This method is thread safe related to calls to (assuming those copy operations are happening over ranges already added), but concurrent calls to should not be attempted. Intended usage is that one thread will call this method, while multiple threads may access previously added ranges from , concurrently with this method or themselves. Copies the subarray starting from index of length to the destination array . Concurrent calls to this method is valid even with one single concurrent call to . This should normally be just (a + b) / 2, except in one interesting case: If a and b are 'consecutive' floats (they differ only in the least significant bit), the above expression is possible to be rounded to a. This can lead to bin bounds that are not strictly increasing! Also note that the simple (a + b) / 2 can overflow, that's the reason for a / 2 + b / 2. This should normally be just (a + b) / 2, except in one interesting case: If a and b are 'consecutive' floats (they differ only in the least significant bit), the above expression is possible to be rounded to a. This can lead to bin bounds that are not strictly increasing! Also note that the simple (a + b) / 2 can overflow, that's the reason for a / 2 + b / 2. Initial placement of the pegs. Places pegs one by one and always picks the largest existing segment to split. Gets the best split for a peg between min and max. In case of a tie, use 'pos' to pick the location closer to the more natural split. Computes the energy reduction for splitting segment [min, max] at 'split' point Returns true if 'i' is a better peg placement than 'j', which means it is closer to the ideal position for peg 'pos' After the initial peg placement, For the remaining bins, compute the best energy distribution and the worst energy distribution The best energy distribution: - make the distances as equal as possible; - some of them will be 'ave' and the rest will be 'ave+1' The worst energy distribution: - all except one distance will be 1 Environment variable containing optional resources path. Attempts to find a file that is expected to be distributed with a TLC component. Searches in the following order: 1. In the customSearchDir directory, if it is provided. 2. In the custom search directory specified by the environment variable. 3. In the root folder of the provided assembly. 4. In the folder of this assembly. In each case it searches the file in the directory provided and combined with folderPrefix. If any of these locations contain the file, a full local path will be returned, otherwise this method will return null. File name to find folder prefix, relative to the current or customSearchDir Custom directory to search for resources. If null, the path specified in the environment variable will be used. Assembly type to search the path of. Path to the existing file. Null if not found. Given a folder path, create it if it doesn't exist. Fails if the folder name is empty, or can't create the folder. Similar to Stream.CopyTo but takes a length rather than assuming copy to end. Returns amount copied. Source stream to copy from Destination stream to copy to Number of bytes to copy Size of buffer to use when copying, default is 81920 to match that of Stream number of bytes copied Writes a length prefixed span of ints. Writes a span of ints without the length prefix. Writes a length prefixed span of uints. Writes a span of uints without the length prefix. Writes a length prefixed array of bytes. Writes a length prefixed span of bytes. Writes a length prefixed array of bytes. Writes an array of bytes without the length prefix. Writes a length prefixed span of Floats. Writes a length prefixed array of Floats. Writes a span of Floats without the length prefix. Writes a length prefixed span of doubles. Writes a span of doubles without the length prefix. Writes a length prefixed span of bools as bytes with 0/1 values. Writes a span of bools as bytes with 0/1 values, without the length prefix. Writes a length prefixed span of chars. Writes a length prefixed array of packed bits. Writes what Microsoft calls a UTF-7 encoded number in the binary reader and writer string methods. For non-negative integers this is equivalent to LEB128 (see https://en.wikipedia.org/wiki/LEB128). The number of bytes that would be written if one were to attempt to write the value in LEB128. If this return it will read exactly length bytes, and unlike the regular read method fails if it cannot. The stream The buffer into which to write the data. The offset of the output array into which to write. The number of bytes to read. If this return it will try to read exactly length bytes. The stream The buffer into which to write the data. The offset of the output array into which to write. The number of bytes to read. Reads a LEB128 encoded unsigned integer. A convenience method to open a stream writer, by default with no-BOM UTF-8 encoding, buffer size of 1K, and the stream left open. Checks that the directory of the file name passed in already exists. This is meant to be called before calling an API that creates the file, so the file need not exist. An absolute or relative file path, or null to skip the check (useful for optional user parameters) The user level parameter name, as exposed by the command line help Naive thread pool focused on reducing the latency to execution of chunky work items as much as possible. If a thread is ready to process a work item the moment a work item is queued, it's used, otherwise a new thread is created. This is meant as a stop-gap measure for workloads that would otherwise be creating a new thread for every work item. How long should threads wait around for additional work items before retiring themselves. The queue of work items. Also used as a lock to protect all relevant state. The number of threads currently waiting for work to arrive. Queues an delegate to be executed immediately on another thread, and returns a that represents its eventual completion. The task will always end in the state; if the delegate throws an exception, it'll be allowed to propagate on the thread, crashing the process. Queues an delegate and associated state to be executed immediately on another thread, and returns a that represents its eventual completion. The task will always end in the state; if the delegate throws an exception, it'll be allowed to propagate on the thread, crashing the process. Copies the values from src to dst. This can be removed once we have the APIs from https://github.com/dotnet/corefx/issues/33006. Assumes input is sorted and finds value using BinarySearch. If value is not found, returns the logical index of 'value' in the sorted list i.e index of the first element greater than value. In case of duplicates it returns the index of the first one. It guarantees that items before the returned index are < value, while those at and after the returned index are >= value. Assumes input is sorted and finds value using BinarySearch. If value is not found, returns the logical index of 'value' in the sorted list i.e index of the first element greater than value. In case of duplicates it returns the index of the first one. It guarantees that items before the returned index are < value, while those at and after the returned index are >= value. Assumes input is sorted and finds value using BinarySearch. If value is not found, returns the logical index of 'value' in the sorted list i.e index of the first element greater than value. In case of duplicates it returns the index of the first one. It guarantees that items before the returned index are < value, while those at and after the returned index are >= value. Akin to FindIndexSorted, except stores the found index in the output index parameter, and returns whether that index is a valid index pointing to a value equal to the input parameter value. Akin to FindIndexSorted, except stores the found index in the output index parameter, and returns whether that index is a valid index pointing to a value equal to the input parameter value. Assumes input is sorted and finds value using BinarySearch. If value is not found, returns the logical index of 'value' in the sorted list i.e index of the first element greater than value. In case of duplicates it returns the index of the first one. It guarantees that items before the returned index are < value, while those at and after the returned index are >= value. Assumes input is sorted and finds value using BinarySearch. If value is not found, returns the logical index of 'value' in the sorted list i.e index of the first element greater than value. In case of duplicates it returns the index of the first one. It guarantees that items before the returned index are < value, while those at and after the returned index are >= value. Assumes input is sorted and finds value using BinarySearch. If value is not found, returns the logical index of 'value' in the sorted list i.e index of the first element greater than value. In case of duplicates it returns the index of the first one. It guarantees that items before the returned index are < value, while those at and after the returned index are >= value. Assumes input is sorted and finds value using BinarySearch. If value is not found, returns the logical index of 'value' in the sorted list i.e index of the first element greater than value. In case of duplicates it returns the index of the first one. It guarantees that items before the returned index are < value, while those at and after the returned index are >= value. Assumes input is sorted and finds value using BinarySearch. If value is not found, returns the logical index of 'value' in the sorted list i.e index of the first element greater than value. In case of duplicates it returns the index of the first one. It guarantees that items before the returned index are < value, while those at and after the returned index are >= value. Finds the unique index for which func(input[i]) == false whenever i < index and func(input[i]) == true whenever i >= index. Callers should guarantee that there is such an index. Uses binary search. Finds the unique index for which func(input[i], value) == false whenever i < index and func(input[i], value) == true whenever i >= index. Callers should guarantee that there is such an index. Uses binary search. Checks that an input IList is monotonically increasing. An array of values True if the array is monotonically increasing (if each element is greater than or equal to previous elements); false otherwise. ILists containing NaN values are considered to be not monotonically increasing. Checks that an input array is monotonically increasing. An array of values True if the array is monotonically increasing (if each element is greater than or equal to previous elements); false otherwise. Checks that an input array is monotonically increasing. An array of values True if the array is monotonically increasing (if each element is greater than or equal to previous elements); false otherwise. Arrays containing NaN values are considered to be not monotonically increasing. Returns whether an input integer vector is sorted and unique, and between an inclusive lower and exclusive upper bound for the first and last items, respectively. Returns whether an input integer vector up to is sorted and unique, and between an inclusive lower and exclusive upper bound for the first and last items, respectively. Create an array of specified length, filled with a specified value Given a predicate, over a range of values defined by a limit calculate first the values for which that predicate was true, and second an inverse map. The input schema where the predicate can check if columns are active. The predicate to test for various value An ascending array of values from 0 inclusive to exclusive, holding all values for which is true Forms an inverse mapping of , so that [[i]] == i, and for other entries not appearing in , [i] == -1 Given a predicate, over a range of values defined by a limit calculate first the values for which that predicate was true, and second an inverse map. Indicates the exclusive upper bound on the tested values The predicate to test for various value An ascending array of values from 0 inclusive to exclusive, holding all values for which is true Forms an inverse mapping of , so that [[i]] == i, and for other entries not appearing in , [i] == -1 Given the columns needed, over a range of values defined by a limit calculate first the values for which the column is present was true, and second an inverse map. Indicates the exclusive upper bound on the tested values The set of columns the calling component operates on. An ascending array of values from 0 inclusive to exclusive, holding all values for which are present. (The respective index appears in the collection). Forms an inverse mapping of , so that [[i]] == i, and for other entries not appearing in , [i] == -1 Resizes the array if necessary, to ensure that it has at least elements. The array to resize. Can be null. The minimum number of items the new array must have. True means that the old array is preserved, if possible (Array.Resize is called). False means that a new array will be allocated. The new size, that is no less than . Resizes the array if necessary, to ensure that it has at least and at most elements. The array to resize. Can be null. The minimum number of items the new array must have. The maximum number of items the new array can have. True means that the old array is preserved, if possible (Array.Resize is called). False means that a new array will be allocated. The new size, that is no less than and no more that . Returns the number of set bits in a bit array. Given a generic method with a single type parameter, re-create the generic method on a new type, then reinvoke the method and return the result. A common pattern throughout the code base is to have some sort of generic method, whose parameters and return value are, as defined, non-generic, but whose code depends on some sort of generic type parameter. This utility method exists to make this common pattern more convenient, and also safer so that the arguments, if any, can be type checked at compile time instead of at runtime. Because it is strongly typed, this can only be applied to methods whose return type is known at compile time, that is, that do not depend on the type parameter of the method itself. The type of the receiver of the instance method. The type of the return value of the method. A delegate that should be a generic method with a single type parameter. The generic method definition will be extracted, then a new method will be created with the given type parameter, then the method will be invoked. The target of the invocation. The new type parameter for the generic method The return value of the invoked function A static version of . A one-argument version of . A one-argument version of . A one-argument, three-type-parameter version of . A one-argument, three-type-parameter version of . A two-argument version of . A two-argument version of . A two-argument, two-type-parameter version of . A two-argument, two-type-parameter version of . A two-argument, three-type-parameter version of . A two-argument, three-type-parameter version of . A three-argument version of . A four-argument version of . A five-argument version of . A six-argument version of . A seven-argument version of . An eight-argument version of . A nine-argument version of . A ten-argument version of . This is akin to , except applied to instead of . A delegate that should be a generic method with a single type parameter. The generic method definition will be extracted, then a new method will be created with the given type parameter, then the method will be invoked. The new type parameter for the generic method A one-argument version of . A two-argument version of . A three-argument version of . A four-argument version of . Get indented version of command line or same string if we unable to produce it. command line indented version of command line(if possible) Try produce indented string for command line. command line indenting text writer true if we was able to produce indented string without any problem Result codes from Parse methods. Note that the order matters. No issues. Empty or only whitespace Extra non-whitespace characters after successful parse Parsing error This produces zero for an empty string, or NaN depending on the used. This produces zero for an empty string, or NaN depending on the used. A fixed-length circular array. Items are added at the end. If the array is full, adding an item will result in discarding the least recently added item. Represents the for a generic function corresponding to , with the following characteristics: The method is an instance method on an object of type . One generic type argument. A return value of . The type of the receiver of the instance method. The type of the return value of the method. Creates a representing the for a generic instance method. This helper method allows the instance to be created prior to the creation of any instances of the target type. The following example shows the creation of an instance representing the method: FuncInstanceMethodInfo1<object, int>.Create(obj => obj.GetHashCode) The expression which creates the delegate for an instance of the target type. A representing the for the generic instance method. Represents the for a generic function corresponding to , with the following characteristics: The method is an instance method on an object of type . One generic type argument. A return value of . The type of the receiver of the instance method. The type of the parameter of the method. The type of the return value of the method. Creates a representing the for a generic instance method. This helper method allows the instance to be created prior to the creation of any instances of the target type. The following example shows the creation of an instance representing the method: FuncInstanceMethodInfo1<object, object, int>.Create(obj => obj.Equals) The expression which creates the delegate for an instance of the target type. A representing the for the generic instance method. Represents the for a generic function corresponding to , with the following characteristics: The method is an instance method on an object of type . One generic type argument. A return value of . The type of the receiver of the instance method. The type of the first parameter of the method. The type of the second parameter of the method. The type of the return value of the method. Creates a representing the for a generic instance method. This helper method allows the instance to be created prior to the creation of any instances of the target type. The following example shows the creation of an instance representing the method: FuncInstanceMethodInfo1<object, object, int>.Create(obj => obj.Equals) The expression which creates the delegate for an instance of the target type. A representing the for the generic instance method. Represents the for a generic function corresponding to , with the following characteristics: The method is an instance method on an object of type . Two generic type arguments. A return value of . The type of the receiver of the instance method. The type of the first parameter of the method. The type of the second parameter of the method. The type of the return value of the method. Creates a representing the for a generic instance method. This helper method allows the instance to be created prior to the creation of any instances of the target type. The following example shows the creation of an instance representing the method: FuncInstanceMethodInfo1<object, object, int>.Create(obj => obj.Equals) The expression which creates the delegate for an instance of the target type. A representing the for the generic instance method. Represents the for a generic function corresponding to , with the following characteristics: The method is an instance method on an object of type . Three generic type arguments. A return value of . The type of the receiver of the instance method. The type of the parameter of the method. The type of the return value of the method. Creates a representing the for a generic instance method. This helper method allows the instance to be created prior to the creation of any instances of the target type. The following example shows the creation of an instance representing the method: FuncInstanceMethodInfo1<object, object, int>.Create(obj => obj.Equals) The expression which creates the delegate for an instance of the target type. A representing the for the generic instance method. Represents the for a generic function corresponding to , with the following characteristics: The method is an instance method on an object of type . Three generic type arguments. A return value of . The type of the receiver of the instance method. The type of the first parameter of the method. The type of the second parameter of the method. The type of the return value of the method. Creates a representing the for a generic instance method. This helper method allows the instance to be created prior to the creation of any instances of the target type. The following example shows the creation of an instance representing the method: FuncInstanceMethodInfo1<object, object, int>.Create(obj => obj.Equals) The expression which creates the delegate for an instance of the target type. A representing the for the generic instance method. Represents the for a generic function corresponding to , with the following characteristics: One generic type argument. A return value of . The type of the return value of the method. Represents the for a generic function corresponding to , with the following characteristics: One generic type argument. A return value of . The type of the parameter of the method. The type of the return value of the method. Represents the for a generic function corresponding to , with the following characteristics: One generic type argument. A return value of . The type of the first parameter of the method. The type of the second parameter of the method. The type of the return value of the method. Represents the for a generic function corresponding to , with the following characteristics: Two generic type arguments. A return value of . The type of the first parameter of the method. The type of the second parameter of the method. The type of the return value of the method. Represents the for a generic function corresponding to , with the following characteristics: Three generic type arguments. A return value of . The type of the parameter of the method. The type of the return value of the method. Represents the for a generic function corresponding to , with the following characteristics: Three generic type arguments. A return value of . The type of the first parameter of the method. The type of the second parameter of the method. The type of the return value of the method. Represents the for a generic function corresponding to , with the following characteristics: The method is static. One generic type argument. A return value of . The type of the return value of the method. Represents the for a generic function corresponding to , with the following characteristics: The method is static. One generic type argument. A return value of . The type of the parameter of the method. The type of the return value of the method. Represents the for a generic function corresponding to , with the following characteristics: The method is static. One generic type argument. A return value of . The type of the first parameter of the method. The type of the second parameter of the method. The type of the return value of the method. Represents the for a generic function corresponding to , with the following characteristics: The method is static. Two generic type arguments. A return value of . The type of the first parameter of the method. The type of the second parameter of the method. The type of the return value of the method. Represents the for a generic function corresponding to , with the following characteristics: The method is static. Three generic type arguments. A return value of . The type of the parameter of the method. The type of the return value of the method. Represents the for a generic function corresponding to , with the following characteristics: The method is static. Three generic type arguments. A return value of . The type of the first parameter of the method. The type of the second parameter of the method. The type of the return value of the method. A Hash Array that implements both val -> index lookup and index -> val lookup. Also implements memory efficient sorting. Note: Supports adding and looking up of items but does not support removal of items. Find the index of the given val in the hash array. Returns a bool representing if val is present. Index outputs the index that the val is at in the array, -1 otherwise. Make sure the given value has an equivalent TItem in the hashArray and return the index of the value. Make sure the given value has an equivalent TItem in the hashArray and return the index of the value. Adds the value as a TItem. Does not check whether the TItem is already present. Returns the index of the added value. Appropriately fills _rgnext and _rgit based on _rgt and _rghash. Copies all items to the passed in span. Requires the passed in span to be at least the same length as Count. Creates a combined hash of possibly heterogeneously typed values. The leading hash, incorporated into the final hash A variable list of objects, where null is a valid value The combined hash incorporating a starting hash, and the hash codes of all input values Creates a combined hash of multiple homogeneous typed non-null values. The type of items to hash The leading hash, incorporated into the final hash A variable list of non-null values The combined hash incorporating a starting hash, and the hash codes of all input values Hash the characters in a of . This MUST produce the same result as the other overloads (with equivalent characters). Hash the characters in a string builder. This MUST produce the same result as HashString(sb.ToString()). Combines the given hash value with a uint value, using the murmur hash 3 algorithm. Make certain to also use or on the final hashed value, if you depend upon having distinct bits. Implements the murmur hash 3 algorithm, using a mock UTF-8 encoding. The UTF-8 conversion ignores the possibilities of unicode planes other than the 0th. That is, it simply converts char values to one, two, or three bytes according to the following rules: * 0x0000 to 0x007F : 0xxxxxxx * 0x0080 to 0x07FF : 110xxxxx 10xxxxxx * 0x0800 to 0xFFFF : 1110xxxx 10xxxxxx 10xxxxxx NOTE: This MUST match the StringBuilder version below. Implements the murmur hash 3 algorithm, using a mock UTF-8 encoding. The UTF-8 conversion ignores the possibilities of unicode planes other than the 0th. That is, it simply converts char values to one, two, or three bytes according to the following rules: * 0x0000 to 0x007F : 0xxxxxxx * 0x0080 to 0x07FF : 110xxxxx 10xxxxxx * 0x0800 to 0xFFFF : 1110xxxx 10xxxxxx 10xxxxxx NOTE: This MUST match the string version above. Performs a MurmurRound on each int in the sequence The final mixing ritual for the Murmur3 hashing algorithm. Most users of will want to close their progressive building of a hash with a call to this method. Implements a heap. A Heap structure gives efficient access to the ordered next element. A delegate that takes two T objects, and if it returns true then the second object should be popped before the first A Heap structure gives efficient access to the ordered next element. A delegate that takes two T objects, and if it returns true then the second object should be popped before the first The initial capacity of the heap Func tests true if first element should be after the second Current count of elements remaining in the heap Discard all elements currently in the heap Peek at the first element in the heap Remove and return the first element in the heap The first element in the heap Add a new element to the heap The item to add For the heap to allow deletion, the heap node has to derive from this class. Implements a heap. A Heap structure gives efficient access to the ordered next element. A delegate that takes two T objects, and if it returns true then the second object should be popped before the first A Heap structure gives efficient access to the ordered next element. A delegate that takes two T objects, and if it returns true then the second object should be popped before the first The initial capacity of the heap Func tests true if first element should be after the second Current count of elements remaining in the heap Remove all elements currently in the heap Peek at the first element in the heap Remove and return the first element in the heap Add a new element to the heap Remove an element from the heap This is a read-write stream that, at or below a length threshold given in the constructor, works purely in memory, but if it ever grows larger than that point backs off to use the file system. This can be useful if we have intermediate operations that require streams. The temporary file will be destroyed if the object is properly disposed. Constructs an initially empty read-write stream. Once the number of bytes in the stream exceeds , then we back off to disk. The maximum length we will accommodate in memory A common usecase of the hybrid memory stream is to create a persistent readable (not necessarily writable) copy of a stream whose source is very transient and temporary. This utility method makes that creation of a copy somewhat easier. A stream that can be opened The maximum length we will accommodate in memory A readable copy of the data stream Creates the overflow stream if it does not exist, switching all content over to the file-based stream and disposing of the memory stream. If the overflow stream already exists, this method has no effect. we count the number of values first to allocate a single array with of proper size Implements a least recently used cache. Returns the keys of the items stored in the cache, sorted from the most recently used to the least recently used. Initializes a new LRU cache with a given size. The class is not thread safe. Looks up and returns an item. If the item is found, mark it as recently used. Adds a new item in the cache. It will be marked as recently used. If the cache would grow over the max size, the least recently used item is removed. Some useful math methods. Finds the best least-squares fit of y = ax + b The x values. The y values. The coefficent a. The intercept b. The product of elements in a an array the product of a's elements Find the max element of a an array the max element Find the minimum element of a an array the minimum element Finds the first index of the max element of the span. NaNs are ignored. If all the elements to consider are NaNs, -1 is returned. The caller should distinguish in this case between two possibilities: 1) The number of the element to consider is zero. 2) All the elements to consider are NaNs. The span of floats. the first index of the max element Finds the first index of the minimum element of the span. NaNs are ignored. If all the elements to consider are NaNs, -1 is returned. The caller should distinguish in this case between two possibilities: 1) The number of the element to consider is zero. 2) All the elements to consider are NaNs. The span of floats. the first index of the minimum element computes the "softmax" function: log sum_i exp x_i Span of numbers to softmax the softmax of the numbers may have slightly lower roundoff error if inputs are sorted, smallest first computes "softmax" function of two arguments: log (exp x + exp y) true if two float values are close (using relative comparison) Returns the log of the gamma function, using the Stirling approximation Argument of function Log Gamma(x) Accurate to eight digits for all x. Computes the log factorial of n, using fast methods The number to compute the factorial of The log factorial of n Returns the two-tailed p-value given a t statistic from a distribution parameterized by the provided number of degrees of freedom. The t-statistic The degrees of freedom The corresponding two-tailed p-value Lentz's algorithm for evaluating the continued fraction b0 + a1 / (b1 + a2 / (b2 + a3 / (b3 + a4 / ...) ) ) The a function mapping positive integers to a sequence term The b function mapping non-negative integers to a sequence term Calculate the continued fraction to this tolerance The evaluation of the continued fraction The logistic sigmoid function: 1 / (1 + e^(-x)). Hyperbolic tangent. The logistic sigmoid function: 1 / (1 + e^(-x)). Hyperbolic tangent. The exponential function: e^(x). The logistic sigmoid function: 1 / (1 + e^(-x)). The hyperbolic tangent function. The exponential function: e^(x). Apply a soft max on an array of floats. Note that src and dst may be the same array. Apply a soft max on a range within an array of floats. Note that src and dst may be the same array. Entropy of a given probability Cross-entropy of two distributions Given a set of values Ln(a1), Ln(a2), ... Ln(an), return Ln(a1+a2+...+an). This is especially useful when working with log probabilities and likelihoods. Math.Sin returns the input value for inputs with large magnitude. We return NaN instead, for consistency with Math.Sin(infinity). Math.Cos returns the input value for inputs with large magnitude. We return NaN instead, for consistency with Math.Cos(infinity). Returns the smallest integral value that is greater than or equal to the result of the division. Number to be divided. Number with which to divide the numerator. Swap the first m*n elements within a given array so that, for any non-negative i and j less than m and n respectively, dst[i*n+j] == src[j*m+i]. This variant of the function is single threaded, and useful in that case where the workflow is otherwise single threaded. Elements of the array are this type Where to write the transpose. Note that dst cannot be the same as src. Swap the first m*n elements within a given array so that, for any non-negative i and j less than m and n respectively, dst[i*n+j] == src[j*m+i] Elements of the array are this type The source elements of the transpose. Must contain at least m*n elements. Where to write the transpose. Note that dst cannot be the same as src. Must contain at least m*n elements. The major index. The minor index. Elements are currently stored in "m" blocks of "n" items. Swap the first m*n elements within a given array so that, for any non-negative i and j less than m and n respectively, b[i*n+j] == a[j*m+i] Where to write the transpose. Note that dst cannot be the same as src. Swap the first m*n elements within a given array so that, for any non-negative i and j less than m and n respectively, b[i*n+j] == a[j*m+i] Where to write the transpose. Note that dst cannot be the same as src. A synchronization primitive meant to address situations where you have a set of entities of known count, where you want to iteratively provide critical sections for each depending on which comes first, but you do not necessarily know what constitutes "first" until all such entities tell you where they stand in line. The anticipated usage is that whatever entity is using the to synchronize itself, will register itself using so as to unblock any "lower" waiters as soon as it knows what value it needs to wait on, perform whatever local work it can, and when it needs to, wait on the event it got when it registered itself. It may then repeat the cycle by registering itself for a new event (or, finally, retiring itself through ). This is an event-line pair. The intended usage is, when the line is the minimum at a point when all waiters have registered, the event will be signaled. Creates a minimum waiter. The initial number of waiters Indicates to the waiter that we want to, at some future point, wait at a given position. This object will return a reset event that can be waited on, at the point when we actually want to wait. This method itself has the potential to signal other events, if by registering ourselves the waiter becomes aware of the maximum number of waiters, allowing that waiter to enter its critical state. If multiple events are associated with the minimum value, then only one will be signaled, and the rest will remain unsignaled. Which is chosen is undefined. Retires one of the waiters, and return the current maximum number of waiters. If it so happens that by retiring this waiter the number of waiters reaches the maximum, the appropriate waiter will be signaled as described in . Normalized string type. For string pooling. NormStr's can only be created by the Pool. Find the given string in the pool. If not found, returns null. Make sure the given string has an equivalent NormStr in the pool and return it. Make sure the given string has an equivalent NormStr in the pool and return it. Make sure the given string builder has an equivalent NormStr in the pool and return it. Adds the item. Does NOT check for whether the item is already present. The primary use case for this structure is to impose ordering among multiple producer threads, in case one is producing output that one wishes to have ordered. More specifically, the ordered waiter allows a thread to wait on a particular position. So if three threads wait on 0, 1, then 2 (in any order), the first thread to clear the wait will be 0, then 1 will be cleared once incremented, then 2 will be cleared once incremented. This is an event-line pair. The intended usage is, when the line is hit by the containing ordered waiter, the thread will be hit. Creates an ordered waiter. If true, then the first position (that is, position 0) will be considered already free to proceed. If not something will need to hit increment. Creates an ordered waiter. If startPos is >= 0 then waiter starts from that position. If not something will need to hit increment |startPos| times. Wait on a given position. This will block, until this object has called up to the position indicated. This accepts cancellation tokens, but the default cancellation token also works. Moves the waiter to the next position, and signals any waiters waiting at or before that position. Signals all waiters. No more calls to should be attempted. This will signal all the waiters, but cause them to throw an exception. The exception that will be the inner exception, of an exception that will throw for all current and subsequent waiters. Contains extension methods that aid in building cross platform. This is an interface for creating samples of a requested size from a stream of data of type . The sample is created in one pass by calling for every data point in the stream. Implementations should have a delegate for getting the next data point, which is invoked if the current data point should go into the reservoir. If the number of elements sampled is less than the reservoir size, this should return the number of elements sampled. Otherwise it should return the reservoir size. Returns the number of elements sampled so far. Sample the next data point from the stream. This must be called before any calls to , and no subsequent calls to can be made after that. Return the elements in the sample. This class produces a sample without replacement from a stream of data of type . It is instantiated with a delegate that gets the next data point, and builds a reservoir in one pass by calling for every data point in the stream. In case the next data point does not get 'picked' into the reservoir, the delegate is not invoked. Sampling is done according to the algorithm in this paper: https://epubs.siam.org/doi/pdf/10.1137/1.9781611972740.53. Gets the reservoir sample. This class produces a sample with replacement from a stream of data of type . It is instantiated with a delegate that gets the next data point, and builds a reservoir in one pass by calling for every data point in the stream. In case the next data point does not get 'picked' into the reservoir, the delegate is not invoked. Sampling is done according to the algorithm in this paper: https://epubs.siam.org/doi/pdf/10.1137/1.9781611972740.53. Returns the cache. Users should not change the elements of the returned array. Access only elements up to the logical length of the array, which is . Gets a reservoir sample with replacement of the elements sampled so far. Users should not change the elements returned since multiple elements in the reservoir might be pointing to the same memory. This class takes care of downloading resources needed by ML.NET components. Resources are located in a pre-defined location, that can be overridden by defining Environment variable . The location of the resources to download from. Uses either the default location or a location defined in an Environment variable. An environment variable containing a timeout period (in milliseconds) for downloading resources. If defined, overrides the timeout defined in the code. Environment variable containing optional url to download resources from. Generates a url from a suffix. Returns a that tries to download a resource from a specified url, and returns the path to which it was downloaded, and an exception if one was thrown. The function checks whether or not the absolute URL with the default host "aka.ms" formed from redirects to the default Microsoft homepage. As such, only absolute URLs with the host "aka.ms" is supported with . The host environment. A channel to provide information about the download. The relative url from which to download. This is appended to the url defined in . The name of the file to save. The directory where the file should be saved to. The file will be saved in a directory with the specified name inside a folder called "mlnet-resources" in the directory. An integer indicating the number of milliseconds to wait before timing out while downloading a resource. The download results, containing the file path where the resources was (or should have been) downloaded to, and an error message (or null if there was no error). Returns the error message if an error occurred, null if download was successful. Get the path where the resource should be downloaded to. If the environment variable is defined, download to the location defined there. Otherwise, download to the "dir" directory inside \mlnet-resources\. This method checks whether or not the provided aka.ms url redirects to Microsoft's homepage, as the default faulty aka.ms URLs redirect to https://www.microsoft.com/?ref=aka . The provided url to check A class containing common statistical functions Returns a number uniformly sampled from 0...(rangeSize-1) Size of range to sample from, between 0 and int.MaxValue^2 Random number generator Sampled value Returns a number sampled from a zero-mean, unit variance Gaussian A Random to use for the sampling a sample uses Joseph L. Leva's algorithm from "A fast normal random number generator", 1992 Returns a sample from the gamma distribution with scale parameter 1, shape parameter alpha Shape parameter The random number generator to use Sample from gamma distribution Uses Marsaglia and Tsang's fast algorithm Generates a beta-distributed random variable Random generator to use first parameter second parameter Sample from distribution Generates a dirichlet-distributed random variable Random generator to use array of parameters array in which to store resulting sample Sample from a standard Cauchy distribution: https://en.wikipedia.org/wiki/Lorentzian_function Returns a number sampled from the binomial distribution with parameters n and p Random generator to use Parameter N of binomial Parameter p of binomial Should be robust for all values of n, p Returns a "view" stream, which appears to be a possibly truncated version of the stream. Reads on this containing stream will also advance the wrapped stream. If truncated, reads will not progress beyond the indicated length, and writes will fail beyond the indicated length. This stream supports seeking (and associated operations) if the underlying stream supports seeking, where it is supposed that the returned SubsetStream instance has position 0 during creation, corresponding to whatever the position of the enclosed stream was during creation, so this stream will act as an "offset" version of the enclosed stream. As this is intended to operate over a subset of a stream, during closing or disposal of the subset stream, the underlying stream will always remain open and undisposed. Construct the view stream. The underlying stream The maximum length this containing stream should appear to have, or null if unbounded The running (unweighted) count of elements added to this object. The running (weighted) count of elements. The running count of non-zero elements. The running weight of non-zero elements. The running arithmetic mean. Thes sample variance. The sample standard deviation. Returns the standard error of the mean. The maximum. The minimum. Accumulates one more value, optionally weighted. This accumulation procedure is based on the following, with adjustments as appropriate for weighted instances: https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance The value The weight given to this value Amount of appearance of this value Adds a stats object with another type of stats object. The result should be equivalent, up to the effects of numerical imprecision, as if in addition to all the values this object has seen, it has also seen the values added to the other object. A convenient way to combine the observations of two Stats objects into a new Stats object. The first operand The second operand A class for one-pass accumulation of weighted summary statistics, up to the fourth moment. The accumulative algorithms used here may be reviewed at https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance All quantities are weighted, except for RawCount. The running count of non-zero elements. The sample skewness. The sample kurtosis. Z-test value for a normality test based on the skew. Under the null hypothesis of normality this quantity will itself be approximately normally distributed. Z-test value for a normality test based on the kurtosis. Under the null hypothesis of normality this quantity will itself be approximately normally distributed. Omnibus K2 unifying the skew and kurtosis Z-tests. Under the null hypothesis of normality this quantity will be approximately chi-squared distributed. D'Agostino, Ralph B.; Albert Belanger; Ralph B. D'Agostino, Jr (1990). "A suggestion for using powerful and informative tests of normality". The American Statistician 44 (4): 316–321. JSTOR 2684359. Accumulates one more value, optionally weighted. This accumulation procedure is based on the following, with adjustments as appropriate for weighted instances: https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance The value The weight given to this value Amount of appearance of this value Adds a stats object with another type of stats object. The result should be equivalent, up to the effects of numerical imprecision, as if in addition to all the values this object has seen, it has also seen the values added to the other object. A convenient way to combine the observations of two Stats objects into a new Stats object. The first operand The second operand This class performs discretization of (value, label) pairs into bins in a way that minimizes the target function "minimum description length". The algorithm is outlined in an article "Multi-Interval Discretization of Continuous-Valued Attributes for Classification Learning" [Fayyad, Usama M.; Irani, Keki B. (1993)] https://ijcai.org/Past%20Proceedings/IJCAI-93-VOL2/PDF/022.pdf The class can be used several times sequentially, it is stateful and not thread-safe. Both Single and Double precision processing is implemented, and is identical. Finds the bins for Single values (and integer labels) Maximum number of bins Minimum number of values per bin (stopping condition for greedy bin splitting) Cardinality of the labels The feature values The corresponding label values An array of split points, no more than total (but maybe less), ending with PositiveInfinity Finds the bins for Double values (and integer labels) Maximum number of bins Minimum number of values per bin (stopping condition for greedy bin splitting) Cardinality of the labels The feature values The corresponding label values An array of split points, no more than total (but maybe less), ending with PositiveInfinity Calculate the entropy and label cardinality for a given interval within the data Finds the optimum bins with respect to The sorted array of indices that are exclusive upper bounds of the respective bins A readable that is backed by a . Because text readers strip line breaks from the end of their lines, this compensates by inserting \n line feed characters at the end of every input line, including the last one. Create a stream wrapping the given text reader, using the encoding. the reader to wrap Create a stream wrapping the given text reader, using the given encoding. The class assumes that the encoding is distributive, that is, the concatenation of the byte encodings of different strings, is a valid byte encoding of the single encoding of the concatenation of those strings. (I believe all standard encodings obey this property.) The reader to wrap. The encoding to use. A helper method that will either ensure that is less than (so there are at least some characters), or that is set. An object that serves as a source of a cancellation token, as well as having the ability for anything to push an exception into the message, to trigger the cancellation. The general intended usage is that, prior to creating a bunch of collaborating thread workers, this object is created and made accessible to them, somehow. Each thread worker will work as they would be engineered without this class, except they will wrap their contents in a try-catch block to push any exceptions (hopefully none) into this marshaller, using . Further, any potentially blocking operation of the thread workers must be changed to use as the cancellation token (this token is canceled iff is ever called). The controlling thread, whatever that may be, once it is either sure has been called (possibly by receiving the cancellation) or is sure somehow that the workers have finished by its own means, will call to throw the set exception as an inner exception, in the wrapping thread. A cancellation token, whose source will be canceled if is ever called. Any thread blocking operation of a family of thread workers using this structure must use this cancellation token, or else there is a strong possibility for threads to stop responding if an exception is thrown at any point. Store an exception and set the cancellation token. If this was already called, this newly passed in exception is ignored. (Important, since a common source of exceptions would be the exceptions relating to the cancellation.) The type of worker that threw the exception, used in the description of the wrapping exception The exception that will become the inner exception If an exception was ever set through , raise it as an appropriate inner exception. This should only be called just prior to dispose, when the workers have already finished. If there is no stored exception, this will do nothing. Note that this does not "expire" the exception, that is, if you were to call this again, it would throw the same exception. The tree structure is simultaneously a tree, and a node in a tree. The interface to get children occurs through the tree itself implementing itself as a dictionary. This tree is not terribly efficient, and is meant to be accommodate additions, deletions, and change of values. Because it is implemented as a dictionary, there is an unfortunate collision in naming between the dictionary type of "values" (which in this case are child trees) and the tree type of values, called "node values." Children are keyed with values of this type The type of the node value Either the node value, or the default of the value type, if is false. This is the key for this child node in its parent, if any. If this is not a child of any parent, that is, it is the root of its own tree, then The parent for this tree, or null if it has no parent. All child keys for this node. All children for this node. The number of children with this node as a parent. Initializes a tree with no node value, and no children. Tries to get the subtree for a child key. The key of the child to find The child, if any, or null if no child with this key is present Whether a child with this key was present Sees whether a child with a given key is present. The key of the child to find Adds a new child to this dictionary. The key / child node pair to add Adds a node as a child of this node. This will disconnect the Removes this node and all its descendants from a tree, leading it to be the root of its own tree. Following this, will be null, and the previous parent (if any) will no longer have this node present as a child. Remove a child with a specified key. The key of the child to remove Convenience utilities for vector operations on . Applies to every explicitly defined element of the vector, in order of index. Applies the to each corresponding pair of elements where the item is implicitly defined in the vector. By explicitly defined, we mean that for a given index i, both vectors have an entry in corresponding to that index. The first vector The second vector Delegate to apply to each pair of non-zero values. This is passed the index, and two values Applies the ParallelVisitor to each corresponding pair of elements where at least one is non-zero, in order of index. a vector another vector Function to apply to each pair of non-zero values - passed the index, and two values Sets all values in the vector to the default value for the type, without changing the density or index structure of the input array. That is to say, the count of the input vector will be the same afterwards as it was before. A delegate for functions that can change a value. Index of entry Value to change A predicate on some sort of value. The value to test The result of some sort of test from that value Applies the to every explicitly defined element of the vector. Applies some function on a value at a particular slot value, changing that slot value. This function will, wherever possible, not change the structure of . If the vector is sparse, and the corresponding slot is not explicitly represented, then this can involve memory copying and possibly memory reallocation on . However, if the item is explicitly represented, even if the item is set to the default value of it will not change the structure of , in terms of sparsifying a dense array, or dropping indices. The vector to modify The slot of the vector to modify The manipulation function A predicate that returns true if we should skip insertion of a value into sparse vector if it was default. If the predicate is null, we insert any non-default. Given a vector, turns it into an equivalent dense representation. Given a vector, ensure that the first slots are explicitly represented. Creates a maybe sparse copy of a VBuffer. Whether the created copy is sparse or not is determined by the proportion of non-default entries compared to the sparsity parameter. A delegate for functions that access an index and two corresponding values, possibly changing one of them. Slot index of the entry. Value from first vector. Value from second vector, which may be manipulated. A delegate for functions that access an index and two corresponding values, stores the result in another vector. Slot index of the entry. Value from first vector. Value from second vector. The value to store the result. Applies the to each pair of elements where is defined, in order of index. If there is some value at an index in that is not defined in , that item remains without any further modification. If either of the vectors are dense, the resulting will be dense. Otherwise, if both are sparse, the output will be sparse iff there is any slot that is not explicitly represented in either vector. Argument vector, whose elements are only read Argument vector, that could change Function to apply to each pair of elements Applies the to each pair of elements where is defined, in order of index. It stores the result in another vector. If there is some value at an index in that is not defined in , that slot value is copied to the corresponding slot in the result vector without any further modification. If either of the vectors are dense, the resulting will be dense. Otherwise, if both are sparse, the output will be sparse iff there is any slot that is not explicitly represented in either vector. Argument vector, whose elements are only read Argument vector, whose elements are read in most cases. But in some cases may be densified. Result vector Function to apply to each pair of elements Applies the to each pair of elements where either or , has an element defined at that index. If either of the vectors are dense, the resulting will be dense. Otherwise, if both are sparse, the output will be sparse iff there is any slot that is not explicitly represented in either vector. Argument vector, whose elements are only read Argument vector, that could change Function to apply to each pair of elements Applies the to each pair of elements where either or , has an element defined at that index. It stores the result in another vector . If either of the vectors are dense, the resulting will be dense. Otherwise, if both are sparse, the output will be sparse iff there is any slot that is not explicitly represented in either vector. Argument vector, whose elements are only read Argument vector, whose elements are read in most cases. But in some cases may be densified. Result vector Function to apply to each pair of elements The actual implementation of and , that has internal branches on the implementation where necessary depending on whether this is an inner or outer join of the indices of on . The actual implementation of and , that has internal branches on the implementation where necessary depending on whether this is an inner or outer join of the indices of on . Applies a function to explicitly defined elements in a vector , storing the result in , overwriting any of its existing contents. The contents of do not affect calculation. If you instead wish to calculate a function that reads and writes , see and . Post-operation, will be dense iff is dense. Applies a function to two vectors, storing the result in , whose existing contents are discarded and overwritten. The function is called for every index value that appears in either or . If either of the two inputs is dense, the output will necessarily be dense. Otherwise, if both are sparse, the output will be sparse iff there is any slot that is not explicitly represented in either vector. Copy from a source list to the given VBuffer destination. Updates the logical length and number of physical values to be represented in , while preserving the underlying buffers. Tausworthe hybrid random number generator.