Releases: tensorflow/model-analysis
Releases · tensorflow/model-analysis
Release 0.13.1
Major Features and Improvements
- Added support for squared pearson correlation (R squared) post export
metric.
Bug fixes and other changes
- Cast / convert labels for precision / recall at K so that they work even if
the label and the classes Tensors have different types, as long as the types
are compatible. - Post export metrics will now also search for prediction keys prefixed by
metric_tag if it is specified. - Added support for precision/recall @ k using canned estimators provided
label vocab not used. - Preserve unicode type of slice keys when serialising to and deserialising
from disk, instead of always converting them to bytes. - Use
__slots__in accumulators.
Breaking changes
- Expose Python 3 types in the code (this will break Python 2 compatibility)
Release 0.13.0
Release 0.13.0
Major Features and Improvements
- Python 3.5 is supported.
Bug fixes and other changes
- Added support for fetching additional tensors at prediction time besides
features, predictions, and labels (predict now returns FetchedTensorValues
type). - Removed internal usages of encoding.NODE_SUFFIX indirection within dicts
in the eval_saved_model module (encoding.NODE_SUFFIX is still used in
FeaturesPredictionLabels) - Predictions are now returned as tensors (vs dicts) when "predictions" is the
only output (this is consistent with how features and labels work). - Depends on
apache-beam[gcp]>=2.11,<3. - Depends on
protobuf>=3.7,<4. - Depends on
scipy==1.1.0.
Breaking changes
- Post export metrics for precision_recall_at_k were split into separate
fuctions: precision_at_k and recall_at_k. - Requires pre-installed TensorFlow >=1.13,<2.
Deprecations
Release 0.12.1
Release 0.12.1
Major Features and Improvem 8000 ents
Bug fixes and other changes
- Fixed documentation formatting issues.
Breaking changes
Deprecations
Release 0.12.0
Release 0.12.0
Major Features and Improvements
- Python 3.5 readiness complete (all tests pass). Full Python 3.5
compatibility is expected to be available with the next version of Model
Analysis (after Apache Beam 2.11 is released). - Added support for customizing the pipeline (via extractors, evaluators, and
writers). See architecture for more details. - Added support for excluding the default metrics from the saved model graph
during evaluation. - Added a mechanism for performing evaluations via post_export_metrics without
access to a Tensorflow EvalSavedModel. - Added support for computing metrics with confidence intervals using the
Poisson bootstrap technique.
To use, set the num_bootstrap_samples to a number greater than 1--20 is
recommended for confidence intervals.
Bug fixes and other changes
- Fixed bugs where TFMA was incorrectly modifying elements in DoFns, which
violates the Beam API. - Fixed correctness issue stemming from TFMA incorrectly relying on evaluation
ordering that TF doesn't guarantee. - We now store feature and label Tensor information in SignatureDef inputs
instead of Collections in anticipation of Collections being deprecated in TF
2.0. - Add support for fractional labels in AUC, AUPRC and confusion matrix at
thresholds. Previously the labels were being passed directly to TensorFlow,
which would cast them tobool, which meant that all non-zero labels were
treated as positive examples. Now we treat a fractional labellin[0, 1]as two examples, a positive example with weightland a negative
example with weight1 - l. - Depends on
numpy>=1.14.5,<2. - Depends on
scipy==0.19.1. - Depends on
protobuf==3.7.0rc2. - Chicago Taxi example is moved to tfx repo (https://github.com/tensorflow/tfx/tree/master/examples/chicago_taxi)
Breaking changes
- Moved tfma.SingleSliceSpec to tfma.slicer.SingleSliceSpec.
Deprecations
Release 0.11.0
Release 0.11.0
Major Features and Improvements
- We now support unsupervised models which have
model_fns that do not take a
labelsargument. - Improved performance by using
make_callableinstead of repeated
session.runcalls. - Improved performance by better choice of default "combine" batch size.
- We now support passing in custom extractors in the model_eval_lib API.
- Added support for models which have multiple examples per raw input (e.g.
input is a compressed example which expands to multiple examples when parsed
by the model). For such models, you must specify anexample_refparameter
to yourEvalInputReceiver. This 1-D integer Tensor should be batch aligned
with features, predictions and labels and each element in it is an index in
the raw input tensor to identify which input each feature / prediction /
label came from. See
eval_saved_model/example_trainers/fake_multi_examples_per_input_estimator.py
for an example. - Added support for metrics with string
value_ops. - Added support for metrics whose
value_ops return multidimensional arrays. - We now support including your serving graph in the EvalSavedModel. You can
do this by passing aserving_input_receiver_fntoexport_eval_savedmodel
or any of the TFMA Exporters.
Bug fixes and other changes
- Depends on
apache-beam[gcp]>=2.8,<3. - Depends on
tensorflow-transform>=0.11,<1. - Requires pre-installed TensorFlow >=1.11,<2.
- Factor our utility fun 10000 ctions for adding sliceable "meta-features" to FPL.
- Added public API docs
- Add an extractor to add sliceable "meta-features" to FPL.
- Potentially improved performance by fanning out large slices.
- Add support for assets_extra in
tfma.exporter.FinalExporter. - Add a light-weight library that includes only the export-related modules for
TFMA for use in your Trainer. See docstring in
tensorflow_model_analysis/export_only/__init__.pyfor usage. - Update
EvalInputReceiverso the TFMA collections written to the graph only
contain the results of the last call if multiple calls toEvalInputReceiver
are made. - We now finalize the graph after it's loaded and post-export metrics are added,
potentially improving performance. - Fix a bug in post-export PrecisionRecallAtK where labels with only 1 dimension
were not correctly handled. - Fix an issue where we were not correctly wrapping SparseTensors for
features
andlabelsintf.identity, which could cause TFMA to encounter
TensorFlow issue #17568 if there were control dependencies on thesefeatures
orlabels. - We now correctly preserve
dtypeswhen splitting and concatenating
SparseTensors internally. The failure to do so previously could result in
unexpectedly large memory usage if string values were involved due to the
inefficient pickling of NumPy string arrays with a large number of elements.
Breaking changes
- Requires pre-installed TensorFlow >=1.11,<2.
- We now require that
EvalInputReceiver,export_eval_savedmodel,
make_export_strategy,make_final_exporter,FinalExporterand
LatestExporterbe called with keyword arguments only. - Removed
check_metric_compatibilityfromEvalSavedModel. - We now enforce that the
receiver_tensorsdictionary forEvalInputReceiver
contains exactly one key namedexamples. - Post-export metrics have now been moved up one level to
tfma.post_export_metrics. They should now be accessed via
tfma.post_export_metrics.aucinstead of
tfma.post_export_metrics.post_export_metrics.aucas they were before. - Separated extraction from evaluation.
EvaluteAndWriteResultsis now called
ExtractEvaluateAndWriteResults. - Added
EvalSharedModeltype to encapsulatemodel_pathand
add_metrics_callbacksalong with a handle to a shared model instance.
Deprecations
Release 0.9.2
Release 0.9.2
Major Features and Improvements
- Improved performance especially when slicing across many features and/or
feature values.
Bug fixes and other changes
- Depends on
tensorflow-transform>=0.9,<1. - Requires pre-installed TensorFlow >=1.9,<2.
Release 0.9.1
Release 0.9.1
Bug fixes and other changes
- Depends on
apache-beam[gcp]>=2.6,<3. - Requires pre-installed TensorFlow >=1.10,<2.
- Updated ExampleCount to use the batch dimension as the example count. It
also now tries a few fallbacks if none of the standard keys are found in the
predictions dictionary: the first key in sorted order in the predictions
dictionary, or failing that, the first key in sorted order in the labels
dictionary, or failing that, it defaults to zero. - Fix bug where we were mutating an element in a DoFn - this is prohibited in
the Beam model and can cause subtle bugs. - Fix bug where we were creating a separate Shared handle for each stage in
Evaluate, resulting in no sharing of the model across stages.
Release 0.9.0
Major Features and Improvements
- Add a TFMA unit test library for unit testing your the exported model and
associated metrics computations. - Add
tfma.export.make_export_strategywhich is analogous to
tf.contrib.learn.make_export_strategy. - Add
tfma.exporter.FinalExporterandtfma.exporter.LatestExporterwhich
are analogous totf.estimator.FinalExporterand
tf.estimator.LastExporter. - Add
tfma.export.build_parsing_eval_input_receiver_fnwhich is analogous to
tf.estimator.export.build_parsing_serving_input_receiver_fn. - Add integration testing for DNN-based estimators.
- Add new post export metrics:
- AUC (
tfma.post_export_metrics.post_export_metrics.auc) - Precision/Recall at K
(tfma.post_export_metrics.post_export_metrics.precision_recall_at_k) - Confusion matrix at thresholds
(tfma.post_export_metrics.post_export_metrics.confusion_matrix_at_thresholds)
- AUC (
Bug fixes and other changes
- Peak memory usage for large DataFlow jobs should be lower with a fix in when
we compact batches of metrics during the combine phase of metrics
computation. - Remove batch size override in
chicago_taxiexample. - Added dependency on
protobuf>=3.6.0<4for protocol buffers. - Updated SparseTensor code to work with SparseTensors of any dimension.
Previously on SparseTensors with dimension 2 (batch_size x values) were
supported in the features dictionary. - Updated code to work with SparseTensors and dense Tensors of variable
lengths across batches.
Breaking changes
- EvalSavedModels produced by TFMA 0.6.0 will not be compatible with later
versions due to the following changes:- EvalSavedModels are now written out with a custom "eval_saved_model"
tag, as opposed to the "serving" tag before. - EvalSavedModels now include version metadata about the TFMA version that
they were exported with.
- EvalSavedModels are now written out with a custom "eval_saved_model"
- Metrics and plot outputs now are converted into proto and serialized.
Metrics and plots produced by TFMA 0.6.0 will not be compatible with later
versions. - Requires pre-installed TensorFlow >=1.9,<2.
- TFMA now uses the TensorFlow Estimator functionality for exporting models of
different modes behind the scenes. There are no user-facing changes
API-wise, but EvalSavedModels produced by earlier versions of TFMA will not
be compatible with this version of TFMA. - tf.contrib.learn Estimators are no longer supported by TFMA. Only
tf.estimator Estimators are supported. - Metrics and plot outputs now include version metadata about the TFMA version
that they were exported with. Metrics and plots produced by earlier versions
of TFMA will not be compatible with this version of TFMA.
Release 0.6.0
Initial release of TensorFlow Model Analysis.