Fix a minor bug where we read a field from None

PiperOrigin-RevId: 656112311

docs/guide/fairness_indicators.md

@@ -25,6 +25,14 @@ In particular, Fairness Indicators includes the ability to:
 *   Dive deep into individual slices to explore root causes and opportunities
     for improvement
 
+This
+[case study](https://developers.google.com/machine-learning/practica/fairness-indicators),
+complete with [videos](https://www.youtube.com/watch?v=pHT-ImFXPQo) and
+programming exercises, demonstrates how Fairness Indicators can be used on one
+of your own products to evaluate fairness concerns over time.
+
+[![](http://img.youtube.com/vi/pHT-ImFXPQo/0.jpg)](http://www.youtube.com/watch?v=pHT-ImFXPQo)
+
 The pip package download includes:
 
 *   **[Tensorflow Data Validation (TFDV)](https://www.tensorflow.org/tfx/data_validation/get_started)**

docs/tutorials/tfx/components.ipynb

@@ -164,8 +164,7 @@
       },
       "outputs": [],
       "source": [
-        "# TFX has a constraint of 1.16 due to the removal of tf.estimator support.\n",
-        "!pip install \"tfx\u003c1.16\""
+        "!pip install tfx"
       ]
     },
     {

docs/tutorials/tfx/components_keras.ipynb

@@ -949,7 +949,7 @@
       },
       "source": [
         "### Trainer\n",
-        "The `Trainer` component will train a model that you define in TensorFlow.\n",
+        "The `Trainer` component will train a model that you define in TensorFlow. Default Trainer support Estimator API, to use Keras API, you need to specify [Generic Trainer](https://github.com/tensorflow/community/blob/master/rfcs/20200117-tfx-generic-trainer.md) by setup `custom_executor_spec=executor_spec.ExecutorClassSpec(GenericExecutor)` in Trainer's contructor.\n",
         "\n",
         "`Trainer` takes as input the schema from `SchemaGen`, the transformed data and graph from `Transform`, training parameters, as well as a module that contains user-defined model code.\n",
         "\n",

docs/tutorials/tfx/neural_structured_learning.ipynb

@@ -71,15 +71,6 @@
         "\u003c/table\u003e"
       ]
     },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "-niht8EPmUUl"
-      },
-      "source": [
-        "\u003e Warning: Estimators are not recommended for new code.  Estimators run \u003ca href=\\\"https://www.tensorflow.org/api_docs/python/tf/compat/v1/Session\\\"\u003e\u003ccode\u003ev1.Session\u003c/code\u003e\u003c/a\u003e-style code which is more difficult to write correctly, and can behave unexpectedly, especially when combined with TF 2 code. Estimators do fall under our [compatibility guarantees](https://tensorflow.org/guide/versions), but will receive no fixes other than security vulnerabilities. See the [migration guide](https://tensorflow.org/guide/migrate) for details."
-      ]
-    },
     {
       "cell_type": "markdown",
       "metadata": {
@@ -173,9 +164,8 @@
       },
       "outputs": [],
       "source": [
-        "# TFX has a constraint of 1.16 due to the removal of tf.estimator support.\n",
         "!pip install -q \\\n",
-        "  \"tfx\u003c1.16\" \\\n",
+        "  tfx \\\n",
         "  neural-structured-learning \\\n",
         "  tensorflow-hub \\\n",
         "  tensorflow-datasets"

docs/tutorials/tfx/template_local.ipynb

@@ -45,24 +45,15 @@
         "id": "XdSXv1DrxdLL"
       },
       "source": [
-        "\u003cdiv class=\"devsite-table-wrapper\"\u003e\u003ctable class=\"tfo-notebook-buttons\" align=\"left\"\u003e\n",
-        "\u003ctd\u003e\u003ca target=\"_blank\" href=\"https://www.tensorflow.org/tfx/tutorials/tfx/template_local\"\u003e\n",
-        "\u003cimg src=\"https://www.tensorflow.org/images/tf_logo_32px.png\"/\u003eView on TensorFlow.org\u003c/a\u003e\u003c/td\u003e\n",
-        "\u003ctd\u003e\u003ca target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/tfx/blob/master/docs/tutorials/tfx/template_local.ipynb\"\u003e\n",
-        "\u003cimg src=\"https://www.tensorflow.org/images/colab_logo_32px.png\"\u003eRun in Google Colab\u003c/a\u003e\u003c/td\u003e\n",
-        "\u003ctd\u003e\u003ca target=\"_blank\" href=\"https://github.com/tensorflow/tfx/tree/master/docs/tutorials/tfx/template_local.ipynb\"\u003e\n",
-        "\u003cimg width=32px src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\"\u003eView source on GitHub\u003c/a\u003e\u003c/td\u003e\n",
-        "\u003ctd\u003e\u003ca href=\"https://storage.googleapis.com/tensorflow_docs/tfx/docs/tutorials/tfx/template_local.ipynb\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/download_logo_32px.png\" /\u003eDownload notebook\u003c/a\u003e\u003c/td\u003e\n",
-        "\u003c/table\u003e\u003c/div\u003e"
-      ]
-    },
-    {
-      "metadata": {
-        "id": "4PC7GThinsMw"
-      },
-      "cell_type": "markdown",
-      "source": [
-        "\u003e Warning: Estimators are not recommended for new code.  Estimators run \u003ca href=\\\"https://www.tensorflow.org/api_docs/python/tf/compat/v1/Session\\\"\u003e\u003ccode\u003ev1.Session\u003c/code\u003e\u003c/a\u003e-style code which is more difficult to write correctly, and can behave unexpectedly, especially when combined with TF 2 code. Estimators do fall under our [compatibility guarantees](https://tensorflow.org/guide/versions), but will receive no fixes other than security vulnerabilities. See the [migration guide](https://tensorflow.org/guide/migrate) for details."
+        "<div class=\"devsite-table-wrapper\"><table class=\"tfo-notebook-buttons\" align=\"left\">\n",
+        "<td><a target=\"_blank\" href=\"https://www.tensorflow.org/tfx/tutorials/tfx/template_local\">\n",
+        "<img src=\"https://www.tensorflow.org/images/tf_logo_32px.png\"/>View on TensorFlow.org</a></td>\n",
+        "<td><a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/tfx/blob/master/docs/tutorials/tfx/template_local.ipynb\">\n",
+        "<img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\">Run in Google Colab</a></td>\n",
+        "<td><a target=\"_blank\" href=\"https://github.com/tensorflow/tfx/tree/master/docs/tutorials/tfx/template_local.ipynb\">\n",
+        "<img width=32px src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\">View source on GitHub</a></td>\n",
+        "<td><a href=\"https://storage.googleapis.com/tensorflow_docs/tfx/docs/tutorials/tfx/template_local.ipynb\"><img src=\"https://www.tensorflow.org/images/download_logo_32px.png\" />Download notebook</a></td>\n",
+        "</table></div>"
       ]
     },
     {
@@ -88,7 +79,7 @@
         "## Prerequisites\n",
         "\n",
         "* Linux / MacOS\n",
-        "* Python \u003e= 3.5.3\n",
+        "* Python >= 3.5.3\n",
         "\n",
         "You can get all prerequisites easily by [running this notebook on Google Colab](https://colab.sandbox.google.com/github/tensorflow/tfx/blob/master/docs/tutorials/tfx/template_local.ipynb).\n"
       ]
@@ -112,7 +103,7 @@
         "virtualenv -p python3 venv\n",
         "source venv/bin/activate\n",
         "# Install python packages.\n",
-        "python -m pip install --upgrade \"tfx\u003c2\"\n",
+        "python -m pip install --upgrade \"tfx<2\"\n",
         "```\n",
         "If you are using colab:\n"
       ]
@@ -126,8 +117,7 @@
       "outputs": [],
       "source": [
         "import sys\n",
-        "# TFX has a constraint of 1.16 due to the removal of tf.estimator support.\n",
-        "!{sys.executable} -m pip install --upgrade \"tfx\u003c1.16\""
+        "!{sys.executable} -m pip install --upgrade \"tfx<2\""
       ]
     },
     {
@@ -138,7 +128,7 @@
       "source": [
         "NOTE: There might be some errors during package installation. For example,\n",
         "\n",
-        "\u003eERROR: some-package 0.some_version.1 has requirement other-package!=2.0.,\u0026lt;3,\u0026gt;=1.15, but you'll have other-package 2.0.0 which is incompatible.\n",
+        ">ERROR: some-package 0.some_version.1 has requirement other-package!=2.0.,&lt;3,&gt;=1.15, but you'll have other-package 2.0.0 which is incompatible.\n",
         "\n",
         "Please ignore these errors at this moment."
       ]
@@ -408,13 +398,13 @@
         "\n",
         "We will modify copied pipeline definition in `pipeline/pipeline.py`. If you are working on your local environment, use your favorite editor to edit the file. If you are working on Google Colab, \n",
         "\n",
-        "\u003e**Click folder icon on the left to open `Files` view**.\n",
+        ">**Click folder icon on the left to open `Files` view**.\n",
         "\n",
-        "\u003e**Click `my_pipeline` to open the directory and click `pipeline` directory to open and double-click `pipeline.py` to open the file**.\n",
+        ">**Click `my_pipeline` to open the directory and click `pipeline` directory to open and double-click `pipeline.py` to open the file**.\n",
         "\n",
-        "\u003eFind and uncomment the 3 lines which add `StatisticsGen`, `SchemaGen`, and `ExampleValidator` to the pipeline. (Tip: find comments containing `TODO(step 5):`).\n",
+        ">Find and uncomment the 3 lines which add `StatisticsGen`, `SchemaGen`, and `ExampleValidator` to the pipeline. (Tip: find comments containing `TODO(step 5):`).\n",
         "\n",
-        "\u003e Your change will be saved automatically in a few seconds. Make sure that the `*` mark in front of the `pipeline.py` disappeared in the tab title. **There is no save button or shortcut for the file editor in Colab. Python files in file editor can be saved to the runtime environment even in `playground` mode.**\n",
+        "> Your change will be saved automatically in a few seconds. Make sure that the `*` mark in front of the `pipeline.py` disappeared in the tab title. **There is no save button or shortcut for the file editor in Colab. Python files in file editor can be saved to the runtime environment even in `playground` mode.**\n",
         "\n",
         "You now need to update the existing pipeline with modified pipeline definition. Use the `tfx pipeline update` command to update your pipeline, followed by the `tfx run create` command to create a new execution run of your updated pipeline.\n",
         "\n",
@@ -459,7 +449,7 @@
         "\n",
         "In this step, you will add components for training and model validation including `Transform`, `Trainer`, `Resolver`, `Evaluator`, and `Pusher`.\n",
         "\n",
-        "\u003e **Open `pipeline/pipeline.py`**. Find and uncomment 5 lines which add `Transform`, `Trainer`, `Resolver`, `Evaluator` and `Pusher` to the pipeline. (Tip: find `TODO(step 6):`)\n",
+        "> **Open `pipeline/pipeline.py`**. Find and uncomment 5 lines which add `Transform`, `Trainer`, `Resolver`, `Evaluator` and `Pusher` to the pipeline. (Tip: find `TODO(step 6):`)\n",
         "\n",
         "As you did before, you now need to update the existing pipeline with the modified pipeline definition. The instructions are the same as Step 5. Update the pipeline using `tfx pipeline update`, and create an execution run using `tfx run create`.\n",
         "\n",
@@ -558,13 +548,13 @@
         "id": "MhClPWEuuOaP"
       },
       "source": [
-        "\u003e **Open `pipeline/pipeline.py`**. Comment out `CsvExampleGen` and uncomment the line which create an instance of `BigQueryExampleGen`. You also need to uncomment `query` argument of the `create_pipeline` function.\n",
+        "> **Open `pipeline/pipeline.py`**. Comment out `CsvExampleGen` and uncomment the line which create an instance of `BigQueryExampleGen`. You also need to uncomment `query` argument of the `create_pipeline` function.\n",
         "\n",
         "We need to specify which GCP project to use for BigQuery again, and this is done by setting `--project` in `beam_pipeline_args` when creating a pipeline.\n",
         "\n",
-        "\u003e **Open `pipeline/configs.py`**. Uncomment the definition of `BIG_QUERY__WITH_DIRECT_RUNNER_BEAM_PIPELINE_ARGS` and `BIG_QUERY_QUERY`. You should replace the project id and the region value in this file with the correct values for your GCP project.\n",
+        "> **Open `pipeline/configs.py`**. Uncomment the definition of `BIG_QUERY__WITH_DIRECT_RUNNER_BEAM_PIPELINE_ARGS` and `BIG_QUERY_QUERY`. You should replace the project id and the region value in this file with the correct values for your GCP project.\n",
         "\n",
-        "\u003e **Open `local_runner.py`**. Uncomment two arguments, `query` and `beam_pipeline_args`, for create_pipeline() method.\n",
+        "> **Open `local_runner.py`**. Uncomment two arguments, `query` and `beam_pipeline_args`, for create_pipeline() method.\n",
         "\n",
         "Now the pipeline is ready to use BigQuery as an example source. Update the pipeline and create a run as we did in step 5 and 6."
       ]

tfx/dsl/component/experimental/component_utils.py

@@ -189,9 +189,7 @@ def create_tfx_component_class(
   )
 
   for fn in (pre_execution, post_execution):
-    if fn is not None:
-      _type_check_execution_function_params(tfx_component_spec_class, fn)
-      utils.assert_no_private_func_in_main(fn)
+    _type_check_execution_function_params(tfx_component_spec_class, fn)
   try:
     pre_execution_spec, post_execution_spec = [
         _convert_function_to_python_executable_spec(fn)

tfx/dsl/component/experimental/utils.py

@@ -151,24 +151,6 @@ def assert_is_top_level_func(func: types.FunctionType) -> None:
     )
 
 
-def assert_no_private_func_in_main(func: types.FunctionType) -> None:
-  """Asserts the func is not a private function in the main file.
-
-
-  Args:
-    func: The function to be checked.
-
-  Raises:
-    ValueError if the func was defined in main and whose name starts with '_'.
-  """
-  if func.__module__ == '__main__' and func.__name__.startswith('_'):
-    raise ValueError(
-        'Custom Python functions (both @component and pre/post hooks) declared'
-        ' in the main file must be public. Please remove the leading'
-        f' underscore from {func.__name__}.'
-    )
-
-
 def _create_component_spec_class(
     func: types.FunctionType,
     arg_defaults: Dict[str, Any],
@@ -271,7 +253,12 @@ def _create_executor_spec_instance(
     an instance of `executor_spec_class` whose executor_class is a subclass of
     `base_executor_class`.
   """
-  assert_no_private_func_in_main(func)
+  if func.__module__ == '__main__' and func.__name__.startswith('_'):
+    raise ValueError(
+        'Custom Python @components declared in the main file must be public. '
+        f'Please remove the leading underscore from {func.__name__}.'
+    )
+
   executor_class_name = f'{func.__name__}_Executor'
   executor_class = type(
       executor_class_name,

tfx/dsl/component/experimental/utils_test.py

@@ -30,10 +30,6 @@ def top_level_func() -> None:
   pass
 
 
-def _private_func() -> None:
-  pass
-
-
 class UtilsTest(tf.test.TestCase):
   # pylint: disable=g-error-prone-assert-raises
   # pylint: disable=unused-argument
@@ -44,14 +40,6 @@ def func() -> str:
 
     utils.assert_is_functype(func)
 
-  def test_assert_no_private_func_in_main_succeeds(self):
-
-    with self.assertRaisesRegex(
-        ValueError,
-        r'Custom Python functions \(both @component and pre/post hooks\)',
-    ):
-      utils.assert_no_private_func_in_main(_private_func)
-
   def test_assert_is_func_type_raises_error(self):
     with self.assertRaisesRegex(
         ValueError, 'Expected a typehint-annotated Python function'

tfx/orchestration/experimental/core/env.py

@@ -113,17 +113,6 @@ def update_pipeline_run_status(
   ) -> None:
     """Updates orchestrator storage backends with pipeline run status."""
 
-  @abc.abstractmethod
-  def create_pipeline_run_node_executions(
-      self,
-      owner: str,
-      pipeline_name: str,
-      pipeline: pipeline_pb2.Pipeline,
-      node_id: str,
-      executions: Sequence[metadata_store_pb2.Execution],
-  ) -> None:
-    """Creates (sub-)pipeline run node executions in the storage backend."""
-
   @abc.abstractmethod
   def record_orchestration_time(self, pipeline_run_id: str) -> None:
     """Records the orchestration time for a pipeline run."""
@@ -222,16 +211,6 @@ def update_pipeline_run_status(
   ) -> None:
     pass
 
-  def create_pipeline_run_node_executions(
-      self,
-      owner: str,
-      pipeline_name: str,
-      pipeline: pipeline_pb2.Pipeline,
-      node_id: str,
-      executions: Sequence[metadata_store_pb2.Execution],
-  ) -> None:
-    pass
-
   def record_orchestration_time(self, pipeline_run_id: str) -> None:
     pass
 

tfx/orchestration/experimental/core/env_test.py

@@ -84,16 +84,6 @@ def update_pipeline_run_status(
   ) -> None:
     raise NotImplementedError()
 
-  def create_pipeline_run_node_executions(
-      self,
-      owner: str,
-      pipeline_name: str,
-      pipeline: pipeline_pb2.Pipeline,
-      node_id: str,
-      executions: Sequence[metadata_store_pb2.Execution],
-  ) -> None:
-    raise NotImplementedError()
-
   def record_orchestration_time(self, pipeline_run_id: str) -> None:
     raise NotImplementedError()