Change dot notation in add column documentation to tuple

[jeppe-dos]

A tuple must be used to make columns in structs as described in add_column:
"Because "." may be interpreted as a column path separator or may be used in field names, it is not allowed to add nested column by passing in a string. To add to nested structures or to add fields with names that contain "." use a tuple instead to indicate the path."
This PR corrects the documentation to use tuples instead of dot notation.

From issue 1407

[Fokko]

Thanks @jeppe-dos for fixing this 🙌

[kevinjqliu]

Looks like there might be a bug with this change. I tried to follow the docs

from pyiceberg.catalog.sql import SqlCatalog
from pyiceberg.schema import Schema
from pyiceberg.types import NestedField, StringType, DoubleType, LongType

warehouse_path = "/tmp/warehouse"
catalog = SqlCatalog(
    "default",
    **{
        "uri": f"sqlite:///{warehouse_path}/pyiceberg_catalog.db",
        "warehouse": f"file://{warehouse_path}",
    },
)

schema = Schema(
    NestedField(1, "city", StringType(), required=False),
    NestedField(2, "lat", DoubleType(), required=False),
    NestedField(3, "long", DoubleType(), required=False),
)
catalog.create_namespace_if_not_exists("default")
try:
    catalog.drop_table("default.locations")
except:
    pass

table = catalog.create_table("default.locations", schema)

# with table.update_schema() as update:
#     # In a struct
#     update.add_column("details.confirmed_by", StringType(), "Name of the exchange")

with table.update_schema() as update:
    update.add_column(("details", "confirmed_by"), StringType(), "Name of the exchange")

errors

Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "/Users/kevinliu/repos/iceberg-python/pyiceberg/table/update/schema.py", line 192, in add_column
    parent_field = self._schema.find_field(parent_full_path, self._case_sensitive)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/kevinliu/repos/iceberg-python/pyiceberg/schema.py", line 215, in find_field
    raise ValueError(f"Could not find field with name {name_or_id}, case_sensitive={case_sensitive}")
ValueError: Could not find field with name details, case_sensitive=True

[kevinjqliu]

Heres where the errors happens

iceberg-python/pyiceberg/table/update/schema.py

Lines 184 to 192 in b0ea716

	name = path[-1]
	parent = path[:-1]
	full_name = ".".join(path)
	parent_full_path = ".".join(parent)
	parent_id: int = TABLE_ROOT_ID
	if len(parent) > 0:
	parent_field = self._schema.find_field(parent_full_path, self._case_sensitive)

And some debugging statements:

(Pdb) path
('details', 'confirmed_by')
(Pdb) name
'confirmed_by'
(Pdb) parent
('details',)
(Pdb) parent_full_path
'details'
(Pdb) parent_id
-1
(Pdb) len(parent) > 0
True

parent_field = self._schema.find_field(parent_full_path, self._case_sensitive)

is where it errors.

Seems like we're missing the case where no "parent" is present

[jeppe-dos]

Looks like there might be a bug with this change. I tried to follow the docs

from pyiceberg.catalog.sql import SqlCatalog
from pyiceberg.schema import Schema
from pyiceberg.types import NestedField, StringType, DoubleType, LongType

warehouse_path = "/tmp/warehouse"
catalog = SqlCatalog(
    "default",
    **{
        "uri": f"sqlite:///{warehouse_path}/pyiceberg_catalog.db",
        "warehouse": f"file://{warehouse_path}",
    },
)

schema = Schema(
    NestedField(1, "city", StringType(), required=False),
    NestedField(2, "lat", DoubleType(), required=False),
    NestedField(3, "long", DoubleType(), required=False),
)
catalog.create_namespace_if_not_exists("default")
try:
    catalog.drop_table("default.locations")
except:
    pass

table = catalog.create_table("default.locations", schema)

# with table.update_schema() as update:
#     # In a struct
#     update.add_column("details.confirmed_by", StringType(), "Name of the exchange")

with table.update_schema() as update:
    update.add_column(("details", "confirmed_by"), StringType(), "Name of the exchange")

errors

Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "/Users/kevinliu/repos/iceberg-python/pyiceberg/table/update/schema.py", line 192, in add_column
    parent_field = self._schema.find_field(parent_full_path, self._case_sensitive)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/kevinliu/repos/iceberg-python/pyiceberg/schema.py", line 215, in find_field
    raise ValueError(f"Could not find field with name {name_or_id}, case_sensitive={case_sensitive}")
ValueError: Could not find field with name details, case_sensitive=True

Yes, the struct has to exist before you can insert anything into it. This can be adjusted in the code to automatically create the parent. For now, it is detailed in the documentation changes. Should I write more explicitly?

[kevinjqliu]

Yes, the struct has to exist before you can insert anything into it.

ah i see, that makes sense. in that case, can we edit the example so that it works out of the box?

Also i think its valuable to move the comment to the top level docs of "Add Column". We can include both the details about dot notation and struct parent

[kevinjqliu]

i found another dot notion in Move column, do we need to change this too?
https://py.iceberg.apache.org/api/#move-column