Apache Data Lakehouse Weekly: September 3-9, 2026

Three things ran through the dev lists this week. Projects kept redrawing their own boundaries, with the Iceberg Rust DataFusion integration voted out of Iceberg and into DataFusion on both lists at once. Catalogs kept turning informal metadata into spec text, with labels landing in the Iceberg REST spec and tags being argued down to the encoding of a query parameter in Polaris. And format governance got a lot more explicit, with Parquet debating what a version number in a footer actually promises a reader. Six projects, one week. Here is what happened and why it matters. Apache Iceberg The largest thread of the week was not a spec argument at all. Sung Yun opened the call for the Iceberg Summit 2027 Selection Committee after the PMC approved the event, and the volunteer replies stacked up fast. Russell Spitzer, Ryan Blue, Kevin Liu, Dipankar Mazumdar, Neelesh Salian, Roy Hasson, John Zhuge, Talat Uyarer, Nathan Yee, Hongyue Zhang, Arnav Balyan and others put their names in within days. The committee will have 11 members, no more than one per company, and at least three from the Iceberg PMC. Volunteering closes Friday, September 11 at 11:59 PM PDT, and the PMC votes on the roster after that. The work is real but bounded, mostly program structure plus CFP review, which ran three to six hours last year. That per-company cap is worth pausing on. Iceberg's conference program is now selected by a committee designed so no single vendor can shape the agenda, and the volunteer list already spans most of the major players in the ecosystem. For a project this commercially contested, that structure does more for trust than any code of conduct document. On the spec side, the catalog surface kept expanding. Prashant Singh's finer grained read restrictions vote passed with binding +1s from Honah J., Kevin Liu and Amogh Jahagirdar, and he posted the result thread on September 3. The change lets a REST catalog hand back a restricted view of table data rather than an all-or-nothing load, which is the piece most enterprise deployments have been faking with a proxy in front of the catalog. Singh also pointed out a detail buried in the spec examples. The masked email address in the sample is iceberg16112018@apache.org, and 16112018 is the date Iceberg entered incubation. Right behind it, Andrei Tserakhau restarted the vote on catalog-provided labels in the REST read path. The proposal is deliberately small, adding an optional flat key-value map to LoadTableResult and LoadViewResult, with structured tag entities and write APIs pushed to follow-ups. Daniel Weeks, Fokko Driesprong, Ryan Blue, Anoop Johnson, Amogh Jahagirdar, Kevin Liu, Sung Yun and Eduard Tudenhöfner all weighed in, and the result thread closed it out. Scoping the vote to the read path was the smart move here. Labels are the kind of feature that grows a governance model, a write API and a permissions story if you let the first version carry all of it, and splitting the spec change from the implementation PRs kept the discussion on one question at a time. The view spec got the most interesting design debate of the week. Alex Stephen asked why creating a view with multiple engine representations is so hard in practice. The spec allows one view to carry Spark, Trino and Flink dialects, but there is no way to append a representation to an existing view, only to replace one, so a direct REST call is the only real path. He proposed a new view property, replace.append-dialect.allowed. Ryan Blue's first question was whether the intent was for CREATE VIEW to append automatically. Stephen argued the real pain is the user journey, where a query fails because the engine picked the wrong representation and the only recourse is calling REST endpoints by hand. Daniel Weeks pushed the discussion toward ALTER VIEW rather than a property flag, and he made the argument that carried the thread. An engine should only add or update its own dialect, because updating another engine's representation would require parsing SQL it cannot parse. Weeks also objected to toggling behavior with properties, since that makes a declarative statement depend on session configuration. Prashant Singh backed that and pointed at StarRocks, which already has explicit ADD DIALECT and MODIFY DIALECT grammar. Péter Váry raised the consistency question that has no clean answer. If a Flink user adds a Flink representation to a view created in Spark, how does anyone know the two return the same rows? His position is that the project accepts the user's authority on that. Talat Uyarer proposed a workflow that needs no spec change at all. Engines should display every representation with its dialect label in DESCRIBE EXTENDED, since the SQL strings are already in the metadata and showing another engine's text requires no translation. The user reads the Flink SQL, writes the Spark equivalent, and issuing ALTER VIEW is itself the consistency assertion. Váry still prefers defining both statements at creation time so the view is never partially defined. The thread has not settled, but it moved from "add a property" to "fix the presentation and use ALTER," which is a better place to be. Two v3 and v4 discussions produced concrete decisions. Marco Kroll's thread on the _pos column in efficient column updates asked whether the spec has any precedent for storing data purely for debugging. He searched and found none, and Russell Spitzer agreed the check belongs at write time, since a writer that sees a mismatch between origin row position and actual position should just fail. Anurag Mantripragada closed it: the Efficient Column Updates sync decided _pos will not be required, and writing it will not be allowed. That is the right call. Every optional field in a spec becomes a compatibility question for every reader that follows. Ryan Blue's field ID tracking for non-materialized columns in v4 went deeper into transform metadata. The design requires a result data type when the output cannot be derived, uses bound id-based references, and skips a special case for identity transforms. Russell Spitzer asked for sort orders to get the same treatment and raised the lineage question, whether a table now carries both a SchemaID and an ExpressionsID. Gianluca Graziadei pressed on Hilbert clustering over heterogeneous columns, where feeding raw values to the curve wastes bits and degrades clustering along a dimension. Blue's answer draws the line clearly. Whatever parameterization a function needs must be fully captured by the expression stored in metadata, so hilbert(zvalue(col1, 0, 1), zvalue(col2, 100, 50)) is fine and hidden inputs are not. Péter Váry connected it to his index work, where a Cluster spec already lists materialized and non-materialized fields, and suggested redefining sort order so the index can reuse it. Russell Spitzer also restarted the File Type URI relativization debate after a long sync call, keeping the discussion on the list between meetings so people who could not attend can follow the argument. The open question is how the URI subfield of the new File type behaves, since Parquet allows more than one persistence form. EJ Song opened a related performance thread on row-granular concurrency for V3 deletion vectors. With one DV per data file, validateAddedDVs fails a commit at file granularity even when two operations deleted rows that do not overlap, and the resulting ValidationException is not retryable, so the whole operation is recomputed. Xiening Dai backed the idea and reminded the thread that a commit loop should retry until either a real conflict appears or the version conflict clears. Releases moved on three fronts. Neelesh Salian posted the 1.12.0 status, grouped the remaining PRs by how close they are to merge, and said he wants RC0 cut in days, with a longer vote window because of the US long weekend. Danny Jones called the vote on Iceberg Rust 0.11.0 RC1, and the verification reports are worth reading as a template. L. C. Hsieh checked signatures, 489 license headers, 2,209 passing tests and the Python bindings on macOS arm64, and Anoop Johnson ran the full suite including the minio, REST, HMS and Spark integration tests on Ubuntu. The Terraform provider v0.1.0 RC3 vote also passed, giving teams a first supported path to manage Iceberg catalog resources as infrastructure code. Smaller items still worth your attention. Rahul Mahadev's proposal to standardize a User-Agent format for REST clients got bumped by Micah Kornfield after it landed in his spam folder, which is its own small comment on how much good work gets lost to mail filters. Someone posted a 10x faster Z-order bit interleaving implementation using a lookup table and asked for review. And the European community keeps growing, with meetups announced for London and Warsaw in October. Apache Polaris Polaris spent the week on performance, protocol contracts and the health of its own review process. Dmitri Bourlatchkov opened the most productive thread by questioning whether InMemoryEntityCache earns its keep. His observation was simple. With JDBC persistence, even a cache hit still issues a query to confirm the entity version, so what is the cache actually buying? Prithvi S answered with numbers instead of opinion. He traced Resolver.resolveAll() with JDBC and compared a warm cache against a null cache on the same catalog, using a loadTable-shaped resolve down a two-level namespace path. On a warm cache the resolve issued a single SELECT against the ENTITIES table pulling id, catalog_id, entity_version and grant_records_version. His framing is the key point: InMemoryEntityCache is not a "skip the database" cache, it is a "skip the expensive load" cache, with the version check as the invalidation path. Yufei Gu backed that, noting the version check is what keeps a multi-pod Polaris deployment consistent, because one pod cannot see another pod's writes. Robert Stupp then found the real problem hiding in the data. A cold resolve of that same simple path issued 23 SELECTs. He asked whether that is the expected cold-path shape or whether the hierarchy, grants and versions can be fetched in a bounded number of operations, and he proposed folding the version fence into the final conditional write for mutations, so an UPDATE with a version predicate that touches zero rows becomes the conflict signal. Jean-Baptiste Onofré traced the 23 queries to their source. AtomicOperationMetaStoreManager.loadResolvedEntityById() issues one or two grant-record queries per entity and gets called once per entity in a resolved path with no batching, which is a classic N+1 and has nothing to do with caching. Bourlatchkov added a second angle, asking whether grant record lookups are needed at all when an external authorizer like Ranger or OPA is in play, since those queries return empty results anyway. That thread is a good model for how performance discussions should go on a dev list. A question, a traced measurement, a correction to the framing, and then a split into independent fixes. The Iceberg Catalog Migrator release did not make it. Ajantha Bhat called the vote on 1.1.0 RC0 and collected careful verification from JB Onofré, Ayush Saxena, Prithvi S and Robert Stupp. Onofré caught that the binary distribution and CLI jar do not document jquery.jstree.js, an MIT-licensed dependency that needs its license inline. Bhat then cancelled the vote, and said he will fix the jars to include META-INF/LICENSE and META-INF/NOTICE and exclude webapps content from the Hadoop dependencies before cutting a new RC. A cancelled vote over a missing license file looks like friction from the outside. It is actually the ASF release process working exactly as designed. Data sharing came back to life. Jean-Baptiste Onofré resumed the Open Sharing APIs proposal and said he would open a draft PR for an /api/shares/v1 endpoint to make the design concrete. Dennis Huo had a competing draft ready in PR 5446, favoring /api/management/ for share administration and /api/shares/ for the consumer data plane, plus a companion document walking through end-to-end user journeys for each design choice. Prithvi S read both and endorsed the shape: a first-class share with enumerated members, a restricted consumer, a listing that serves as the binding and audit unit, and a segregated read-only Iceberg REST surface with credential vending. Keeping stock Iceberg REST as the consumer protocol and leaving Flight and Polaris-to-Polaris federation out of v1 also drew agreement. Huo laid out the current mental model, where everything considered data plane lives under /api/catalog/ and everything administrative lives under /api/management/, while agreeing the privileges themselves should separate catalog, principal and sharing concerns. The Tag Spec review turned into a lesson in API contract design. Robert Stupp raised four contract questions while the spec is still separate from the implementation, starting with the encoding of identifier elements in target query parameters. His point goes past the unit separator. Namespace elements and object names can contain ampersands, question marks, equals signs, plus signs and percent signs, and those need to survive rather than be read as query syntax. Prithvi S argued all four should be resolved before PR 5366 merges, since they are contract questions rather than storage layout questions. Dmitri Bourlatchkov went further and said Polaris should adopt a stricter namespace representation in its native APIs instead of inheriting the Iceberg REST convention, which has produced recurring issues on the Iceberg list. EJ Wang updated the spec doc with the clarifications and proposed keeping Iceberg's namespace query convention for v1, with explicit supported-name, encoding and decoding rules written into the spec, and a replacement codec handled separately since the issue affects existing APIs too. Storage support keeps widening. Austen Tomek from Chicago Trading Company introduced himself and proposed Cloudflare R2 support with scoped credential vending, which is a first contribution done the right way, on the list before the PR. R2 speaks S3-compatible APIs but has no STS, so Cloudflare issues short-lived scoped credentials through locally signed JWTs while the server holds the parent API token. Yufei Gu read that as STS-style vending where Polaris performs issuance without calling out to the object store, and leaned toward treating R2 as S3-compatible storage rather than a new config type. Sushant Raikar pushed back on the middle ground and asked whether R2 should be fully first-class, with its own config, its own credential vending and its own R2FileIO, which would sidestep the one-FileIO-to-one-storage-type question entirely at the cost of a new FileIO to maintain. That trade is one every catalog faces as object stores multiply, and it is better settled once than per vendor. Two more threads deserve a mention. Vignesh A's soft-roll authorization for the Iceberg REST /v1/config endpoint produced a clean distinction from Bourlatchkov: GET_CATALOG_CONFIG_PROPERTIES is an operation, not a privilege, because the SPI is written in terms of operations for non-native authorizers like Ranger, and OPA has no concept of privileges at all. And Yufei Gu's thread on PR review and committership named something every project is now living with. LLMs make it easy for contributors to submit large PRs, large PRs are harder to review well, and across many communities they sit unmerged while both authors and reviewers get frustrated. Onofré split the metric in two. Time to initial response matters a lot and should be fast, because it keeps contributors engaged. Time to merge matters much less than project quality and long-term maintainability, and iteration on a PR is normal committer work, not failure. Elsewhere on the list, a GitHub discussion on export and apply for tables and views surfaced a real bootstrap gap. Users want to export a realm including tables, views and their privileges, rewrite the file paths, and apply the result in another environment. MonkeyCanCode flagged the hard parts, including agreeing on a table creation syntax without SQL, handling dialect-specific views, and the fact that an export full of filesystem paths cannot be reused across environments without rewriting. The community also announced a Bay Area meetup on September 30. Apache Arrow Arrow's week was about pruning the format and welcoming people. The vote to informally deprecate Tensor and SparseTensor in IPC, started by Raúl Cumplido, passed with binding +1s from Weston Pace, Joris Van den Bossche and David Li plus support from Benjamin Kietzman, and Cumplido posted the result. Max Burke brought a useful pointer into the thread, an open flatbuffers proposal for a deprecated-readonly attribute that would keep generating getters while dropping setters, so existing data still reads and new data cannot be written with the field. Antoine Pitrou said it might help later, while noting Arrow can rely on conventions and let each implementation decide what it exposes. Informal deprecation is the pragmatic path here. The types stay readable, nobody's archived data breaks, and implementers get a clear signal to stop investing. The more consequential format discussion was Mandukhai Alimaa and Rok Mihevc's proposal for a canonical extension type for the FILE type. Parquet recently added a FILE logical type, Parquet C++ reader and writer support is underway, and without a matching Arrow representation file fields lose their semantics crossing IPC, the C Data Interface, Flight and language bindings. Antoine Pitrou immediately questioned the name and the ownership. Should it be parquet.file rather than arrow.file, and is it Arrow's job to standardize a type that Parquet defined? Mihevc agreed the namespace should show where the spec came from and pointed at geoarrow as precedent for non-Arrow-namespaced types. Gang Wu and Neelesh Salian backed parquet.file, and Matt Topol noted the existing type shipped as arrow.parquet.variant, so symmetry argues for arrow.parquet.file. The naming is small. The principle is not. As formats borrow types from each other, the namespace is what tells a reader three years later which specification governs the semantics. Andrew Lamb announced Kosta Tarasov as a new Arrow committer, and the congratulations thread ran long, with Neelesh Salian, Ruoxi Sun, Raúl Cumplido, Kevin Gurney and Jeffrey Vo among the repliers. Tarasov's name showed up on the Parquet list in the same week as a co-author of the merged Rust ALP implementation, which is a nice illustration of how contributions compound across sibling projects. Matt Topol shipped a release the hard way. His Arrow Go 18.8.0 RC1 vote hit a bad signature error that David Li caught in verification, Topol re-signed and re-uploaded, and then the checks passed for Neelesh Salian, David Li on Ubuntu 25.04 with Go 1.27 and Raúl Cumplido on Debian 14 with Go 1.27.1. The result and the release announcement followed within a day. The most interesting outside contribution came from Prateek Singh, who announced ArrowMetal 0.1.0, Arrow compute kernels running on Apple silicon GPUs through Metal. The design detail that matters is memory. Arrow buffers live in shared-storage Metal memory, so an array is a valid CPU Arrow buffer and a valid GPU buffer at the same time and nothing gets uploaded or downloaded. Data crosses through the C Data Interface, the C Stream Interface and the C Device Data Interface with ARROW_DEVICE_METAL, which means it works with pyarrow, Polars, DuckDB, pandas, arrow-rs, Arrow Go, Arrow JS, the R package and arrow-swift. Curt Hagenlocher's reaction captured the significance: this might breathe new life into the Device Data interface. Matt Topol agreed and said he will look at the Arrow Go issues, possibly saving some for the hackathon at Community Over Code in October. The zero-copy device interface has been in the spec for a while without a headline consumer. A laptop-class GPU backend that works across eight language bindings is exactly the kind of thing that pulls a dormant interface into use. The project also held its community meeting on September 9. Apache Parquet Parquet had the busiest technical week of the six projects, and nearly all of it circled one question: what does a format version promise? The versioning proposal thread ran past 45 messages. Micah Kornfield pulled it back from mechanics to requirements, arguing the community should agree on the rules before arguing about what goes in the footer. His two anchors are worth quoting in spirit. A file written with a preview feature must be readable by any reader that supports the preview features used, and by a reader that supports the major version where the feature was fully adopted. A reader must never return incorrect data when it does not understand a preview feature, which puts the burden on writers to communicate preview usage in a way that makes silent misreads impossible. That second rule is the one that shapes everything else. Parquet files outlive the software that wrote them, and a reader that quietly returns wrong values is worse than a reader that refuses to open the file. The related threads all inherit from it: whether to write the version number in the footer, how readers should behave when they meet an unsupported format version, what forward compatibility means when files get rewritten, and a useful comparison with how Arrow IPC handles the same problem. One piece of that already shipped as spec text. Divjot Arora's vote on handling unrecognized logical and physical type combinations passed with binding +1s from Antoine Pitrou, Micah Kornfield, Fokko Driesprong and Gang Wu, plus non-binding support from Russell Spitzer and Matt Topol, and Arora posted the result. Pitrou described it as short and useful, which is the highest praise a spec clarification gets. The build side got simpler. Divjot Arora's proposal to inline parquet.thrift into parquet-java drew a full options review. Ryan Blue approved the PR, which uses a local copy plus a script that pulls new copies from parquet-format by commit hash or ref and records the resolved version in a parquet-format.version file. Blue argued against writing scripts to diff and validate the local copy, since the file is in version control and git already does that well. Gang Wu floated a git submodule pointing at a commit hash, then clarified his reply was not blocking. Fokko Driesprong had the same thought and rejected it on ergonomics, calling submodules clunky, easy to leave stale and awkward in daily use, and he opened a PR to reinstate nightly parquet-format snapshots as an alternative. Arora documented all the considered options in the thread for posterity, and Russell Spitzer landed the closing argument: option one is not worth debating against a better solution until the simplest one is in place. Releases are moving on both format and Java. Fokko Driesprong opened the 2.14.0 format release discussion in the spirit of releasing more often, carrying chronological ordering of INT96 timestamps, the FILE logical type with its self-reference follow-up, and Adaptive Lossless Floating-Point encoding. Arora asked to slip in his type-combination PR once its vote closed, Driesprong added it to the milestone, and Gang Wu and Andrew Lamb both backed the release, with Wu noting it unblocks a queue of waiting PRs. The 2.14.0 RC1 vote is now open, alongside the Parquet Java 1.18.1 RC1 vote. Arora also drove the vote on extended precision nanosecond timestamps, which lets TimestampType annotate FIXED_LEN_BYTE_ARRAY(12) columns so the full SQL timestamp range fits in nanoseconds, with all three time units supported and reference implementations in Java and C++ plus a parquet-testing file. Daniel Weeks, Micah Kornfield and Ryan Blue voted binding +1, with Stevo Mitrić and Alkis Evlogimenos non-binding. Twelve bytes for a timestamp sounds extravagant until you have tried to store pre-1677 dates at nanosecond precision in 64 bits. The vector type debate is the one to watch. Rok Mihevc summarized a focused call on the physical representation of a numeric vector type, convened because vector database storage needs differ from the fixed-size-list discussions so far, and because Iceberg needs a vector type too. The call converged on two options without picking one. Antoine Pitrou challenged the premise, asked for a real explanation of the different storage needs, called option A a short-term fix with severe encoding limitations, and said he is only lukewarm on option C while conceding it does not paint the project into a corner. Mihevc conceded the framing was too broad and restated the requirement as a contract rather than a layout: a fixed number of numeric elements, elements that cannot be null and are finite, and room for future vector-specific properties like normalization guarantees or specialized encodings. He noted Lance uses FixedSizeList with specialized physical encodings while Hudi stores a vector as a single FLBA. Will Edwards made the strongest counterargument, that efficiency here is a software problem rather than a format problem, since a reader can expose flat typed memory and nothing in the format forces a List allocation per row. He reframed the real question as what SHOW CREATE TABLE should say when a Parquet file is the only source of schema. Daniel Weeks focused on semantics instead, arguing the point of a logical vector type is to differentiate it from a fixed-size list, so vectors should prohibit nulls, NaN and infinity as elements rather than merely detecting them in statistics. This is a genuinely hard design call, and the fact that Iceberg is waiting on the outcome makes it one of the most consequential decisions in the ecosystem right now. Encoding work kept pace. Prateek Gaur's ALP encoding thread turned into a cross-language progress report. Andrew Lamb reported the Rust implementation from Kosta and Devan merged, Vinoo Ganesh is addressing Gang Wu's comments on the Java PR, the C++ PR is in another review round, and Gaur and Arnav Balyan are starting the Go implementation. With ALP riding in Parquet Format 2.14.0, floating-point columns are about to get materially cheaper across four language stacks at once. PFOR encoding and an extensible decimal floating-point type are moving behind it. One security item needs action. Gidon Gershinsky published CVE-2026-73334, a moderate-severity issue in the org.apache.parquet.crypto.keytools package affecting parquet-hadoop 1.12 through 1.18.0. When a writer sets the optional KMS URL parameter, that URL is stored in the file, and a reader configured to trust file-controlled KMS URLs can forward it to a pluggable KmsClient that skips host validation. If you use Parquet envelope encryption with a custom KmsClient, read the advisory and check whether your reader configuration trusts file-controlled URLs. The community also kept its sync cadence, with notes from the September 9 sync posted the same day. Apache DataFusion DataFusion gained a repository and a maintainer roster this week. The headline is the vote to accept the Iceberg DataFusion integration into the DataFusion project, opened by Andrew Lamb on September 9 with a proposed PR from Gabriel Musat to move the code into apache/datafusion-iceberg. Andy Grove and L. C. Hsieh voted binding +1, with Kevin Liu, Kumar Ujjawal and Shekhar Rajak non-binding, and Liu cross-linked the matching Iceberg vote so both communities are voting on the same move. The discussion that led there ran 30 messages across both lists. Lamb's argument came from the field. At VLDB he spoke to at least three companies adding Iceberg support to their products, and every one of them had forked iceberg-rust for some reason. His conclusion is that giving the DataFusion integration access to maintainers who have deep DataFusion context helps everyone, and that the code should stay in the ASF under DataFusion governance rather than drift into a vendor repo. Kurtis Nusbaum, writing on the Iceberg list, agreed the DataFusion internals knowledge does not belong in an Iceberg-specific package, while flagging two honest counterarguments. Anyone forking to add a feature now has to fork in two places, and arrow-rs and parquet already show that one repo can hold two projects. He judged neither strong enough to outweigh the maintenance case. Kevin Liu agreed and removed the Iceberg Python binding that exported DataFusion's TableProvider, one less coupling to carry across the split. Renjie Liu and Gabriel Musat backed the plan, Musat volunteered to port the commit history, and Lamb filed for the new repository. The history port PR and a compile-and-test PR are already open. Two releases are in flight. Tim Saucer's DataFusion 55.1.0 RC1 vote collected binding +1s from L. C. Hsieh, Adrian Garcia Badaracco, Andrew Lamb, Marko Milenković and Oleks V., verified across Apple silicon and macOS 15 with rustc 1.98.1. The sqlparser-rs 0.63.0 RC1 vote is moving in parallel, which matters well beyond DataFusion, since sqlparser-rs is the SQL front end for a long list of Rust data tools. Andrew Lamb also announced Luca Cappelletti as a new DataFusion committer, with congratulations from Bruce Ritchie, Kumar Ujjawal, Bhargava Vadlamani and Jeffrey Vo. And Comet got its own meeting slot. The dedicated Comet weekly sync now runs Fridays at 10:30 AM Pacific, with Bhargava Vadlamani adding the meeting link to the Comet docs and Manu Zhang asking for recordings so contributors in other time zones are not shut out. The project is also crowdsourcing its September ASF board report. Apache Ossie Ossie, the incubating semantic layer interchange project, had its most substantive week yet, and it is the list to start reading if you have not. Justin Talbot opened a PR adding a Relational Query Interface specification to core-spec, coming out of the expression language working group. It defines Layer 2 of the proposed layered query interface, covering how a semantic layer exposes an Ossie model to SQL-native BI and AI tools as related SQL relations, and what correctness guarantees hold when measures are queried through ordinary SQL with a MEASURE() extension. Some of the core ideas trace back to Hyde and Fremlin's Measures in SQL paper. That paper's first author is on the list. Julian Hyde, in his introduction thread, laid out the position that will shape this spec if it holds. The query language must be closed, meaning query outputs have the same shape as their inputs, so queries can be composed on queries. If the language also subsumes relational algebra including joins and aggregation, and can define measures, then no separate modeling language is needed, because models are definable on base tables with the equivalent of CREATE VIEW. That is a strong claim, and it cuts against how most semantic layer products are built today, where the model is a YAML dialect sitting above SQL rather than an extension of it. Jakub Moravec brought the equivalence problem in from a different angle. If Ossie converts a model from tool A to tool B, who guarantees the two are semantically equivalent? He argued at least one dimension of that is a lineage problem, drawing on his OpenLineage experience, and noted that evaluating equivalence by running queries depends on both implementations being correct and on the test data being complete. His related point in the how do we expect OSI to be used discussion is the one that should worry spec authors. In OpenLineage it is easy for a payload to be schema-valid and still useless for cross-vendor integration, because checking syntactic validity is simple while checking whether what was documented is sufficient is not. Modeling proposals stacked up. A proposal for shared filters, shared dimensions and metric references drew a strong argument from wanggaohang for model-level filters. Enterprise filters get shared by many metrics, and inline-only definitions make every business-definition change expensive to propagate, while filters and metrics are often owned by different roles, with domain experts maintaining the definitions. Mario De Felipe argued in make relationship cardinality explicit that cardinality belongs in two places because it means two different things. At the ontology layer it is a business rule that holds regardless of how tables are laid out. At the dimensional and metric layer it describes the physical data, which is what join safety and path selection need. There are also live proposals for hierarchy support, where fabrice-etanchaud pointed at Mondrian's long-standing treatment as prior art, plus semantic filters and a dedicated thread for downstream agent field reports. Vendors are showing up with code rather than opinions. Damian Waldron announced a bidirectional ThoughtSpot converter between TML and Ossie, ThoughtSpot's first code contribution to the project. Ossie datasets map to a ThoughtSpot model's tables plus a Table or SQL View document per dataset, fields and metrics map to model columns, relationships map to joins, and anything with no Ossie equivalent rides in custom_extensions under a vendor entry. Timextender also introduced itself and a converter it plans to contribute. Converters are how an interchange format proves it is real, and that custom_extensions escape hatch is both the pragmatic choice and the thing to watch, since every vendor-specific field that lands there is a piece of semantics the spec has not yet standardized. The project is also learning to ship. The first release discussion settled the scope question quickly. Markus Weimer asked whether the first release needs to line up with the 1.0 spec and suggested a 0.3 instead, noting from past incubators that building the release muscle matters as much as the technical agreement. Yufei Gu confirmed the community sync reached the same conclusion, Jean-Baptiste Onofré said he is building the release machinery for a source-only 0.3.0 to verify plumbing and run a full legal check, and Russell Spitzer endorsed treating it as an exercise rather than loading it with meaning. Kurt Stirewalt asked the practical follow-up: the ontology working group wants PR 332 in the release, and is there a formal process for requesting that, or for deciding which in-review features make a given release? Governance questions like that are exactly what a first release surfaces. The project also welcomed Josh Klahr to the PPMC, and worked through build questions including Just versus Makefile and consolidating Python code under one uv workspace. Cross-Project Themes Code is moving to where the maintainers are. The Iceberg Rust DataFusion integration voting its way out of Iceberg and into DataFusion is the clearest case, but the same instinct shows up in Arrow debating whether a Parquet-defined type should carry a parquet namespace, and in Parquet inlining parquet.thrift into parquet-java instead of coordinating two repos on every change. Three years ago the ecosystem grew by adding integrations inside each project. Now it grows by putting each integration under the governance of the people who actually maintain that side of it. Andrew Lamb's VLDB observation, that every company adding Iceberg support had forked iceberg-rust, is the market signal behind the governance change. Catalog metadata is becoming spec, and the arguments are about contracts rather than storage. Iceberg voted labels into the REST read path and passed finer-grained read restrictions. Polaris spent the week on tag spec encoding rules, on whether a config-endpoint check is an operation or a privilege, and on what a share is as an audit unit. In every one of those threads the productive move was the same: separate the wire contract from the implementation, resolve the contract first, and let the storage layout follow. Robert Stupp's insistence on a reversible encoding for namespace elements containing ampersands and percent signs is not pedantry. It is the difference between a spec that survives its second implementation and one that does not. Version and compatibility semantics are the new center of gravity. Parquet's versioning thread, its unsupported-version reader behavior thread and its footer-version thread are all one question. Iceberg's decision to forbid writing _pos is the same question from the other side, refusing to add an optional field that every future reader would have to reason about. Arrow's informal deprecation of Tensor takes the same position, keeping old data readable while cutting off new writes. Formats that outlive their software have to be explicit about what a reader is allowed to assume, and three projects converged on that independently in one week. Semantic modeling has an Apache home, and it is drawing the people who wrote the theory. Ossie has Julian Hyde arguing for a closed query language, ThoughtSpot and Timextender shipping converters, and OpenLineage veterans warning about schema-valid but useless payloads. Its numeric vector question is being decided in the Parquet list. Its query interface leans on Measures in SQL. If you build on the lakehouse and you have been treating the semantic layer as a vendor concern, that assumption is expiring. Looking Ahead Iceberg 1.12.0 RC0 should appear within days, with a longer vote window than usual. The Iceberg Summit 2027 Selection Committee call closes September 11 and the PMC vote on the 11 members follows. Both DataFusion and Iceberg votes on moving the DataFusion integration run at least seven days, so expect the apache/datafusion-iceberg repo to become real in mid-September. Parquet Format 2.14.0 RC1 and Parquet Java 1.18.1 RC1 are open, and the vector type physical representation is the discussion most likely to produce a decision with ecosystem-wide consequences. Polaris will cut a new Catalog Migrator RC once the license packaging fix merges, and the Open Sharing draft PRs are the ones to read. Ossie is building release machinery for a source-only 0.3.0. Community Over Code lands in October, and Matt Topol has already flagged the Arrow device interface work as hackathon material. Want to go deeper on Apache Iceberg, Polaris, Arrow, Parquet, DataFusion and the rest of the open lakehouse stack? I write books on all of it, from beginner guides to architecture deep dives. You can browse the full catalog at books.alexmerced.com. Alex Merced, Data Lakehouse and AI Evangelist