Watch our highlights video describing how Apache XTable (Incubating) - formerly OneTable - brings interoperability to all major data table formats. This video comes from a session at Open Source Data Summit in November 2023.
Transcript
Tim Brown
All right, so to kick things off, I think we need to start from a shared understanding of what are
these table formats. And at their core, these formats are really just great abstractions that allow
us to think about large sets of data like tables. We can do inserts, updates, deletes, manage the
tables, and really just make our lives a lot easier when dealing with this data at scale. So there's
three main players in this space right now. But when you think about it at rest, when these
tables are just existing in your S3 bucket or GCS or your Azure account, they're not all that
different. They're metadata in a set of Parquet files or whatever other format you're using to
store that data. So with that in mind, the OneTable project is really just this omnidirectional
interop for the table format.
It's a lightweight translation layer of this metadata that allows us to, you know, read,
fundamentally read the tables as any format and work with any sort of tools that maybe require
a certain format. The project itself is not a separate table format. It's not adding any new
metadata. It's just building on top of what's already present there by these three existing
projects.
So there's a lot of engineering decisions that have to be made when you're building out your
data lake, and a lot of time can be spent tinkering with different formats, trying to understand the
features that exist in each of these formats that fit your use cases best.
It could be possible that one format is not going to fit every use case, and maybe there are also
vendors that you're considering using that only support one of these three formats. You don't
want to be laying awake at night wondering what life would have been like if you had just
chosen the other format. So with that, we present OneTable, which helps you write once and
query everywhere.
Ashvin Agrawal
But this is just the beginning. There are a lot of good interesting problems yet to be solved which
includes merge on read, delete, vectors, Apache Paimon which is a new project coming from
Alibaba team that is getting integrated. We are super focused on improving the
performance using the footprint of OneTable so that it can live hidden behind in the infra and
automatically generate these target table formats. We want to work on various deployment modes we and also take it more closer to the engine integration as possible. And in the long run even,
there are many more exciting features both for researchers and for industry. For example, active
writers synchronize, commit timestamps and so on.
I'm super excited and there are a lot of problems which we want to work on with you. I
welcome you all to join the project. Come and share, contribute, provide us insight.