Recordly at dbt Coalesce 2022

dbt Coalesce is the analytics engineering conference hosted by dbt Labs, the company behind dbt. This year, Coalesce was held for the first time face-to-face in New Orleans, Louisiana, from the 17th to the 21st of October. Coalesce 2022 focused on two announcements: the Semantic Layer, and dbt support for Python models. Python is an interesting opening for sure and allows choosing the best language for a given case. But the Semantic Layer. That is going to be a game-changer.

Python as the second language!

First, Python support. As with the Semantic Layer announcement, this wasn't a surprise. dbt Core is an open-source tool, and following open-source practices, Python support has been widely discussed in dbt Github.

Why now? This year, the dependent timelines came together: Snowflake supports Python with Snowpark, BigQuery with DataProc, and Databricks has greatly improved the SQL experience with Warehouses and Unity Catalog. And why Python? It gives SQL-oriented analytics engineers a flexible language with a long history to solve data problems, and some problems are just easier with Python. And it brings Python-oriented Data Analysts / Scientists closer to Data Engineers, something that both will surely appreciate.

The really short story of dbt-py models is that you can do pretty much everything you are expecting you can do with "data python", as long as you return a data frame. You can import packages, and define and use functions. Having all of the possibilities in the same tool and the same lineage together with SQL really brings the platform together.

With the Football World Cup 2022 coming up, we are actually forecasting the winner with Python, Snowflake, and Snowpark using dbt. Follow our channels for its release.

Semantic Layer and other cloud capabilities

Now, the ground-breaking announcement: the Semantic Layer. How many times have you defined "Sales rolling 12 Months" in different tools? In Power BI, in the API layer, in Snowflake for those straight-to-csv cases? Did you remember to remove those discounted sales? And then someone brings in Tableu along with Power BI, and you do it again.

This is where dbt Semantic Layer comes in. It's a layer for centrally defining metrics (and later more) and querying them consistently from anywhere.

This opens the field for truly headless BI, not just the embedded visualizations we have now. Separating the analytics backend from the analytics frontend. You could have the data, metrics, aggregations, and dimensions defined centrally in an open and standardized way. Then, on top of this, you could have whatever tooling suits the use case best.

Cloud IDE has seen a lot of improvements and polish, with many quality-of-life improvements on the roadmap. What you’ll see right now is vastly improved cold (opening a project) and hot (refreshing the window) startup times, which are independent of project size. There’s lots of quality-of-life improvements in the UI, navigation, and so on. Some interesting picks from the roadmap are for example the integration of sqlfmt (http://sqlfmt.com/) to the Cloud IDE, and generic webhooks for notifications.

How to attend a conference AT Recordly?

What about the conference itself? How was it being face-to-face in New Orleans? Last year's online conference was packed with interesting talks and announcements. The dbt Slack community channels were just lit with discussions, both on- and off-topic. This year, when they announced they'd be doing a live conference in New Orleans, I was really excited and wanted to attend. So how does one go to a brand new conference from a year-old company?

In Recordly, we follow a "4x2" thinking in decision-making (this is borrowed from our parent company Futurice). Following the 4x2 framework, I thought about what attending Coalesce in person would mean to our people, clients, numbers, and the World, both now and in the future. Sounds intimidating but it really helps to position and frame the different relevant viewpoints. I then presented my case to my colleagues on a Slack channel. After a round of comments and discussion, I got the go-ahead. After that, it was just a matter of confirming the active projects, and booking the conference, flights, and hotel with a company credit card.

This was my first international conference, and I really enjoyed it. dbt Coalesce was a compact event, all the talks and events were in the same conference center. This doesn't mean it was small: it was sold out, and approximately 1500 people were there to share the experience. The different tracks had interesting talks varying from modeling the real world to implementing a data diff tool in your dbt project.

Besides the different presentation tracks, the "hallway" track of meeting different people from different backgrounds all with the same kind of problems in the data space was a learning experience.

And hey, Coalesce 2023 is in San Diego, California. Perhaps you'd like to represent Recordly there (given that 4x2 still stands)? Check our open positions, and join our growing band!