The Check with Joseki Tech | Episode 8: A Conversation with Optimove (Part 1)

I: INTRODUCTION AND OPTIMOVE OVERVIEW

Adam Steidley

Hello, everyone, and welcome to The Check from Joseki Tech. Today, joining me are Lior, Amelia, and Brittany from Optimove. We will talk about some of the data and processes around Optimove today. So, can we all introduce ourselves? Do you want to start us off, Lior?

Lior Grubert

Yeah, sure. Thank you, Adam, for the introduction. I'm Lior Grubert, the senior team lead on the customer data engineering team. I have been with Optimove for about five years now. Excited to record this today.

Amelia Dahm

My name is Amelia, Amelia Dahm, and I am a customer data engineer team leader at Optimove. I have been with Optimove for exactly three years now, almost to the date. I'm so excited to be here and talk a little bit about it.

Brittany Crowson

I am Brittany Crowson. I am a technical project manager and have only been with Optimove for about seven months, so I am the baby on the team. We have a lovely team, and I'm super excited to be here to discuss it.

Adam Steidley

Excellent. So, for those who don't know, Optimove is a multi-channel marketing hub used for sending emails, texts, and direct mail, among other things. It also serves as a customer data platform, so you upload your customer data—things about your customers like their tier in the loyalty program, names, addresses, etc., and your sales information about them. And then Optimove, their secret sauce, is they can learn based on a customer's micro-segments what is best to send to them versus someone else. So we can pick the right campaign for the right person at the right time.

II: DATA INTEGRATION AND PROCESSING WITH OPTIMOVE

Adam Steidley

So, Lior and I have been working on the same large enterprise client for four or five years now, and they've got tens of millions of customer records and hundreds of thousands of sales records every day that go over. Today, we want to give you a little intro to what some of that data process looks like and then how we do the care and feeding of that because, you know, it's a large client, and they've got new data all the time, and for years. I think we've really gotten good at it. The meeting that the three of us have every other week is, you know, pretty pleasant, pretty fun, because we all know what we're doing and when things move along at a nice pace. So, Lior, do you want to introduce how some data processes work?

Lior Grubert

Yeah, definitely. First, I wanted to distinguish between the batch and real-time processes. Today, we're going to talk about the batch process. So, every day, we get data from this client and process it on our end automatically. Everything is set up in advance, and the data could come from various sources. Optimove and the client use Snowflake, and we can integrate Snowflake into the data very smoothly. Other data sources we use at Optimove include flat files, semi-structured files like JSON or Parquet, and structured files like CSV.

Lior Grubert

We can also connect to any given database, whether a cloud or a non-cloud database, and we can also get batch data via API. Specifically, with this client, we use Snowflake to utilize their process, and then we also use flat files. After we ingest those data, we have what we call a single customer view where we calculate all the different parameters, all the different attributes, as we call it, or data points, and let our product do its magic. So afterward, you'll be able to send all the different campaigns to the right, like Adam mentioned, to the right customers with the right channel at the right time. 

III: CHANGE MANAGEMENT AND DEPLOYMENT SCHEDULING

Adam Steidley

Amelia—we have a cadence that we use to look at and deploy new changes. Can you talk us through how we've been doing that lately?

Amelia Dahm

Yes, of course. So we meet biweekly, as we mentioned, in these super pleasant meetings and talk about the upcoming changes we expect. So with the process that Lior refers to, this batch process, we're constantly making, integrating changes, adding new tables, adding new attributes, adding new things and adjustments as this business evolves. So, within these meetings, we will discuss new sources and what the source will look like. We will also do some testing within these meetings to understand how we will roll this out and make it live on the Optimove site. We also talk through different use cases. If there's something we can do or are unsure how to do, we'll all come together to discuss exactly how we can utilize the Optimove product to have that within the site.

Adam Steidley

And we're doing deploys every other week at this point, and that's also gotten pretty steady. One of the things we do—I think we can all attest it works well—is waiting until the end of the daily process when all of that day's emails have gone out. And then, at that point, you guys roll out the new code so you can rerun today's data to ensure the process looks good. So the whole process is intraday, which everyone loves, not having to wake up in the middle of the night for these things. We get to see it run nice and smoothly once, and then we run with the real data that evening. So that's gone well for us. We've gotten into that groove for a while now; we also picked the day of the week. We're typically doing a Tuesday these days because Tuesdays into Wednesdays are the least important email days for the client. So, we found it to be a sweet spot for us. 

IV: STANDARDIZING CHANGE REQUESTS AND DOCUMENTATION

Adam Steidley

Brittany, you're our project manager, and I know you're new to the team. We've developed our own habits of doing things, and from what you've told me, all of your other clients have adopted a more standardized process based on how we have been doing things for a while. Do you want to walk us through what that looks like?

Brittany Crowson

Yeah, absolutely, and it goes back to what you just said, where there is a huge collaboration between us and our clients. And, I'm about to show you that the form we use to align all these details is really important to the organization. We have all of the really important parts of the data that need to flow into our UI available to us without having to do too much back and forth or potentially missing something important. So, I'll share my screen really quickly and walk through this. All right. It's pretty straightforward and simple, but we now utilize this with all of our clients for any change requests or changes in the data that will go forward in our sprint process.

Brittany Crowson

So, the clients would come in and fill this out themselves. So they give their instance name, project manager (they usually put my name here), email, and their name if this change will be for more than one instance. An instance would be if they have a QA instance and then a production instance, for example. So they would say yes or no to that. Is this related to any kind of consent or opt-in or opt-out features? And then, down here, you get into the actual requests. So we have all of these laid out here: adding new tables, adding columns to existing tables, updating customer attributes or activity history, and updating purchase history attributes. 

Brittany Crowson

Let's say we're going to add a new data table. When you click on it, it takes you down here to this really pretty, again, very detail-oriented section where the client would put in all of the examples, very specific examples of what they need and what we will be looking for in the data that gets shared in the future. In order to make this change happen, they can also add files and notes here. Basically, when they submit this ticket, it comes directly to me, and then I work with our data team to implement this change. As you said, we meet biweekly with the client and then internally to ensure that we're aligned and that what we are communicating back to the client is accurate for that sprint.

Adam Steidley

Cool. Yeah, this does look a lot like the confluence page we've been building for years, and the process we've been on is at the client site. We have a confluence instance and a sample page with the same exact information. Then, we share a PDF of that with the Optimove team for discussion. The other thing that is valuable from both sides is the notion of living documentation. Once we're a year down the road, we say, "Oh, what did we do with that field?" It's good to have the documentation around. You can always pull it up, reference it, and bring it up from sometimes before.

Brittany Crowson

And with our really big enterprise-level clients, we're dealing with a lot of data and a lot of data points and records. So keeping all of this very detailed and organized is extremely important for us.

Adam Steidley

Yeah, excellent.

Amelia Dahm

I do want to say this form is relatively new at Optimove. We've used a different version of the confluence page with this enterprise client for years. We saw how well it worked and decided it was applicable to all our clients. Adam, you were providing all the details. You know Optimove very well; we also needed to ensure all the other clients do. There are a couple of specific processes within Optimove that are super important. Let's say a file is missing from the batch process. We need to understand if we're allowed to execute campaigns with or without that file. We need that set-in-stone detail for each table; if that gets lost in translation, there could be issues down the line.

Amelia Dahm

The documentation is super important, and the conception of it was derived from our working relationship, which is awesome.

Adam Steidley

Now, when we move on, you guys will build out the code. To me, you're like the development team, and then everything gets handed off to the ops team. So, are these forms also available to the Ops team so they can better understand how the data is set up should they have problems in the middle of the night?

Lior Grubert

So this is accessible, but they don't really use it. They point it out to us, and we have a broader team (not just in the US, but we also have a 24/7 team that operates in Tel Aviv, India, and the UK) that can provide full support. Once those engineers get to see those forms, they're able to identify the issue if needed or perform any fixes on the spot. So, it doesn't necessarily mean they have to implement it or have the US team wake up in the middle of the night to perform it. It can be ready by the time the team wakes up.

V: CONCLUSION AND FUTURE DISCUSSIONS

Adam Steidley

Excellent. I hope that was instructive for those. I know we've got some new features coming in Optimove for self-service for adding data. We can talk about that in a later call. We've also got a bunch of tips that we'll want to share with people, but we'll save that for the next podcast. So anyway, thank you, Lior, thank you, Amelia, thank you, Brittany, and we'll talk again soon.