I: INTRODUCTION AND OPTIMOVE OVERVIEW
Adam Steidley
Hi, and welcome to The Check with Joseki Tech today. Joining us again are Lior and Amelia from Optimove. Hi, guys.
Lior Grubert
Hi, Adam. Hi, Amelia.
Amelia Dahm
Hello.
Adam Steidley
Would you guys like to introduce yourselves real quick?
Lior Grubert
Yeah, I can get us started. I'm Lior Grubert, the Senior Team Lead in the Customer Data Engineering team. I've been with Optimove for about five years now.
Adam Steidley
Excellent.
Amelia Dahm
My name is Amelia Dahm, and I am also a Team Lead in Customer Data Engineering. I'm so happy to be here, Adam. Thank you for inviting us.
Adam Steidley
Thanks for joining. We've all worked on a shared enterprise client that uses Optimove as their multi-channel marketing hub. All the customer and sales data are loaded into Optimove, and then there's a nightly process to update the customer records. We then use that to segment and send the right offers to the right people at the right time. We've seen a lot of success with this product, and today, we want to talk about some of the tricks we use on the data side to make things easier.
II: THE 'ISMAIN' FIELD
Adam Steidley
For instance, when we upload all our customer data to Optimove, we ensure that our customer records are unique by email address. When sending an email, our most common channel, we ensure we don't accidentally send the same email twice. However, since we also use this as a multi-channel hub, we deal with SMS, phone numbers, and direct mail, where addresses can be duplicated across the dataset. Amelia, would you like to talk about how we've been working this out?
Amelia Dahm
Yes, of course. Within the Optimove product, we essentially add what we call the 'ismain' field. This field indicator shows which customer record, whether there are five or ten, should be indicated as the main one. For example, with SMS, if we have duplicated phone numbers, we identify which one we want to send a campaign to. We use a multitude of different fields to determine if they're a customer who's opted in or a frequent purchaser, among other criteria, because we want to send the campaign to the correct account and ensure all their future purchases are attributed to the same account. It's worth noting that we have 'ismain' for SMS, but we can also apply it across various brands or channels. For the client we work with, we have it for SMS and email. I've seen this across the board; many clients have come to me with issues regarding multiple emails sent to the same customer, who is stored as two separate customers on our backend. The concept of 'ismain,' which I first heard about through the account we worked with, may seem small technically, but its impact is significant.
Adam Steidley
I think we've had a lot of good luck with it. And we use it extensively for direct mail, ensuring that if the same address line and zip code appear more than once, we pick one of those records as the 'winner'—the specific 'ismain' doesn't matter. You're going to send that mail piece to the same physical address one way or another. But it might have a different first name—is it 'Hello, Adam' or 'Hello, Adam's wife'? Another aspect for Optimove is when the sale record comes back, and the sale is attributed to that customer record. We want to ensure we're picking the record we think most likely made the purchase. So, when we're using these 'ismain' operators, we first check to see if one of them can receive the message, and then we lean towards whoever has the most recent sale, which tends to bubble up toward the top of the list. It's worked pretty well for us. A crucial caveat here is not to forget the 'ismain' operator on your target group because then you'll end up sending duplicates. It's a mistake that newcomers make once and only once.
III. RANDOM SLICES AND HOLDOUT GROUPS
Adam Steidley
Okay, so one of the other things we use, which doesn't involve the data team as much, is random slices. These are super handy. Let's just look at these real quick in the tool. We have the 'random customer slice' and the 'random customer percentage.' A slice is a decile assigned to your customer record when it's first created. Then, it stays the same for the customer's entire lifetime. It doesn't change from one week to the next or from one campaign to the next. The 'random percentage' works similarly, ranging from one to one hundred. Why are these interesting?
We often hold out a control group for a given campaign, which tells you how well that particular campaign performed. Sometimes, we need to do a long-term holdout. We just finished a big analysis with that client. They're using the 'risk of churn' feature within Optimove and wanted to see how well that helped them improve their sales. So, we took the two lowest slices, the bottom 20%, and excluded them from all the 'risk of churn' campaigns so that we could then look at how that 20% performed over a 30-day period versus how the people in the 'risk of churn' campaigns performed. We saw a nice lift there over the month, and then we looked into the sub-slices for the 'risk of churn' to see how one worked better than another. That worked out really well and is a trick we use all the time.
IV. BACKGROUND DATA UPDATES
Adam Steidley
Another common problem is that some days, data doesn't run; one of the tables fails, or something on the backend prevents the data from getting to you. Lior, what's one of our tricks to help alleviate that problem?
Lior Grubert
In the data team at Optimove, we have something we call a 'background,' which is essentially a historical data update that happens daily. It allows clients to send retroactive data within a specific time frame. After that, you can see it incorporated into the site in sections like 'activity history,' 'purchase history,' and 'campaign analysis.' Everything happens during the batch process. So, as you mentioned, Adam, if there's a delay in the data or something happened along the way, and the data from a specific day didn't get included in that day's batch process, Optimove has the capability to capture retroactive data within a predefined time frame. We can define it as a seven-day, a five-day, or a thirty-day period. Many of our clients use this feature, which calculates the data retroactively. So, it's kind of like a rolling period. If we define a seven-day period, every single day, we're going to calculate seven days back. This allows us to be more tolerant of data issues that our clients might encounter.
Adam Steidley
That's proven to be really useful. There are a lot of times when we calculate something like the lifecycle stage, which is based on the previous lifecycle stage. So, you really do have to make sure you're processing every day carefully. And if you miss one, it can be a bit of a nightmare.
Lior Grubert
Exactly.
V. CONDITIONAL LANGUAGE AND CUSTOM ATTRIBUTES
Adam Steidley
Cool. So, there's another data tool that we'll share quickly here, but one of the things we use all the time is conditional language in our template. For instance, if the customer is a rewards member, show them their current number of points. If they're not, show them a join link. You can build this easily based on whether they have a rewards number or not. But what if you wanted to do something more complicated? For example, if their favorite store is one of a list of twenty, then they'll see special treatment A; another list, treatment B; another list, treatment C; and then default with treatment D. This can get too complicated to do inside the template itself because the conditional language we're able to use there is usually just pretty straightforward.
But we can create a new attribute for ourselves to help. So, over in the Data Studio, we can add a new attribute and make it a custom attribute. We can do an if-else condition here based on the attributes, kind of to our little heart's content. So, if their address is one of these values, we can now do multiple conditions. We can say that if it's this and their age is greater than 18, then they get the adult variant, location A. We can add the else conditions, repeat for other conditions, and add a default. So now we can get around many of the limitations we might have in the template by putting all the logic here in an attribute.
Once it's all done, this will be published and made available the next day. You can add these all on your own as long as you have admin capabilities. This has proven to be really effective for us in a number of places. One of the other things that's really good about it is that the person on the team who understands this data may be very different from the guy building out the templates. So, the template team just wants to have nice, simple conditions. Okay, treatment A, I'll show this; treatment B, I'll show that, and all the logic behind that they don't want to know. Then, they can build out the template in a more straightforward way. Likewise, they can come back, and if we need to change that logic, we can do it on the data side and not have to deal with the template. So, it's a good separation. It's worked pretty well for us in a number of cases.Â
VI. AD HOC ATTRIBUTES
Adam Steidley
So, the last trick we had was the ad hocs. This is a process that we started five years ago, and the idea is we'll have data in our database that we need to send, but we might only need to use it for one campaign. So, I don't want to create a new field, and you can go back and, you know, listen to our last podcast. Creating a new field is pretty easy but still requires work from the data team. It's going to take a couple of weeks to turn it around. So, we do ad hocs. And you, Lior, how have you found using those from your side?
Lior Grubert
Yeah, this has been super helpful for our clients and a relatively smooth process for us to implement. Basically, we get a specific data point or a table from our clients, and then we give a custom attribute or something ad hoc; we just add it, and then it will always be available as an attribute on the site. Our clients can then choose whatever goes there. The way it works on our site is that all we need to do is have a specific type to join it into the customer record.
It could be a rewards number, an email, or any other data point that will tie into the customer record, and then we can truncate all the data every day. Our clients can upload it every single day according to their business needs without the data team having to do anything to accommodate that.
Adam Steidley
Yeah, so this is the way we're doing it now, just for those who are a bit more data-minded: In our Snowflake, we've got a table and a view, and the table just has a campaign name, which is really just to track what's going on an available date, which is when the data should start being available, and a launch date, which will be the last day you need the data. We've got six different ad hocs, so we didn't want to do six tables; we just did ad hoc numbers one through six. The priority: How important is this campaign versus another one? We're joining the customer record to the email address and the value.
So, on any given day, we just want to show the highest priority ad hoc value for that ad hoc number for any given email address. So, we create a view that will then be partitioned by email and ordered by priority in the campaign, but only where the current date is between the availability and launch dates. Then we're sharing this with Optimove, and they just merrily load all that data daily. And then, for segmentation, the data team can just work in Snowflake, where they live all the time; they just populate the table and then go and create a target group using ad hoc attribute one, or it's equal to a given value, so it works pretty smoothly. We also use the same idea except where it's not one ad hoc value; it's ten.
We did this because we had a use case, the annual membership status email, which needed a bunch of different inserts we didn't want to add to the customer record, ten fields for one send a year. So, we said, okay, we'll just do this, populate a bunch of fields, and use it for that one time. And this has proven useful for, you know, probably three or four campaigns that go out every month are using it one way or the other, but we do the same idea with the view and the same with the table and the view. That's then looking over the values to take the best one.
Amelia Dahm
Yeah, I definitely think this is a super useful trick. I know that within Optimove, we put a lot of emphasis on the process running time, and we don't want to add attributes continually if they're only going to be used once. So, in terms of time to value, you get the data on the Optimove site very quickly and at your own pace. You can have it on the site when you want to do it. And then, from a batch process running time, we're not going to have all these additional unused fields on the site that were only relevant one time. So, I do find a lot of value in these from a lot of different aspects.
Adam Steidley
Yeah. The other useful thing is that the business often has a new idea they're going to want to do. This gives us a way to add the field immediately to the market while we take two or three weeks to complete the process. And as well, a lot of the campaigns and ideas don't work out and don't wind up being things that you wind up wanting to keep around.Â
VII. CLEANUP AND OPTIMIZATION
Adam Steidley
So, as our last item cleanup -- we all got to worry about our cleanup. So, Amelia, you want to talk us through real quick how we look for attributes that aren't being used and how we just sort of try to get rid of them?
Amelia Dahm
Yeah, of course. I would be happy to. We've been referring to the batch process for the client that we work with. It runs at 02:00 a.m. It will run for about 6 hours until you can actually execute campaigns. We want to make sure that the process is as smooth as possible and as quick as possible. Just so whatever campaigns need to be sent out can be on time. So, there is some responsibility on the data side to ensure our process is clean. So, a lot of the time, we run an automated process built where we can understand on the back end what clients are using and what they're not using. So, what attributes are being used in target groups today? What attributes haven't been used in four years?
Amelia Dahm
It shouldn't take too much time during that batch process if it's not being used. It's only adding more to it. And I think from a user perspective as well, it's never helpful to have too much clutter on your site. When you're getting multiple marketers looking through hundreds of attributes, having only the ones you know are clean and viable to use within the Optimove site is better.
Adam Steidley
Yeah, and the process that we go through, and we try to do this about once a quarter. So Amelia will come back with some attributes that aren't being used, and then I'll go to the business team to double-check if they're being used, like if anyone remembers why we would be using them or not. Usually, we thought it would be a good idea, but then it never came true. Or someone who doesn't work here anymore and thought it was a good idea, those kinds of processes. Our first trick is to have our customer service rep rename all those fields. Do not use that; we're going to deprecate. Then, we can pull them out of the process to save the processing time, and then when they're no longer referenced, we can remove them from the instance. Exactly. And Lior, we saw some really good improvements with our processing time through this, right?
Lior Grubert
Yeah, definitely. Throughout the entire time, I think it's been like five years now. We've gotten from, you know, hours of hours to a short amount of time using those processes, making sure that our code is extra clean, being efficient with our queries, just to make sure that we indeed send the campaigns as soon as possible. And just so you know, we're dealing with a lot of different, you know, data points, more than 500 attributes, 80 tables on a single every single day. So that's a lot of data, as well as a lot of customer records. So, it's important, like Amelia mentioned, to make sure that everything stays as clean as possible.
VIII. CONCLUSION AND FUTURE DISCUSSIONS
Adam Steidley
Excellent. Well, I think we got a lot of good tips for people today, so thanks, Lior. Thanks, Amelia. And we will be in touch. I'm sure we will have another podcast soon. Thanks.
Comments