I’ve been thinking a lot – about my relationship with technology, what it means to be a semitechnical, and how AI is changing the way people conduct themselves at work.

It’s taken some time for me to adjust to this new reality in which it’s becoming increasingly common for people to default to using AI when starting to work on a word artifact – docs, memos, articles, social posts, and even internal messages.

I too have used AI for all of the above tasks and have found that in cases where the stakes are low, letting AI do the thinking for me frees me up to continue thinking about the things I do want to think about.

The problem I run into is this:

Most times, I tend to care a little too much

I’ve always cared about what I create.

Whether it’s a work artifact, a newsletter post, or even a weekly update sent in a Slack channel. And this habit dates back to 1997 (I was eight).

Funny that I happened to sift through my primary school report cards and came across this one where my first-grade teacher wrote, “Arpit, you are very painstaking and particular about your work. Keep it up.”

It’s starting to all make sense now.

I’ve always been particular about what I create, and have paid the price for my obsession. Starting middle school, I could never finish my examination papers because I had to make every word sound good and every diagram look great.

I distinctly remember that I’d be on page 2 of 4 of my answer sheet while my peers would already walk up to grab an extra sheet. They could go on spilling words without the need to pause and think. And it served them well because many teachers graded based on volume rather than quality.

I did alright though. I was always told that I needed to work faster but I struggled with that. Below, my fourth-grade teacher writes, “Carry on with your good work Arpit but try to increase your speed a bit.”

I stuck to the notion that it’s better to do less but do a good job. And doing a good job takes time because one has to think things through.

Eventually, you get faster at the things you become really good at, and that’s how you stay ahead. But speed alone is not enough. What you create has to be worth consuming – you want people to keep coming back for more.

Rethinking my technology use

I’m in the camp that believes AI will completely change how people use software – both B2B tools and consumer apps. We will increasingly talk to our machines and spend less time interacting with pixels.

But what won’t change is the need to create and consume word artifacts at work – from internal memos and messages to external posts and positioning. And what separates the average, good, and great is the time spent in thinking mode.

As more people default to AI-generated artifacts for pretty much everything, the real value will be created in organic artifacts – ones involving humans spending time doing the thinking.

Thinking horizontally

When embarking on any new project of consequence, I like to spend time thinking horizontally – lying down, tapping into lived experiences and accumulated knowledge.

Once I have generated a thesis or a set of ideas, I might run them by AI to identify blind spots or find similar ideas. This step is useful, but it’s also easy to get carried away; practicing restraint and challenging the AI’s output helps.

Before AI, I’d find myself reading through lots of sources, ingesting a lot of information, and building upon my own ideas. I think there’s value in this organic transformation of information and I’m not giving up on it – quite the contrary, in fact.

There’s immense value in spending time in organic thinking mode before consulting AI. Depending on how fast your brain is able to process information, you might end up spending hours, days, or weeks. But that mental exercise is tremendously valuable because that’s where original ideas emerge.

The context that lives within our brains cannot be replicated inside a machine (yet) and therefore, defaulting to AI right at the start can never lead to the same outcome.

What it means to be a semitechnical

I’ve been spending an inordinate amount of time thinking about this. Because being semitechnical is not being halfway between non-technical and technical. It’s not just about building without code. And it’s certainly not about learning how to use a tool or two.

I think of it less as a skillset and more as a mindset that molds one’s identity.

Of course you need some technical skills to operate some tools – just like in any job. However, what sets knowledge work apart is that one’s output is a clear reflection of one’s thinking.

It’s becoming increasingly easy to spot when the thinking is outsourced to the thinking mode inside ChatGPT, Claude, Gemini, etc. And letting AI do the thinking is a clear reflection of how much one cares about the artifacts they create – very little, obviously.

But that’s not it.

When one chooses to let AI create artifacts, they also choose to care less about the experience of the people consuming those artifacts.

And if they choose not to care much about people’s experience consuming their work, they are unlikely to care much about the work itself, let alone take pride in it.

And one not taking pride in their work leads to mediocrity.

Finally, in the age of AI – the epitome of mediocrity – there will be no place for humans creating mediocre work.

Staying ahead

The obvious answer is to shun mediocrity and set a high bar for yourself. Nobody else can do that for you, and unfortunately, most managers will expect you to be part of their performance theatre rather than push you towards excellence.

People say they want to work with people smarter than them, but when they find themselves in that situation, most cannot handle it. Humans have evolved into control-seeking animals and independently-minded people are perceived as threats by most. And those who settle for mediocrity are the easiest to control because they are the easiest to replace.

On the contrary, those who consistently produce top-notch work – not because it’s expected but because they choose not to settle for anything less – are always at an advantage.

But the question remains: How do you pursue excellence when surrounded by mediocrity?

Well, once you set a high bar for yourself, you can tap into your intrinsic motivation – one that doesn’t rely on external validation. You need to decide what good looks like and work towards greatness – you will know once you get there.

Most people only talk about it but very few experience it because getting there takes years of effort (plus a healthy dose of good luck).

The Semitechnical Cake

In my quest to get more people to understand what it really means to be a semitechnical and identify as one, I’ve come up with this visual model called The Semitechnical Cake:

The model depicts the 18 core traits of a semitechnical. The bottom layer lays down the “technical skills” that need to be learned – everything else is an attitude, part of one’s mindset.

As you can probably tell, I think about this stuff a little too much and it’s taken several iterations to arrive at this model. There’s very little overlap between the traits; you start at the bottom and work your way up the layers, adopting the attitudes one by one.

You might be nodding and saying to yourself, “Well, I already possess these traits”, to which I’d say, “Well then, my friend, you’re already a semitechnical and you must stop calling yourself non-technical.”

Hear me out

You're better off spending your limited time and mental capacity learning the basics of any given piece of technology that you intend to use every day – instead of every other tool built on the same foundations.

LMK what you think on LinkedIn

It’s funny how sometimes you start doing something you really want to, but then get presented with an opportunity that would be silly to pass on, only to find yourself years later wanting to go back to that thing you left incomplete.

And when you finally do go back to that thing, it feels as though a part of you has been waiting all along for the other parts to come back from the detour, so that you can get going towards the final destination.

Back in 2019, when I was leading growth at Make (then Integromat), the no-code trend was in full swing. And yes, it was a trend and not a “movement” as some liked to call it.

My biggest point of annoyance was that instead of helping newcomers acquire foundational knowledge about software development, data, and automation, individuals with large audiences in our space only cared about pushing tools they were affiliated with. They made their bucks and moved on to the next hot trend – rinse and repeat.

The same thing is happening today.

Too much focus on learning specific tools rather than learning the underlying technologies.

“Experts” are pushing the same false narrative: learn this tool – and not that one – if you want to do this work.

People who are new to the space, looking to acquire monetizable skills to stay relevant in this mad economy, are overwhelmed. They are spending their limited capacity deciding which tool to learn and jumping from one to the other based on recommendations from anonymous gurus on Reddit.

The part of me that was left behind is finally reunited with the other parts, and together with their combined knowledge, are now headed to the final destination – a place for all of us semitechnicals.

We’re not “non-technical”

Crazy how even in 2025, people like you and me are often labelled “non-technical” because we’re not code-writing engineers.

Well, I refuse to accept that label and so should you – especially if the following sound like you:

• Thinks in systems

• Builds with software

• Understands the constraints

• Documents the little things

• Automates the tedious

• Thinks critically

• Communicates clearly

• Chooses creativity

• Doesn’t settle for mediocrity

• Thrives in uncertainty

• Attracted to problems

• Makes incremental improvements

• Chooses to figure, not fret

• Permissionless doer

• Helps others thrive

• Cares about their craft

• Practices Delayed Gratification

• Infinitely curious

Sounds like you?

Then for god’s sake, stop calling yourself “non-technical”. Where you live on the technicality spectrum is for you to decide, but keep in mind that your position isn’t static – you’re moving every day. And not necessarily to the right → but also to the ← left.

There are days when I spend hours making diagrams in Miro, playing around with every little element (and bugging my wife to help me decide between variants that often look the same to her).

What you see below is an outcome of me going through this process – one that I won’t give up for the sake of moving faster:

The primary idea behind the technicality spectrum is that you don’t need to hit a milestone to call yourself a semitechnical. Similarly, even if you already have technical skills in a particular domain, you can identify as a semitechnical in an adjacent one.

In fact, the more I think about it, the more I’m convinced that the world already has more semitechnicals than non-technicals.

If you’ve ever seen a live movie set, you’ll know what I’m getting at.

The people operating those insane cameras or the people obsessing over lighting positions are super technical people, aren’t they? But the moment they step out of the movie business, they are labelled “non-technical” lol.

I’m convinced that this is true of every industry, and not just tech:

The very-technicals tend to think less of the less-technicals, even when the same people might be way more technical than the very-technicals in an adjacent context.

Think about this:

Who is more technical? The engineer building AI tools to generate videos or the filmmaker who is tinkering with those tools to create something worth watching?

Engineers thinking of users of their software as non-technical is not very different from a luthier (instrument maker) thinking the same of a rock star. Imagine Orville Gibson, the inventor of the Gibson guitar, thinking “less” of Jimi Hendrix. Crazy right?

So, the next time you see someone use the term “non-technical”, let them know about the technicality spectrum and help them understand what it means to be a semitechnical!

Happy Holidays! 🎅

Before you go, you gotta check out this video by my friend William, who’s helped me understand that being a <semi> can mean so much more!

Share

I’m not an engineer.

I don’t know how to code.

I love using software though.

In fact, I’m obsessed with software that enables Semitechnicals to do stuff previously only possible with code.

Does that make me “non-technical”?

Well, in my opinion, everybody, especially if they work in tech, has some level of technical aptitude and sits somewhere on the technicality spectrum.

Semitechnicals act as a bridge between the more technical teams – typically data and engineering – and the less technical teams, particularly GTM teams, as they’re the ones putting data to use in their everyday tools.

Today, many different teams act as that bridge team, and they go by many different names. However, Ops still remains the easiest term that encapsulates the work done by bridge teams.

You can call them RevOps, BizOps, MarketingOps, GTM Engineering, Value Engineering, or whatever else you fancy; the job remains pretty much the same, and the people executing it are all, well, semitechnicals.

Semitechnicals don’t build features or orchestrate data pipelines – instead, they set up processes that enable more technical and less technical teams to collaborate effectively.

Semitechnicals understand the priorities, constraints, and workflows of both. They have a sound understanding of all the systems and can translate requirements into automated, well-documented workflows.

Most importantly though, semitechnicals know what’s good for the business.

They help GTM teams utilize technical assets (like internal tools and data models) in their everyday tools to drive growth. At the same time, semitechnicals help data and engineering teams understand the impact of their work on the business.

So, who is a Semitechnical?

• A semitechnical is someone who uses software and data to solve everyday business problems.

• A semitechnical doesn’t code like an engineer, but they do think like one — enough to architect, automate, and document critical workflows.

• They have a sound understanding of data foundations.

• They can read and write enough HTML, SQL, and JSON to get their way around whatever tools and data they need to.

Moreover, throw any piece of software at them, and they will use the heck out of it!

But…

That’s just one flavor of a semitechnical

I know this guy called William Classic. He’s one of the few people I know who uses AI every day to get better at his craft – telling stories and uplifting brands via amazing videos.

He not only has the technical ability to work with video equipment of all types, but also knows how to amplify the impact of his work by incorporating software in his process.

And in this line of work, process is everything.

William might not know what SQL stands for, but he is damn fucking technical when it comes to using powerful creative tools.

He has this rare ability to take the content he consumes and blend the ideas with his own to create masterful work.

So, would you consider William “non-technical”?

Wanna share your semitechnical perspective? Leave a comment here or on LinkedIn.

I’m not an engineer.

I don’t know how to code.

Even though I learned programming in high school and aced the computer exam, I wasn’t quite interested in writing code.

Today, I don’t pretend to be even remotely interested in dabbling with code.

I’m too impatient. I love interacting with well-designed interfaces. And I’ve got nothing to prove to anybody.

Growing up, I had two very different kinds of role models – besides my mother, of course, who was relentless in her pursuit of making me excel at everything I did.

My father, who worked independently and used his math skills to make money. And my mother’s brother, a renowned engineer (the real kind) who has dedicated his life to building technology in India for the benefit of the masses.

Both of these men have held the highest possible standard for integrity – they’ve only cared about making an impact, have never been money-motivated, and have had zero tolerance for bullshit.

How do you expect me to be any different?

Moreover, having lost both parents fairly early, I value nothing more than time.

And I have a lot left to do with my limited time.

But this time, it’s different

Of all the hype cycles in tech over the last two decades, this one’s different.

Firstly, unlike Crypto and VR, GenAI has already hit mass adoption; people like you and I use it every day.

Secondly, and I think this is a key difference, GenAI is amplifying the benefits of existing tech rather than forcing people to use something completely foreign.

We have a better browser. We spend less time scrolling through web pages plastered with ads. And what I find most exciting is the renewed interest in automating the mundane.

Workflow automation isn’t new – far from it. It’s been around a long time and we’ve had access to really good automation tools for the better part of the last decade.

However, before GenAI, it took significantly longer to build and troubleshoot workflows. It was normal to spend fifteen minutes writing a data transformation formula inside a Make scenario. It would take even longer when working with Airtable because they didn’t have the best docs back then. Today, I use Claude to generate, understand, and test a formula in less than a minute. Clay has embedded AI-based formula-generation in the product, making the process even easier and faster.

Similarly, back in the day, learning how to write Regular Expressions (Regex) opened up many new possibilities and made data extraction from a long piece of text a breeze. But I had to go through the learning curve and spend a lot of time writing and testing expressions. Today, I use Claude to…you know the rest.

So you see, I use AI to move faster at things I’m already very good at.

Had I been a software engineer, I would absolutely be spending my days inside Cursor or Claude Code or Codex or whatever’s next (Jules?).

But I’m not.

So you think I’m “non-technical” huh?

Even back in the day, I found the distinction between “technical” and “non-technical” rather odd.

For the longest time, my bio said, “A non-engineer who loves APIs” to make it evident that even though I don’t code, I know how to work with APIs. It’s absolutely delightful to be able to do the things I’ve been doing for the last 10 years – without the ability to write code.

The idea of an API wasn’t new for me when I started dabbling with automation tools back in 2017. I had spent significant time working with software engineers and by then, had a good understanding of the software development lifecycle.

I understood how databases work, had seen real-world applications of APIs, and was comfortable reading and writing JSON.

As I got deeper into the automation space, it quickly became evident that, ultimately, automating workflows is all about moving data between tools. And having a good understanding of data structures and data types is key to building robust workflows and troubleshooting them quickly.

So I got deeper into data.

So deep that I spent 5 years – from 2019 to 2024 – deep in the data space. Sure I learned SQL, but I also learned how data is collected, transformed, moved, stored, analyzed, and acted upon.

I became obsessed with the tools that enabled all of these data workflows, documented everything I learned, sold my first blog to Amplitude, built a community, wrote my first book, and collaborated with almost a dozen data startups.

Even though I can read and write basic HTML, SQL, and JSON, I still can’t write code from scratch or troubleshoot code generated by AI.

Does that make me “non-technical”?

I’m a Semitechnical 💪

The reason I never managed to learn to code is simple – I love what I do, and I don’t love the idea of spending time inside an IDE. Heck, I don’t even like the term “Integrated Development Environment”.

I love using software though – like a lot.

But the idea of building software? Nope, not for me.

I also love being really good at whatever it is I decide to do – like the best – and it was evident early on that there’s no way in hell that I would become a 10x engineer because I was always more interested in customer-facing work.

I’ve always enjoyed data work and have been obsessed with workflow automation – these things come to me naturally (not sure if you ever noticed that my handle across channels is iCanAutomate 🤓).

Lastly, not to gloat, but being a semitechnical has paid off really well, and has given me the opportunity to work with the hottest software companies in the world – all while living in India.

All this to say that there’s never been a better time to be a semitechnical – or a <semi>.

What’s next?

Semitechnical is not just an idea. It’s an identity.

Just like people say, “I’m an engineer”, I want people to say, “I’m a semitechnical”.

That’s the future I’m working towards.

That’s my new mission.

Hit me up on LinkedIn if you want in!

What about databeats?

databeats continues to live on as a free resource with over a hundred pieces of neatly organized material for anyone looking to learn data fundamentals.

That said, one of my goals is to further simplify the core concepts so that anyone building with software can apply them in their everyday workflows (rather than just the ones working with large volumes of data at fast-growing companies).

Finally, what to expect from this newsletter?

Thought-provoking ideas and actionable tips to help you thrive as a Semitechnical in an automation-forward world.

And of course, updates on all the cool shit we’re working on.

Wait, are you in India?

Join me at an I can automate with Make event – check out the event calendar and reserve your spot!

🛠 1. Software Development Lifecycle (SDLC) Awareness

Understands how software is built, deployed, and maintained — enough to collaborate effectively with engineers.

• Basics of frontend vs backend (e.g. client-side rendering, server-side logic): How user interfaces (websites, apps) connect to server-side systems and databases

• Understanding the basics of REST APIs (endpoints, auth, rate limits): How different software systems communicate with each other through Application Programming Interfaces or APIs

• Basics of application database design (tables, keys, indexes): How data is structured and stored in a production database (SQL or NoSQL) to enable product functionality

• Awareness of DevOps concepts (environments, CI/CD, staging vs prod): Familiarity with deployment, version control systems like Git, and maintenance processes that keep software running smoothly

⚙️ 2. Workflow Automation

Can build and troubleshoot no-code/low-code systems that move data and trigger actions across tools.

• Concepts of event-based triggers and webhooks: How systems can automatically respond to specific events or changes

• Understanding data structures and types (objects, arrays, strings, booleans, etc.): How inputs and outputs in a workflow are organized (JSON, arrays, objects) and the data format or type of each data point

• Basic JavaScript logic: For formula-based filtering and data manipulation, and creating dynamic outputs (use AI to write/test formulas for the tool you're working in)

• Understanding JSON or JavaScript Object Notation: How to read outputs structured as JSON and draft inputs as JSON when calling APIs

• Working with APIs using visual tools like Make, Zapier, Airtable, Clay, n8n: HTTP Methods or Verbs (POST, GET, etc), Query Parameters, Headers, Tokens, Authorization methods, rate limits, and of course, reading API docs

• Understand RegEx or Regular Expressions: Ability to read and use Regex patterns for data extraction and validation (use AI to write/test RegEx as needed)

• Error handling and dead-letter queue (DLQ): Irrespective of the tools in use, understand the root cause of an error via the DLQ, and account for future errors by incorporating error handling in the workflow.

📊 3. Data Fundamentals

Understands how data is captured, structured, stored, moved, queried, and used across systems.

• Basics of first-party data collection: How to gather customer data generated when users interact with a website or an app

• Understanding product instrumentation: How to specify exactly what data within an application must be tracked for analysis and activation purposes

• Familiarity with data types and how they're stored (e.g. timestamps, booleans, nested objects)

• Knowledge of events vs entities and how they’re modeled: Understanding different categories of information and how they relate to business objects and user actions

• Understanding application data models (used in production systems) vs analytical models (used in BI tools): Distinguishing between data structured for running applications versus data organized for analysis and reporting

• Understanding SQL: Reading and writing simple database queries (often AI-assisted) to retrieve, filter, and analyze data

• Understanding how to move data between databases and third-party tools (e.g. from product → warehouse → CRM/tool) for analysis and activation

• Familiarity with tools that support this movement: CDPs and ETL tools like Segment, Hightouch, Census, Fivetran, etc.

✅ Key Characteristics: The (AI-Native) Semitechnical

A modern <semi> doesn’t need to master any one language or system — they need to:

• Learn and practice “Systems Thinking”

• Understand how things fit together

• Use visual tools and AI wherever possible

• Configure systems to talk to each other

• Know enough to collaborate deeply with product, data, and engineering

• Technical skills applied specifically to solve business problems and improve operational efficiency

  ---

I've been looking for one, straightforward, concrete answer to that question for a while as I wasn't convinced that the primary reason to collect data was to make decisions or deliver personalized experiences.

So earlier this year, I started mapping out the data lifecycle – from collection to insights to action and beyond – hoping to find the answer.

It took some time but I did find it.

And then the task at hand was to deliver the message in the most effective way.

If you know me, you know that I really believe in the power of words; but as the saying goes, "A picture is worth a thousand words".

However, it wasn't easy to draw a picture – a visual model in this case – depicting the thinking and the reasoning behind the straightforward answer to the elusive question, "Why does an organization collect data?"

Every time I thought "This is it", someone managed to poke a hole or two in the model. Often that someone was me but not always, and I'm grateful to all the beautiful minds who helped shape my thinking and pushed me to keep iterating.

And after 27 iterations, The Good Data To Growth Pyramid is ready for prime time!

Enough said.

🥁 🥁

Introducing The GDG Pyramid

The GDG Pyramid is a visual model depicting how an organization can:

• Build a foundation for Good Data

• Derive good insights from the data

• Turn the insights into Growth

And the one, straightforward, concrete answer to "Why does an organization collect data?" is right there in the model:

An organization collects data to enable teams to derive insights that further enable them to make decisions and run experiments to ultimately, drive growth for the business.

It's worth noting that "growth" is a fuzzy word and can mean so many things. Therefore, for the sake of simplicity, let's give it a definition:

Growth refers to any measurable outcome that’s good for an organization.

Examples of growth include more revenue, more efficiency, more profit or impact, higher CSAT, and so on.

I know what I'm positing is not necessarily a popular opinion.

I even ran a week-long poll on LinkedIn and as you can see below, growth won by an inch:

As Vin Diesel once said, "It don't matter if you win by an inch or a mile, winning's winning.

Just kidding.

This isn't a popularity contest and the only reason I ran the poll was to initiate a discussion and share the results here, which I would have irrespective of the outcome.

My goal here, as with all my efforts, is not to prescribe or preach, but to offer a perspective – one that gives you something to think about and something to act upon.

The only difference this time around, as you might already know, is that I'm turning the GDG thesis into a book called "Good Data To Growth: Make Data Fuel Your Organization’s Growth Engine".

And I've decided to release it as a WIP digital book, the first paid product by databeats that will be available for purchase in two weeks.

In the meantime, I'd encourage you to read the book introduction which offers a complete breakdown of the GDG model.

Read The GDG Intro

It takes about 15 minutes and I recommend reading it on your computer as the diagrams are a bit heavy on detail.

I’m not a data engineer, analyst, or scientist, and I strongly believe that you don’t need a background in data to understand the data lifecycle or to improve the outcomes of your everyday efforts using data.

Understanding what data is collected how is a prerequisite for anyone who works with data in their day-to-day.

However, with a plethora of tooling options and the complexities of adhering to privacy regulations, companies need to be very intentional about what data they collect, and how they make that data available in the tools used by various teams.

Keeping that in mind, I wanted to share this guide (originally published on the Airbyte blog) that offers an in-depth overview of the technologies available to collect behavioral data from primary and secondary data sources.

Ready to take notes? Cool, let’s dive in!

So your company is launching a new product and you’ve been tasked with setting up the behavioral data infrastructure? Or maybe you need to revamp the existing setup using modern tools? 

There are a few different technologies (CDI, CDP, ELT/ETL) that can be used to collect behavioral data, and at the same time, there are several tools with capabilities that span multiple technologies.

Navigating this maze and making an informed decision is daunting and time-consuming — this guide aims to change that.

Before going into tools and technologies though, I’d like to shed some light on why collecting behavioral data is important and where exactly behavioral data comes from.

Why collect behavioral data?

Behavioral data is collected when users perform actions or events while interacting with a product.

Behavioral data, also referred to as event data or product-usage data, serves two main purposes for teams — understanding how the product is being used or not used (user behavior) and building personalized customer experiences across various touchpoints to influence user behavior.

Understanding product usage requires prior instrumentation of the features whose usage you’d like to measure — tracking the events a user performs and sending those events to third-party tools for analysis. Additionally, events also help trigger campaigns and experiences via downstream activation tools.

Launching new features without instrumenting them beforehand is a classic mistake — it takes away the opportunity to analyze how those features are used (if at all) and to trigger in-app experiences or messages when relevant events take place (or don’t).

Where does behavioral data come from? 

Although the events I’m referring to take place within your product, the actual source of behavioral data can be an external tool or service that’s embedded within your product.

For the love of simplicity, I like to categorize behavioral data sources as primary and secondary.

Primary data sources

Your core product — web app, mobile apps, a smart device, or a combination — powered by proprietary code is a primary or first-party behavioral data source.

If your product is built using no-code tools, you won’t have a primary source for your behavioral data — you’d rely on the no-code tools to make behavioral data available to you (either via webhooks or integrations with data collection tools).  

To collect data from your primary sources, you can use the client and server-side SDKs or the APIs provided by data collection tools. 

Secondary data sources

Third-party tools that your customers interact with directly or indirectly — tools used for authentication, payments, in-app experiences, support, feedback, engagement, and advertising are secondary data sources.

Customers interact with third-party tools indirectly or unknowingly when they are embedded within your core product experiences.

Examples include Auth0 for authentication, Stripe for payments, and Userflow for in-app experiences — from a user’s point of view, they are using your product even when interacting with these external tools.

Customers also interact with external tools that are evidently not part of the core product experience but are integral touchpoints.

Creating a support ticket via Zendesk, leaving feedback via Typeform, opening an email sent via Intercom, or engaging with an ad on Facebook — these are all interactions that help understand the customer journey. 

It’s also helpful to keep in mind that third-party tools generate a lot of data but not all of it is event data. What exactly you can collect in terms of events and objects depends on the integrations offered by the data collection tool you use.

To collect data from secondary sources, you can either use source integrations offered by data collection tools or write your own code.

Technologies and tools to collect behavioral data

Just like all the layers of the modern data landscape, the data collection layer has experienced a lot of activity in the last couple of years, with the launch of several open-source products that have become popular very quickly.

The overlap between products is also increasing as core capabilities are being extended to cover adjacent use cases.

Customer Data Infrastructure or CDI

CDI is a less common term that’s often confused with CDP (Customer Data Platform).

A platform cannot exist without infrastructure, and CDP is essentially a layer on top of CDI — an additional component that offers a visual interface to do some stuff with data (collected via the CDI). 

CDI is a standalone solution that can exist without a CDP, whereas a CDP is sold as an add-on by some CDI vendors. Learn more about the differences between the two.

Key aspects of a CDI are as follows:

1. CDI is purpose-built to collect behavioral data from primary or first-party data sources but some solutions also support a handful of secondary data sources (third-party tools).

2. Data is typically synced to a cloud data warehouse like Snowflake, BigQuery, or Redshift, but most CDI solutions have the ability to sync data to third-party tools as well.

3. All CDI vendors offer a variety of data collection SDKs and APIs

4. Some CDI solutions store a copy of the data, some make it optional, and some don’t.

5. CDI solutions that store a copy of the data also offer out-of-the-box identity resolution. 

The core capabilities of a CDP, on the other hand, include identity resolution and the ability for users to build and sync audiences to external tools using a drag-and-drop UI (without writing SQL). 

CDI and CDP solutions

Segment offers multiple products — Connections is their CDI offering, Profiles is an identity resolution add-on, and Twilio Engage includes CDP capabilities. Segment also offers Protocols, a data governance tool.

mParticle takes a slightly different approach — it offers CDI capabilities along with identity resolution in its Standard edition whereas audience building is available on the Premium plan. mPartice also offers additional products for product analytics (Indicative) and predictive analytics (Cortex).

Both Segment and mParticle support data warehouses and a host of third-party tools as destinations, as well as store a copy of your data that can be accessed later if needed. 

RudderStack (Event Stream) and Jitsu are open-source CDI solutions positioned as alternatives to Segment Connections. Both products support warehouses and third-party tools but RudderStack offers a more extensive catalog of destinations.

Snowplow is the only CDI solution that literally calls itself a behavioral data platform. It is also open-source and unlike the others, Snowplow doesn’t support third-party tools as — it is focused on warehouses and a few open-source projects as destinations.

Other CDI solutions worth looking into are Freshpaint which offers codeless tracking and MetaRouter which is a server-side CDI that only runs in a private cloud instance. 

The links below will take you to the integration catalogs of the respective tools: 

• Segment: Warehouses and third-party destinations

• mParticle: All integrations (note that Feed refers to data sources)

• RudderStack: Warehouses and third-party destinations

• Jitsu: All destinations

• Snowplow: All integrations

• Freshpaint: All integrations

• MetaRouter: All destinations

ELT solutions

ELT solutions are purpose-built to extract all types of data from a large number of third-party tools (secondary sources) and load the data into cloud data warehouses. That said, not all integrations offered by ELT tools support behavioral data or event data. 

ELT tools don’t store any data and don’t support third-party tools as destinations. 

Airbyte is an open-source ELT tool that offers source connectors with 150+ tools like Zendesk, Intercom, Stripe, Typeform, and Facebook Ads, many of which generate event data. Airbyte also offers a Connector Development Kit (CDK) that you can use to build integrations that are maintained by Airbyte’s community members.

Other ELT vendors include Fivetran, Stitch, and Meltano (also open-source). 

As mentioned earlier, CDI solutions also offer source integrations with a few third-party tools but those are not as comprehensive and deep as the integrations offered by ELT tools. 

When contemplating whether to use an ELT tool or a source integration of a CDI tool to extract data from a third-party tool, consider the following:

• CDI is best-in-class to collect behavioral data from primary or first-party data sources — web and mobile apps, and IoT devices

• ELT is best-in-class to collect all types of data including behavioral data from secondary data sources — third-party tools that power various customer experiences.

Product analytics tools 

Amplitude, Mixpanel, Indicative (by mParticle), Heap, and PostHog (open-source) are product analytics tools purpose-built for behavioral data analysis. At the same time though, all of these offer SDKs and APIs to collect data from your primary data sources. 

Product analytics tools by nature store a copy of your data and allow you to export the data via APIs. Additionally, if you’d like to export data from these tools to your data warehouse, you can either use native integrations that some of these tools offer, or leverage Airbyte’s integrations with Amplitude, Mixpanel, or PostHog. 

However, it’s important to keep in mind that beyond analysis, there are plenty of activation use cases for behavioral data. 

Custom tracking solutions

If readymade solutions are not for you, you can always build a custom tracking service that collects data from your apps and syncs it to your warehouse and downstream applications. That said, having first-hand experience with such a solution, I can tell you that maintenance and troubleshooting are not trivial and the frustration is real.

More importantly, with so many different flavors of CDI and ELT solutions available, building one’s own is just not the best use of engineering resources. In fact, engineers generally hate building integrations — if you’re one, let me know if I’m wrong. 

Conclusion

Using purpose-built data collection tools (CDI and ELT) is more efficient, prevents vendor lock-in, and just makes more sense.

I recommend adopting a CDI to collect data from primary or first-party data sources, and sticking to your ELT tool to collect data from secondary or third-party sources. 

Now that you have a better picture of the tools needed to collect behavioral data for analysis and activation, don’t forget to collaborate with stakeholders from various teams when it comes to deciding which events to track and what data to send to which destination. 

Continue Learning

data beats

How To Decide Which Events to Track?

This guide is part five of the series on understanding customer data for product analytics; it was originally published on the Amplitude blog. To decide which events to track, you need to first list down all the questions you’d like to answer about your users and their product usage…

Read more

4 years ago · 1 like · Arpit Choudhury

Here’s the karaoke-ready instrumental soundtrack:

And below are the lyrics with timestamps. I enjoy them more with the soundtrack, hope you do the same!

[From 0:13]

It starts with..

Ingestion, from a data source,
To a destination, where the data goes.
Your website or app is your first-party source,
Every API or SaaS is a third-party source.

All I know..

Data is a valuable thing,
Watch it fly by and face vendor lock-in.
Store a copy so when there is a need,
You’ve got access, you don't need to plead.

[9-second pause]

🎵🎵🎵🎵

[From 0:40]

The Modern Data Stack is no secret,
You need data, so you need tools to collect it, store it, analyze it, and activate it.
It's kinda sad that it seems so complicated,
Alright alright, lemme try and explain it..

The separation of Compute and Storage in the cloud,
Made warehousing cheaper, more accessible, much faster.
Queries take seconds to process in parallel,
Pay as you go, storage is infinitely scalable.

ETL became ELT
Because Extracting and Loading without Transforming is easy,
Once in the warehouse, whenever the need be,
Data can be transformed by writing a SQL query.

Once transformed, your data models are ready,
To derive insights, they need to be visualized,
To drive action, they need to be synchronized,
And that is all, the MDS isn't so complicated after all.

[10-second pause]

🎵🎵🎵🎵

[From 1:28 — slow down]

S-Q-L can get you far,
But if you don't know SQL, it doesn't even matter.
Modern data tools can do it all,
It's early days, they're already getting better.

Ingestion, from a data source,
To a destination, where the data goes.
CDI for your first-party source,
ELT for every third-party source.

It's not so hard..

Warehouse the data, you now know why,
You gotta know where your data lies.
When it's time to derive insights,
Go build them models — analyze and visualize!

But insights alone are not enough.
You gotta take action, it's not so tough.
Reverse ETL or CDP?
It doesn't matter, choose whatever's easy.

[Don’t pause, it gets faster]

I know why you lookin' at me.
I know why you lookin' at me.
Now I know why you lookin' at me.

You wanna know about — Data Quality,
And you want to know about — Observability,
Metrics — Repository,
Cataloging — and Discovery.

[2-second pause]
[From 2:32 — slow down]

Well I would say, don't worry,
Just start small, there's no hurry,
The perfect stack
Is no good if used by nobody-eeee.

[From 2:53]

Data tools, you can buy them all,
But in the end, people are what matter.
Data people and non-data people,
Let's beat the gap, let's do it together-rrrrr!

🎵🎵🎵🎵

Want to listen to the MDT?👇

https://suno.com/@arpit

Setting up a data practice is a lot less straightforward and unless you have a strong understanding of all the moving parts of a data stack, you can spend a ridiculous amount of time and effort trying to figure out the skills you’re looking for in your first data hire. 

When people talk about data talent, they typically refer to technical roles in data — analyst, engineer, or scientist. However, there are a few data-adjacent roles that are less technical in nature but are quintessential to the successful implementation and adoption of a data stack. 

Technical Data Talent

Hiring for a technical role in data is challenging for two reasons, the obvious being that the demand for data talent has never been higher and companies are competing fiercely to hire experienced professionals. 

The less obvious reason is that a majority of companies, especially the ones in the early stages of figuring out how to go about building a data practice, have little idea about the people they need to hire.

Companies often use the titles Analyst, Engineer, and Scientist interchangeably to attract candidates to apply for data roles with unclear, often unreasonable expectations. I’ve even heard people say that using the Data Scientist title is a proven method to attract a ton of applicants — whether they’d be relevant for the role or not is a different story. 

Even if you know what you’re looking for in an early data hire, it’s not easy to describe the role in a manner that will attract a decent number of relevant candidates. 

Most people gravitate towards hiring a data generalist — often referring to them as a data scientist — and hoping that this one person alone can solve all data woes and make the organization “data-driven” (a term I have come to dislike for a variety of reasons; data-led makes more sense to me). 

In any case, hiring technical data people is not enough, and organizations of all sizes need semi-technical data talent — more than they ever have before. 

Semitechnical Data Talent

I dislike the term non-technical because everybody, especially if they work in tech, has some level of technical aptitude. Referring to a product/growth/marketing person as non-technical because they don’t write code is, in my opinion, naive.  

And now that a growing number of non-data folks are learning how to write SQL, does it even make sense to call them non-technical?

Why not start referring to these people as semi-technical instead?

Semi-technical data people play a crucial role in the process of making data available to go-to-market (GTM) teams in the tools they rely on.

Semi-technical data people often carry ops in their title and act as a bridge between the data team and the rest of the organization.

Product Ops, Marketing Ops, Sales Ops, Revenue Ops, or Data Ops — these are bridge teams that essentially enable the rest of the organization to utilize or consume data across a bunch of different tools. 

Bridge teams don’t orchestrate pipelines or write SQL — instead, they set up processes that enable data and GTM teams to collaborate effectively and understand each others’ priorities, constraints, workflows, and goals. 

Needless to say, folks who make all this happen are definitely not non-technical and are not easy to find. The requisite skills are acquired by working at the intersection of data, engineering, and GTM, and the number of people who’ve been fortunate to have that experience is relatively small.

That said, a shortage of ops talent also presents a unique opportunity for data-inclined GTM folks to become ops pros by making sure that tools are implemented properly, integrations follow best practices and are well-documented, and the right data is collected and made available in the tools used by GTM teams.

Therefore, a solution to the problem of finding semi-technical data talent is to foster data literacy among GTM teams.

Data Literacy Is The Key

I have played the role of that GTM person at a fast-growing company without a dedicated data or ops team where I helped set up the customer data infrastructure — everything I do to bridge the gap between data people and non-data people can be attributed to this experience.

Subscribe now

A typical conversation about data often brings up the privacy practices of big tech — the fact that they gather too much data and the growing concerns over the opaqueness of their data policies have given birth to stringent privacy laws such as the EU’s GDPR and California’s CCPA. 

Privacy laws and the fact that browsers are making third-party cookies obsolete are making companies more accountable, forcing them to take a hard look at their data collection practices. As a result, a snowball effect is taking place — companies are embracing transparency and creativity while trying to stay compliant, and the awareness about data is increasing amongst individuals. 

Customer data is the centerpiece that enables personalization and automation at scale — it provides context on the user as well as the user’s behavior in terms of using a product.

Customer data is best described when broken into the following two types: 

• User data: It provides context on a user and their traits, and is also referred to as entity data (user being the entity)

• Interaction data: It provides context on how the user interacts with a product and is also referred to as event data, behavioral data, or product-usage data

Customer data is also gathered when users interact with your brand outside of your core product experience via secondary data sources or third-party tools used for advertising, engagement, and support, to name a few. However, this guide focuses on customer data that comes from a primary or first-party data source — a website, app, smart device, or a combination of these — and comprises entity data and event data. 

P.S. The “customer” in “customer data” includes free users of a paid product as well as users who pay with personal data to use a product. 

Entity data

Entity data includes personally identifiable information (PII) such as name, email, and phone number, as well as other details such as age, country, and preferences. 

It is often referred to as user data since a user is the main entity or object. It comprises user properties or user attributes, each of which stores information or traits about a user. 

Entity data is stored in tables where columns represent user properties like name and email, while each row represents a user. One of the properties acts as an identifier and has to contain a unique value for each row (user).

In the table above, email can act as an identifier by ensuring no two users have the same email. However, it is a better practice to assign a unique ID to each user since an email address can change but the user_id remains fixed. 

Accounts or groups as entities

A group of users or an account is also an entity with distinct attributes generally referred to as organizations or workspaces in the case of B2B SaaS products.

From a hierarchical point of view, accounts are groups that comprise users. Thus, the data about an account or group comprises group properties that store information about an account such as the subscription type or the number of users. If accounts are known as organizations, the associated properties should be referred to as organization properties.

It is common to gather data about both users and groups at the same time. This is, once again, particularly true for B2B SaaS tools where a user is part of an account or organization with multiple users.

How do you collect entity data?

Entity data where user is the entity is gathered as a result of users sharing data directly or indirectly. 

Users share data directly when they input details in a form, respond to an email or a survey, or when they interact with conversational interfaces like chatbots and voice bots.

On the other hand, users share data indirectly when they use a product. When listening to music on Spotify, a user shares data about their music preferences including genres, artists, and even specific songs they like. Similarly, when a user creates reports on Amplitude, they share data about the type of reports they find useful. 

Since Amplitude is a B2B SaaS tool where multiple users are part of an organization, the number of reports created under an organization is data associated with an organization and not a particular user. Hence, in this case, Organization is another entity, number_of_reports is an organization property, and the value of this property changes when any user in an organization creates or deletes a report.

It’s important to not confuse entity data that changes as a result of product usage (number of reports) with event data that is generated when a user interacts with a product (report created)

Event data

An event refers to a unique action performed by a user while interacting with a product, and the data generated in the process is called event data or interaction data. 

Clicks and hovers on the web, taps and swipes on mobile, and text or voice commands on chat and voice interfaces — all such interactions are actions performed by a user or events that take place inside an app. 

Event data enables you to understand user behavior and is therefore often referred to as behavior data. Additionally, event data enables you to take action on data or activate the data in external tools where the data is made available. 

A common use case is event-based contextual messaging (in-app or email) where a campaign is triggered when a certain event X takes place. Or when a certain event Y doesn’t take place within a specified timeframe after X takes place — the possibilities are endless. 

Event data comprises three key elements: 

• The action or the event that took place

• The timestamp or the precise date and time when the event took place

• The state or all other properties associated with the event (known as event properties)

Add to Cart, Buy Now, and Complete Payment are all actions or events. The exact moment when an event takes place is recorded as a timestamp.

The properties that provide more context about the event Add to Cart could be user_id, product_id, price, and quantity—all of which provide information related to the event or the state of the event. 

How do you collect event data?

Collecting event data requires you to create a tracking plan specifying the events to track and the associated properties for each event. Then you get your data engineering team to implement the tracking plan using either of the following: 

• CDI (customer data infrastructure) or CDP (customer data platform)

• Custom tracking service

Once event tracking is implemented, event data collected is made available in the configured destinations (product analytics and engagement tools) and typically, a copy of this data is stored in a data warehouse. 

Learn more about event data or behavioral data collection tools in this guide.

What data to track vs how to track it

While it is good to know about the event-tracking process, as a data-led professional, you should focus on what to track rather than how to do it.

Why so? 

I won’t disagree if you argue that defining what data is to be tracked and the tracking process itself are equally important. However, these two activities should ideally be owned by different people and depending on the size of your organization, maybe even different teams that collaborate closely. 

To engineer or not to engineer

Typically, a data engineer takes care of implementing the tracking and collaborates with product and growth teams to decide which tools and technologies to use. The company stage, the scope of work and rework, available resources, priorities, and several other factors influence this decision.

Many companies, however, leave the entire tracking process to the data/engineering team, keeping marketing and product folks completely out of the loop—doing so invariably results in data that is inaccurate, inconsistent, and often redundant when too many events are tracked just for the sake of tracking.

Deciding what to track is simply not an engineer’s job and expecting engineers to know how other teams wish to use what data is, well, disastrous.

Putting customer data into action

Now that you know what customer data is and what your role is in the process of gathering it, the next step is to be able to answer the following questions:

• What purpose does event data serve?

• How Event Data Enables Data-powered Growth?

• What do event and entity data look like in the context of customer data?

• How to decide which events to track and what data to gather?

Lucky for you, each question above links to a guide in the 5-part series on customer data. 

Once you have answers to the above, you will be equipped to gain a clear understanding of how to create a data tracking plan and will be able to do the following with confidence:

• Lead the implementation of event-driven analytics and engagement tools with confidence

• Gather clean and consistent customer data and overcome challenges that crop up along the way

• Ask the right questions of your data in order to better understand user behavior

• Identify opportunities to collect and act upon data to elevate the customer experience

• Build better products, provide better experiences, and have better conversations

Lastly, it is incredibly useful to have a good understanding of various data types before you begin working on your tracking plan, so whenever that is, this guide on data types will be helpful.

Now move on to part 2 to understand the purpose of collecting event data.

The good thing today is that there are so many different tools and technologies available to solve a certain problem. The bad thing is that there are so many different tools and technologies available to solve a certain problem. 

Go-to-market or GTM teams — product, growth, sales, marketing, and customer success — are the biggest beneficiaries of a sound data stack.

The buying decisions, however, are often taken by data engineering (DE) teams, leading to misalignment between the two camps.

Both camps want more control over their workflows — GTM wants a GUI to do more without relying on DE, but DE despises GUIs and wants to ensure that they don’t have to do the same thing twice. The divide is stark at B2B orgs where GTM relies heavily on product-usage data to personalize experiences at every customer touchpoint. 

What GTM (non-data) wants

At the very least, GTM wants to be able to quickly visualize how the product is being used —  where users are dropping off, what’s leading to activation, etc. — and personalize their outreach based on those factors.

To do this effectively and without relying on DE, besides a product analytics tool, GTM needs a visual segmentation tool to create and sync audiences to their activation tools, which they can do using a CDP or to an extent, a reverse ETL (rETL) tool with visual querying capabilities. 

There’s another emerging category called PLG CRM that is trying to address this problem by enabling GTM to combine data from multiple sources, build segments, and trigger actions in activation tools based on those segments. 

There are many companies in this category so I might be wrong but unlike CDP or rETL, PLG CRM doesn’t actually move the data downstream and is better suited to alerting use cases. Also, there’s evident overlap between the use cases addressed by PLG CRM and rETL.

Irrespective of the technologies powering the tools, being able to analyze and activate data is what GTM teams want, which is a reasonable ask. 

What DE (data) wants

It’s important to acknowledge that DE caters to the needs of all teams and not just GTM teams — the scope of work for DE is larger which definitely influences the data tools they choose.

That said, DE is generally averse to tools that don’t neatly fit into their existing workflows and require them to build and maintain additional data pipelines — CDPs and product analytics tools are the most common ones that fall into this category.

The ideal scenario for DE is to maintain a single pipeline that starts with collecting data from all possible sources in a data warehouse, writing SQL to transform or model all types of data for all purposes, and making this data available in all the tools used for analysis and activation. 

The data warehouse is the centerpiece of everything data teams do — setting one up is not negotiable and for good reason. Without a data warehouse in place, there’s really not much for a data engineer or analyst to do, especially if they wish to stick to best practices and use best-of-breed tools. 

While it doesn’t take long to spin up a cloud data warehouse, ingesting data from various sources, transforming/modeling the data, and syncing the data to analysis and activation tools is not trivial — the entire process can take weeks or longer depending on the number of data sources and available resources.

Consequences for GTM

It’s reasonable to say that sooner or later, every company has to invest in the above process if they wish to maximize the value derived from data. 

However, prioritizing the ideal process over the immediate needs of GTM teams can leave them in a state of flux.

They have little control over their daily workflows and every new analysis or segmentation becomes a request for the data team to fulfill. 

They need to become SQL-proficient in order to experiment and iterate.

And if there’s a broken pipeline affecting data quality, which further distorts the customer experience, they might never know or discover it only after the damage is done.

The Middle Ground

There are many factors at play when evaluating tools — implementation, scalability, extensibility, and interoperability. However, the most important is value. A powerful tool with insane performance and capability is no good if nobody uses it or derives value from it.

Therefore, companies need to embrace a middle ground by scoping the needs of their teams and investing in tools that can fulfill those needs.

For smaller companies without a dedicated data engineer, it just makes sense to empower GTM with the tools they can use themselves. 

Or even for bigger non-tech companies with large GTM teams. 

Or even enterprises that have the resources to keep both camps happy and to invest in processes to ensure that data quality isn’t compromised.

If your organization’s priority is to build a data team, by all means, invest in best-in-class tools for every layer of the data stack. On the other hand, if the goal is to empower GTM teams to analyze and act upon data using the tools they’re comfortable with, let them have those tools and work backward to figure out the challenges that emerge.

After all, the modern data stack is about enabling people to use data to do their best work, isn’t it?

Continue Reading