Instagram story viewer> @the.big.data.guy> Posts

Shrujan K | Data Engineer 💻

@the.big.data.guy

185

followers

16

following

Data Engineering, explained simply
Big Data • AI • Systems
Calm content. No fluff.
IIIT-B Alum

POSTS STORIES REELS TAGGED

Download All

💳 Every time you pay using PhonePe, Google Pay or any UPI app, the payment feels almost instant.

But behind that single tap, multiple distributed systems are working together to make sure your money is never duplicated, never lost and always stays consistent. ⚙️

That’s where the famous ACID properties come in:

⚛️ Atomicity
⚖️ Consistency
🚦 Isolation
💾 Durability

These four principles silently protect millions of UPI transactions every single day.

The best engineering is often the engineering we never notice. 🚀

👇 Which hidden system should I explain next?

Netflix 🍿
Google Maps 🗺️
Amazon 📦
Swiggy 🍔

Calm explanations. No fluff.

[data engineering] [big data] [apache spark] [distributed systems] [database] [acid properties] [atomicity] [consistency] [isolation] [durability] [upi] [phonepe] [google pay] [payment systems] [software engineering] [databricks] [system design] [backend engineering]

#DataEngineering #BigData #DistributedSystems #SoftwareEngineering #UPI by @the.big.data.guy

17

0

2 months ago

You pause a YouTube video for just a second.

Behind that simple click, a complete Big Data pipeline starts working—an event is created, streamed, partitioned, replicated, and your exact watch position is saved so you can continue seamlessly across any device.

That’s the engineering most people never see.

Follow @the.big.data.guy for more “Behind the System” explanations.

Calm explanations. No fluff.

[data engineering] [big data] [apache spark] [pyspark] [sql] [github] [open source] [data engineer] [ai] [gen ai] [machine learning] [analytics engineering] [databricks] [azure] [software engineering] by @the.big.data.guy

30

0

2 months ago

Most developers know GitHub.

Very few know which repositories are actually worth following.

I spent time filtering through hundreds of repositories to shortlist the ones that are consistently recommended by data engineers and the open-source community.

Whether you’re just starting your Data Engineering journey or already working in the industry, these repositories are packed with real-world projects, production-ready examples, and high-quality learning resources.

📌 Save this post so you never have to search for these repositories again.

💬 Which GitHub repository has helped you the most? Share it in the comments—I might feature it in Part 2.

Calm explanations. No fluff.

[data engineering] [big data] [artificial intelligence] [AI] [Apache Spark] [PySpark] [SQL] [GitHub] [open source] [Databricks] [Azure] [ETL] [data pipelines] [analytics engineering] [software engineering]

#DataEngineering #BigData #claude #aivideo #GitHub by @the.big.data.guy

9

0

2 months ago

Most developers open a GitHub repository… and close it within 30 seconds.

Not because it’s difficult.
Because nobody teaches where to look first.

In this reel, I’ll show you the exact order I use to understand any GitHub repository without feeling overwhelmed.

Once you know this framework, you’ll spend less time randomly clicking folders and more time actually understanding the project.

Have you ever opened a repo and instantly felt lost? 👇

[data engineering] [GitHub] [GitHub repository] [Git] [AI] [Claude] [software engineering] [open source] [developer tools]

#DataEngineering #GitHub #OpenSource #Programming #TechEducation

Calm explanations. No fluff. by @the.big.data.guy

46

25

3 months ago

Learning Git doesn’t have to mean memorizing commands.

One of the best free resources I’ve come across is learninggitbranching.js.org.

Instead of passively watching tutorials, you actually type Git commands and instantly see how your branches change. It makes concepts like branching and merging much easier to understand.

If you’re planning to become a Data Engineer, Software Engineer, or just want to get comfortable with Git, this is a great place to start.

💾 Save this reel so you can come back to it when you begin learning Git.

Calm explanations. No fluff.

[Data Engineering, Big Data, Git, GitHub, Version Control, Software Engineering, Programming, AI]

#DataEngineering #GitHub #Git #Programming #softwareengineering by @the.big.data.guy

32

3

3 months ago

Skipped the “what I ate” part.

Work vlog. Because my Big Data content wasn’t getting any views anyway. 🤷‍♂️

[Work Vlog | Big Data | Data Engineering | Corporate Life | Tech Life | Office Vlog | Day in the Life | Software Engineer | Data Analytics]

#WorkVlog #BigData #DataEngineering #CorporateLife #TechLife OfficeVlog DayInTheLife SoftwareEngineer DataAnalytics TechReels WorkLife DataScience by @the.big.data.guy

24

5

3 months ago

⚙️ 3 tools that make my life easier as a Data Engineer.

These aren’t the core technologies I work with every day like SQL, Spark, or Databricks. Instead, they’re the tools that help me code faster, automate repetitive tasks, and keep my workflows running smoothly.

🤖 GitHub Copilot – An AI coding assistant that helps me write code faster and debug issues more efficiently.

🔄 Airflow / Dagster / Prefect – Orchestration tools that schedule, manage, and monitor data pipelines so everything runs reliably.

📦 Docker – Packages your code and its dependencies, ensuring it works the same on your laptop, a server, or in the cloud.

Sometimes it’s the supporting tools—not just the core tech—that make the biggest difference.

💬 Which tool has saved you the most time at work? Let me know in the comments. I might feature your suggestion in a future reel.

📌 Follow @the.big.data.guy for practical Big Data, Data Engineering, and AI content explained in a simple way.

[Data Engineering, Big Data, AI, AI Agents, GitHub Copilot, Claude, Claude Code, Docker, Apache Airflow, Dagster, Prefect, Data Pipelines, ETL, ELT, PySpark, Apache Spark, Databricks, Azure Data Factory, Microsoft Fabric, Azure, Python, SQL, Data Engineer, Data Engineering Tools, Developer Productivity]
#ai #claude #software #techlife #likesharecomment by @the.big.data.guy

23

1

3 months ago

Big Data, Simplified | Episode 10: Execution

A single SQL query might look simple.

But behind that one query, hundreds of machines are processing different parts of the data, combining partial results, and returning one final answer.

That’s the power of Big Data.

What looks like one answer is actually hundreds of machines solving one problem together.

📌 Series completed

✅ Episode 1: What is Big Data?
✅ Episode 2: Volume
✅ Episode 3: Velocity
✅ Episode 4: Variety
✅ Episode 5: Distributed Systems
✅ Episode 6: Scaling
✅ Episode 7: Replication
✅ Episode 8: Partitioning
✅ Episode 9: Parallel Processing
✅ Episode 10: Execution

Thank you for being part of this journey. More Data Engineering concepts coming soon.

[Big Data] [Data Engineering] [Distributed Systems] [SQL] [Apache Spark]

#BigData #DataEngineering #DataScience #ArtificialIntelligence #MachineLearning by @the.big.data.guy

23

0

3 months ago

Big Data, Simplified | Episode 9: Parallel Processing
Splitting data is only half the solution.
The real speed comes from Parallel Processing.
Instead of one machine processing every partition, multiple machines work on different partitions at the same time.
This drastically reduces processing time and allows Big Data systems to analyze massive datasets efficiently.
Partitioning divides the data. Parallel Processing finishes the job.
📌 Series roadmap
✅ Episode 1: What is Big Data? ✅ Episode 2: Volume ✅ Episode 3: Velocity ✅ Episode 4: Variety ✅ Episode 5: Distributed Systems ✅ Episode 6: Scaling ✅ Episode 7: Replication ✅ Episode 8: Partitioning ✅ Episode 9: Parallel Processing
Follow along if you’re learning Big Data and Data Engineering from scratch. #bigdata #ai #cloud #aitools #datascience by @the.big.data.guy

10

0

3 months ago

Big Data, Simplified | Episode 8: Partitioning

When Big Data is spread across multiple machines, how does each machine know which data to process?

That’s where Partitioning comes in.

Partitioning divides a large dataset into smaller pieces, and each machine is responsible for only one partition.

When a query is executed, every machine processes its own partition instead of scanning the entire dataset.

This reduces unnecessary work and makes Big Data systems much faster and more efficient.

📌 Series roadmap

✅ Episode 1: What is Big Data?
✅ Episode 2: Volume
✅ Episode 3: Velocity
✅ Episode 4: Variety
✅ Episode 5: Distributed Systems
✅ Episode 6: Scaling
✅ Episode 7: Replication
✅ Episode 8: Partitioning

Follow along if you’re learning Big Data and Data Engineering from scratch.
#data #ai #cloud #aitools #datascience by @the.big.data.guy

15

0

3 months ago

Big Data, Simplified | Episode 7: Replication

What happens if one machine storing your data suddenly fails?

That’s where Replication comes in.

Instead of keeping just one copy of the data, Big Data systems store multiple copies across different machines.

If one machine goes offline, another machine with the same data can immediately take over.

This keeps applications available and minimizes downtime, even when hardware fails.

📌 Series roadmap

✅ Episode 1: What is Big Data?
✅ Episode 2: Volume
✅ Episode 3: Velocity
✅ Episode 4: Variety
✅ Episode 5: Distributed Systems
✅ Episode 6: Scaling
✅ Episode 7: Replication

Follow along if you’re learning Big Data and Data Engineering from scratch.
#aitools
#bigdata #ai #cloud #data by @the.big.data.guy

17

0

3 months ago

Big Data, Simplified | Episode 6: Scaling

As data grows, companies have two ways to increase computing power.

Vertical Scaling means upgrading an existing machine with more CPU, RAM, or storage.

Horizontal Scaling means adding more machines and distributing the workload across them.

For a personal computer or a small application, Vertical Scaling is often enough.

But for Big Data systems handling massive workloads, Horizontal Scaling is usually the preferred approach because it’s easier to scale and more reliable.

📌 Series roadmap

✅ Episode 1: What is Big Data?
✅ Episode 2: Volume
✅ Episode 3: Velocity
✅ Episode 4: Variety
✅ Episode 5: Distributed Systems
✅ Episode 6: Scaling

Follow along if you’re learning Big Data and Data Engineering from scratch.

#BigData #DataEngineering #datascience #ai #data by @the.big.data.guy

18

0

3 months ago