NVIDIA GPU Deep Dive : Tensor and Ray Tracing Cores

Nvidia GPU Tensor Core Architecture
Nvidia GPU Tensor Core Architecture

This blog is Part 3 in our Nvidia GPU blog series. In Parts 1 and 2, we explored the fundamentals of NVIDIA GPU architecture: from CUDA cores to streaming multiprocessors (SMs) and warp schedulers. Today, we transition into what sets modern NVIDIA GPUs apart-the Tensor Cores and Ray Tracing (RT) Cores-two hardware innovations that fuel cutting-edge advancements in AI computation and real-time rendering.

NVIDIA Tensor and Ray Tracing Cores have revolutionized , making GPUs the backbone of deep learning workloads. Meanwhile, RT Cores enabled real-time photorealistic graphics, redefining visual realism in games and simulations. In this post, we dissect their architectures, working principles, and how they seamlessly integrate into the GPU pipeline.

🧠 Tensor Cores – Unleashing AI Performance

Architectural Evolution and Integration

Tensor Cores were introduced in the Volta (GV100) architecture, setting a new benchmark for AI acceleration. Each Streaming Multiprocessor (SM) houses multiple Tensor Cores, tightly coupled with CUDA cores and other functional units. This co-location allows Tensor Cores to offload matrix-heavy operations like convolutions and fully connected layers, while CUDA cores handle scalar/vector instructions.

Across architectures:

  • Volta: First appearance; FP16 precision support.
  • Turing: Added INT8 & INT4 precision for inference workloads.
  • Ampere: Introduced TF32 and enhanced sparsity support.
  • Hopper: Revolutionized with FP8 and the Transformer Engine.

Each SM includes hardware schedulers that coordinate the execution of Tensor Core operations alongside traditional CUDA threads, ensuring maximal occupancy and throughput.

Nvidia GPU Tensor Cores

Precision Modes: FP32, FP16, TF32, INT8, FP8, and FP4

Precision is critical in balancing performance and numerical stability.

  • FP32: Full precision; used in scientific computing.
  • TF32: Ampere’s innovation, offering FP32 dynamic range but with FP16 compute efficiency-achieves up to 10× speedup in deep learning training.
  • FP16 (Half Precision): Ideal for neural nets tolerant to low precision; critical in deep learning inference.
  • INT8 & INT4: Efficient for low-latency inference, especially on edge devices.
  • FP8 & FP4: Next-gen precision; FP8 is central to Hopper’s Transformer Engine, while FP4 is emerging for ultra-low-power inference.

Tensor Cores dynamically downscale or upscale precision based on kernel requirements, facilitated by NVIDIA’s cuDNN and TensorRT libraries.

NVIDIA GPU - mixed precision training
NVIDIA GPU – mixed precision training

In-Depth: Matrix Multiply-Accumulate (MMA) Engine

At the heart of Tensor Cores is the Matrix Multiply-Accumulate (MMA) Engine. The MMA pipeline is designed to perform:

D=A×B+C

For matrix tiles (commonly 16×16). The Tensor Core parallelizes these tiles across multiple hardware lanes, enabling:

  • 16×16×16 FP16 MMAs in a single cycle (Volta-Ampere).
  • Dynamic 8×8 tiles with FP8 in Hopper for optimized attention layers.

Tensor Cores rely on warp-level primitives, where threads in a warp collaborate on MMA operations. NVIDIA’s CUDA WMMA API provides direct access to these functionalities.

Transformer Engine & Sparsity Acceleration

Transformer models-which power LLMs like GPT-have unique computational patterns. Hopper’s Transformer Engine:

  • Detects per-layer precision requirements (FP16 vs FP8).
  • Dynamically adjusts execution precision at runtime.
  • Leverages context-aware sparsity, which identifies zero-value weights and skips redundant computation.

This enables 50%+ reductions in compute/memory without sacrificing accuracy. Ampere introduced 4:2 structured sparsity, doubling throughput for sparse models.

🎮 Ray Tracing Cores – Enabling Cinematic Real-Time Ray Tracing

The Shift from Rasterization to Path Tracing

Rasterization dominates traditional graphics pipelines by projecting 3D geometry into 2D fragments. However, it fails to simulate true light behavior like global illumination and caustics. Ray tracing simulates:

  • Primary rays (camera → scene)
  • Reflection/refraction rays (mirror/glass)
  • Shadow rays (light occlusion)

RT Cores were introduced in Turing, transforming ray tracing from a batch process to a real-time application.

Rasterization vs Raytracing
Rasterization vs Raytracing

BVH Traversal and Intersection Hardware

The core of RT Core acceleration lies in BVH (Bounding Volume Hierarchy) traversal.

  • BVH Tree: Scene geometry is hierarchically partitioned into boxes.
  • Traversal: RT Cores quickly determine if rays intersect bounding boxes-rejecting entire regions early.

Once inside a leaf node, Ray-Triangle Intersection units calculate precise hit points using barycentric coordinates and optimized hardware pipelines.

Performance: RTX 3090 achieves 10 billion BVH tests/sec, enabling complex environments at playable frame rates.

Multi-Stage Ray Shading Pipeline

The shading pipeline includes:

  • Ray Generation Shader: Initiates rays.
  • Closest Hit Shader: Determines material/shader logic on hit.
  • Any Hit Shader: Early exits for transparency effects.
  • Miss Shader: Background/environment mapping.

Each ray potentially spawns secondary rays, making recursive traversal necessary. Tensor Cores assist by AI-denoising noisy samples-enhancing visual quality without full sampling.

🤝 Tensor + RT: DLSS and Hybrid Workloads

DLSS (Deep Learning Super Sampling) epitomizes the synergy of Tensor + RT Cores. The workflow:

  1. RT Cores: Render base frame (lower resolution).
  2. Tensor Cores: Upscale + reconstruct via AI models.

DLSS 3.0 (Ada Lovelace) even predicts in-between frames using motion vectors + optical flow estimation-massively boosting FPS in ray-traced environments.

This hybrid method achieves 4K quality at 1080p costs, revolutionizing graphics performance.

🧪 Developer Toolkit: Leveraging Tensor & RT Cores

For AI:

  • CUDA WMMA API
  • cuBLAS/cuDNN
  • TensorRT

For Graphics:

  • Vulkan/DXR
  • NVIDIA OptiX
  • Nsight Visual Profiler

Profiling tools allow warp-occupancy tracking, Tensor Core utilization, and RT core tracing, helping developers fine-tune performance.

📈 Conclusion

NVIDIA’s Tensor and Ray Tracing Cores are cornerstones of modern GPU architecture, unlocking unprecedented capabilities across AI, graphics, and simulation. These specialized cores complement general CUDA compute, enabling applications that were infeasible a decade ago.

In Part 4, we’ll unravel the mysteries of GPU memory subsystems, cache hierarchies, and NVLink/NVSwitch-crucial for multi-GPU scaling.

Deep Dive into GPU Compute Hierarchy

Modern NVIDIA GPUs are feats of hierarchical design, optimized to maximize parallelism, minimize latency, and deliver staggering computational throughput. Building upon Part 1, which introduced the high-level architecture of NVIDIA GPUs, this is Part2 – a Deep dive into GPU Compute Hierarchy: Graphics Processing Clusters (GPCs), Texture Processing Clusters (TPCs), Streaming Multiprocessors (SMs), and CUDA cores. Understanding this hierarchy is essential for anyone looking to write optimized CUDA code or analyze GPU-level performance.

1. Graphics Processing Clusters (GPCs)

1.1 Overview

At the top of NVIDIA’s compute hierarchy lies the Graphics Processing Cluster. A GPC is an independently operating unit within the GPU, responsible for distributing workloads efficiently across its internal resources. Each GPC contains a set of Texture Processing Clusters (TPCs), Raster Engines, and shared control logic.

📊 GPC Block Diagram:

Nvidia Turing TU102 Full GPU with 72 SM Units.
Nvidia Turing TU102 Full GPU with 72 SM Units. Image : developer.nvidia.com
Internal Die layout of Nvida Turing TU102 GPU .
Internal Die layout of Nvida Turing TU102 GPU . Image : developer.nvidia.com

1.2 GPC Architecture

Each GPC includes:

  • One or more Raster Engines
  • Several TPCs (typically 2 to 8 depending on the GPU tier)
  • A Geometry Engine (in graphics workloads)

📘 Example GPC layout in RTX 30 series:

Nvidia RTX30 GPC block Diagram
Nvidia RTX30 GPC block Diagram. Image: Nvidia Ampere GA102 GPU Architecture document

1.3 Scalability Role

More GPCs generally equate to more parallel compute and graphics capability. High-end GPUs like the H100 feature many GPCs to support large-scale AI workloads, while mobile GPUs may only include one or two.

2. Texture Processing Clusters (TPCs)

2.1 Role of TPCs

TPCs are the next level down. A TPC groups together Streaming Multiprocessors (SMs) and a set of fixed-function texture units, providing both compute and graphics acceleration. Originally optimized for texture mapping and rasterization, TPCs in modern GPUs support general-purpose compute as well.

2.2 Components of a TPC

Each TPC typically contains:

  • Two Streaming Multiprocessors (SMs)
  • Shared L1 cache
  • Texture units (for graphics and compute shaders)
  • A PolyMorph Engine (responsible for vertex attribute setup and tessellation)

📊 TPC Diagram with SMs and Texture Units:

NVIDIA Ampere GA104 architecture showing GPC, TPC, SM.
NVIDIA Ampere GA104 architecture showing GPC, TPC, SM. Image: wolfadvancedtechnology.com

2.3 Texture Mapping

Texture units in the TPC fetch texels from memory, perform filtering (e.g., bilinear, trilinear), and handle texture addressing. These units have been extended to support texture sampling for compute workloads, such as in scientific visualization.

3. Streaming Multiprocessors (SMs)

3.1 Importance of SMs

Streaming Multiprocessors are the core programmable units of NVIDIA GPUs. They execute the majority of instructions, including floating-point arithmetic, integer operations, load/store instructions, and branch logic.

3.2 SM Internal Structure

A modern SM (e.g., in the Hopper H100 or Blackwell B100) consists of:

  • Multiple CUDA cores (up to 128 per SM)
  • Load/Store Units (LSUs)
  • Integer and Floating Point ALUs
  • Tensor Cores (for matrix operations)
  • Special Function Units (SFUs)
  • Warp schedulers and dispatch units
  • Register files
  • Shared memory and L1 cache

📘 SM Layout Reference (Volta/Hopper SMs):

Nvidia Volta Streaming Multiprovessor (SM) block diagram
Nvidia Volta Streaming Multiprovessor (SM) block diagram. Image: Nvidia Volta Architecture Document

3.3 Warp Scheduling

The warp scheduler picks a ready warp from a warp pool and issues an instruction every clock cycle. Techniques like GTO (Greedy Then Oldest), Round-Robin, or Two-Level scheduling are used.

Key Benefits:

  • Latency hiding: Warps can be swapped out when memory access stalls occur.
  • Concurrency: Independent warps can issue instructions simultaneously.
How varying the block size while holding other parameters constant would affect the theoretical Warp occupancy.
How varying the block size while holding other parameters constant would affect the theoretical Warp occupancy. Image: docs.nvidia.com

4. CUDA Cores

4.1 Role of CUDA Cores

CUDA cores, also called SPs (Streaming Processors), are the smallest execution units. Each core executes a single thread from a warp, performing basic arithmetic and logic operations.

4.2 Arithmetic Logic Units (ALUs)

Each CUDA core consists of:

  • FP32 FPU (Floating Point Unit)
  • INT ALU (Integer Arithmetic Unit)
  • Optional support for FP64, depending on SM design

4.3 SIMD Execution under SIMT Model

NVIDIA employs a SIMT (Single Instruction, Multiple Thread) model. Each warp executes one instruction at a time across 32 CUDA cores. Despite the SIMT term, the execution model is close to SIMD, with divergence managed by disabling inactive lanes.

4.4 Register Files and Local Storage

Each thread gets a set of registers from the SM’s register file. Efficient register usage is critical to avoiding spills to slower local memory.

5. Specialized Units within SMs

5.1 Tensor Cores

Tensor cores are designed to accelerate matrix multiplications—key in deep learning. They support:

  • FP16, TF32, INT8, and FP4 (in Blackwell)
  • Mixed-precision compute
  • Fused Multiply-Add (FMA) operations on tiles of 4×4, 8×8 matrices

5.2 Special Function Units (SFUs)

SFUs compute transcendental functions like sine, cosine, exp, log, and square root. These are not time-critical in AI workloads but crucial in graphics.

5.3 Load/Store Units (LSUs)

LSUs manage memory operations between registers, shared memory, and L1/L2 caches. Optimizing memory throughput requires understanding how LSUs queue and coalesce memory transactions.

6. Summary and Practical Takeaways

Understanding the hierarchical breakdown of GPC → TPC → SM → CUDA core helps:

  • Optimize kernel launch configurations
  • Maximize warp occupancy
  • Minimize divergence and memory stalls
  • Align workloads with hardware capabilities

In the next part of the series, we’ll explore Tensor Cores and RT Cores in-depth—covering how NVIDIA has fused graphics and AI acceleration into a unified pipeline.

✅ Introduction to NVIDIA GPU Architecture: Hierarchy, Cores, and Parallelism

👋 Welcome to GPU Architecture 101

In this first post in five-part series, where I introduce you to the NVIDIA GPU architecture—a foundation for parallel computing used in gaming, scientific computing, artificial intelligence, and more.

This guide is designed for engineering students and beginner developers who want to understand how modern GPUs work—from their evolution to core architectural blocks like GPCs, TPCs, SMs, and CUDA Cores.

🧠 What is a GPU and Why Does It Matter?

A GPU (Graphics Processing Unit) is a processor specialized in performing many operations simultaneously. Unlike CPUs, which handle one or a few tasks at a time, GPUs contain thousands of smaller cores to process data in parallel.

A Quick Comparison: GPU vs CPU

Source: Datacamp.com
FeatureCPUGPU
FocusSerial processingParallel processing
Cores4–32 large cores100s–1000s of smaller cores
UsageOS, logic, light appsGraphics, AI, simulations

📊 GPUs are ideal for matrix multiplications, image processing, 3D rendering, and training AI models.

🚀 NVIDIA GPU Architecture Hierarchy

Understanding the hierarchy inside a GPU is key to mastering performance tuning and CUDA programming.

Hernandez Fernandez, Moises & Guerrero, Ginés & Cecilia, José & García, José & Inuggi, Alberto & Jbabdi, Saad & Behrens, Timothy & Sotiropoulos, Stamatios. (2015). Erratum: Accelerating fibre orientation estimation from diffusion weighted magnetic resonance imaging using GPUs (PLoS ONE (2015) 10:6 (e0130915) 10.1371/journal.pone.0130915). PLoS ONE. 10. 10.1371/journal.pone.0130915.

Levels of NVIDIA GPU Architecture (2025)

  1. Graphics Processing Cluster (GPC): Top-level cluster that contains TPCs and manages workload distribution.
  2. Texture Processing Cluster (TPC): Contains Streaming Multiprocessors (SMs) and texture units.
  3. Streaming Multiprocessor (SM): The computational engine with CUDA cores, registers, and cache.
  4. CUDA Cores: The smallest processing unit in NVIDIA GPUs.

Each layer is optimized for massive parallelism and throughput.

🔁 The Grid: How GPU Threads Are Organized

In CUDA, threads are organized into:

  • Threads: Single instruction executors
  • Warps: 32 threads grouped for SIMT (Single Instruction, Multiple Threads) execution
  • Blocks: Collections of warps
  • Grids: Collections of blocks
Chapuis, Guillaume & Eidenbenz, Stephan & Santhi, Nandakishore. (2015). “GPU Performance Prediction Through Parallel Discrete Event Simulation and Common Sense”. 10.4108/eai.14-12-2015.2262575.

📌 SIMT is not the same as SIMD. In SIMT, threads may diverge, allowing for more flexible execution.

This structure is why GPUs scale so well—from a GTX 1650 to an A100 or H100—by just increasing the number of SMs and CUDA cores.

⚙️ CUDA Cores: The Heart of the NVIDIA GPU

Source: cudocompute.com

Each CUDA Core performs:

  • Integer operations via ALU
  • Floating-point operations (FP32, FP16)
  • Memory access operations via load/store units
  • Instruction decoding and execution

Example structure inside a CUDA core:

  • ALU (Arithmetic Logic Unit)
  • Register file
  • Instruction decoder
  • Control logic

These cores execute in parallel across warps under SIMT, boosting throughput for matrix-heavy tasks like image filters or neural network inference.

👨‍💻 Programming with CUDA (Hello World)

NVIDIA provides CUDA, a C/C++-like language for writing GPU code. Here’s a simple CUDA kernel to add two vectors:

cCopyEdit__global__ void vectorAdd(float *A, float *B, float *C, int N) {
    int i = threadIdx.x + blockIdx.x * blockDim.x;
    if (i < N) C[i] = A[i] + B[i];
}
  • __global__: Marks a function as a GPU kernel.
  • threadIdx, blockIdx, and blockDim: Built-in variables that locate threads in the grid.

This model maps well to the GPU’s architecture, where thousands of threads execute in parallel.

📌 Summary

  • NVIDIA GPUs are designed for parallel processing using a hierarchical architecture.
  • The architecture scales from GPC → TPC → SM → CUDA cores.
  • CUDA enables direct access to GPU hardware through a thread-block-grid model.
  • Understanding SIMT, warps, and memory layout is key to efficient GPU programming.

I blog about latest technologies including AI / ML, Audio / Video, WebRTC, Enterprise Networking , automotive and more. In the next blog, we’ll explore GPCs, TPCs, and SMs in-depth—including scheduling, caches, and warp control. Follow my blog here and on Linkedin.

🔄 Google’s Agent2Agent (A2A) Protocol: A New Era of AI Agent Interoperability

Google has introduced the Agent2Agent (A2A) protocol, a revolutionary open standard designed to enable seamless communication between AI agents. This innovation, backed by over 50 major tech partners, allows autonomous agents to collaborate across platforms, applications, and organizations—even when built using different technologies.

🌐 Why the Agent2Agent Protocol Is a Game-Changer

As AI agents become central to productivity and automation, enterprises face a major challenge: AI agents often can’t talk to each other. Without a standard protocol, they’re siloed, and their potential is limited.

Google’s A2A protocol solves this by providing a secure, scalable, and open way for agents to:

  • Exchange structured tasks and results
  • Work asynchronously across platforms
  • Handle multiple data formats including text, audio, video, and files

🧠 Core Concepts of the Agent2Agent Protocol

👤 Key Roles in Agent2Agent protocol

A2A defines three main participants:

  • User – A human or service initiating a request
  • Client Agent – Acts on behalf of the user
  • Remote Agent – Fulfills the task via A2A

This structure supports both human-in-the-loop and fully automated workflows.

🔌 Transport and Communication in the Agent2Agent Protocol

  • Transport: HTTP(S) + Server-Sent Events (SSE)
  • Protocol Format: JSON-RPC 2.0
  • Discovery: Agent Cards using standard JSON hosted at .well-known/agent.json

🪪 Agent Cards: Advertise Capabilities

Agent Cards describe what an agent can do, its supported modalities, and authentication requirements. Here’s an example:

{
  "name": "Google Maps Agent",
  "description": "Plan routes and generate directions",
  "url": "https://maps-agent.google.com",
  "skills": [
    {
      "id": "route-planner",
      "name": "Route planning",
      "description": "Helps plan routing between two locations",
      "tags": ["maps", "routing"]
    }
  ]
}

These Agent Cards help other agents choose the best partner to complete a task.

🚀 Sending and Completing Tasks

A2A tasks represent a unit of work between agents. Here’s how a task is created:

{
  "jsonrpc": "2.0",
  "method": "tasks/send",
  "params": {
    "id": "uuid-task-id",
    "message": {
      "role": "user",
      "parts": [{
        "type": "text",
        "text": "Book a conference room"
      }]
    }
  }
}

A response might include artifacts, such as:

{
  "status": {
    "state": "completed"
  },
  "artifacts": [{
    "parts": [{
      "type": "text",
      "text": "Room booked for 10 AM tomorrow."
    }]
  }]
}

A task can be:

  • Immediate
  • Long-running
  • Require multiple agent interactions

🔄 Multi-Turn Conversations

Agents can pause and wait for input. This allows dynamic, back-and-forth workflows. Example:

  1. User: “Request a new phone.”
  2. Agent: “Select a phone type (iPhone or Android)”
  3. User: “Android”
  4. Agent: “Ordered. Your request number is R12443.”

🧵 Streaming Support with SSE

For long tasks, agents can stream updates using tasks/sendSubscribe:

{
  "method": "tasks/sendSubscribe",
  "params": {
    "message": {
      "parts": [{ "type": "text", "text": "Write a report" }]
    }
  }
}

You’ll receive real-time updates like:

data: {
  "artifact": {
    "parts": [
      { "type": "text", "text": "Intro section complete." }
    ]
  }
}

This reduces latency and improves interactivity for complex tasks.

📡 Push Notifications for Offline Updates

Clients can set up push notification endpoints:

{
  "method": "tasks/pushNotification/set",
  "params": {
    "id": "task-id",
    "pushNotificationConfig": {
      "url": "https://example.com/callback",
      "authentication": { "schemes": ["jwt"] }
    }
  }
}

This is ideal for enterprise-grade systems needing alerts on disconnected tasks.

🎨 Multi-Modal Support

A2A supports:

  • Text
  • Audio
  • Video
  • Files (PDFs, images, etc.)
  • Structured data (JSON schema)

Example of a request with a file:

"parts": [
  { "type": "text", "text": "Summarize this document" },
  { "type": "file", "file": { "mimeType": "application/pdf", "data": "&lt;base64>" } }
]

⚖️ A2A vs Anthropic’s MCP

FeatureA2A (Google)MCP (Anthropic)
FocusAgent-to-agent interoperabilityAgent context-sharing
TransportHTTP + SSEJSON over HTTP
Multi-modalityYesText only (mostly)
Push NotificationsYesNo
Long TasksBuilt-in state managementNot native
Agent DiscoveryAgent CardsManual or non-standard
Open SourceYesNot fully

While MCP enriches a single agent’s decision-making, A2A enables real collaboration between independent agents.

🛠 Real-World Applications

🔧 IT Automation

  • A helpdesk agent detects a ticket
  • Another agent gathers logs from a system
  • A third agent takes corrective action

👔 HR and Recruitment

  • The hiring agent sources candidates
  • The scheduler agent books interviews
  • The compliance agent runs background checks

A2A glues these together into one seamless workflow.

❌ Error Handling Made Simple

A2A uses JSON-RPC standard errors like:

  • -32700: JSON parse error
  • -32601: Method not found
  • -32001: Task not found

Agents return meaningful messages and even suggest retries or alternate modalities.

🔍 Conclusion: Why A2A is the Future

The A2A protocol sets a new standard for AI agent collaboration. With support for streaming, structured output, rich media, and open discovery—it’s built for the real world of enterprise AI.

If you’re building multi-agent ecosystems, A2A is the protocol you’ve been waiting for.

📚 Further Reading

If you liked this blog, you will also like my blog on Timeline from Transformers to LLM and Agentic AI and the most popular ML basics – Supervised Machine Learning for beginners

Timeline from Transformers to LLMs and Agentic AI

Since the groundbreaking 2017 paper “Attention is All You Need” introduced the Transformer architecture, the field of artificial intelligence has undergone a rapid and transformative evolution. This blog post will explore the chronology of important events that have shaped the AI landscape, leading up to the current era of Large Language Models (LLMs) and agentic AI. Let us visit the timeline from Transformers to LLM and Agentic AI

2017: The Transformer Revolution

The journey begins with the publication of “Attention is All You Need” by Google scientists in 2017. This paper introduced the Transformer architecture, which relied solely on attention mechanisms, dispensing with recurrence and convolutions entirely. The new model demonstrated superior translation quality and efficiency in machine translation tasks, setting the stage for a paradigm shift in natural language processing.

2018: BERT and the Rise of Bidirectional Models

Building on the success of Transformers, 2018 saw the introduction of BERT (Bidirectional Encoder Representations from Transformers) by Google researchers. BERT’s innovation lay in its bidirectional nature, allowing it to capture context from both directions in text data. This breakthrough significantly improved performance across various language tasks, from question-answering to sentiment analysis.

2019-2020: The GPT Era Begins

OpenAI’s release of GPT-3 (Generative Pre-trained Transformer 3) in 2020 marked a significant milestone in the development of large language models. With 175 billion parameters, GPT-3 demonstrated unprecedented capabilities in natural language understanding and generation, capturing the imagination of researchers and the public alike.

2021-2022: AI Goes Mainstream

During this period, AI technologies began to permeate various industries and applications:

  • AI in Healthcare: The healthcare and pharmaceutical sectors emerged as early adopters of AI, leveraging it for tasks such as appointment scheduling, patient care, and personalized treatment.
  • Self-Driving Vehicles: AI agents moved beyond software into the physical world, making real-time, high-stakes decisions in autonomous vehicles.
  • Code Generation: AI systems like GitHub Copilot began assisting developers in writing code, hinting at the potential for AI to transform software development.
Timeline from Transformers to LLM and Agentic AI

2023: The Year of Generative AI

2023 saw an explosion in generative AI applications, with tools like DALL-E, Midjourney, and ChatGPT capturing public attention. These models demonstrated the ability to generate high-quality text, images, and even code, sparking discussions about the future of creative work and knowledge work.

2024: The Dawn of Agentic AI

As we moved into 2024, the concept of Agentic AI began to take shape. This new paradigm represented a shift from isolated AI tasks to specialized, interconnected agents capable of more autonomous operation. Key developments included:

  • Multi-Agent Systems: AI agents began working collaboratively to solve complex problems, simulating human teamwork in digital environments.
  • Small Language Models (SLMs): The adoption of SLMs alongside LLMs offered new possibilities for efficient, task-specific AI solutions.
  • AI Orchestration: Frameworks for coordinating multiple AI agents emerged, allowing for more complex problem-solving approaches.

2025: The Year of Agentic AI

As we stand in 2025, Agentic AI has become the new frontier in artificial intelligence. This evolution is characterized by several key trends:

  1. Autonomous Decision-Making: AI agents now operate with greater independence, capable of long-term planning and adapting to changing conditions without constant human oversight.
  2. AI Engineers: Systems like Devin AI are now capable of debugging and writing code on their own, pushing the boundaries of what AI can achieve in software development.
  3. Industry Transformation: Agentic AI is revolutionizing various sectors, with the potential to take over entire departments in organizations. For example:
    • In healthcare, AI agents manage tasks from appointment scheduling to personalized treatment plans.
    • In customer service, AI-driven virtual assistants provide increasingly sophisticated and personalized support.
  4. Multi-Agent Collaboration: OpenAI’s introduction of “Swarm,” an experimental framework for coordinating networks of AI agents, has opened new possibilities for complex problem-solving.
  5. Enhanced Personalization: Advanced learning algorithms enable AI agents to tailor services and products to individual needs, creating highly personalized experiences across industries.
  6. Scalable Automation: AI agents are driving automation at an unprecedented scale, from small businesses to large enterprises, significantly reducing costs and operational inefficiencies.
  7. Continuous Learning and Adaptation: Agentic AI systems demonstrate the ability to learn autonomously and adapt to dynamic environments, enabling faster growth and efficiency across sectors.

As we look to the future, the potential of Agentic AI seems boundless. From enhancing decision-making processes to revolutionizing entire industries, these intelligent agents are poised to transform the way we work, create, and solve problems. However, this rapid advancement also brings new challenges in ethics, privacy, and workforce adaptation that society must address.

We saw and have actually lived through this timeline from Transformers to LLM and Agentic AI. The journey has been remarkably swift, showcasing the exponential pace of innovation in artificial intelligence. As we explore the vast potential of Agentic AI, the broader quest for Artificial General Intelligence (AGI) remains a captivating goal. AGI aims to create intelligent systems capable of performing any intellectual task that humans can and represents the ultimate frontier in artificial intelligence. Another interesting article on this topic can be found here. For a deeper dive into the most basic concepts on AI and Machine Learning please visit my other blog pages.

A Deep Dive into PyTorch’s GPU Memory Management

Here is an error I got when using an image generation deep learning model. It is a common error Engineers get when using PyTorch on GPU. To solve this error, a deep dive into PyTorch’s GPU Memory management is needed. So fasten your seat belts 🙂

torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 58.00 MiB. GPU 0 has a total capacity of 3.71 GiB of which 57.00 MiB is free. Including non-PyTorch memory, this process has 3.64 GiB memory in use. Of the allocated memory 3.51 GiB is allocated by PyTorch, and 74.06 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

This error message provides valuable insights:

  • Memory Exhaustion: The GPU’s available memory (3.71 GiB) has been depleted.
  • Allocation Attempt: PyTorch attempted to allocate 20.00 MiB, but there wasn’t enough free space.
  • Memory Usage: 3.68 GiB is in use, with 3.61 GiB allocated by PyTorch and 5.27 MiB reserved but unallocated.
  • Fragmentation Hint: The message suggests that memory fragmentation might be contributing to the issue, and setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True might help.

PyTorch’s Memory Management Strategies

PyTorch employs a sophisticated memory management system to optimize GPU resource utilization. Here’s a detailed breakdown:

  • Caching Allocator: PyTorch uses a caching allocator to reduce the overhead of frequent memory allocations and deallocations. This improves performance but can also contribute to memory fragmentation if not managed effectively.
  • Memory Pooling: PyTorch pools memory into larger blocks to reduce fragmentation and improve allocation efficiency.
  • Automatic Deallocation: PyTorch automatically deallocates memory for tensors that are no longer needed, reducing the risk of memory leaks.
  • torch.cuda.empty_cache(): This function manually clears the cached memory, potentially freeing up unused resources.
  • PYTORCH_CUDA_ALLOC_CONF: This environment variable allows you to fine-tune memory allocation behavior. Experimenting with different configurations can help address fragmentation issues.

Profiling Tools for Deep Insights

To gain a granular understanding of memory usage and identify bottlenecks, profiling tools are indispensable:

NVIDIA System Management Interface (NVIDIA-smi):

  • Real-time monitoring of GPU utilization, temperature, and memory usage.
  • Provides detailed information about processes and applications consuming GPU resources.
  • Example usage in Bash: nvidia-smi
    watch -n0.1 nvidia-smi

    PyTorch Memory Profiler

    • Records memory allocations and deallocations during program execution.
    • Visualizes memory usage patterns over time.
     enable memory history, which will
# add tracebacks and event history to snapshots
torch.cuda.memory._record_memory_history()

run_your_code()
torch.cuda.memory._dump_snapshot("my_snapshot.pickle")

    Open pytorch.org/memory_viz and drag/drop the pickled snapshot file into the visualizer. The visualizer is a javascript application that runs locally on your computer. It does not upload any snapshot data.

    Active Memory Timeline in PyTorch Memory visualizer

    Allocator State History in PyTorch Memory visualizer

    • Integrates seamlessly with PyTorch models and training scripts.
    • Example usage:
    import torch.profiler as profiler
    with profiler.profile() as prof:
        # Your PyTorch code here 
        # ... 

    # Print the profiling results print(prof.key_metrics())

    Nsight Systems:

    • A powerful profiling tool that provides detailed insights into GPU utilization, memory usage, and performance bottlenecks.
    • Offers visualizations for performance analysis.
    • Example usage in Bash
    nsight-systems --profile-gpu --launch-command="python your_script.py"

    Debugging and Optimization Strategies

    1. Reduce Model Size: If possible, use a smaller or optimized version of the Stable Diffusion model to reduce memory requirements.
    2. Adjust Batch Size: Experiment with different batch sizes to find the optimal balance between performance and memory usage.
    3. Optimize Data Loading: Ensure your data loading pipeline is efficient and avoids unnecessary memory copies.
    4. Monitor Memory Usage: Use profiling tools to track memory consumption and identify areas for optimization.
    5. Consider Memory-Efficient Techniques: Explore techniques like gradient checkpointing or quantization to reduce memory usage.
    6. Leverage Cloud-Based GPUs: If your local hardware is constrained, consider using cloud-based GPU instances with larger memory capacities.

    Additional Considerations:

    • GPU Driver Updates: Ensure you have the latest GPU drivers installed to avoid performance issues or memory leaks.
    • Operating System Configuration: Check your operating system’s memory management settings to see if they can be optimized for better GPU performance.
    • TensorFlow vs. PyTorch: If you’re using TensorFlow, explore its memory management features and best practices.

    Advanced Memory Optimization Techniques

    For more advanced scenarios, consider the following techniques:

    • Memory Pooling: Manually create memory pools to allocate and reuse memory blocks efficiently. This can be helpful for specific use cases where memory allocation is frequent.
    • Custom Memory Allocators: If you have deep knowledge of CUDA and memory management, you can create custom memory allocators to address specific memory usage patterns.
    • Profiling and Benchmarking: Use profiling tools to identify performance bottlenecks and benchmark different memory optimization strategies to measure their effectiveness.

    Beyond the Code: A Deeper Dive into Memory Management

    While we’ve covered the essential aspects of PyTorch’s memory management, it’s worth exploring the underlying mechanisms in more detail.

    • CUDA Memory Allocator: CUDA, the underlying framework for NVIDIA GPUs, provides its own memory allocator. PyTorch interacts with this allocator to allocate and manage memory on the device.
    • Memory Fragmentation: When memory is allocated and deallocated frequently, it can lead to fragmentation, where small, unused memory blocks are scattered throughout the memory space. This can make it difficult for PyTorch to allocate larger contiguous blocks of memory.
    • Memory Pooling: PyTorch’s memory pooling strategy involves creating larger memory pools and allocating memory from these pools. This can help reduce fragmentation and improve memory allocation efficiency.
    • Automatic Deallocation: PyTorch uses reference counting to track memory usage and automatically deallocates memory for tensors that are no longer needed. However, it’s important to be aware of potential memory leaks if tensors are not properly managed.
    • Profiling Tools: Profiling tools like Nsight Systems can provide detailed insights into memory usage patterns, including memory allocations, deallocations, and access patterns. This information can be invaluable for identifying memory-related bottlenecks and optimizing your code.

    Conclusion

    Overcoming the “CUDA out of memory” error requires a deep understanding of PyTorch’s memory management strategies and the ability to leverage profiling tools effectively. By following the techniques outlined in this blog post, you can optimize your PyTorch applications for efficient GPU memory usage and unlock the full potential of your NVIDIA GPU

    🚀 The Evolution of YOLO 🚀

    The YOLO (You Only Look Once) series is a real-time object detection algorithm that uses convolutional neural network (CNN). It has dramatically shaped the landscape of real-time computer vision. Each iteration of YOLO brings something unique to the table, enhancing the capabilities and applications of object detection. Let’s dive into the evolution of Yolo, details of each major YOLO model, the companies and organizations behind them, and how they contribute to the evolution of AI.

    🌐 Timeline of Key YOLO Models and Their Innovators:

    1️⃣ Joseph Redmon 🧠:

    • YOLO V1 (2016): Redmon introduced the first YOLO model, which revolutionized object detection by framing it as a single regression problem instead of a classification task. This approach allowed YOLO to detect objects in images at unprecedented speeds, making it suitable for real-time applications.
    • YOLO V2 & YOLO V3: These versions refined the detection process, improving accuracy with techniques like anchor boxes and multi-scale predictions. YOLO V3 was especially known for its balance between speed and accuracy, making it a benchmark for real-time object detection.

    2️⃣ AlexeyAB & WongKinYiu 🔧:

    • YOLO V4: Building on Redmon’s work, AlexeyAB introduced YOLO V4, which incorporated advanced techniques like CSPDarknet53 as the backbone, PANet for path aggregation, and various other improvements that significantly boosted detection accuracy while maintaining speed.
    • Scaled YOLO V4: WongKinYiu extended YOLO V4 by introducing scaling capabilities, allowing the model to adapt to different sizes depending on the computational resources available.
    • YOLO-R & YOLO V7: These versions continued to refine the architecture, focusing on edge-device efficiency without compromising accuracy, and further solidifying YOLO’s role in lightweight, real-time applications.

    3️⃣ Ultralytics (Glenn Jocher) 💻:

    • YOLO V5: Ultralytics’ YOLO V5 made a significant impact by offering an easier-to-use version of YOLO with extensive support for PyTorch. It’s known for its ease of training, deployment, and integration into various projects. YOLO V5 became the go-to model for many practitioners due to its flexibility and performance.
    • YOLO V8: The latest from Ultralytics, YOLO V8, pushes the envelope with state-of-the-art performance, integrating the latest research advancements, and focusing on deployment efficiency in various environments, from cloud to edge devices.

    4️⃣ Meituan Technical Team 🌟:

    • YOLO V6: Aimed at balancing speed and accuracy, YOLO V6 from Meituan was developed with a focus on real-world applications where inference speed on edge devices is critical. It leverages modern techniques like EfficientNet as the backbone to optimize performance.
    • YOLO V6 3.0: This update introduced further refinements in the model architecture, allowing it to perform even better on resource-constrained devices, making it ideal for mobile and embedded applications.

    5️⃣ Baidu 🧬:

    • PP-YOLO Series: Baidu’s PP-YOLO and its successors (V2 and beyond) are optimized for PaddlePaddle, an AI framework developed by Baidu. PP-YOLO models integrate many of the latest research advancements in object detection, providing a powerful tool for various commercial applications. Baidu’s focus on enhancing speed and accuracy makes PP-YOLO particularly well-suited for industrial AI applications where efficiency is key.

    6️⃣ Megvii Technology 🏢:

    • YOLOX: Megvii’s YOLOX introduces a new paradigm by decoupling the head of the network into classification and regression branches, improving performance and making it easier to adapt to different tasks. It’s optimized for versatility, robustness, and ease of deployment, particularly in scenarios requiring high accuracy and low latency. YOLOX’s advancements make it a strong contender in the commercial AI space.

    7️⃣ Alibaba DAMO Academy 🏛️:

    • DAMO YOLO: Alibaba’s DAMO Academy has taken YOLO to new heights with its DAMO YOLO models, which focus on specialized applications requiring high precision. These models leverage PyTorch and Apache licensing and are designed for integration into Alibaba’s vast ecosystem, ensuring scalability and robustness in production environments.

    8️⃣ Deci AI 🛠️:

    • YOLO-NAS: Deci AI introduces a novel approach with YOLO-NAS, utilizing Neural Architecture Search (NAS) to automatically optimize YOLO models for specific tasks. This results in highly efficient, custom-tailored models that excel in specific applications, providing a significant edge in performance and deployment flexibility.
    History of Yolo computer vision model development

    Image Credit : OpenCV.ai

    📜 Licensing Overview:

    • MIT License: Used in the early YOLO versions, allowing for broad use and modification with few restrictions.
    • GPL License: Encourages collaboration while ensuring that derivative work remains open source.
    • Apache License: Offers a balance between open-source freedom and commercial use, widely adopted in enterprise environments.

    💡 Frameworks:

    • Darknet: The original framework used in early YOLO versions, known for its speed and efficiency.
    • PyTorch: Dominates recent YOLO models, providing flexibility and a rich ecosystem for development and deployment.
    • PaddlePaddle: Baidu’s in-house framework, optimized for PP-YOLO models, ensuring tight integration with Baidu’s AI infrastructure.

    As we continue to push the boundaries of AI, the evolution of YOLO to what it is today has been of immense help to computer vision developers. Each new iteration not only refines performance but also expands the possibilities of what AI can achieve in real-world applications. Whether you’re developing on the edge or deploying in the cloud, the YOLO family offers a model for every need. 🌍

    Basic Machine Learning Optimization Algorithms

    Keeping up with my tradition of posting my old handwritten notes, here are my notes on basic Machine Learning optimization algorithms. Optimization algos in ML help minimize the cost function, thereby helping to reduce the error between predicted value and actual value

    1) Most popular – Gradient Descent

    learning gradient descent equation as part of machine learning optimization

    Gradient Descent is used in linear regression, logistic regression & early implementation of neural networks

    Basic Machine learning optimization algorithms
    Gradient Descent ML algorithm

    2) ADAM = ADAptive Moment estimation

    Adam optimization algorithm does not use single global learning rate , but a different learning rate for every single parameter

    Adam Optimization ML algorithm
    Adam Optimization ML algorithm

    I hope you liked my notes on basic machine learning optimization algorithms. Feel free to comment here or on my Linkedin post.

    Hand written notes on Neural Networks and ML course by Andrew Ng

    About 2018 when I started working on Machine learning I took many courses. Here are my hand written notes on Neural Networks and ML course by Andrew Ng. It focuses on the fundamental concepts covered in the course, including Logistic Regression, Neural Networks, and Softmax Regression. Buckle up for some equations and diagrams!

    Part 1: Logistic Regression – The Binary Classification Workhorse

    Logistic regression reigns supreme for tasks where the target variable (y) can only take on two distinct values, typically denoted as 0 or 1. It essentially calculates the probability (a) of y belonging to class 1, given a set of input features (x). Here’s a breakdown of the process:

    1. Linear Combination: The model calculates a linear score (z) by taking a weighted sum of the input features (x) and their corresponding weights (w). We can represent this mathematically as:

      z = w_1x_1 + w_2x_2 + … + w_nx_n

      (where n is the number of features)
    2. Sigmoid Function: This linear score (z) doesn’t directly translate to a probability. The sigmoid function (σ) steps in to transform this score into a value between 0 and 1, representing the probability (a) of y belonging to class 1. The sigmoid function is typically defined as:

    a = \sigma(z) = \frac{1}{1 + e^{-z}}

    Sigmoid Function Plot / Logistic Curve


    Key takeaway: 1 – a represents the probability of y belonging to class 0. This is because the sum of probabilities for both classes must always equal 1.

    Part 2: Demystifying Neural Networks – Building Blocks and Forward Propagation

    1. Perceptrons – The Basic Unit: Neural networks are built using perceptrons, the fundamental unit inspired by biological neurons. A perceptron takes weighted inputs (just like logistic regression), performs a linear transformation, and applies an activation function to generate an output.
    2. Activation Functions: While sigmoid functions are common in logistic regression and the initial layers of neural networks, other activation functions like ReLU (Rectified Linear Unit) can also be employed. These functions introduce non-linearity, allowing the network to learn more complex patterns in the data.
    3. Layering Perceptrons: Neural networks are not limited to single perceptrons. We can stack multiple perceptrons into layers, where each neuron in a layer receives outputs from all the neurons in the previous layer. This creates a complex network of interconnected units.
    4. Forward Propagation: Information flows through the network in a forward direction, layer by layer. In each layer, the weighted sum of the previous layer’s outputs is calculated and passed through an activation function. This process continues until the final output layer produces the network’s prediction.

    Part 3: Unveiling Backpropagation – The Learning Algorithm

    But how do these neural networks actually learn? Backpropagation is the hero behind the scenes! It allows the network to adjust its weights and biases in an iterative manner to minimize the error between the predicted and actual outputs.

    1. Cost Function: We define a cost function that measures how well the network’s predictions align with the actual labels. A common cost function for classification problems is the cross-entropy loss.
    2. Error Calculation: Backpropagation calculates the error (difference between prediction and actual value) at the output layer and propagates it backward through the network.
    3. Weight and Bias Updates: Based on the calculated errors, the weights and biases of each neuron are adjusted in a way that minimizes the overall cost function. This process is repeated iteratively over multiple training epochs until the network converges to a minimum error state.

    Part 4: Softmax Regression – Expanding Logistic Regression for Multi-Class Classification

    Logistic regression excels in binary classification, but what happens when we have more than two possible class labels for the target variable (y)? Softmax regression emerges as a powerful solution!

    1. Generalizing Logistic Regression: Softmax regression can be viewed as an extension of logistic regression for multi-class problems. It calculates a set of class scores (z_i) for each possible class (i).
    2. The Softmax Function: Similar to the sigmoid function, softmax takes these class scores (z_i) and transforms them into class probabilities (a_i) using the following formula:

    a_i = \frac{e^{z_i}}{\sum\limits_{j=1}^{C} e^{z_j}}

    (where Σ represents the sum over all possible classes j)
    Key takeaway: This function ensures that all the class probabilities (a_i) sum up to 1, which is a crucial requirement for a valid probability distribution. Intuitively, for a given input (x), only one class can be true, and the softmax function effectively distributes the probability mass across all classes based on their corresponding z_i scores.

    Softmax Function Curve

    1. Interpretation of Class Probabilities: Each class probability (a_i) represents the model’s estimated probability of the target variable (y) belonging to class i, given the input features (x). This probabilistic interpretation empowers us to not only predict the most likely class but also gauge the model’s confidence in that prediction.

    Part 5: Putting It All Together – Training and Cost Function for Softmax Regression

    Part 5: Putting It All Together – Training and Cost Function for Softmax Regression

    While we’ve focused on the mechanics of softmax, training a softmax regression model involves a cost function. Here’s a brief overview:

    1. Negative Log-Likelihood Cost Function: Softmax regression typically employs the negative log-likelihood cost function. This function penalizes the model for assigning low probabilities to the correct class and vice versa. Mathematically, the cost function can be represented as:Cost=−Σ(yi∗log(ai))
      (where y_i is 1 for the correct class and 0 otherwise)
    2. Model Optimization: During training, the model aims to minimize this cost function by adjusting its weights and biases through backpropagation. As the cost function decreases, the model learns to produce class probabilities that better reflect the underlying data distribution.

    Conclusion: A Stepping Stone to Deep Learning

    These blog and hand written notes on Neural Networks and ML has provided a condensed yet detailed exploration of logistic regression, neural networks, and softmax regression, concepts covered in Andrew Ng’s Advanced Learning Algorithms course. Understanding these fundamental building blocks equips you to delve deeper into the fascinating world of Deep Learning and explore more advanced architectures like Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs).Remember, this is just the beginning of your Deep Learning journey!

    I hope these detailed hand written notes on Neural Networks and ML with diagrams prove helpful for your Deep Learning studies!

    How to Compile Linux kernel

    There were 22 Billion internet connected devices in the world at the end of 2018. Therefore much more total computing devices including the ones that are not connected. Plus, by 2024 the number must be substantially higher. These devices run various programs written in different programming languages. Some run boring mainframe code, while others run trendy AI and ML models. Something really really fundamental to all these devices is that they run an OS – an Operating System. Majority of them run Linux. Lets go back to basics today and do something really fundamental. Let’s learn how to compile Linux kernel.

    Step 1: Install Dependencies Before compiling the kernel, you’ll need to install some dependencies. These may include development tools, compilers, and libraries. The required packages vary depending on your distribution. For example, on Debian-based systems, you can install the necessary packages with the following command:

    sudo apt-get install build-essential libncurses-dev bison flex libssl-dev libelf-dev

    Step 2: Download the Kernel Source Code You can download the kernel source code from the official Linux kernel website (https://www.kernel.org/). Choose a long term supported version e.g. linux-6.6.24.tar.xz and download the corresponding tarball.

    Step 3: Extract the Source Code Navigate to the directory where you downloaded the tarball and extract it using the following command:

    tar xvf linux-6.6.24.tar.xz

    Step 4: Configure the Kernel Change into the kernel source directory:

    cd linux-6.6.24
    Steps to compile Linux kernel
    Steps to compile Linux kernel

    Run the following command to start the kernel configuration:

    make menuconfig

    This command opens a text-based menu where you can configure various kernel options. You can navigate through the menu using the arrow keys and select options using the spacebar. Once you’re done configuring, save your changes and exit the menu.

    Compile the Kernel Once you’ve configured the kernel, you’re ready to compile it. Run the following commands:

    make -j$(nproc)

    This command starts the compilation process. The “-j$(nproc)” option tells make to use as many parallel processes as there are CPU cores, which can speed up the compilation process significantly.

    Install the Kernel Modules After the compilation is complete, you can install the kernel modules using the following command:

    sudo make modules_install

    Install the Kernel To install the newly compiled kernel, run the following command:

    sudo make install

    This command installs the kernel image, kernel modules, and other necessary files.

    Step 5: Update Boot Loader Configuration Finally, you need to update your boot loader configuration to include the new kernel. The procedure for doing this varies depending on your boot loader (e.g., GRUB, LILO).

    Reboot Once you’ve updated the boot loader configuration, reboot your system to boot into the newly compiled kernel.

    That’s it! You’ve successfully compiled and installed the Linux kernel.