Dzone Scraper

Building agentic AI systems fundamentally changes how we handle application security. We are no longer just securing our own code. We are securing our infrastructure against code written dynamically by an LLM and executed on the fly. When building a multi-tenant AI platform, allowing an agent to run arbitrary scripts is a massive escape vector waiting to happen. Google recently made the GKE Agent Sandbox generally available on their custom Arm-based Axion N4A instances. This gives us a highly efficient, hardware-optimized path to run untrusted code safely. Under the hood, this relies on gVisor to intercept application kernel calls and run them in a heavily restricted user-space kernel. In this blueprint, we will build a secure multi-tenant execution environment. We will containerize the agent runtime using Docker, provision a GKE cluster with Axion nodes, isolate the network, and orchestrate the execution layer using a robust Java backend. Step 1: Containerizing the Agent Runtime The first step is establishing a baseline execution environment. We want this Docker image to be as lightweight as possible to reduce the attack surface, while containing the necessary runtimes for the LLM to execute its logic. Dockerfile # Use a minimal Alpine base image to reduce attack surface FROM python:3.11-alpine # Create a non-root user for execution RUN addgroup -S agentgroup && adduser -S agentuser -G agentgroup WORKDIR /sandbox # Copy the execution wrapper script COPY --chown=agentuser:agentgroup execute_payload.py /sandbox/ # Enforce non-root execution USER agentuser # Prevent Python from writing pyc files and buffering stdout ENV PYTHONDONTWRITEBYTECODE=1 ENV PYTHONUNBUFFERED=1 CMD ["python", "execute_payload.py"] To make this functional, we need an entrypoint script that safely reads the LLM-generated code from an injected environment variable or a mounted volume, executes it, and captures the output. Here is a simplified execute_payload.py implementation: Python import os import sys import traceback def main(): # In a production environment, this payload might be injected via # a Kubernetes Secret or a secure sidecar proxy. encoded_payload = os.environ.get("AGENT_PAYLOAD", "") if not encoded_payload: print("Error: No payload provided.") sys.exit(1) try: # Execute the untrusted code within this isolated process # Security constraints are handled by the container and gVisor layers exec(encoded_payload, {"__builtins__": __builtins__}, {}) except Exception as e: print(f"Execution Error: {str(e)}") traceback.print_exc() sys.exit(1) if __name__ == "__main__": main() Even if a malicious script breaks out of the Python runtime, it will find itself as an unprivileged user inside a minimal Alpine container. Step 2: Provisioning GKE With Axion and Agent Sandbox Google Axion (N4A) processors provide excellent performance per watt, making them ideal for running hundreds of concurrent, lightweight agent tasks. We will create a cluster and explicitly enable the sandbox feature. Shell # Create the GKE cluster with Sandbox enabled gcloud container clusters create agent-sandbox-cluster \ --region us-east4 \ --enable-sandbox \ --sandbox type=gvisor \ --release-channel regular # Create a dedicated node pool using Axion N4A instances gcloud container node-pools create axion-agent-pool \ --cluster agent-sandbox-cluster \ --region us-east4 \ --machine-type n4a-standard-4 \ --num-nodes 3 \ --node-labels dedicated=untrusted-agents \ --tags untrusted-workload Applying node labels ensures that trusted core microservices do not accidentally end up on the same physical infrastructure as untrusted agent execution environments. Step 3: Enforcing Network Isolation Compute isolation is useless if the untrusted code can scan your internal network or exfiltrate data to the public internet. We must deploy a strict NetworkPolicy to default-deny all egress traffic from our sandboxed namespace. YAML apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: default-deny-agent-egress namespace: isolated-agents spec: podSelector: matchLabels: app: agent-executor policyTypes: - Egress egress: # Only allow DNS resolution - ports: - port: 53 protocol: UDP - port: 53 protocol: TCP # Allow outbound only to a specific internal API gateway if needed # - to: # - ipBlock: # cidr: 10.0.0.50/32 Step 4: Deploying the Sandboxed Workload With the network secured, we define the Kubernetes deployment. By setting the runtimeClassName to gvisor, Kubernetes routes the container lifecycle through the GKE Agent Sandbox rather than the standard container runtime. YAML apiVersion: apps/v1 kind: Pod metadata: generateName: dynamic-agent-task- namespace: isolated-agents labels: app: agent-executor spec: # Instruct GKE to use the Agent Sandbox (gVisor) runtimeClassName: gvisor # Ensure these pods only land on our Axion node pool nodeSelector: dedicated: untrusted-agents restartPolicy: Never containers: - name: execution-environment image: your-registry/agent-runtime:v1.0.0 env: - name: AGENT_PAYLOAD valueFrom: secretKeyRef: name: task-payload-secret key: payload # Drop all unnecessary Linux capabilities securityContext: runAsUser: 1000 runAsNonRoot: true allowPrivilegeEscalation: false readOnlyRootFilesystem: true capabilities: drop: - ALL resources: requests: memory: "256Mi" cpu: "250m" limits: memory: "512Mi" cpu: "500m" volumeMounts: - name: temp-storage mountPath: /tmp volumes: - name: temp-storage emptyDir: {} Step 5: Orchestrating the Execution via Java Spring Boot To bring this architecture together, the control plane must dynamically spin up these sandboxed pods whenever an AI agent decides it needs to run code. In a modern distributed system, this is typically handled by a core backend microservice. Using the Fabric8 Kubernetes Client in a Java Spring Boot application provides a highly resilient way to orchestrate these ephemeral workloads programmatically. Java import io.fabric8.kubernetes.api.model.Pod; import io.fabric8.kubernetes.client.KubernetesClient; import org.springframework.stereotype.Service; @Service public class AgentOrchestratorService { private final KubernetesClient kubernetesClient; public AgentOrchestratorService(KubernetesClient kubernetesClient) { this.kubernetesClient = kubernetesClient; } public String executeUntrustedCode(String tenantId, String pythonCode) { // 1. Create a Kubernetes Secret containing the code payload String secretName = createPayloadSecret(tenantId, pythonCode); // 2. Load the sandbox Pod template and inject the specific payload secret Pod sandboxedPod = kubernetesClient.pods() .inNamespace("isolated-agents") .load(getClass().getResourceAsStream("/k8s/agent-pod-template.yaml")) .item(); // 3. Launch the pod dynamically via the API server Pod runningPod = kubernetesClient.pods() .inNamespace("isolated-agents") .create(sandboxedPod); // 4. Await completion and extract the logs safely kubernetesClient.pods() .inNamespace("isolated-agents") .withName(runningPod.getMetadata().getName()) .waitUntilCondition(pod -> pod.getStatus().getPhase().equals("Succeeded")pod.getStatus().getPhase().equals("Failed"), 30, java.util.concurrent.TimeUnit.SECONDS); String executionLogs = kubernetesClient.pods() .inNamespace("isolated-agents") .withName(runningPod.getMetadata().getName()) .getLog(); // 5. Clean up the ephemeral resources kubernetesClient.pods().delete(runningPod); kubernetesClient.secrets().withName(secretName).delete(); return executionLogs; } } The Defense in Depth Strategy This architecture relies on a strict defense in depth model. If an LLM hallucinates a malicious payload or a user deliberately attempts prompt injection to compromise the platform, the attacker faces multiple independent barriers. The code executes as a non-root user in a minimal Alpine environment with a read-only filesystem. Network access is completely blocked by native Kubernetes policies. Finally, any attempt to exploit kernel vulnerabilities is intercepted by the gvisor runtime boundary running on dedicated Axion hardware. By combining these layers, engineering teams can build and scale trustworthy Agentic AI platforms without risking the integrity of their core cloud infrastructure.Practical QA Workflow Showing How Teams Integrate LLM Testing into Real CI/CD PipelinesGenerative artificial intelligence introduces unprecedented unpredictability into software development pipelines. Traditional software returns predictable outputs for exact inputs. Large language models generate varied responses for the exact same prompt. QA teams face a massive challenge scaling quality checks for these probabilistic systems. Manual validation falls short during fast deployment cycles. Implementing LLM testing in CI/CD has become mandatory for any modern engineering team. This article outlines the exact workflow teams use to validate AI applications inside continuous integration pipelines. The Shift to Probabilistic Evaluation Models Testing deterministic applications relies on exact assertions. You pass an input and assert a literal string match or a numerical value. Testing generative AI applications requires a completely different approach. QA engineers must evaluate semantic meaning, tone, factual accuracy, and safety constraints. A simple string-matching test fails when the model rewrites a correct answer using different synonyms. Establishing Semantic Metrics Teams need a comprehensive test automation built exclusively for natural language evaluations. Integrating these specialized frameworks directly into the deployment pipeline catches hallucinations early. Evaluating AI outputs involves a mix of quantitative metrics (like mathematical scores) and qualitative assessments (such as human review and LLM-based evaluation). QA teams calculate a similarity score between the generated text and a known good reference. A cosine similarity check determines how close the meanings are. Defining the Deployment Gate Exact word matching fails frequently with language models. The testing pipeline relies on semantic similarity to determine whether a build passes or fails. This mathematical approach significantly reduces human subjectivity from the testing process, instead of removing it completely. Engineering teams set hard thresholds for these similarity scores within the deployment configuration. Step-by-Step Workflow for LLM Testing in CI/CD Setting up LLM testing in CI/CD starts with defining clear evaluation metrics. Teams typically measure faithfulness, answer relevance, and context precision. Faithfulness checks if the model bases its answer strictly on the provided context. Answer relevance evaluates if the response directly addresses the user's prompt without going off-topic. Phase 1: Version Control and Prompt Linting The QA workflow begins the moment a developer commits code. Modern AI applications store prompts as version-controlled assets alongside application logic. The automated sequence triggers an initial static analysis phase upon the pull request. Linter tools check the prompt templates for missing variables or formatting errors. This step prevents broken prompts from reaching the model API. Minimizing Early Execution Costs Catching basic structural errors early saves significant API costs and execution time. Executing live model calls during every single unit test creates unacceptable delays and unpredictable overhead. A mature test automation uses mocked responses for basic functional validation. To ensure these mocks remain accurate, teams often implement schema and contract validation. The pipeline injects predefined JSON responses to verify the application's parsing logic while simultaneously validating them against a formal contract (e.g., OpenAPI or JSON Schema). Phase 2: Mocked Unit Tests QA engineers validate that the application handles API timeouts, rate limits, and malformed outputs correctly. This isolates the application code from the model's unpredictability. Fast feedback loops at this stage keep developers productive. Developers receive immediate alerts if their code breaks the fundamental integration points. Phase 3: Automated Evaluation Runs The core of LLM testing in CI/CD happens during the integration phase. The pipeline deploys the application code to an ephemeral staging environment. The testing script pulls a curated dataset of diverse prompts representing real-world user queries. The system sends these prompts to the live endpoint and records the generated responses. Evaluation frameworks score these responses against the predefined metrics. Pipeline Configuration Example Implementing this layer demands deep large language model optimization to balance test execution speed with evaluation accuracy. Let us look at a practical implementation snippet for GitHub Actions. This configuration installs the required Python dependencies first. The sequence then executes the automated AI evaluations and generates an XML report. YAML name: LLM Pipeline Evaluation on: [push] jobs: evaluate-llm: runs-on: ubuntu-latest steps: - uses: actions/checkout@v3 - name: Setup Python environment uses: actions/setup-python@v4 with: python-version: '3.10' cache: 'pip' # Enables dependency caching - name: Install evaluation dependencies run: pip install -r requirements-test.txt - name: Run automated AI evaluations env: OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY } # Added --instafail and --retries for flaky API stability run: | pytest tests/llm_evaluations/ \ --junitxml=reports/result.xml \ --force-sugar - name: Upload Evaluation Report if: always() # Ensures reports upload even if tests fail uses: actions/upload-artifact@v4 with: name: llm-eval-report path: reports/result.xml Phase 4: Guardrail and Toxicity Checks Security and safety form a mandatory layer of testing generative AI applications. The pipeline executes specialized tests to attempt prompt injection and jailbreak scenarios. The test suite verifies that the model rejects malicious requests and refuses to generate toxic content. PII scanners analyze the outputs to prevent data leaks. Any failure in this security phase triggers an immediate halt. Protecting Brand Integrity Securing the application boundary protects the brand reputation and user privacy. Malicious actors constantly look for vulnerabilities in AI endpoints. Automated security gating stops these vulnerabilities from reaching production. Phase 5: Regression Tracking and Deployment The pipeline aggregates the evaluation scores and compares them against previous test runs. A sudden drop in the faithfulness metric indicates a regression in the model prompt or the retrieval logic. QA teams use specialized dashboards to visualize these trends over time. Only when the new build meets the historical baseline does the process proceed. Closing the Feedback Loop The code is then merged and deployed to the production environment automatically. Continuous monitoring takes over to track live user interactions. Production telemetry feeds directly back into the QA workflow. Engineering teams extract challenging edge cases from live user logs. QA engineers format these edge cases into new test scenarios for the automated suite. Architecting the Testing Infrastructure This continuous feedback loop drives ongoing optimization of large language models. The test dataset grows more comprehensive with every deployment iteration. Refining the dataset guarantees the test automation framework remains highly relevant to actual user behavior. Provisioning Dedicated Hardware Running AI evaluations demands significant computational resources. Standard runners often lack the memory to process large datasets quickly. Engineering teams provision specialized runner instances equipped with high memory capacity. These dedicated machines execute the scripts much faster than standard nodes. Faster execution times prevent bottlenecks during the integration phase. Addressing Cost and API Latency Optimizing the hardware infrastructure directly supports seamless workflows. Teams parallelize test execution across multiple runners to reduce total build time. Executing hundreds of live calls in a CI/CD pipeline costs money and takes time. Teams must optimize their testing strategies to manage these constraints. Tiered Execution Strategies Running a smaller smoke test suite during regular commits provides immediate feedback. The full regression suite runs overnight or before major releases. Caching previous model responses for identical prompts reduces unnecessary API calls. Engaging with QA teams for AI applications helps teams effectively architect tiered testing strategies. Resolving Non-Deterministic Test Failures Traditional automated tests pass or fail consistently. Generative models introduce flakiness into the pipeline by their very nature. A prompt might pass the similarity threshold on nine attempts and fail on the tenth. QA engineers implement retry logic explicitly for these edge cases. Smart Retry Mechanisms The test script requests a regeneration from the model up to three times before registering a hard failure. Analyzing the failure logs helps teams identify poorly phrased prompts. Continuous refinement of both the application code and the test scripts eliminates instability. Scaling the Workflow and Advanced Techniques Integrating LLM testing in CI/CD requires constant adaptation. New foundational models are released frequently. API endpoints change, and token limits shift. The QA architecture must remain modular to enable seamless swapping out of underlying models. Isolating the evaluation logic from the application code prevents vendor lock-in. Shadow Deployments A well-architected pipeline treats the AI model as just another replaceable microservice. Advanced teams implement shadow deployments for major model upgrades. The pipeline routes a small percentage of live traffic to the new model version alongside the existing one. Automated scripts compare the responses from both versions in real-time. This provides statistical confidence before fully committing to the update. Managing Hallucinations at Scale Testing generative AI applications in production via shadow traffic uncovers nuances missed in staging environments. Monitoring actual user interactions reveals unexpected model behaviors. Hallucinations remain the primary risk factor for AI deployments. The automated sequence must include dedicated datasets designed exclusively to trigger known hallucination patterns. Optimizing the RAG Pipeline Modern AI relies on retrieval-augmented generation (RAG) for accurate context, meaning the language model isn't the only component needing testing. The CI/CD workflow must evaluate the retrieval database independently to ensure performance. QA teams use tests to verify that vector databases return the correct document chunks for specific queries. By measuring metrics like Precision (relevance) and Recall (comprehensiveness), teams can fine-tune embeddings and retrieval parameters to ensure the most accurate data reaches the model. Conclusion Integrating LLM testing in CI/CD transforms unpredictable AI experiments into dependable enterprise software. Implementing structured metrics, mocked unit tests, and automated evaluation scripts secures the deployment pipeline. Teams that adopt specialized test automation ship high-quality AI features with complete confidence. Automating the evaluation of semantic accuracy and security creates a predictable release cadence. Engineering organizations must embrace these testing paradigms to lead the generative AI market. Adopting this structured workflow guarantees consistent product quality across every release.TensorFlow vs PyTorch: The Real Difference Isn’t AccuracyA few days ago, I set out to build a simple image classification model using convolutional neural networks (CNNs). The task itself wasn’t particularly complex, but choosing the right framework proved more challenging than expected. I found myself choosing between TensorFlow and PyTorch, two powerful frameworks for building high-performance CNNs. To explore this, I implemented the same CNN in both frameworks under identical conditions and compared them across key aspects like learning curve, flexibility, debugging, and performance. A Quick Look at the Frameworks Before deep-diving into the comparison, it’s worth briefly understanding the two frameworks used throughout this experiment. 1. TensorFlow TensorFlow is an open-source deep learning framework developed by Google. It is widely known for its strong ecosystem and production-ready capabilities. One of its key strengths is its integration with high-level APIs such as Keras, which simplifies model building and training. TensorFlow is commonly used in large-scale applications, offering tools for deployment across web, mobile, and edge devices. Overall, it is often preferred when moving models from experimentation to production environments. 2. PyTorch PyTorch is an open-source deep learning framework developed by Meta Platforms. It has gained significant popularity, especially in the research community, due to its simplicity and flexibility. PyTorch uses a dynamic computation graph, which makes it feel more like standard Python code. This makes model development more intuitive and debugging significantly easier. It is often the preferred choice for experimentation, rapid prototyping, and research-driven projects. Experiment Setup To ensure a fair and meaningful comparison between TensorFlow and PyTorch, both implementations were designed under identical conditions. 1. Dataset The models were trained and evaluated on the CIFAR-10 dataset, a widely used benchmark for image classification tasks.It consists of 60,000 color images across 10 classes, making it suitable for evaluating CNN performance.CIFAR-10 is publicly available for research purposes and is commonly distributed under a permissive academic license, allowing free use for educational and non-commercial applications. 2. Model Architecture A simple yet effective Convolutional Neural Network (CNN) architecture was used in both frameworks. The structure includes: Convolutional layers for feature extractionReLU activation functionsMax-pooling layers for dimensionality reductionFully connected layers for classification Care was taken to ensure that the architecture remained identical in both implementations. 3. Training Configuration To maintain consistency, the following hyperparameters were used across both frameworks: Optimizer: AdamLearning rate: 0.001Batch size: 64Number of epochs: 10Loss function: Cross-Entropy Loss 4. Environment All experiments were conducted using Google Colab. Both TensorFlow and PyTorch implementations were executed in the same runtime environment. The configuration used includes: Runtime Type: GPU-enabled environmentPython Version: 3.xDeep Learning Libraries: TensorFlow and PyTorch (latest stable versions) The experiments were run on the same Colab runtime session to maintain consistency in resource allocation. Implementation To ensure a fair comparison, the same CNN architecture and training configuration were implemented using both TensorFlow and PyTorch. While the underlying model remains identical, the implementation approach differs significantly across the two frameworks. 1. CNN Implementation in TensorFlow The model was first implemented using TensorFlow with its high-level Keras API, which provides a concise and structured way to define deep learning models. Model Definition Python model = models.Sequential([ layers.Conv2D(32, (3,3), activation='relu', input_shape=(32,32,3)), layers.MaxPooling2D((2,2)), layers.Conv2D(64, (3,3), activation='relu'), layers.MaxPooling2D((2,2)), layers.Flatten(), layers.Dense(64, activation='relu'), layers.Dense(10, activation='softmax') ]) The Sequential API allows layers to be stacked in a linear fashion, making the architecture easy to read and implement. This significantly reduces boilerplate code and is especially helpful for beginners. Model Compilation and Training Python model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy']) history = model.fit(x_train, y_train, epochs=10, batch_size=64, validation_data=(x_test, y_test)) Training in TensorFlow is handled using a single high-level function. It automatically manages the training loop, backpropagation, and metric tracking, making the process highly streamlined. Observation: TensorFlow offers a compact and beginner-friendly implementation. With minimal code, it handles most of the underlying complexity, making it ideal for rapid development and production-oriented workflows. 2. CNN Implementation in PyTorch The same CNN architecture was implemented using PyTorch, which follows a more explicit and flexible approach. Model Definition Python class CNN(nn.Module): def __init__(self): super().__init__() self.conv1 = nn.Conv2d(3, 32, 3) self.pool = nn.MaxPool2d(2,2) self.conv2 = nn.Conv2d(32, 64, 3) self.fc1 = nn.Linear(64*6*6, 64) self.fc2 = nn.Linear(64, 10) In PyTorch, models are defined using Python classes. This provides greater flexibility but requires a more detailed understanding of how each component works. Forward Pass Python def forward(self, x): x = self.pool(torch.relu(self.conv1(x))) x = self.pool(torch.relu(self.conv2(x))) x = x.view(-1, 64*6*6) x = torch.relu(self.fc1(x)) x = self.fc2(x) return x The forward pass must be explicitly defined, giving full control over how data flows through the network. This makes it easier to customize and debug complex models. Training Loop Python for inputs, labels in trainloader: optimizer.zero_grad() outputs = model(inputs) loss = criterion(outputs, labels) loss.backward() optimizer.step() Unlike TensorFlow, PyTorch requires a manual training loop. While this increases the amount of code, it also provides complete transparency and control over the training process. Observation: PyTorch offers a more flexible and transparent approach. Although it requires more code, it allows finer control over model behavior, making it a preferred choice for experimentation and research. With both implementations in place, the next step is to evaluate their performance and analyze how they compare across different metrics. Results and Analysis With both implementations completed under identical conditions, we now compare TensorFlow and PyTorch using empirical results and practical observations. 1. Accuracy The image illustrates the Accuracy and Training Time for TensorFlow and PyTorch. (Image by Author) Both frameworks achieved nearly identical performance on the CIFAR-10 dataset: TensorFlow Accuracy: 68.78% PyTorch Accuracy: 68.95% The difference (0.17%) is extremely small and falls within normal training variation. When architecture, data, and hyperparameters are controlled, the choice of framework has virtually no impact on model accuracy. Additionally, both models show: Consistent improvement across epochsNo signs of severe overfittingStable generalization on test data The image illustrates the Train and Test accuracy for TensorFlow and PyTorch. (Image by Author) 2. Loss Convergence The image illustrates the Loss Convergence for TensorFlow and PyTorch in Logarithmic Scale. (Image by Author) TensorFlow exhibits a smooth and gradually decreasing loss, both for training and validation.PyTorch shows a similar downward trend, but with slightly larger values. The higher loss values in PyTorch are due to loss accumulation across batches, whereas TensorFlow reports average loss per epoch. Despite differences in scale, both frameworks demonstrate stable and consistent convergence behavior, indicating effective training. 3. Model Training Performance Training Speed TensorFlow: 715.23 secondsPyTorch: 723.31 seconds TensorFlow is slightly faster (~1% difference), but the gap is minimal For moderate-sized datasets like CIFAR-10, training speed differences are negligible and unlikely to influence framework selection, but TensorFlow provides strong tooling for large-scale deployment, while PyTorch is equally capable in training large models. 4. Scalability and Flexibility TensorFlow follows a more structured and predefined approach, but provides robust tools such as distributed training and deployment pipelines. It also holds an advantage in large-scale production environments, while PyTorch continues to close the gap. PyTorch uses a dynamic computation graph, allowing runtime modifications, which makes custom modifications easy. It is better suited for research and experimentation, where flexibility is critical. 5. Learning Curve From an implementation standpoint: TensorFlow (via Keras) allows model creation with minimal and structured code; hence, it is easier to start with.PyTorch requires explicit definitions for model architecture, forward passes, and training loops; this results in lengthier code and greater initial effort. Ultimately, the choice between TensorFlow and PyTorch is less about performance and more about how you prefer to design, experiment with, and deploy deep learning models. Choosing Between TensorFlow and PyTorch TensorFlow is better suited when working on production-ready systems, where scalability, deployment tools, and a structured workflow are important. Its high-level APIs make it easy to develop models quickly and integrate them into real-world applications, including mobile and edge environments.PyTorch is more appropriate for research and experimentation, where flexibility and control are critical. Its dynamic nature and seamless debugging experience make it ideal for testing new ideas and building custom architectures. Conclusion: Choosing the Right Framework Through this hands-on comparison of TensorFlow and PyTorch using a CNN on the CIFAR-10 dataset, one key insight becomes clear: both frameworks perform almost identically when it comes to core metrics. The experimental results showed: Nearly identical accuracy (~68–69%)Comparable training timesSimilar loss convergence patterns This highlights an important takeaway: The choice of framework has little to no impact on model performance when architecture and training conditions are kept consistent. However, the real difference lies not in performance, but in how you build, debug, and deploy models. Ultimately, the best framework is not the one that performs slightly better on benchmarks, but the one that aligns with your workflow, problem domain, and development style. Connect with me for more updates: MediumLinkedINWhy LLM Pipelines Fail in Production and How Temporal and Kafka Fix ThemA production LLM pipeline is rarely just a prompt and a response. It typically combines retrieval, prompt rendering, model inference, output shaping, validation, persistence, and downstream actions. That broader shape is why many systems look stable in a demo and then become fragile under live traffic. The model call is only one component; the operational problem is the workflow around it. Provider APIs impose rate limits, structured outputs still need application-level checks, and external calls introduce failure ambiguity that ordinary request-response code does not handle well. Where the Breakage Starts Most production failures happen between steps, not inside the prompt. A request enters an API, context is loaded, a model call is sent, the response is parsed, a downstream action is triggered, and a record is written. If the provider generated output but the network dropped before the caller saw it, the system no longer has a clean answer to whether the operation should be retried or treated as complete. Kafka’s default delivery model is at least once, and Temporal’s documentation is explicit that activities may be retried and therefore should be idempotent. That combination makes duplicate side effects the default risk unless the pipeline is designed around durable state and idempotent writes. Duration creates the second breakage pattern. Ingestion may fan out across thousands of chunks, while a risky action may need approval hours later. Temporal workflows can receive external write events through Signals, and durable timers persist across worker and service downtime, so a workflow can pause without collapsing into callback code and scheduled cleanups. Temporal also requires workflow logic to remain deterministic during replay and provides versioning methods so new executions can adopt new code while long-running executions remain on compatible paths. Those concerns are not edge cases in LLM systems; they are normal once the pipeline extends beyond a single synchronous call. Output shape is another common source of confusion. OpenAI’s Structured Outputs guide exists because unconstrained text is not a reliable contract; the feature is designed to enforce a supplied JSON Schema and avoid missing required keys or invalid enum values. But schema compliance is only the first gate. A response can be structurally valid and still be semantically wrong, stale, or unsafe to automate. Production failures happen when formatting success is mistaken for business correctness. Why Kafka solves only part of it Kafka is a strong fit at the ingestion boundary because it turns synchronous pressure into a durable stream of work. Kafka topics are partitioned, ordering is guaranteed within a partition, and each partition is consumed by exactly one consumer in a consumer group at a given time. Consumers also control offsets and can rewind to replay records. That combination is well suited to bursty LLM demand, key-based ordering, and reprocessing after a model or prompt change. Java public void submitRequest(LlmRequest request) { LlmRequestEvent event = new LlmRequestEvent( request.requestId(), request.tenantId(), request.documentId(), request.templateId() ); kafkaTemplate.send("llm.requests", request.requestId(), event); } This pattern keeps the API narrow. The service records intent by publishing an event keyed by requestId; Kafka’s default partitioning uses the key hash, so related records land on the same partition and preserve that partition’s order. Kafka’s producer is also optimized for batching, and its pull-based consumer model lets downstream services fall behind and catch up instead of being overwhelmed by broker-driven push traffic. That is useful when inference latency varies, and demand arrives in bursts. But Kafka only states that work was published and later consumed. It does not know whether retrieval already succeeded, whether a model provider timed out after actually producing output, or whether persistence ran before a crash. Offsets capture consumption position, not business completion. Kafka is excellent for transport, buffering, replay, and fan-out, but insufficient as the sole control plane for a multi-step inference process. Why Temporal Changes the Outcome Temporal addresses the state problem directly. Its model is durable execution: workflows advance through an event history stored by the Temporal service, and that history is what allows an execution to recover from a crash and continue making progress. Worker crashes, network interruptions, and infrastructure outages are handled differently from ordinary application failures because the workflow state is not reconstructed from logs after the fact; it is already part of the execution record. Java @KafkaListener(topics = "llm.requests") public void onRequest(LlmRequestEvent event, Acknowledgment ack) { InferenceWorkflow workflow = workflowClient.newWorkflowStub( InferenceWorkflow.class, WorkflowOptions.newBuilder() .setWorkflowId(event.requestId()) .setTaskQueue("llm-inference") .build() ); try { WorkflowClient.start(workflow::run, event); } catch (WorkflowExecutionAlreadyStarted ex) { log.info("Workflow already started for {}", event.requestId()); } ack.acknowledge(); } The important detail is the workflow identifier. Temporal guarantees workflow ID uniqueness within a namespace and prevents another open workflow with the same ID from starting, which turns duplicate Kafka deliveries into a safe re-entry case instead of parallel duplicate execution. The Kafka listener acknowledges the record after the workflow start is accepted, not after the entire inference path finishes. Kafka remains the transport layer; Temporal becomes the durable execution layer for the request. Inside that workflow, each external operation should sit in an activity with explicit timeout and retry policy rather than inside scattered retry loops. Temporal’s retry model is declarative; activities retry by default, and the platform documentation recommends making activities idempotent and granular because a retry re-executes the whole activity. That fits LLM systems unusually well, since retrieval, prompt rendering, model invocation, validation, and persistence rarely fail for the same reason. Java private final LlmActivities activities = Workflow.newActivityStub( LlmActivities.class, ActivityOptions.newBuilder() .setStartToCloseTimeout(Duration.ofSeconds(45)) .setRetryOptions(RetryOptions.newBuilder() .setMaximumAttempts(5) .setInitialInterval(Duration.ofSeconds(2)) .build()) .build() ); public InferenceResult run(LlmRequestEvent event) { Context context = activities.loadContext(event.documentId()); Prompt prompt = activities.renderPrompt(event.templateId(), context); ModelResponse response = activities.callModel(prompt, event.requestId()); return activities.validateAndPersist(event.requestId(), response); } That separation also makes the final validation step honest. Structured output may guarantee a schema, but business rules still need to decide whether the result is usable, and persistence still needs an idempotent write keyed by requestId so a retry cannot create duplicate approvals, tickets, or rows. In practice, this is the difference between a pipeline that merely retries and a pipeline that resumes safely. Why the Handoff Works in Production The pattern that holds up is a clean handoff. Kafka should own ingress, buffering, replay, and downstream fan-out. Temporal should own the lifecycle of one logical request. When a workflow completes, it can publish a result event for indexing, notifications, analytics, or billing, and each downstream system can consume that event independently. Kafka moves facts through the platform. Temporal ensures the process that produced those facts reaches a correct terminal state. This split also improves the cases that usually appear after launch. Human approval can arrive as a Temporal Signal without keeping compute hot. A workflow can pause for days using a durable timer and continue after downtime. Workflow code can be versioned so new executions take a new path while long-running executions stay on compatible logic, which is important because replay depends on deterministic workflow behavior. The reason LLM pipelines fail in production is not that language models are impossible to operationalize. The reason is that they are often deployed as request handlers when they are actually distributed workflows with uncertain latency, replay risk, and side effects. Kafka fixes the transport problem by providing durable, replayable streams with partition ordering and load decoupling. Temporal fixes the execution problem by persisting progress, surviving worker failure, and making retries, timeouts, and human pauses part of the design instead of a patch applied after incidents. When those two responsibilities are separated cleanly, an LLM pipeline stops behaving like an experimental chain of API calls and starts behaving like a production system.Securing Branch Networks With Firewalls, VPNs, IDS/IPS, and Identity-Based AccessBranch networks no longer behave like quiet extensions of a single headquarters LAN. They terminate local user traffic, break out directly to the internet for SaaS, maintain persistent connections back to core systems, and increasingly host devices that are operationally important even when central resources are unavailable. NIST notes that the enterprise network landscape has shifted because of cloud services, geographic dispersion, and changes in application design, while zero trust guidance emphasizes that network location is no longer the primary signal of trust. In practice, that means a branch cannot be secured by treating the site-to-site tunnel as a blanket trust boundary. The branch edge has to make explicit policy decisions about which flows are allowed, which flows are encrypted, which flows are inspected, and which identities are entitled to touch which resources. Beyond the Old Perimeter The older perimeter model assumed that most meaningful risk arrived from outside the network and that internal traffic was comparatively trustworthy. That assumption breaks down quickly in distributed environments. NIST’s current network guidance explicitly calls out the limitations of perimeter-centric protection and VPN-centric access in environments that include cloud services, remote users, and branch offices, while NSA’s zero trust guidance frames lateral movement as a primary post-compromise technique that segmentation and granular policy are meant to contain. A modern branch design therefore needs layered control points close to the resource and close to the user, not just a tunnel back to a core firewall. That shift also changes how edge devices are treated operationally. Branch firewalls, VPN gateways, and routers are no longer simple plumbing. They are security control planes, and they are common targets. CISA issued Binding Operational Directive 23-02 specifically to reduce the risk from internet-exposed management interfaces, and NSA recommends encrypted administration, ACL-restricted management access, and dedicated management segments rather than broad reachability from production networks. Securing the branch therefore starts with the idea that the branch edge itself must be hardened, isolated, and observable before it is entrusted to enforce policy for anything else. Firewalls Define Intent A branch firewall is most effective when it expresses business intent instead of accumulating ad hoc port exceptions. NIST’s firewall guidance is still the right mental model: block inbound and outbound traffic unless it is expressly permitted, use stateful inspection to track valid sessions, and apply egress filtering so that spoofed or unexpected source traffic cannot leave the site. Where application awareness is needed, NIST also notes that application-proxy gateways can inspect protocol content and, in some cases, decrypt and re-encrypt selected traffic before forwarding it. That combination turns the firewall from a coarse packet filter into a policy engine that knows the difference between permitted business traffic and merely possible traffic. A concise nftables policy for a small branch can be deliberately narrow: Plain Text table inet filter { chain forward { type filter hook forward priority 0; policy drop; ct state established,related accept iifname "lan" oifname "wan" ip saddr 10.20.30.0/24 ip daddr 10.10.0.0/16 tcp dport 443 accept iifname "lan" oifname "wan" ip saddr 10.20.30.0/24 udp dport 53 accept iifname "lan" oifname "wan" ip saddr 10.20.30.0/24 tcp dport { 80, 443 } accept } } The shape of that ruleset matters more than the exact addresses. The first line admits only established or related traffic, which keeps return paths fast without making the policy permissive. The next rule allows a very specific branch-to-core application path over HTTPS. DNS is explicitly separated because name resolution is usually treated as infrastructure rather than open internet access. The final rule allows only web egress from the branch subnet, and the chain-wide policy drop turns every other flow into an intentional denial instead of an accidental omission. That aligns with NIST’s deny-by-default and egress-filtering guidance, and it scales far better than a firewall that starts from “allow any” and slowly adds patches. VPNs Protect the Path VPNs remain essential in branch networking, but their role is precise: protect traffic in transit across untrusted transport, not grant broad implied trust to the attached network. NIST’s IPsec guidance identifies gateway-to-gateway VPNs as the common model for linking a branch office to headquarters and notes that the model is operationally simple because it is largely transparent to end users. The same guidance recommends IKEv2 over IKEv1 because IKEv2 is simpler, faster, and more secure, and it lists modern algorithm choices such as AES-GCM and SHA-2 families as recommended options. It also states that tunnel mode is used for gateway-to-gateway deployments and that perfect forward secrecy should be used when resources allow. A stripped-down strongSwan configuration shows the right shape for a branch-to-core tunnel: Plain Text connections { branch-hq { version = 2 remote_addrs = 198.51.100.10 proposals = aes256gcm16-prfsha384-ecp384 local { auth = pubkey; certs = branch-gw.pem; id = branch-gw.example } remote { auth = pubkey; id = hq-gw.example } children { corp { local_ts = 10.20.30.0/24 remote_ts = 10.10.0.0/16 esp_proposals = aes256gcm16-ecp384 rekey_time = 50m start_action = trap } } dpd_delay = 30s } } The important details are the constrained traffic selectors and the modern cryptographic profile. local_ts and remote_ts keep the tunnel scoped to known subnets instead of turning it into a default route for every packet. rekey_time shortens the lifetime of key material, while dpd_delay enables liveness checking so dead peers do not leave stale state behind. strongSwan’s configuration model exposes exactly those selectors, proposals, and peer-liveness controls, which map cleanly onto NIST’s guidance for tunnel mode, IKEv2, and periodic key refresh. Just as important, NIST’s broader network guidance warns that VPN-based access has limits in the current enterprise landscape. A secure tunnel does not solve segmentation, visibility, or granular authorization by itself. IDS and IPS Reveal Drift Firewalls and VPNs are excellent at enforcing expected paths, but they are not enough to detect misuse inside those paths. That is where network IDS and IPS become decisive. NIST’s IDPS guidance recommends products that combine signature-based detection, anomaly-based detection, and stateful protocol analysis because each method compensates for the others. Signature-based methods are efficient for known threats but weak against novel variants and evasion; anomaly-based methods can detect unknown abuse but are noisy without careful profiling; stateful protocol analysis helps distinguish legitimate protocol behavior from malformed or abusive sequences. NIST also stresses that these systems require tuning and that prevention actions should often be tested in simulation or learning modes before being enforced inline. A practical Suricata rule can be very small while still expressing a meaningful branch policy: Plain Text drop tls $HOME_NET any -> $EXTERNAL_NET any ( msg:"Deprecated TLS from branch host"; tls.version:1.0; sid:1001001; rev:1; ) The rule follows Suricata’s standard structure of action, header, and rule options. In IPS mode, drop blocks the flow and generates an alert, while tls.version:1.0 turns a broad “bad crypto” idea into an enforceable control that stops unsafe client negotiations at the branch edge. That kind of rule is useful because it binds transport hygiene to observable protocol behavior instead of relying on application owners to update every endpoint perfectly. The placement of the sensor still matters. NIST explicitly warns that network-based IDPS cannot inspect payloads inside encrypted traffic such as VPN, HTTPS, or SSH unless traffic is analyzed before encryption or after decryption. In a branch, that usually means placing inspection logically behind the VPN gateway for branch-to-core traffic and beside the egress path for direct internet breakout. Identity Turns Access into Policy The most important change in branch security is that authorization can no longer be inferred from attachment alone. NIST’s zero trust architecture states that access to enterprise resources should be granted on a per-session basis with least privilege, and that policy decisions can vary by identity, device status, network location, time, and other environmental signals. NIST’s secure network landscape guidance pushes the same idea further by arguing that user identity alone is not sufficient and that contextual information about devices and services must be part of the decision. CISA’s zero trust maturity model reinforces that direction by describing automated access controls that consider identity, device risk, application, and data category, and that are time-limited. At the branch edge, the most practical implementation is usually 802.1X with EAP-TLS backed by RADIUS. IEEE 802.1X defines mutual authentication for LAN-attached clients and ports, while EAP-TLS provides certificate-based mutual authentication and key derivation. Once that identity has been established, RADIUS can return standard attributes that place the endpoint into the correct VLAN and attach the correct ACL. RFC 3580 specifies the exact tunnel attributes used for VLAN assignment, and a FreeRADIUS users file can express the authorization response very compactly: Plain Text [[emailprotected]](https://dzone.com/cdn-cgi/l/email-protection) Tunnel-Type := VLAN, Tunnel-Medium-Type := IEEE-802, Tunnel-Private-Group-Id := "120", Filter-Id := "finance-restricted" That snippet is intentionally small, but the effect is powerful. A successful 802.1X session for the named identity receives a VLAN and an access filter rather than broad branch connectivity. The same pattern can be extended from a named user to directory-driven roles, device classes, posture states, and time-bounded administrative sessions. It is also the reason identity-based access belongs in the network discussion rather than only in the IdP discussion: the branch switch or wireless edge becomes the first enforcement point where verified identity is translated into concrete packet-level reachability. Conclusion A secure branch is not created by stacking appliances and hoping that defense in depth emerges automatically. It is created by dividing responsibility cleanly across controls that complement one another. The firewall establishes a deny-by-default policy and limits what can traverse the site. The VPN protects selected traffic across untrusted transport without pretending that encryption is the same thing as trust. IDS and IPS expose misuse, drift, and protocol abuse that still occur inside permitted paths. Identity-based access ensures that branch attachment results in the minimum reachability justified by the authenticated subject and device, not by the convenience of a subnet. When those controls are composed deliberately, the branch stops being a soft edge and becomes a constrained, observable, and policy-driven part of the enterprise security fabric.The Retry Budget Pattern: How to Stop Retry Storms in API-Led and Microservice Systems