Senior Software Engineer, GPU Inference & Kernel Systems

About Runara

Runara is building the inference serving engine for a world where AI compute is scarce, fragmented, and unevenly priced. While everyone else fights over the same scarce GPUs, we are building the systems kernels, compilers, and schedulers that turn any available hardware into a cost-efficient platform for serving models at scale.

We're a small, technically dense team. If you join, your kernels and your design decisions will help to directly determine the future of inference unit economics.

What You'll Work On

Speculative Workload across Heterogeneous Compute

• Tune acceptance rates, batching strategy, and draft/verify scheduling for maximum throughput.
• Design, build, and optimize speculative decoding pipelines. Extend speculative workloads beyond a single machine, routing prefill and decode across heterogeneous hardware classes based on real-time cost and availability.
• Design the scheduling and orchestration logic that spans different xPU hardware, treating a mixed fleet as one fungible compute pool.
• Develop a working model of token economics, cost per token, throughput per dollar, and use it to drive design decisions.

Inference Serving Compiler

• Write and optimize CUDA kernels that make inference viable on every GPU SKU — including quantization, memory layout, and kernel strategies tuned to each xPU target.
• Optimize for memory bandwidth and compute across inference stacks such as TensorRT-LLM, vLLM, and SGLang, adapting each to the constraints of the underlying hardware.
• Work on tensor parallelism and LLM sharding, and on the compiler/runtime layer that takes a model graph and produces hardware-aware execution plans across a wide, messy diversity of GPU architectures and driver/runtime versions.
• Integrate NGC compute availability while continuously probing dozens of cloud providers for capacity, so routing decisions are always based on live pricing and supply.

What We're Looking For

• Strong CUDA kernel programming experience. You've written and optimized custom kernels (attention, GEMM, quantization, fusion, etc.) and understand memory hierarchies, warp-level parallelism, and occupancy trade-offs at a level beyond “I called a library.”
• Experience with LLM inference internals, KV cache management, batching (continuous/dynamic), attention variants, quantization schemes (INT8/INT4/FP8), or speculative/draft-verify decoding.
• Comfort working across heterogeneous hardware, not just the newest datacenter GPUs — you're excited by, not annoyed by, constraints like limited memory bandwidth, older compute capability, or inconsistent driver support.
• Systems-level thinking: you can reason about where time and cost actually go in a serving pipeline (network, scheduling, memory, compute) and optimize the right layer.
• Proficiency in C++ and Python, and familiarity with at least one inference/serving framework (vLLM, TensorRT-LLM, Triton, TGI, SGLang, or an internal equivalent).
• Bonus: experience building or contributing to a compiler, graph rewriter, or hardware-abstraction layer for ML workloads.
• Bonus: experience with distributed scheduling, spot/preemptible compute markets, or cost-aware routing systems.

Why Runara

• You'll own real, load-bearing infrastructure from day one — not a side project.
• The problems here are genuinely unsolved. No other serving platform treats heterogeneous compute as a first-class citizen.
• Direct impact on the company's core economics: every kernel you optimize and every routing decision you improve shows up in our margins.
• Small team, high trust, high autonomy.