On this tutorial, we discover tips on how to run OpenAI’s open-weight GPT-OSS fashions in Google Colab with a robust deal with their technical habits, deployment necessities, and sensible inference workflows. We start by organising the precise dependencies wanted for Transformers-based execution, verifying GPU availability, and loading openai/gpt-oss-20b with the right configuration utilizing native MXFP4 quantization, torch.bfloat16 activations. As we transfer via the tutorial, we work instantly with core capabilities equivalent to structured era, streaming, multi-turn dialogue dealing with, software execution patterns, and batch inference, whereas conserving in thoughts how open-weight fashions differ from closed-hosted APIs when it comes to transparency, controllability, reminiscence constraints, and native execution trade-offs. Additionally, we deal with GPT-OSS not simply as a chatbot, however as a technically inspectable open-weight LLM stack that we are able to configure, immediate, and prolong inside a reproducible workflow.
print(“🔧 Step 1: Putting in required packages…”)
print(“=” * 70)
!pip set up -q –upgrade pip
!pip set up -q transformers>=4.51.0 speed up sentencepiece protobuf
!pip set up -q huggingface_hub gradio ipywidgets
!pip set up -q openai-harmony
import transformers
print(f”✅ Transformers model: {transformers.__version__}”)
import torch
print(f”n🖥️ System Info:”)
print(f” PyTorch model: {torch.__version__}”)
print(f” CUDA accessible: {torch.cuda.is_available()}”)
if torch.cuda.is_available():
gpu_name = torch.cuda.get_device_name(0)
gpu_memory = torch.cuda.get_device_properties(0).total_memory / 1e9
print(f” GPU: {gpu_name}”)
print(f” GPU Reminiscence: {gpu_memory:.2f} GB”)
if gpu_memory < 15:
print(f”n⚠️ WARNING: gpt-oss-20b requires ~16GB VRAM.”)
print(f” Your GPU has {gpu_memory:.1f}GB. Think about using Colab Professional for T4/A100.”)
else:
print(f”n✅ GPU reminiscence ample for gpt-oss-20b”)
else:
print(“n❌ No GPU detected!”)
print(” Go to: Runtime → Change runtime sort → Choose ‘T4 GPU'”)
elevate RuntimeError(“GPU required for this tutorial”)
print(“n” + “=” * 70)
print(“📚 PART 2: Loading GPT-OSS Mannequin (Appropriate Technique)”)
print(“=” * 70)
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline
import torch
MODEL_ID = “openai/gpt-oss-20b”
print(f”n🔄 Loading mannequin: {MODEL_ID}”)
print(” This will likely take a number of minutes on first run…”)
print(” (Mannequin measurement: ~40GB obtain, makes use of native MXFP4 quantization)”)
tokenizer = AutoTokenizer.from_pretrained(
MODEL_ID,
trust_remote_code=True
)
mannequin = AutoModelForCausalLM.from_pretrained(
MODEL_ID,
torch_dtype=torch.bfloat16,
device_map=”auto”,
trust_remote_code=True,
)
pipe = pipeline(
“text-generation”,
mannequin=mannequin,
tokenizer=tokenizer,
)
print(“✅ Mannequin loaded efficiently!”)
print(f” Mannequin dtype: {mannequin.dtype}”)
print(f” Machine: {mannequin.gadget}”)
if torch.cuda.is_available():
allotted = torch.cuda.memory_allocated() / 1e9
reserved = torch.cuda.memory_reserved() / 1e9
print(f” GPU Reminiscence Allotted: {allotted:.2f} GB”)
print(f” GPU Reminiscence Reserved: {reserved:.2f} GB”)
print(“n” + “=” * 70)
print(“💬 PART 3: Primary Inference Examples”)
print(“=” * 70)
def generate_response(messages, max_new_tokens=256, temperature=0.8, top_p=1.0):
“””
Generate a response utilizing gpt-oss with really helpful parameters.
OpenAI recommends: temperature=1.0, top_p=1.0 for gpt-oss
“””
output = pipe(
messages,
max_new_tokens=max_new_tokens,
do_sample=True,
temperature=temperature,
top_p=top_p,
pad_token_id=tokenizer.eos_token_id,
)
return output[0][“generated_text”][-1][“content”]
print(“n📝 Instance 1: Easy Query Answering”)
print(“-” * 50)
messages = [
{“role”: “user”, “content”: “What is the Pythagorean theorem? Explain briefly.”}
]
response = generate_response(messages, max_new_tokens=150)
print(f”Consumer: {messages[0][‘content’]}”)
print(f”nAssistant: {response}”)
print(“nn📝 Instance 2: Code Technology”)
print(“-” * 50)
messages = [
]
response = generate_response(messages, max_new_tokens=300)
print(f”Consumer: {messages[0][‘content’]}”)
print(f”nAssistant: {response}”)
print(“nn📝 Instance 3: Inventive Writing”)
print(“-” * 50)
messages = [
{“role”: “user”, “content”: “Write a haiku about artificial intelligence.”}
]
response = generate_response(messages, max_new_tokens=100, temperature=1.0)
print(f”Consumer: {messages[0][‘content’]}”)
print(f”nAssistant: {response}”)
We arrange the complete Colab atmosphere required to run GPT-OSS correctly and confirm that the system has a appropriate GPU with sufficient VRAM. We set up the core libraries, examine the PyTorch and Transformers variations, and ensure that the runtime is appropriate for loading an open-weight mannequin like gpt-oss-20b. We then load the tokenizer, initialize the mannequin with the right technical configuration, and run just a few fundamental inference examples to substantiate that the open-weight pipeline is working finish to finish.
print(“n” + “=” * 70)
print(“🧠 PART 4: Configurable Reasoning Effort”)
print(“=” * 70)
print(“””
GPT-OSS helps completely different reasoning effort ranges:
• LOW – Fast, concise solutions (fewer tokens, sooner)
• MEDIUM – Balanced reasoning and response
• HIGH – Deep pondering with full chain-of-thought
The reasoning effort is managed via system prompts and era parameters.
“””)
class ReasoningEffortController:
“””
Controls reasoning effort ranges for gpt-oss generations.
“””
EFFORT_CONFIGS = {
“low”: {
“system_prompt”: “You’re a useful assistant. Be concise and direct.”,
“max_tokens”: 200,
“temperature”: 0.7,
“description”: “Fast, concise solutions”
},
“medium”: {
“system_prompt”: “You’re a useful assistant. Suppose via issues step-by-step and supply clear, well-reasoned solutions.”,
“max_tokens”: 400,
“temperature”: 0.8,
“description”: “Balanced reasoning”
},
“excessive”: {
“system_prompt”: “””You’re a useful assistant with superior reasoning capabilities.
For complicated issues:
1. First, analyze the issue completely
2. Think about a number of approaches
3. Present your full chain of thought
4. Present a complete, well-reasoned reply
Take your time to suppose deeply earlier than responding.”””,
“max_tokens”: 800,
“temperature”: 1.0,
“description”: “Deep chain-of-thought reasoning”
}
}
def __init__(self, pipeline, tokenizer):
self.pipe = pipeline
self.tokenizer = tokenizer
def generate(self, user_message: str, effort: str = “medium”) -> dict:
“””Generate response with specified reasoning effort.”””
if effort not in self.EFFORT_CONFIGS:
elevate ValueError(f”Effort have to be one in every of: {record(self.EFFORT_CONFIGS.keys())}”)
config = self.EFFORT_CONFIGS[effort]
messages = [
{“role”: “system”, “content”: config[“system_prompt”]},
{“position”: “person”, “content material”: user_message}
]
output = self.pipe(
messages,
max_new_tokens=config[“max_tokens”],
do_sample=True,
temperature=config[“temperature”],
top_p=1.0,
pad_token_id=self.tokenizer.eos_token_id,
)
return {
“effort”: effort,
“description”: config[“description”],
“response”: output[0][“generated_text”][-1][“content”],
“max_tokens_used”: config[“max_tokens”]
}
reasoning_controller = ReasoningEffortController(pipe, tokenizer)
print(f”n🧩 Logic Puzzle: {test_question}n”)
for effort in [“low”, “medium”, “high”]:
outcome = reasoning_controller.generate(test_question, effort)
print(f”━━━ {effort.higher()} ({outcome[‘description’]}) ━━━”)
print(f”{outcome[‘response’][:500]}…”)
print()
print(“n” + “=” * 70)
print(“📋 PART 5: Structured Output Technology (JSON Mode)”)
print(“=” * 70)
import json
import re
class StructuredOutputGenerator:
“””
Generate structured JSON outputs with schema validation.
“””
def __init__(self, pipeline, tokenizer):
self.pipe = pipeline
self.tokenizer = tokenizer
def generate_json(self, immediate: str, schema: dict, max_retries: int = 2) -> dict:
“””
Generate JSON output in accordance with a specified schema.
Args:
immediate: The person’s request
schema: JSON schema description
max_retries: Variety of retries on parse failure
“””
schema_str = json.dumps(schema, indent=2)
system_prompt = f”””You’re a useful assistant that ONLY outputs legitimate JSON.
Your response should precisely match this JSON schema:
{schema_str}
RULES:
– Output ONLY the JSON object, nothing else
– No markdown code blocks (no “`)
– No explanations earlier than or after
– Guarantee all required fields are current
– Use right information sorts as specified”””
messages = [
{“role”: “system”, “content”: system_prompt},
{“role”: “user”, “content”: prompt}
]
for try in vary(max_retries + 1):
output = self.pipe(
messages,
max_new_tokens=500,
do_sample=True,
temperature=0.3,
top_p=1.0,
pad_token_id=self.tokenizer.eos_token_id,
)
response_text = output[0][“generated_text”][-1][“content”]
cleaned = self._clean_json_response(response_text)
attempt:
parsed = json.hundreds(cleaned)
return {“success”: True, “information”: parsed, “makes an attempt”: try + 1}
besides json.JSONDecodeError as e:
if try == max_retries:
return {
“success”: False,
“error”: str(e),
“raw_response”: response_text,
“makes an attempt”: try + 1
}
messages.append({“position”: “assistant”, “content material”: response_text})
messages.append({“position”: “person”, “content material”: f”That wasn’t legitimate JSON. Error: {e}. Please attempt once more with ONLY legitimate JSON.”})
def _clean_json_response(self, textual content: str) -> str:
“””Take away markdown code blocks and additional whitespace.”””
textual content = re.sub(r’^“`(?:json)?s*’, ”, textual content.strip())
textual content = re.sub(r’s*“`$’, ”, textual content)
return textual content.strip()
json_generator = StructuredOutputGenerator(pipe, tokenizer)
print(“n📝 Instance 1: Entity Extraction”)
print(“-” * 50)
entity_schema = {
“identify”: “string”,
“sort”: “string (particular person/firm/place)”,
“description”: “string (1-2 sentences)”,
“key_facts”: [“list of strings”]
}
entity_result = json_generator.generate_json(
“Extract details about: Tesla, Inc.”,
entity_schema
)
if entity_result[“success”]:
print(json.dumps(entity_result[“data”], indent=2))
else:
print(f”Error: {entity_result[‘error’]}”)
print(“nn📝 Instance 2: Recipe Technology”)
print(“-” * 50)
recipe_schema = {
“identify”: “string”,
“prep_time_minutes”: “integer”,
“cook_time_minutes”: “integer”,
“servings”: “integer”,
“issue”: “string (simple/medium/onerous)”,
“substances”: [{“item”: “string”, “amount”: “string”}],
“steps”: [“string”]
}
recipe_result = json_generator.generate_json(
“Create a easy recipe for chocolate chip cookies”,
recipe_schema
)
if recipe_result[“success”]:
print(json.dumps(recipe_result[“data”], indent=2))
else:
print(f”Error: {recipe_result[‘error’]}”)
We construct extra superior era controls by introducing configurable reasoning effort and a structured JSON output workflow. We outline completely different effort modes to fluctuate how deeply the mannequin causes, what number of tokens it makes use of, and the way detailed its solutions are throughout inference. We additionally create a JSON era utility that guides the open-weight mannequin towards schema-like outputs, cleans the returned textual content, and retries when the response shouldn’t be legitimate JSON.
print(“n” + “=” * 70)
print(“💬 PART 6: Multi-turn Conversations with Reminiscence”)
print(“=” * 70)
class ConversationManager:
“””
Manages multi-turn conversations with context reminiscence.
Implements the Concord format sample utilized by gpt-oss.
“””
def __init__(self, pipeline, tokenizer, system_message: str = None):
self.pipe = pipeline
self.tokenizer = tokenizer
self.historical past = []
if system_message:
self.system_message = system_message
else:
self.system_message = “You’re a useful, pleasant AI assistant. Bear in mind the context of our dialog.”
def chat(self, user_message: str, max_new_tokens: int = 300) -> str:
“””Ship a message and get a response, sustaining dialog historical past.”””
messages = [{“role”: “system”, “content”: self.system_message}]
messages.prolong(self.historical past)
messages.append({“position”: “person”, “content material”: user_message})
output = self.pipe(
messages,
max_new_tokens=max_new_tokens,
do_sample=True,
temperature=0.8,
top_p=1.0,
pad_token_id=self.tokenizer.eos_token_id,
)
assistant_response = output[0][“generated_text”][-1][“content”]
self.historical past.append({“position”: “person”, “content material”: user_message})
self.historical past.append({“position”: “assistant”, “content material”: assistant_response})
return assistant_response
def get_history_length(self) -> int:
“””Get variety of turns in dialog.”””
return len(self.historical past) // 2
def clear_history(self):
“””Clear dialog historical past.”””
self.historical past = []
print(“🗑️ Dialog historical past cleared.”)
def get_context_summary(self) -> str:
“””Get a abstract of the dialog context.”””
if not self.historical past:
return “No dialog historical past but.”
abstract = f”Dialog has {self.get_history_length()} turns:n”
for i, msg in enumerate(self.historical past):
position = “👤 Consumer” if msg[“role”] == “person” else “🤖 Assistant”
preview = msg[“content”][:50] + “…” if len(msg[“content”]) > 50 else msg[“content”]
abstract += f” {i+1}. {position}: {preview}n”
return abstract
convo = ConversationManager(pipe, tokenizer)
print(“n🗣️ Multi-turn Dialog Demo:”)
print(“-” * 50)
conversation_turns = [
“Hi! My name is Alex and I’m a software engineer.”,
“I’m working on a machine learning project. What framework would you recommend?”,
“Good suggestion! What’s my name, by the way?”,
“Can you remember what field I work in?”
]
for flip in conversation_turns:
print(f”n👤 Consumer: {flip}”)
response = convo.chat(flip)
print(f”🤖 Assistant: {response}”)
print(f”n📊 {convo.get_context_summary()}”)
print(“n” + “=” * 70)
print(“⚡ PART 7: Streaming Token Technology”)
print(“=” * 70)
from transformers import TextIteratorStreamer
from threading import Thread
import time
def stream_response(immediate: str, max_tokens: int = 200):
“””
Stream tokens as they’re generated for real-time output.
“””
messages = [{“role”: “user”, “content”: prompt}]
inputs = tokenizer.apply_chat_template(
messages,
add_generation_prompt=True,
return_tensors=”pt”
).to(mannequin.gadget)
streamer = TextIteratorStreamer(
tokenizer,
skip_prompt=True,
skip_special_tokens=True
)
generation_kwargs = {
“input_ids”: inputs,
“streamer”: streamer,
“max_new_tokens”: max_tokens,
“do_sample”: True,
“temperature”: 0.8,
“top_p”: 1.0,
“pad_token_id”: tokenizer.eos_token_id,
}
thread = Thread(goal=mannequin.generate, kwargs=generation_kwargs)
thread.begin()
print(“📝 Streaming: “, finish=””, flush=True)
full_response = “”
for token in streamer:
print(token, finish=””, flush=True)
full_response += token
time.sleep(0.01)
thread.be a part of()
print(“n”)
return full_response
print(“n🔄 Streaming Demo:”)
print(“-” * 50)
streamed = stream_response(
“Depend from 1 to 10, with a short remark about every quantity.”,
max_tokens=250
)
We transfer from single prompts to stateful interactions by making a dialog supervisor that shops multi-turn chat historical past and reuses that context in future responses. We exhibit how we preserve reminiscence throughout turns, summarize prior context, and make the interplay really feel extra like a persistent assistant as a substitute of a one-off era name. We additionally implement streaming era so we are able to watch tokens arrive in actual time, which helps us perceive the mannequin’s stay decoding habits extra clearly.
print(“n” + “=” * 70)
print(“🔧 PART 8: Perform Calling / Device Use”)
print(“=” * 70)
import math
from datetime import datetime
class ToolExecutor:
“””
Manages software definitions and execution for gpt-oss.
“””
def __init__(self):
self.instruments = {}
self._register_default_tools()
def _register_default_tools(self):
“””Register built-in instruments.”””
@self.register(“calculator”, “Carry out mathematical calculations”)
def calculator(expression: str) -> str:
“””Consider a mathematical expression.”””
attempt:
allowed_names = {
okay: v for okay, v in math.__dict__.objects()
if not okay.startswith(“_”)
}
allowed_names.replace({“abs”: abs, “spherical”: spherical})
outcome = eval(expression, {“__builtins__”: {}}, allowed_names)
return f”Outcome: {outcome}”
besides Exception as e:
return f”Error: {str(e)}”
@self.register(“get_time”, “Get present date and time”)
def get_time() -> str:
“””Get the present date and time.”””
now = datetime.now()
return f”Present time: {now.strftime(‘%Y-%m-%d %H:%M:%S’)}”
@self.register(“climate”, “Get climate for a metropolis (simulated)”)
def climate(metropolis: str) -> str:
“””Get climate data (simulated).”””
import random
temp = random.randint(60, 85)
circumstances = random.selection([“sunny”, “partly cloudy”, “cloudy”, “rainy”])
return f”Climate in {metropolis}: {temp}°F, {circumstances}”
@self.register(“search”, “Seek for data (simulated)”)
def search(question: str) -> str:
“””Search the online (simulated).”””
return f”Search outcomes for ‘{question}’: [Simulated results – in production, connect to a real search API]”
def register(self, identify: str, description: str):
“””Decorator to register a software.”””
def decorator(func):
self.instruments[name] = {
“operate”: func,
“description”: description,
“identify”: identify
}
return func
return decorator
def get_tools_prompt(self) -> str:
“””Generate instruments description for the system immediate.”””
tools_desc = “You’ve entry to the next instruments:nn”
for identify, software in self.instruments.objects():
tools_desc += f”- {identify}: {software[‘description’]}n”
tools_desc += “””
To make use of a software, reply with:
TOOL:
ARGS:
After receiving the software outcome, present your ultimate reply to the person.”””
return tools_desc
def execute(self, tool_name: str, args: dict) -> str:
“””Execute a software with given arguments.”””
if tool_name not in self.instruments:
return f”Error: Unknown software ‘{tool_name}'”
attempt:
func = self.instruments[tool_name][“function”]
if args:
outcome = func(**args)
else:
outcome = func()
return outcome
besides Exception as e:
return f”Error executing {tool_name}: {str(e)}”
def parse_tool_call(self, response: str) -> tuple:
“””Parse a software name from mannequin response.”””
if “TOOL:” not in response:
return None, None
strains = response.break up(“n”)
tool_name = None
args = {}
for line in strains:
if line.startswith(“TOOL:”):
tool_name = line.change(“TOOL:”, “”).strip()
elif line.startswith(“ARGS:”):
attempt:
args_str = line.change(“ARGS:”, “”).strip()
args = json.hundreds(args_str) if args_str else {}
besides json.JSONDecodeError:
args = {“expression”: args_str} if tool_name == “calculator” else {“question”: args_str}
return tool_name, args
instruments = ToolExecutor()
def chat_with_tools(user_message: str) -> str:
“””
Chat with software use functionality.
“””
system_prompt = f”””You’re a useful assistant with entry to instruments.
{instruments.get_tools_prompt()}
If the person’s request will be answered instantly, accomplish that.
If you might want to use a software, point out which software and with what arguments.”””
messages = [
{“role”: “system”, “content”: system_prompt},
{“role”: “user”, “content”: user_message}
]
output = pipe(
messages,
max_new_tokens=200,
do_sample=True,
temperature=0.7,
pad_token_id=tokenizer.eos_token_id,
)
response = output[0][“generated_text”][-1][“content”]
tool_name, args = instruments.parse_tool_call(response)
if tool_name:
tool_result = instruments.execute(tool_name, args)
messages.append({“position”: “assistant”, “content material”: response})
messages.append({“position”: “person”, “content material”: f”Device outcome: {tool_result}nnNow present your ultimate reply.”})
final_output = pipe(
messages,
max_new_tokens=200,
do_sample=True,
temperature=0.7,
pad_token_id=tokenizer.eos_token_id,
)
return final_output[0][“generated_text”][-1][“content”]
return response
print(“n🔧 Device Use Examples:”)
print(“-” * 50)
tool_queries = [
“What is 15 * 23 + 7?”,
“What time is it right now?”,
“What’s the weather like in Tokyo?”,
]
for question in tool_queries:
print(f”n👤 Consumer: {question}”)
response = chat_with_tools(question)
print(f”🤖 Assistant: {response}”)
print(“n” + “=” * 70)
print(“📦 PART 9: Batch Processing for Effectivity”)
print(“=” * 70)
def batch_generate(prompts: record, batch_size: int = 2, max_new_tokens: int = 100) -> record:
“””
Course of a number of prompts in batches for effectivity.
Args:
prompts: Checklist of prompts to course of
batch_size: Variety of prompts per batch
max_new_tokens: Most tokens per response
Returns:
Checklist of responses
“””
outcomes = []
total_batches = (len(prompts) + batch_size – 1) // batch_size
for i in vary(0, len(prompts), batch_size):
batch = prompts[i:i + batch_size]
batch_num = i // batch_size + 1
print(f” Processing batch {batch_num}/{total_batches}…”)
batch_messages = [
[{“role”: “user”, “content”: prompt}]
for immediate in batch
]
for messages in batch_messages:
output = pipe(
messages,
max_new_tokens=max_new_tokens,
do_sample=True,
temperature=0.7,
pad_token_id=tokenizer.eos_token_id,
)
outcomes.append(output[0][“generated_text”][-1][“content”])
return outcomes
print(“n📝 Batch Processing Instance:”)
print(“-” * 50)
batch_prompts = [
“What is the capital of France?”,
“What is 7 * 8?”,
“Name a primary color.”,
“What season comes after summer?”,
“What is H2O commonly called?”,
]
print(f”Processing {len(batch_prompts)} prompts…n”)
batch_results = batch_generate(batch_prompts, batch_size=2)
for immediate, lead to zip(batch_prompts, batch_results):
print(f”Q: {immediate}”)
print(f”A: {outcome[:100]}…n”)
We prolong the tutorial to incorporate software use and batch inference, enabling the open-weight mannequin to assist extra sensible software patterns. We outline a light-weight software execution framework, let the mannequin select instruments via a structured textual content sample, after which feed the software outcomes again into the era loop to provide a ultimate reply. We additionally add batch processing to deal with a number of prompts effectively, which is beneficial for testing throughput and reusing the identical inference pipeline throughout a number of duties.
print(“n” + “=” * 70)
print(“🤖 PART 10: Interactive Chatbot Interface”)
print(“=” * 70)
import gradio as gr
def create_chatbot():
“””Create a Gradio chatbot interface for gpt-oss.”””
def reply(message, historical past):
“””Generate chatbot response.”””
for user_msg, assistant_msg in historical past:
messages.append({“position”: “person”, “content material”: user_msg})
if assistant_msg:
messages.append({“position”: “assistant”, “content material”: assistant_msg})
messages.append({“position”: “person”, “content material”: message})
output = pipe(
messages,
max_new_tokens=400,
do_sample=True,
temperature=0.8,
top_p=1.0,
pad_token_id=tokenizer.eos_token_id,
)
return output[0][“generated_text”][-1][“content”]
demo = gr.ChatInterface(
fn=reply,
title=”🚀 GPT-OSS Chatbot”,
description=”Chat with OpenAI’s open-weight GPT-OSS mannequin!”,
examples=[
“Explain quantum computing in simple terms.”,
“What are the benefits of open-source AI?”,
“Tell me a fun fact about space.”,
],
theme=gr.themes.Smooth(),
)
return demo
print(“n🚀 Creating Gradio chatbot interface…”)
chatbot = create_chatbot()
print(“n” + “=” * 70)
print(“🎁 PART 11: Utility Helpers”)
print(“=” * 70)
class GptOssHelpers:
“””Assortment of utility features for frequent duties.”””
def __init__(self, pipeline, tokenizer):
self.pipe = pipeline
self.tokenizer = tokenizer
def summarize(self, textual content: str, max_words: int = 50) -> str:
“””Summarize textual content to specified size.”””
messages = [
{“role”: “system”, “content”: f”Summarize the following text in {max_words} words or less. Be concise.”},
{“role”: “user”, “content”: text}
]
output = self.pipe(messages, max_new_tokens=150, temperature=0.5, pad_token_id=self.tokenizer.eos_token_id)
return output[0][“generated_text”][-1][“content”]
def translate(self, textual content: str, target_language: str) -> str:
“””Translate textual content to focus on language.”””
messages = [
{“role”: “user”, “content”: f”Translate to {target_language}: {text}”}
]
output = self.pipe(messages, max_new_tokens=200, temperature=0.3, pad_token_id=self.tokenizer.eos_token_id)
return output[0][“generated_text”][-1][“content”]
def explain_simply(self, idea: str) -> str:
“””Clarify an idea in easy phrases.”””
messages = [
{“role”: “system”, “content”: “Explain concepts simply, as if to a curious 10-year-old. Use analogies and examples.”},
{“role”: “user”, “content”: f”Explain: {concept}”}
]
output = self.pipe(messages, max_new_tokens=200, temperature=0.8, pad_token_id=self.tokenizer.eos_token_id)
return output[0][“generated_text”][-1][“content”]
def extract_keywords(self, textual content: str, num_keywords: int = 5) -> record:
“””Extract key matters from textual content.”””
messages = [
{“role”: “user”, “content”: f”Extract exactly {num_keywords} keywords from this text. Return only the keywords, comma-separated:nn{text}”}
]
output = self.pipe(messages, max_new_tokens=50, temperature=0.3, pad_token_id=self.tokenizer.eos_token_id)
key phrases = output[0][“generated_text”][-1][“content”]
return [k.strip() for k in keywords.split(“,”)]
helpers = GptOssHelpers(pipe, tokenizer)
print(“n📝 Helper Features Demo:”)
print(“-” * 50)
sample_text = “””
Synthetic intelligence has remodeled many industries in recent times.
From healthcare diagnostics to autonomous automobiles, AI programs have gotten
“””
print(“n1️⃣ Summarization:”)
abstract = helpers.summarize(sample_text, max_words=20)
print(f” {abstract}”)
print(“n2️⃣ Easy Clarification:”)
clarification = helpers.explain_simply(“neural networks”)
print(f” {clarification[:200]}…”)
print(“n” + “=” * 70)
print(“✅ TUTORIAL COMPLETE!”)
print(“=” * 70)
print(“””
🎉 You have discovered tips on how to use GPT-OSS on Google Colab!
WHAT YOU LEARNED:
✓ Appropriate mannequin loading (no load_in_4bit – makes use of native MXFP4)
✓ Primary inference with correct parameters
✓ Configurable reasoning effort (low/medium/excessive)
✓ Structured JSON output era
✓ Multi-turn conversations with reminiscence
✓ Streaming token era
✓ Perform calling and gear use
✓ Batch processing for effectivity
✓ Interactive Gradio chatbot
KEY TAKEAWAYS:
• GPT-OSS makes use of native MXFP4 quantization (do not use bitsandbytes)
• Really helpful: temperature=1.0, top_p=1.0
• gpt-oss-20b suits on T4 GPU (~16GB VRAM)
• gpt-oss-120b requires H100/A100 (~80GB VRAM)
• All the time use trust_remote_code=True
RESOURCES:
📚 GitHub: https://github.com/openai/gpt-oss
📚 Hugging Face: https://huggingface.co/openai/gpt-oss-20b
📚 Mannequin Card: https://arxiv.org/abs/2508.10925
📚 Concord Format: https://github.com/openai/concord
📚 Cookbook: https://cookbook.openai.com/matter/gpt-oss
ALTERNATIVE INFERENCE OPTIONS (for higher efficiency):
• vLLM: Manufacturing-ready, OpenAI-compatible server
• Ollama: Simple native deployment
• LM Studio: Desktop GUI software
“””)
if torch.cuda.is_available():
print(f”n📊 Ultimate GPU Reminiscence Utilization:”)
print(f” Allotted: {torch.cuda.memory_allocated() / 1e9:.2f} GB”)
print(f” Reserved: {torch.cuda.memory_reserved() / 1e9:.2f} GB”)
print(“n” + “=” * 70)
print(“🚀 Launch the chatbot by operating: chatbot.launch(share=True)”)
print(“=” * 70)
We flip the mannequin pipeline right into a usable software by constructing a Gradio chatbot interface after which including helper utilities for summarization, translation, simplified clarification, and key phrase extraction. We present how the identical open-weight mannequin can assist each interactive chat and reusable task-specific features inside a single Colab workflow. We finish by summarizing the tutorial, reviewing the important thing technical takeaways, and reinforcing how GPT-OSS will be loaded, managed, and prolonged as a sensible open-weight system.
In conclusion, we constructed a complete hands-on understanding of tips on how to use GPT-OSS as an open-source language mannequin slightly than a black-box endpoint. We loaded the mannequin with the right inference path, avoiding incorrect low-bit loading approaches, and labored via essential implementation patterns, together with configurable reasoning effort, JSON-constrained outputs, Concord-style conversational formatting, token streaming, light-weight software use orchestration, and Gradio-based interplay. In doing so, we noticed the true benefit of open-weight fashions: we are able to instantly management mannequin loading, examine runtime habits, form era flows, and design customized utilities on high of the bottom mannequin with out relying totally on managed infrastructure.
Try the Full Code Implementation. Additionally, be happy to comply with us on Twitter and don’t neglect to affix our 130k+ ML SubReddit and Subscribe to our E-newsletter. Wait! are you on telegram? now you possibly can be a part of us on telegram as properly.
Must companion with us for selling your GitHub Repo OR Hugging Face Web page OR Product Launch OR Webinar and many others.? Join with us
