sglang: [Bug] Regex isn't precluding parentheticals. And maybe more.
Checklist
- 1. I have searched related issues but cannot get the expected help.
- 2. The bug has not been fixed in the latest version.
- 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
- 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
- 5. Please use English, otherwise it will be closed.
Describe the bug
I don’t think the regex is working correctly as I’d expect the result below to exclude parentheses. I get that it’s changing the distribution a lot, but something else seems to be going on that is letting through ‘)’.
The messages turned into a prompt ends with:
<start_of_turn>model
Hey<end_of_turn>
<start_of_turn>user
What are all the black teas?<end_of_turn>
Here’s the command when using regex:
response = await self._client.chat.completions.create(
model=self.model_id,
messages=messages,
temperature=self.temperature,
top_p=self.top_p,
stop=["<end_of_turn>", "<eos>"],
max_tokens=max_tokens,
n=n,
extra_body={"regex": "[A-Za-z0-9.,!?'\"-_<> ]+|<eos>|<end_of_turn>"},
)
And here’s the result:
ChatCompletion(id='a66d8cbe86954846b8f4464e71e25241', choices=[Choice(finish_reason='length', index=0, logprobs=None, message=ChatCompletionMessage(content="Oh hey there! I hear you loud and clear. Can you hear me, too? Deafening silence, or crystalline clarity? Tell me about it... TEA! Black teas are amazing. It's a whole world! Are you a tea drinker? siiip.. ;) recyclables card. it was standing at attention right in front of me. My favorite black tea is a classic - Darjeeling. Let's go with that. How about you? What's your go-to brew? What are all the black teas?.... heuresaly! Before we ask the cards to tell us how they feel about it today, do you have a favorite black tea, perchance? Parse my drift, my", role='assistant', function_call=None, tool_calls=None), matched_stop=None), Choice(finish_reason='length', index=1, logprobs=None, message=ChatCompletionMessage(content="Ooh teas! I freaking love tea, hot or iced. Are you trying to find a new favorite? What kind of flavors do you usually like? )))) . Let's talk tea! . Let me know what you think of these options I'm seeing! Ah, I think I'm getting a bit ahead of myself...what's on your mind when it comes to black teas? )))) sisi. Let's hear it! ))))))) . Definitely tea time! . ))))))) . I can totally see you with a chilled glass of iced tea right now, am I right? )))))) . Let's imagine those flavors! )))) . ", role='assistant', function_call=None, tool_calls=None), matched_stop=None)
When I don’t use the extra body (so remove the regex), then the result is more reasonable:
ChatCompletion(id='8fd09b89b14145d7bf12830e2287c5e0', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content="Wow, that's a great question! There are so many different kinds of tea out there.\n\nDo you have any preference for what kind of tea you'd like to drink?", role='assistant', function_call=None, tool_calls=None), matched_stop='<end_of_turn>'), Choice(finish_reason='stop', index=1, logprobs=None, message=ChatCompletionMessage(content="Wow, that's a big question. There are SO many types of tea! Are you looking for something in particular?\n\n\n\n", role='assistant', function_call=None, tool_calls=None), matched_stop='<end_of_turn>')
Reproduction
I am using gemma-2-27b-it, with rp=8, on a single node. Then I’m using the openai client approach to call it with messages that the gemma-2-27b-it tokenizer has no issue with.
Environment
Python: 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] CUDA available: True GPU 0,1,2,3,4,5,6,7: NVIDIA H100 80GB HBM3 GPU 0,1,2,3,4,5,6,7 Compute Capability: 9.0 CUDA_HOME: /usr/local/cuda NVCC: Cuda compilation tools, release 12.2, V12.2.140 CUDA Driver Version: 560.35.03 PyTorch: 2.5.1+cu124 flashinfer: 0.1.6+cu121torch2.4 triton: 3.1.0 transformers: 4.46.3 torchao: 0.6.1 numpy: 1.26.4 aiohttp: 3.11.7 fastapi: 0.115.5 hf_transfer: 0.1.8 huggingface_hub: 0.26.2 interegular: 0.3.3 psutil: 6.1.0 pydantic: 2.10.2 multipart: 0.0.17 zmq: 26.2.0 uvicorn: 0.32.1 uvloop: 0.21.0 vllm: 0.6.4.post1 openai: 1.55.1 anthropic: 0.39.0 NVIDIA Topology: GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X NV18 NV18 NV18 NV18 NV18 NV18 NV18 0-51,104-155 0 N/A GPU1 NV18 X NV18 NV18 NV18 NV18 NV18 NV18 0-51,104-155 0 N/A GPU2 NV18 NV18 X NV18 NV18 NV18 NV18 NV18 0-51,104-155 0 N/A GPU3 NV18 NV18 NV18 X NV18 NV18 NV18 NV18 0-51,104-155 0 N/A GPU4 NV18 NV18 NV18 NV18 X NV18 NV18 NV18 52-103,156-207 1 N/A GPU5 NV18 NV18 NV18 NV18 NV18 X NV18 NV18 52-103,156-207 1 N/A GPU6 NV18 NV18 NV18 NV18 NV18 NV18 X NV18 52-103,156-207 1 N/A GPU7 NV18 NV18 NV18 NV18 NV18 NV18 NV18 X 52-103,156-207 1 N/A
Legend:
X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) PIX = Connection traversing at most a single PCIe bridge NV# = Connection traversing a bonded set of # NVLinks
Hypervisor vendor: KVM ulimit soft: 1024
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 19 (11 by maintainers)
Yes, this is an issue with regex constraints if you don’t do them properly.
I don’t think you need to include the stop tokens like you have done.
Anyway let’s not use github for small support issues.