Why did I always get errors when running the logs demo?

[binarywang]

Processed prompts:   0%|                | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]WARNING 02-13 18:11:07 scheduler.py:895] Input prompt (4738 tokens) is too long and exceeds limit of 512
Processed prompts: 100%|█| 1/1 [00:00<00:00, 1998.24it/s, est. speed input: 11046477.13 toks/s, output: 0.00 toks/
[rank0]: Traceback (most recent call last):
[rank0]:   File "/home/ubuntu/.local/lib/python3.10/site-packages/pydantic/main.py", line 1187, in parse_raw
[rank0]:     obj = parse.load_str_bytes(
[rank0]:   File "/home/ubuntu/.local/lib/python3.10/site-packages/pydantic/deprecated/parse.py", line 49, in load_str_bytes
[rank0]:     return json_loads(b)  # type: ignore
[rank0]:   File "/usr/lib/python3.10/json/__init__.py", line 346, in loads
[rank0]:     return _default_decoder.decode(s)
[rank0]:   File "/usr/lib/python3.10/json/decoder.py", line 337, in decode
[rank0]:     obj, end = self.raw_decode(s, idx=_w(s, 0).end())
[rank0]:   File "/usr/lib/python3.10/json/decoder.py", line 355, in raw_decode
[rank0]:     raise JSONDecodeError("Expecting value", s, err.value) from None
[rank0]: json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

[rank0]: During handling of the above exception, another exception occurred:

[rank0]: Traceback (most recent call last):
[rank0]:   File "/home/ubuntu/dottxt-ai-logs/example.py", line 85, in <module>
[rank0]:     results = parser.analyze_logs(
[rank0]:   File "/home/ubuntu/dottxt-ai-logs/stressed.py", line 380, in analyze_logs
[rank0]:     analysis = self.logger(prompt, max_tokens=self.token_max)
[rank0]:   File "/home/ubuntu/.local/lib/python3.10/site-packages/outlines/generate/api.py", line 511, in __call__
[rank0]:     return format(completions)
[rank0]:   File "/home/ubuntu/.local/lib/python3.10/site-packages/outlines/generate/api.py", line 497, in format
[rank0]:     return self.format_sequence(sequences)
[rank0]:   File "/home/ubuntu/.local/lib/python3.10/site-packages/outlines/generate/json.py", line 50, in <lambda>
[rank0]:     generator.format_sequence = lambda x: schema_object.parse_raw(x)
[rank0]:   File "/home/ubuntu/.local/lib/python3.10/site-packages/pydantic/main.py", line 1214, in parse_raw
[rank0]:     raise pydantic_core.ValidationError.from_exception_data(cls.__name__, [error])
[rank0]: pydantic_core._pydantic_core.ValidationError: 1 validation error for LogAnalysis
[rank0]: __root__
[rank0]:   Expecting value: line 1 column 1 (char 0) [type=value_error.jsondecode, input_value='', input_type=str]

[just-cameron]

The context length you have is 512, whereas it is 32k in the original demo:

https://github.com/dottxt-ai/demos/blob/main/logs%2Fexample.py#L36

Have you changed the model or the vLLM settings?

[binarywang]

yes， I have changed it back to 32k，but got anothor error：

<LOGS END><｜Assistant｜>
Processed prompts:   0%|             | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s][rank0]: Traceback (most recent call last):
[rank0]:   File "/home/milos/ai-logs/STRESSED-logs/example.py", line 85, in <module>
[rank0]:     results = parser.analyze_logs(
[rank0]:   File "/home/milos/ai-logs/STRESSED-logs/stressed.py", line 380, in analyze_logs
[rank0]:     analysis = self.logger(prompt, max_tokens=self.token_max)
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/outlines/generate/api.py", line 503, in __call__
[rank0]:     completions = self.model.generate(
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/outlines/models/vllm.py", line 105, in generate
[rank0]:     results = self.model.generate(
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/utils.py", line 1063, in inner
[rank0]:     return fn(*args, **kwargs)
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/llm.py", line 353, in generate
[rank0]:     outputs = self._run_engine(use_tqdm=use_tqdm)
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/llm.py", line 879, in _run_engine
[rank0]:     step_outputs = self.llm_engine.step()
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/llm_engine.py", line 1389, in step
[rank0]:     outputs = self.model_executor.execute_model(
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/executor/gpu_executor.py", line 134, in execute_model
[rank0]:     output = self.driver_worker.execute_model(execute_model_req)
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/worker/worker_base.py", line 303, in execute_model
[rank0]:     inputs = self.prepare_input(execute_model_req)
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/worker/worker_base.py", line 291, in prepare_input
[rank0]:     return self._get_driver_input_and_broadcast(execute_model_req)
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/worker/worker_base.py", line 253, in _get_driver_input_and_broadcast
[rank0]:     self.model_runner.prepare_model_input(
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/worker/model_runner.py", line 1586, in prepare_model_input
[rank0]:     model_input = self._prepare_model_input_tensors(
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/worker/model_runner.py", line 1196, in _prepare_model_input_tensors
[rank0]:     return builder.build()  # type: ignore
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/worker/model_runner.py", line 867, in build
[rank0]:     attn_metadata = self.attn_metadata_builder.build(
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/attention/backends/utils.py", line 215, in build
[rank0]:     input_block_tables[i, :len(block_table)] = block_table
[rank0]: ValueError: could not broadcast input array from shape (513,) into shape (512,)
Processed prompts:   0%|             | 0/1 [00:36<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]

[binarywang]

@cpfiffer any ideas on this？

[just-cameron]

Could you show us more information about your environment + model? I'm not sure how much you've changed the underlying model. I'm quite surprised this is happening.

[binarywang]

Could you show us more information about your environment + model? I'm not sure how much you've changed the underlying model. I'm quite surprised this is happening.

Thanks, I am using deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B, and below is the info showing in nvitop

Thu Feb 20 16:27:46 2025                        (Press h for help or q to quit)
╒═════════════════════════════════════════════════════════════════════════════╕
│ NVITOP 1.4.2       Driver Version: 525.60.11      CUDA Driver Version: 12.0 │
├───────────────────────────────┬──────────────────────┬──────────────────────┤
│ GPU  Name        Persistence-M│ Bus-Id        Disp.A │ Volatile Uncorr. ECC │
│ Fan  Temp  Perf  Pwr:Usage/Cap│         Memory-Usage │ GPU-Util  Compute M. │
╞═══════════════════════════════╪══════════════════════╪══════════════════════╪═══════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╕
│   0  Tesla T4            Off  │ 00000000:09:00.0 Off │                    0 │ MEM: ███████████████████████████████████████████████████████████████████████████████████████▍ 80.9%                   │
│ N/A   72C    P0     32W / 70W │  12429MiB / 15360MiB │      0%      Default │

[just-cameron]

1. Can you give us your Outlines version?
2. Is the nvitop output from when the model is running, or just in memory? You're pretty close to your memory limit and I'd be worried about it maxing out. Though, this doesn't really seem like an obvious memory issue to me.

[binarywang]

1. Can you give us your Outlines version?
2. Is the nvitop output from when the model is running, or just in memory? You're pretty close to your memory limit and I'd be worried about it maxing out. Though, this doesn't really seem like an obvious memory issue to me.

1. outline versions are unchanged, as below:

outlines==0.0.46
outlines_core==0.1.14

2. Yes , the nvitop output from when the model is running.

[just-cameron]

I updated the requirements.txt file for the logs demo. Can you try installing from there? The outlines version here is quite old.