Skip to content

Commit ac3c3b8

Browse files
authored
Merge pull request llm-d-incubation#207 from waltforme/launcher
Update howto.md for the launcher
2 parents cccb44f + b4ce7e2 commit ac3c3b8

1 file changed

Lines changed: 29 additions & 3 deletions

File tree

inference_server/launcher/howto.md

Lines changed: 29 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -17,18 +17,30 @@ Start the service:
1717
uvicorn --port 8001 --log-level info launcher:app
1818
```
1919

20-
Send commands (sung HTTPie or cURL):
20+
Send commands (using HTTPie or cURL) with the following payload:
2121
```json
2222
{
2323
"options": "--model TinyLlama/TinyLlama-1.1B-Chat-v1.0 --port 8005",
24-
"env_var": {
24+
"env_vars": {
2525
"VLLM_USE_V1": "1",
2626
"VLLM_LOGGING_LEVEL": "DEBUG"
2727
}
2828
}
2929
```
30+
For example, if using cURL, the command will be something like
31+
```shell
32+
curl -X POST http://localhost:8001/v2/vllm/instances \
33+
-H "Content-Type: application/json" \
34+
-d '{
35+
"options": "--model TinyLlama/TinyLlama-1.1B-Chat-v1.0 --port 8005",
36+
"env_vars": {
37+
"VLLM_USE_V1": "1",
38+
"VLLM_LOGGING_LEVEL": "DEBUG"
39+
}
40+
}'
41+
```
3042

31-
The vLLM will start serving and you can request generations:
43+
The vLLM will start serving and you can request generations with the following payload:
3244
```json
3345
{
3446
"model": "TinyLlama/TinyLlama-1.1B-Chat-v1.0",
@@ -40,3 +52,17 @@ The vLLM will start serving and you can request generations:
4052
"max_tokens": 100
4153
}
4254
```
55+
For example, if using cURL, the command will be something like
56+
```shell
57+
curl -X POST http://localhost:8005/v1/chat/completions \
58+
-H "Content-Type: application/json" \
59+
-d '{
60+
"model": "TinyLlama/TinyLlama-1.1B-Chat-v1.0",
61+
"messages": [
62+
{"role": "system", "content": "You are a helpful assistant."},
63+
{"role": "user", "content": "Tell me a joke about AI."}
64+
],
65+
"temperature": 0.7,
66+
"max_tokens": 100
67+
}'
68+
```

0 commit comments

Comments
 (0)