@@ -17,18 +17,30 @@ Start the service:
1717uvicorn --port 8001 --log-level info launcher:app
1818```
1919
20- Send commands (sung HTTPie or cURL):
20+ Send commands (using HTTPie or cURL) with the following payload :
2121``` json
2222{
2323 "options" : " --model TinyLlama/TinyLlama-1.1B-Chat-v1.0 --port 8005" ,
24- "env_var " : {
24+ "env_vars " : {
2525 "VLLM_USE_V1" : " 1" ,
2626 "VLLM_LOGGING_LEVEL" : " DEBUG"
2727 }
2828}
2929```
30+ For example, if using cURL, the command will be something like
31+ ``` shell
32+ curl -X POST http://localhost:8001/v2/vllm/instances \
33+ -H " Content-Type: application/json" \
34+ -d ' {
35+ "options": "--model TinyLlama/TinyLlama-1.1B-Chat-v1.0 --port 8005",
36+ "env_vars": {
37+ "VLLM_USE_V1": "1",
38+ "VLLM_LOGGING_LEVEL": "DEBUG"
39+ }
40+ }'
41+ ```
3042
31- The vLLM will start serving and you can request generations:
43+ The vLLM will start serving and you can request generations with the following payload :
3244``` json
3345{
3446 "model" : " TinyLlama/TinyLlama-1.1B-Chat-v1.0" ,
@@ -40,3 +52,17 @@ The vLLM will start serving and you can request generations:
4052 "max_tokens" : 100
4153}
4254```
55+ For example, if using cURL, the command will be something like
56+ ``` shell
57+ curl -X POST http://localhost:8005/v1/chat/completions \
58+ -H " Content-Type: application/json" \
59+ -d ' {
60+ "model": "TinyLlama/TinyLlama-1.1B-Chat-v1.0",
61+ "messages": [
62+ {"role": "system", "content": "You are a helpful assistant."},
63+ {"role": "user", "content": "Tell me a joke about AI."}
64+ ],
65+ "temperature": 0.7,
66+ "max_tokens": 100
67+ }'
68+ ```
0 commit comments