Archives de l’auteur : Frederic

Ollama n’utilise pas le GPU de la carte … Misère

En passant

Répondre

Ma version d’OS est « Ubuntu 22.04.5 LTS« .

Ma version de carte/drivers NVIDIA :

# uname -a
Linux 5.15.0-130-generic #140-Ubuntu SMP Wed Dec 18 17:59:53 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

# nvidia-smi -L
GPU 0: Quadro 4000 (UUID: GPU-13797e5d-a72f-4c72-609f-686fa4a8c956)

# nvidia-smi 
Mon Jan 20 16:41:52 2025       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 390.157                Driver Version: 390.157                   |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Quadro 4000         Off  | 00000000:00:10.0 Off |                  N/A |
| 36%   62C   P12    N/A /  N/A |      1MiB /  1985MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

# cat /proc/driver/nvidia/version
NVRM version: NVIDIA UNIX x86_64 Kernel Module  390.157  Wed Oct 12 09:19:07 UTC 2022
GCC version:  gcc version 11.4.0 (Ubuntu 11.4.0-1ubuntu1~22.04)

# nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2024 NVIDIA Corporation
Built on Tue_Oct_29_23:50:19_PDT_2024
Cuda compilation tools, release 12.6, V12.6.85
Build cuda_12.6.r12.6/compiler.35059454_0

# ubuntu-drivers devices
== /sys/devices/pci0000:00/0000:00:10.0 ==
modalias : pci:v000010DEd000006DDsv000010DEsd00000780bc03sc00i00
vendor   : NVIDIA Corporation
model    : GF100GL [Quadro 4000]
driver   : nvidia-driver-390 - distro non-free recommended
driver   : xserver-xorg-video-nouveau - distro free builtin

Journal de ollama :

# journalctl -u ollama -f --no-pager
ollama[2974]: llama_kv_cache_init: kv_size = 8192, offload = 1, type_k = 'f16', type_v = 'f16', n_layer = 32, can_shift = 1
ollama[2974]: llama_kv_cache_init:        CPU KV buffer size =  1024.00 MiB
ollama[2974]: llama_new_context_with_model: KV self size  = 1024.00 MiB, K (f16):  512.00 MiB, V (f16):  512.00 MiB
ollama[2974]: llama_new_context_with_model:        CPU  output buffer size =     0.56 MiB
ollama[2974]: llama_new_context_with_model:        CPU compute buffer size =   560.01 MiB
ollama[2974]: llama_new_context_with_model: graph nodes  = 1030
ollama[2974]: llama_new_context_with_model: graph splits = 1
ollama[2974]: time=2025-01-20T16:30:12.070Z level=INFO source=server.go:594 msg="llama runner started in 4.28 seconds"

Quand je fais l’installation j’ai bien « NVIDIA GPU installed »

# curl -fsSL https://ollama.com/install.sh | shsh
>>> Cleaning up old version at /usr/local/lib/ollama
>>> Installing ollama to /usr/local
>>> Downloading Linux amd64 bundle
######################################################################## 100,0%
>>> Adding ollama user to render group...
>>> Adding ollama user to video group...
>>> Adding current user to ollama group...
>>> Creating ollama systemd service...
>>> Enabling and starting ollama service...
>>> NVIDIA GPU installed.

J’ai un problème de chargement

ollama[3917]: time=2025-01-20T16:52:30.680Z level=INFO source=routes.go:1238 msg="Listening on 127.0.0.1:11434 (version 0.5.7)"
ollama[3917]: time=2025-01-20T16:52:30.681Z level=INFO source=routes.go:1267 msg="Dynamic LLM libraries" runners="[cuda_v12_avx rocm_avx cpu cpu_avx cpu_avx2 cuda_v11_avx]"
ollama[3917]: time=2025-01-20T16:52:30.681Z level=INFO source=gpu.go:226 msg="looking for compatible GPUs"
ollama[3917]: time=2025-01-20T16:52:30.702Z level=INFO source=gpu.go:630 msg="Unable to load cudart library /usr/lib/x86_64-linux-gnu/libcuda.so.390.157: symbol lookup for cuDeviceGetUuid failed: /usr/lib/x86_64-linux-gnu/libcuda.so.390.157: undefined symbol: cuDeviceGetUuid"
ollama[3917]: time=2025-01-20T16:52:30.741Z level=INFO source=gpu.go:392 msg="no compatible GPUs were discovered"
ollama[3917]: time=2025-01-20T16:52:30.742Z level=INFO source=types.go:131 msg="inference compute" id=0 library=cpu variant=avx compute="" driver=0.0 name="" total="61.4 GiB" available="59.4 GiB"

A suivre …

Proxmox : Resize disk on Ubuntu 22

En passant

Répondre

Passage de 98Go à 392G, sans aucun problème.

La première étape se fait via l’IHM de Proxmox, ensuite il faut lancer ses commandes sur Ubuntu.

fdisk -l

Disk /dev/sda: 400 GiB, 429496729600 bytes, 838860800 sectors
Disk model: QEMU HARDDISK
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 0588156E-1871-4A3D-900F-4C8C2758E02E

Device Start End Sectors Size Type
/dev/sda1 2048 4095 2048 1M BIOS boot
/dev/sda2 4096 4198399 4194304 2G Linux filesystem
/dev/sda3 4198400 838860766 834662367 398G Linux filesystem

Disk /dev/mapper/ubuntu--vg-ubuntu--lv: 100 GiB, 107374182400 bytes, 209715200 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

df -h

Filesystem Size Used Avail Use% Mounted on
tmpfs 6,2G 1,2M 6,2G 1% /run
/dev/mapper/ubuntu--vg-ubuntu--lv 98G 89G 4,2G 96% /
tmpfs 31G 4,0K 31G 1% /dev/shm
tmpfs 5,0M 0 5,0M 0% /run/lock

sudo pvdisplay

--- Physical volume ---
PV Name /dev/sda3
VG Name ubuntu-vg
PV Size <398,00 GiB / not usable 16,50 KiB
Allocatable yes
PE Size 4,00 MiB
Total PE 101887
Free PE 76287
Allocated PE 25600
PV UUID kJRjOE-1iPT-CVJQ-7QyB-c8I2-ndQQ-Uzi9VE

pvresize /dev/sda3

Physical volume "/dev/sda3" changed
1 physical volume(s) resized or updated / 0 physical volume(s) not resize

sudo lvextend -l +100%FREE /dev/ubuntu-vg/ubuntu-lv

Size of logical volume ubuntu-vg/ubuntu-lv changed from 100,00 GiB (25600 extents) to <398,00 GiB (101887 extents).
Logical volume ubuntu-vg/ubuntu-lv successfully resized.

sudo lvdisplay

--- Logical volume ---
LV Path /dev/ubuntu-vg/ubuntu-lv
LV Name ubuntu-lv
VG Name ubuntu-vg
LV UUID l8Obv4-PXVy-VEsm-db9B-5yZ8-Ybmi-010JO9
LV Write Access read/write
LV Creation host, time ubuntu-server, 2024-02-08 09:41:27 +0000
LV Status available
# open 1
LV Size <398,00 GiB
Current LE 101887
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:0

sudo resize2fs /dev/ubuntu-vg/ubuntu-lv

resize2fs 1.46.5 (30-Dec-2021)
Filesystem at /dev/ubuntu-vg/ubuntu-lv is mounted on /; on-line resizing required
old_desc_blocks = 13, new_desc_blocks = 50
The filesystem on /dev/ubuntu-vg/ubuntu-lv is now 104332288 (4k) blocks long.

df -h

Filesystem Size Used Avail Use% Mounted on
tmpfs 6,2G 1,2M 6,2G 1% /run
/dev/mapper/ubuntu--vg-ubuntu--lv 392G 89G 286G 24% /
tmpfs 31G 4,0K 31G 1% /dev/shm
tmpfs 5,0M 0 5,0M 0% /run/lock

Proxmox/NVIDIA : Quadro 4000 en mode PCI Passthrough … ca marche !

En passant

Répondre

Configuration :

  • Proxmox : 8.3.2
  • Proxmox kernel : 6.8.12-5-pve
  • VM : Ubuntu 22.04.5 LTS
  • VM kernel : 5.15.0-130-generic

Installation :

# sudo add-apt-repository ppa:graphics-drivers/ppa --yes
# sudo apt update
# sudo apt install nvidia-cuda-toolkit
# sudo apt install nvidia-driver-390
...
reboot
...
# nvidia-smi       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 390.157                Driver Version: 390.157                   |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Quadro 4000         Off  | 00000000:00:10.0 Off |                  N/A |
| 36%   61C   P12    N/A /  N/A |      1MiB /  1985MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

Le driver 4xx ne fonctionne pas avec la « Quadro 4000 » .

Pour finir, blocage des mises à jours

# dpkg-query -W --showformat='${Package} ${Status}\n' | grep -v deinstall | awk '{ print $1 }' | \
    grep -E 'nvidia.*-[0-9]+$' | \
    xargs -r -L 1 sudo apt-mark hold
libnvidia-cfg1-390 passé en figé (« hold »).
libnvidia-common-390 passé en figé (« hold »).
libnvidia-compute-390 passé en figé (« hold »).
libnvidia-decode-390 passé en figé (« hold »).
libnvidia-encode-390 passé en figé (« hold »).
libnvidia-extra-470 passé en figé (« hold »).
libnvidia-fbc1-390 passé en figé (« hold »).
libnvidia-gl-390 passé en figé (« hold »).
libnvidia-ifr1-390 passé en figé (« hold »).
nvidia-compute-utils-390 passé en figé (« hold »).
nvidia-dkms-390 passé en figé (« hold »).
nvidia-driver-390 passé en figé (« hold »).
nvidia-kernel-common-390 passé en figé (« hold »).
nvidia-kernel-source-390 passé en figé (« hold »).
nvidia-utils-390 passé en figé (« hold »).
xserver-xorg-video-nvidia-390 passé en figé (« hold »).

Munin : Correction de problèmes sur les plugins mysqls

En passant

Répondre

Pour tester manuellement j’ai fait :

/usr/sbin/munin-run --debug mysql_
/usr/sbin/munin-run --debug mysql_isam_space_

j’ai pu voir qu’il me manquait des installations en Perl

apt-get install -y libcache-cache-perl
apt-get install libdbd-mysql-perl
apt-get install libgd-gd2-perl
apt-get install libgd-graph-perl