Proxmox

I have a 3 node HA cluster of enterprise hardware which runs a small home business and a bunch of personal VM's backed by iSCSI ZFS LUNS. It's extremely underutilized at this point, but I'm looking to expand my usage.

I have a bit of money to spend on upgrades, and I am looking into the feasibility of procuring some GPU's. My budget is in the neighborhood of $600-1000. The main factor is I have 3 hosts. Because I have HA (and have experimented with ProxLB, which I am not using right now) all of my VM's are built as if they can freely migrate, and that's been fine so far, but all my reading about adding GPU's has left me more confused than when I started. As for starting goals, I want to add GPU transcoding to my Jellyfin instance, dedicate a GPU to immich for doing ML on my library, add a GPU to a VDI VM I use as a virtual workspace, and potentially use local language models for some development which is not yet defined.

I've read a ton on using nvidia GPU's with vGPU's, but I have a lot of concerns. I do not have the funds to buy an actual vgpu license, and doing the fastapi-dls thing feels a little too unreliable to justify a large spend. A P40 would be in the budget range, but without tensor cores and with the licensing complexity I'm not sure it's a good investment. Additionally, the documentation on how that would work with HA hasn't clicked; I don't even see HA mentioned here: https://pve.proxmox.com/wiki/NVIDIA_vGPU_on_Proxmox_VE . Most threads on using non-nvidia GPU's has led me to believe it isn't really worth it.

Another option I though of was using multiple lower-cost GPU's (like a Tesla P4) and just assign full cards to the VM's; my reading though is I'm going to break things with HA, and I'll have to use resource mapping which is more complex and outside my comfort zone: https://pve.proxmox.com/wiki/QEMU/KVM_Virtual_Machines#resource_mapping I also don't have the tensor cores with this option.

It seems like I have a lot of options, but at this point I'm not sure what's the best way to proceed. I don't want to spend $600 and end up with something causing more headaches than it's bringing value. I'd like to hear people's advice, experience, and recommendations especially if there's something I'm missing here that my reading hasn't uncovered.

Bought two Teamgroup 2TB device with SLC "crash". The cache was so tiny it wore out with almost no usage from repeated writes.

Just a warning.

Kein #backup ? Keine Gnade!

Nach diesem Motto die freundliche Erinnerung sich um das Thema auch im privaten Umfeld zu kümmern.

Mit @proxmox ist es aber auch sehr einfach, dafür zu sorgen, dass man die #VM s im Zweifel wieder auf die Beine gekommt.

Ja, ein ordentlicher #Cluster wäre auch eine Lösung. Der kommt auch, aber so geht es auch.

Als 2nd Stage gehts von der Lucy per #veeam auf die USB Platte und die pendelt zum Wohnwagen.

#proxmox #homeassistant

Upgrades to @debian Trixie 13.2 on Lenovo T470 and @proxmox VM successful.

I'm running Proxmox 9 on a server with currently a single LXC running. On no apparent time schedule, within a day but frequently within an hour, the host will stop responding to SSH, pings and the web UI stops updating. However, the LXC continues to run happily. The only way to bring the host back up is to power cycle the server.

Has anyone got any troubleshooting tips?

Sick burn, @proxmox, damn :D

"It is so intuitive that even the helpdesk could perform recoveries."

cross-posted from: https://lemmy.world/post/37886983

Searching for "proxmox releases" in ecosia I accidentally ended up on the site

proxmax (dot) org/download/

Which I wasn't able to download anything from but still looks shady AF.

Stay safe <3

Here is an as rare as great tutorial to use @proxmox to build a small #Kubernetes cluster (#K3S on #LXC) : https://garrettmills.dev/blog/2022/04/18/Rancher-K3s-Kubernetes-on-Proxmox-Container/

I have a room with 30 thinclients currently running Windows with the possibility to open RDP sessions.

The current setup using VMWare is slow even for 2D content. That is why I want to replace it with Proxmox. What can I expect? I suspect the current setup is using SAN, I want to go ZFS on local drives.

I experimented in a homelab on KVM to see how fast the VMs can become. With 8 cores Google Earth becomes somewhat usable over RDP. But imagine 30 students using it on the same VM. The VM is Debian 13 btw.

I also experiemented with spice and 3D acceleration, but it works only locally and does not support multiple logins. What other options do I have. Even when I setup the VM to use virgl it uses software rendering over RDP. I thought of replacing the Windows on the thinclients with Linux, but then I would need individual VMs for every student and a secure spice session. Is that even possible? I would need a potent GPU in the server, maybe more than one. Is a 64 core CPU and 512 GB RAM enough for 30 students?

I've read that proxmox uses temporary .vv files for noVNC in the browser. I hope this can be setup permanently to be accessible over the network.

Any advice or new ideas are welcome!

Can someone help me understand why I can't see one of my NFS shares when installing a VM.

If I choose to install a container it ends up in the proper location on my nas with all the others, but if I choose a VM then it wants to install in another share that is used for other purposes. In the 'node' view I can access the console and see both NFS shares.

I'm sure I'm doing something wrong, but I don't know what.

Hello fellow Proxmox enjoyers!

I have questions regarding the ZFS disk IO stats and hope you all may be able to help me understand.

Setup (hardware, software)

I have Proxmox VE installed on a ZFS mirror (2x 500 GB M.2 PCIe SSD) rpool . The data (VMs, disks) resides on a seperate ZFS RAID-Z1 (3x 4TB SATA SSD) data_raid.

I use ~2 TB of all that, 1.6 TB being data (movies, videos, music, old data + game setup files, ...).

I have 6 VMs, all for my use alone, so there's not much going on there.

Question 1 - costant disk write going on?

I have a monitoring setup (CheckMK) to monitor my server and VMs. This monitoring reports a constant write IO operation for the disks, ongoing, without any interruption, of 20+ MB/s.

I think the monitoring gets the data from zpool iostat, so I watched it with watch -n 1 'sudo zpool iostat', but the numbers didn't seem to change.

It has been the exact same operations and bandwidth read / write for the last minute or so (after taking a while for writing this, it now lists 543 read ops instead of 545).

Every 1.0s: sudo zpool iostat

              capacity     operations     bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
data_raid   2.29T  8.61T    545    350  17.2M  21.5M
rpool       4.16G   456G      0     54  8.69K  2.21M
----------  -----  -----  -----  -----  -----  -----

The same happens if I use -lv or -w flags for zpool iostat.

So, are there really constantly 350 write operations going on? Or does it just not update the IO stats all too often?

Question 2 - what about disk longevity?

This isn't my first homelab-setup, but it is my first own ZFS- and RAID-setup. If somebody has any SSD-RAID or SSD-ZFS experiences to share, I'd like to hear them.

The disks I'm using are:

• 3x Samsung SSD 870 EVO 4TB for data_raid
• 2x Samsung SSD 980 500GB M.2 for rpool

Best regards from a fellow rabbit-hole-enjoyer.

I recently installed proxmox on two 7050 micro pcs, and set up a third box with PBS.

I had planned to set up a proxmox HA pair with the PBS providing quorum.

After digging into this now I'm questioning doing this at all.

I have some docker instances that are monitoring things with node-red.

I'm thinking about setting up home assistant for controlling stuff.

Is messing with proxmox HA worth it?

The solution seems like an after thought, at least from an installation and configuration aspect.

Anyone using HA?

I recently installed proxmox on two 7050 micro pcs, and set up a third box with PBS.

Within proxmox is it best to run docker in a LXC or a VM ?

Should I run one docker image per instance, or all in the same instance ?

I would like to back this stuff up to PBS. Is this a factor in this decision?

I have plex, node-red, influxdb, mosquitto and grafana dockers on an old pc that I wish to rehost to proxmox.

OK, this is really annoying...

I have an intel based PC that I use as a server that is sitting under the lounge TV that I was using to drive the TV with a linux VM using pci pass-through of the iGPU.

Every thing was working well, then all of a sudden there was no output to the TV. The iGPU still works fine during boot.

I have just updated proxmox to the latest, and no better results.

I sometimes will get it to work on a new install of a vm, but on reboot of the vm it all stops.

Any thoughts on where/what to look at?

For the love of god, wait a couple releases before upgrading, or trust your backups.

I am having issues passing through my new GPU to my Windows 11 VM. I previously had another AMD GPU hooked up to this same VM without issue. The new 9070 GPU will display the Proxmox terminal just fine, so I'm relatively sure there is no defect with the device.

CPU: AMD Ryzen 9 5900X
GPU: AMD RX 9070 XT
MB: NZXT N7 B550
PWR: 850 Watt

IOMMU: Enabled
Above 4G Decoding: Enabled
Re-Size Bar Support: Enabled

When I try to start the VM, I get this error:

error writing '1' to '/sys/bus/pci/devices/0000:08:00.0/reset': Inappropriate ioctl for device
failed to reset PCI device '0000:08:00.0', but trying to continue as not all devices need a reset
swtpm_setup: Not overwriting existing state file.
kvm: vfio: Unable to power on device, stuck in D3
kvm: vfio: Unable to power on device, stuck in D3
TASK OK

Device:

08:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 48 [RX 9070/9070 XT] [1002:7550] (rev c0) (prog-if 00 [VGA controller])
Subsystem: XFX Limited Device [1eae:8811]

/etc/default/grub:

GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR='lsb_release -i -s 2> /dev/null || echo Debian'
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt rd.driver.pre=vfio-pci kvm.ignore_msrs=1 vfio-pci.ids=1002:7550,1eae:8811 disable_vga=1 disable_idle_d3=1"
GRUB_CMDLINE_LINUX=""

/etc/modules:

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

/etc/modprobe.d/vfio.conf:

options vfio-pci ids=0000:08:00.0,0000:08:00.1
options kvm ignore_msrs=1
options vfio-pci ids=1002:7550,1eae:8811 disable_vga=1 disable_idle_d3=1
softdep radeon pre: vfio vfio_pci

VM.conf: I downloaded and added the ROM to the VM configuration file.

hostpci0: 0000:08:00,pcie=1,x-vga=1,romfile=9070xt.rom

At this point I feel totally lost in the weeds, and would greatly appreciate any help that could point me in the right direction to fix this. Thanks!

UPDATE: the error message changed somewhere between updating the BIOS and removing the rom entry from the vm.conf.

error writing '1' to '/sys/bus/pci/devices/0000:08:00.0/reset': Inappropriate ioctl for device failed to reset PCI device '0000:08:00.0', but trying to continue as not all devices need a reset swtpm_setup: Not overwriting existing state file.
kvm: ../hw/pci/pci.c:1654: pci_irq_handler: Assertion `0 <= irq_num && irq_num < PCI_NUM_PINS' failed. stopping swtpm instance (pid 10043) due to QEMU startup error TASK ERROR: start failed: QEMU exited with code 1

Not finding a lot of info on using Proxmox import wizard to extract VMs from an ESXi free license host. I can add the host as storage and see the VMs but it looks like it's failing afterwards due to possibly some kind of rate limit error? But I know the free version API is either closed or not available. Any ideas? Thanks!

Went through pve update today and had a message saying

Removable bootloader found at '/boot/efi/EFI/BOOT/BOOTX64.efi', but GRUB packages not set up to update it! Run the following command:

echo 'grub-efi-amd64 grub2/force_efi_extra_removable boolean true' | debconf-set-selections -v -u

Then reinstall GRUB with 'apt install --reinstall grub-efi-amd64'

To anyone facing this message: don’t forget! See here in the Proxmox forums

Latest announcements for Proxmox.

I recently needed to set up Tailscale in an AlmaLinux 9 LXC container running on my Proxmox 8.2 server. Following the official instructions from Tailscale's RHEL 9 guide and even trying their Linux install script, I ran into some issues. The main problem turned out to be missing TUN device support in the LXC container.

I had to edit the LXC container configuration on the Proxmox host:

nano /etc/pve/lxc/<container-id>.conf

And add these lines:

lxc.cgroup2.devices.allow: c 10:200 rwm

lxc.mount.entry: /dev/net/tun dev/net/tun none bind,create=file

Here is the article in case is useful for someone else.

Hello!

I've had Linux server/s at home for hobby and fun for years. My current "main" server is quite old, quite slow, and in need of an upgrade.

I acquired a used Thinkstation Mini SFF, which blows the old one out of the water, being like 15% of the size. The heaviest usage is a Plex server for my and my family. Current setup has worked well but it's unable to transcode properly, without a lot of noise and only some codecs/resolutions.

I installed Proxmox on this Thinkstation device and it works very well. I migrated my WP family blog from a Virtualbox to Proxmox VM, worked perfectly.

I then used the helper scripts to install Plex for testing. I got hardware acceleration working with the i5-8500T/UHD 630. Did some testing and it seems to work great.

I have yet to migrate the server and the data. The Thinkstation has ~320gb SSD disk which is way to small for my video collection.

So I'm what I'm wondering, what is the best next step? I can one or more similar devices for cheap/nothing.

I don't really understand the nodes function of Proxmox properly.

Should I:

A) Use my current one as a "main" node, install large SSD/HDD on 1 or 2 similar devices and connect them as storage nodes?

B) Add large USB SSD/HDD to my existing Proxmox device?

C) Setup a simple NAS device and connect Proxmox to that?

D) Something else?

After that's decided, I could then move the Plex installation/setup from my current server to the Proxmox setup.

The usage is not heavy, 3 concurrent streams are rare. Media is currently mostly 1080p.

My main concern is the Plex setup, but if one setup is more futureproof, smarter, handier, cooler, nicer than the other, I'd greatly appreciate your input.

Pretty new to proxmos but I've been using the helper scripts to stand up some services quickly. Is it possible to rename some of those services in the proxmox UI to something other than the default it chooses?