Description
Large language models can run locally with a C and C++ inference engine accelerated through Vulkan-capable GPUs. It helps developers, researchers, and advanced users test compatible LLaMA-style models without sending prompts to an online service.
Model files are separate and can be large. Local AI output can be wrong, biased, or unsafe for decisions that require expertise, and prompts may still expose private data to anyone with access to the machine.