Skip to content

Debugging Applications

Although we would love everything to work out of the box, the world, the cloud, and software are more complicated than that. In this guide we show you how to debug apps on KraftCloud so that you may fix issues, or, in the worst case, be able to provide enough information for us to do so for you.

Debugging the Build, Packaging and Pushing Steps

The kraft cloud deploy command performs several steps on your local device before actually deploying your app to KraftCloud:

  1. Downloads the runtime image (defined in the Kraftfile) from the KraftCloud registry.
  2. Builds the application filesystem using the Dockerfile via BuildKit.
  3. Packages the application filesystem and the runtime in an OCI image.
  4. Pushes the OCI image to the to the KraftCloud registry under your username’s namespace.

In case any of these fail or report issues you can enable kraft debugging by using the --log-level and --log-type flags:

Terminal window
kraft cloud deploy --log-level debug --log-type basic [...]

You should then see debug output for the 4 steps above, similar to:

D kraftkit 0.8.2
D using metro=fra0
D using token=cm9ib3QkcmF6dmFuLnVuaWtyYWZ0LmlvLnVzZXJzLmtyYWZ0Y2xvdWQ6cml4YllzVVA2Z1VldDdZNGp0RWZUY1ZyOEJlRkxFU2s=
D cannot run because: no image specified deployer=image-name
D using deployer=kraftfile-runtime
D querying catalog local=true remote=false
D found 0/66 matching packages in oci catalog
D using packager=kraftfile-runtime
D querying catalog arch=x86_64 local=true plat=kraftcloud remote=false version=latest
D found 0/66 matching packages in oci catalog
D querying catalog arch=x86_64 local=true plat=kraftcloud remote=true version=latest
i found (kraftcloud/x86_64) (6bd4542, 11c196e, 2.0 MB)
D found 1/67 matching packages in oci catalog
D saving manifest digest=sha256:6bd45428b8ecf667954cb8f829ee3f8f6f6f23fa0a0e7dc1aa27fcae39826f89
i building rootfs
D using buildkit addr=docker-container://buildkitd version=v0.12.3

Hopefully this output will give you better insights into what the issue is; if not, please report the issue to us with the output on our Discord server.

Debugging Running Apps

The most direct way to debug apps on KraftCloud is to use the apps’s console output, which may include kernel output. To see it, after starting the KraftCloud instance (via kraft cloud deploy or kraft cloud inst create), use:

Terminal window
kraft cloud inst <instance_name> logs

In case of a crash, you’ll see a full crash output:

Powered by Unikraft Telesto (0.16.2~9c264902)
[ 0.066949] CRIT: [libukvmem] Cannot handle write page fault at 0x1000bb8024 (ec: 0x2): -12
[ 0.067680] CRIT: [libkvmplat] page fault handler returned error: -12
[ 0.068227] CRIT: [libkvmplat] Unhandled page fault vaddr=0x1000bb8024, error code=0x2
[ 0.068910] CRIT: [appelfloader] Unikraft crash - Telesto (0.16.2~9c264902)
[ 0.069503] CRIT: [appelfloader] Thread "python3"@0x4017e4020
[ 0.069993] CRIT: [appelfloader] RIP: 0008:000000100041fdcb
[ 0.070468] CRIT: [appelfloader] RSP: 0010:000000100017f760 EFLAGS: 00010206 ORIG_RAX: 0000000000000002
[ 0.071269] CRIT: [appelfloader] RAX: 0000001000bb8000 RBX: 00000004018012f0 RCX:0000000000000007
[ 0.072040] CRIT: [appelfloader] RDX: 0000000000000000 RSI: aaaaaaaaaaaaaaab RDI:00000010007bccc0
[ 0.072793] CRIT: [appelfloader] RBP: 000000000000001a R08: 0000000000000001 R09:00000010004e2b99
[ 0.073558] CRIT: [appelfloader] R10: bb9e744ce6503a27 R11: 0000000401861290 R12:00000010007bccc0
[ 0.074312] CRIT: [appelfloader] R13: 00000010007bbd48 R14: 000000100017f890 R15:0000000000000038
[ 0.075076] CRIT: [appelfloader] Stack:
[ 0.075406] CRIT: [appelfloader] 100017f760 00 00 00 00 00 00 00 00 |........|
[ 0.076043] CRIT: [appelfloader] 100017f768 a1 01 00 00 00 00 00 00 |........|
[ 0.076680] CRIT: [appelfloader] 100017f770 48 bd 7b 00 10 00 00 00 |H.{.....|
[ 0.077324] CRIT: [appelfloader] 100017f778 e0 92 7a 00 10 00 00 00 |..z.....|
[ 0.077959] CRIT: [appelfloader] 100017f780 48 bd 7b 00 10 00 00 00 |H.{.....|
[ 0.078596] CRIT: [appelfloader] 100017f788 90 f8 17 00 10 00 00 00 |........|
[ 0.079233] CRIT: [appelfloader] 100017f790 38 00 00 00 00 00 00 00 |8.......|
[ 0.079870] CRIT: [appelfloader] 100017f798 72 fc 41 00 10 00 00 00 |r.A.....|
[ 0.080504] CRIT: [appelfloader] Call Trace:
[ 0.080871] CRIT: [appelfloader] [0x000000100041fdcb]
[ 0.081311] CRIT: [appelfloader] Bad frame pointer
It looks like the instance exited fatally. To see more details about why, run:
kraft cloud instance get http-python312-hb7ij

Use the recommended command for detailed output. You get something like:

uuid: 988c3054-09c0-48ef-ac02-877f8352d93f
name: http-python312-hb7ij
private fqdn: http-python312-hb7ij.internal
private ip:
state: stopped
created: 2024-04-18T18:24:12Z
started: 2024-04-18T18:24:12Z
stopped: 2024-04-18T18:24:13Z
start count: 1
restart count: 0
restart attempts:
next restart:
restart policy: never
stop origin: initiated by kernel (----k)
stop reason: Out of memory. Try increasing instance's memory (see -M flag). (i127 PGFAULT ENOMEM)
app exit code:
image: http-python312@sha256:8b92ed612f8450de94355a0a5b8917a710fd6c902c4a7be80ffb3989b5a99023
memory: 128 MiB
args: /usr/bin/python3 /src/
service group:
boot time: 96.03 ms
up time: 166ms

This gives you a more clear reason of why the error occurred, in the stop reason line. In this case, it’s because of insufficient memory.

Some times the stop reason is insufficient for debugging. The next step is to enable debug tracing for the instance.

To do that, in your application of choice in the examples repository, or in an application directory you created, update the runtime entry in the Kraftfile to reference the debug build of the image you use by adding -dbg to the name of the runtime. For example, if you want to run the http-go1.21 example with debug output, update its Kraftfile as follows:

spec: v0.6
runtime: base:latest-dbg
rootfs: ./Dockerfile
cmd: ["/server"]

That is, simply change base:latest to base:latest-dbg. Now we’re ready to re-deploy:

Terminal window
kraft cloud deploy --name http-go-strace -p 443:8080 .

We can now can now inspect the logs as before and get the system call tracing:

Terminal window
kraft cloud inst logs http-go-strace
[ 0.000000] Info: [libkvmplat] Unikraft Telesto (0.16.2~5b96d531)
[ 0.000000] Info: [libkvmplat] Architecture: x86_64
[ 0.000000] Info: [libkvmplat] Boot loader : unknown-lxboot
[ 0.000000] Info: [libkvmplat] Command line: unikraft netdev.ip="" vfs.fstab="initrd0:/:extract::ramfs=2:" virtio_mmio.device=4K@0xd0001000:5 -- /server
epoll_ctl(0x4, 0x1, ...) = 0x0
epoll_ctl(0x4, 0x1, ...) = 0x0
getsockname(fd:3, <out>sockaddr:{sa_family=AF_0xffffffff901f0002, ...}, <ref:0xc000045a9c>10385019274329587824) = OK
accept4(0x3, 0xc000045ad0, ...) = Resource temporarily unavailable (-11)
nanosleep(0x100052f120, 0x0, ...) = 0x0
clock_gettime(CLOCK_MONOTONIC, <out>timespec:{tv_sec=0, tv_nsec=187025020}) = OK
clock_gettime(CLOCK_MONOTONIC, <out>timespec:{tv_sec=0, tv_nsec=187640540}) = OK
set_robust_list(0x10005839a0, 0x18, ...) = Function not implemented (-38)
rt_sigprocmask(0x2, 0x1000583fb0, ...) = 0x0
mmap(va:0x1050000000, 67108864, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|0x4000, fd:-1, 0) = va:0x1050000000
mprotect(va:0x1050000000, 135168, PROT_READ|PROT_WRITE) = OK
sigaltstack(0x0, 0x1000583158, ...) = 0x0
rt_sigprocmask(0x2, 0x1000583168, ...) = 0x0
gettid() = pid:6
epoll_pwait(0x4, 0x10001803c8, ...) = 0x0
clock_gettime(CLOCK_MONOTONIC, <out>timespec:{tv_sec=0, tv_nsec=191570733}) = OK
epoll_pwait(0x4, 0x100052eb28, ...) = 0x0
nanosleep(0x100052f120, 0x0, ...) = 0x0
clock_gettime(CLOCK_MONOTONIC, <out>timespec:{tv_sec=0, tv_nsec=192861008}) = OK
clock_gettime(CLOCK_MONOTONIC, <out>timespec:{tv_sec=0, tv_nsec=193478343}) = OK

This mechanism should work for all apps/runtimes and we hope will give you enough insights into what the problem is. If not, please contact us on the Discord server and provide this output so we can assist you in debugging.

Learn More