1.8Interview Depth Check

Question 1

Prompt: A CI pipeline that was green for weeks starts failing intermittently with "No space left on device" errors. The CI runners are shared across teams. Walk me through how you would diagnose and fix this using only command-line tools.

What a strong answer should cover:

• Using df -h to confirm which filesystem is full
• Using du -h --max-depth=1 to drill down into the largest directories
• Checking common culprits: Docker images (docker system df), old test artifacts, build caches, /tmp accumulation
• Distinguishing between a one-time cleanup and a systemic fix (scheduled cleanup scripts, artifact retention policies)

Example answer:

• First I would SSH into the runner and run df -h to confirm the disk is genuinely full and identify which mount point is affected.
• Then I would run du -h --max-depth=1 / to see which top-level directories are consuming the most space. Common offenders are /var/lib/docker, /tmp, and user home directories with cached build artifacts.
• I would check Docker specifically with docker system df since dangling images and unused volumes accumulate fast on shared runners.
• For the immediate fix, I would run targeted cleanup -- docker system prune -a, find /tmp -mtime +7 -delete, and remove old test-result directories.
• For the long-term fix, I would add a cleanup step to the CI pipeline configuration and propose artifact retention policies so this does not recur.

Question 2

Prompt: You are debugging a test failure on a staging server. The application writes logs to a non-standard location and you do not know where. How would you find the log files?

What a strong answer should cover:

• Using find with name patterns like *.log or time-based filters (-mmin -30)
• Checking the application's configuration in /etc/ or environment variables for log path settings
• Using lsof to see which files the running process has open
• Checking process information in /proc/<PID>/fd for open file descriptors

Example answer:

• I would start by checking the application's config files in /etc/ or the deployment directory for a LOG_DIR or LOG_PATH setting.
• If that does not yield results, I would find the application's PID with ps aux | grep <app-name> and then check its open file descriptors with ls -la /proc/<PID>/fd or lsof -p <PID> -- this shows every file the process has open, including log files.
• As a broader sweep, I would run find / -name "*.log" -mmin -30 2>/dev/null to find any log file modified in the last 30 minutes.
• I could also check environment variables of the running process with cat /proc/<PID>/environ | tr '\0' '\n' | grep -i log to see if a log path was configured at startup.

Question 3

Prompt: Explain the difference between /tmp, /var/tmp, and a project-level tmp/ directory. When would you use each in a QA context, and what are the risks of each?

What a strong answer should cover:

• /tmp is cleared on reboot (or periodically by systemd-tmpfiles), suitable for truly ephemeral data
• /var/tmp persists across reboots, suitable for data that should survive a restart but is still temporary
• Project-level tmp/ is version-control-aware and scoped to the project, but risks being committed accidentally
• Risk analysis: /tmp data loss on reboot, /var/tmp accumulation without cleanup, project tmp/ polluting git history

Example answer:

• /tmp is OS-managed and cleared on reboot. I use it for throwaway test data that I need only for the duration of a single test run -- staging JSON fixtures, intermediate processing files, or temporary downloads. The risk is that a reboot mid-run loses everything.
• /var/tmp survives reboots. I use it when a long-running process might span a reboot, like a multi-hour soak test generating artifacts. The risk is that nobody cleans it up, so it can accumulate silently.
• A project-level tmp/ directory is useful for test artifacts that should be scoped to the project and easy to find, but I always add it to .gitignore. The risk is forgetting that and committing large binary files to version control.
• In CI specifically, I prefer /tmp because the runner is ephemeral anyway, and it avoids permission issues that sometimes occur with project-level directories when the CI user differs from the checkout user.

Question 4

Prompt: A junior team member accidentally ran rm -rf on a directory they did not intend to delete on a staging server. What is your response plan, and what preventive measures would you put in place?

What a strong answer should cover:

• Immediate assessment: what was deleted, is it recoverable from backups or version control
• Checking if the filesystem supports undelete or if snapshots exist
• Preventive measures: aliases for rm with confirmation, restricted sudo access, filesystem snapshots, immutable infrastructure
• The difference between "we can recover" and "we need to rebuild"

Example answer:

• First I would assess the damage: what directory was deleted and is the data in version control or backed up. If it is application code, a fresh git clone restores it. If it is data or logs, I check for filesystem snapshots or backup schedules.
• For immediate recovery, I would check if the server uses LVM snapshots or ZFS snapshots that can be rolled back. On cloud infrastructure, I would check if there is a recent volume snapshot.
• For prevention, I would implement several layers: add alias rm='rm -i' to shared server profiles so rm always asks for confirmation, restrict sudo access so junior engineers cannot operate as root, use immutable infrastructure where possible so servers are rebuilt from images rather than modified in place, and set up automated backups with tested restore procedures.
• The deeper lesson is that staging servers should be treated as disposable. If a single rm command can cause a crisis, the environment setup is not automated enough.