System & Process Supervisor for Linux
Table of Contents
Introduction
watchdogd(8) is an advanced system and process supervisor daemon, primarily intended for embedded Linux and server systems. By default it periodically kicks the system watchdog timer (WDT) to prevent it from resetting the system. In its more advanced guise it monitors critical system resources, supervises the heartbeat of processes, records deadline transgressions, and initiates a controlled reset if needed.
When a system starts up, watchdogd determines the reset cause by
querying the kernel. In case of system reset, and not power loss, the
reset reason is available already in a file for later analysis by an
operator or network management system (NMS). This information in
turn can be used to put the system in an operational safe state, or
non-operational safe state.
News: as of v4.0, multiple watchdog devices are supported.
What is a watchdog timer?
Most server and laptop motherboards today come equipped with a watchdog timer (WDT). It is a small timer connected to the reset circuitry so that it can reset the board if the timer expires. The WDT driver, and this daemon, periodically "kick" (reset) the timer to prevent it from firing.
Most embedded systems utilize watchdog timers as a way to automatically recover from malfunctions: lock-ups, live-locks, CPU overload. With a bit of logic sprinkled on top the cause can more easily be tracked down.
The Linux kernel provides a common userspace interface /dev/watchdog,
created automatically when the appropriate watchdog driver is loaded.
If your board does not have a WDT, the kernel provides a softdog.ko
module which in many cases can be good enough.
The idea of a watchdog daemon in userspace is to run in the background
of your system. When there is no more CPU time for the watchdog daemon
to run it will fail to "kick" the WDT. This will in turn cause the WDT
to reboot the system. When it does watchdogd has already saved the
reset reason for your post mortem.
As a background process, watchdogd can of course also be used to
monitor other aspects of the system ...
What can watchdogd do?
Without arguments watchdogd runs in the background, monitoring the CPU
and as long as there is CPU time it "kicks" /dev/watchdog every 10
seconds. If the daemon is stopped, or does not get enough CPU time to
run, the underlying WDT hardware will detect this and reboot the system.
This is the normal mode of operation.
With a few lines in watchdogd.conf(5), it can also monitor other aspects of the system, such as:
- File descriptor leaks
- File system usage
- Generic script
- Load average
- Memory leaks
- Process live locks
- Reset counter, e.g., for snmpEngineBoots (RFC 2574)
- Temperature
Read more about Built-in Monitors in the extended documentation.
Build & Install
Note: To enable any of the extra monitors and the process supervisor, see
./configure --help
watchdogd is tailored for Linux systems and builds against most modern
C libraries. However, three external libraries are required: libite,
libuEv, and libConfuse. Neither should present any surprises,
all of them use de facto standard configure scripts and support
pkg-config. The latter is used by the watchdogd configure script
use to locate required libraries and header files.
The common ./configure --some --args --here && make is usually
sufficient to build watchdogd. But, if libraries are installed in
non-standard locations you may need to provide their paths, e.g. with
PKG_CONFIG_PATH. The following also sets the most common install
and search paths for the build:
PKG_CONFIG_PATH=/opt/lib/pkgconfig:/home/ian/lib/pkgconfig \
./configure --prefix=/usr --sysconfdir=/etc --localstatedir=/var
make
If you're not building from a released tarball but instead use the GIT sources, see the Contributing section below.
Contributing
If you find bugs or want to contribute fixes or features, check out the code from GitHub:
git clone https://github.com/troglobit/watchdogd
cd watchdogd
./autogen.sh
The autogen.sh script runs autoconf, automake, et al to create the
configure script and such generated files not part of the VCS tree. For
more details, see the file CONTRIBUTING in the GIT sources.
Origin & References
[watchdogd(8)[] is an improved version of the original, created by Michele d'Amico and adapted to uClinux-dist by Mike Frysinger. It is maintained by Joachim Wiberg collaboratively at GitHub.
The original code in uClinux-dist is available in the public domain, whereas this version is distributed under the ISC license. See the file LICENSE for more details on this.
The logo, "Watch Dog Detective Taking Notes", is licensed for use by
the watchdogd project, copyright © Ron Leishman.
Copyright (C) 2008 Michele d'Amico <michele.damico@fitre.it> Copyright (C) 2008 Mike Frysinger <vapier@gentoo.org> Copyright (C) 2012-2024 Joachim Wiberg <troglobit@gmail.com> Permission to use, copy, modify, and/or distribute this software for any purpose with or without fee is hereby granted, provided that the above copyright notice and this permission notice appear in all copies. THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
Contributing to watchdogd
We welcome any and all help in the form of bug reports, fixes, patches for new features -- preferably as GitHub pull requests. Other methods are of course also possible: emailing the maintainer a patch or even a raw file, or simply emailing a feature request or an alert of a problem. For email questions/requests/alerts there is always the risk of memory exhaustion on the part of the maintainer(s).
If you are unsure of what to do, or how to implement an idea or bugfix,
open an issue with "RFC: Unsure if this is a bug ... ?", or similar,
so we can discuss it. Talking about the code first is the best way to
get started before submitting a pull request.
Either way, when sending an email, patch, or pull request, start by stating the version the change is made against, what it does, and why.
Please take care to ensure you follow the project coding style and the commit message format. If you follow these recommendations you help the maintainer(s) and make it easier for them to include your code.
Coding Style
Tip: Always submit code that follows the style of surrounding code!
First of all, lines are allowed to be longer than 72 characters these days. In fact, there exist no enforced maximum, but keeping it around 100 chars is OK.
The coding style itself is strictly Linux KNF.
Commit Messages
Commit messages exist to track why a change was made. Try to be as clear and concise as possible in your commit messages, and always, be proud of your work and set up a proper GIT identity for your commits:
git config --global user.name "Jane Doe"
git config --global user.email jane.doe@example.com
Example commit message from the Pro Git online book, notice
how git commit -s is used to automatically add a Signed-off-by:
Brief, but clear and concise summary of changes
More detailed explanatory text, if necessary. Wrap it to about 72
characters or so. In some contexts, the first line is treated as
the subject of an email and the rest of the text as the body. The
blank line separating the ummary from the body is critical (unless
you omit the body entirely); tools like rebase can get confused if
you run the two together.
Further paragraphs come after blank lines.
- Bullet points are okay, too
- Typically a hyphen or asterisk is used for the bullet, preceded
by a single space, with blank lines in between, but conventions
vary here
Signed-off-by: Jane Doe <jane.doe@example.com>
Another good counter example is this ...
Code of Conduct
It is expected of everyone engaging in the project to, in the words of Bill & Ted; be excellent to each other.
Contributor Code of Conduct
As contributors and maintainers of this project, and in the interest of fostering an open and welcoming community, we pledge to respect all people who contribute through reporting issues, posting feature requests, updating documentation, submitting pull requests or patches, and other activities.
We are committed to making participation in this project a harassment-free experience for everyone, regardless of level of experience, gender, gender identity and expression, sexual orientation, disability, personal appearance, body size, race, ethnicity, age, religion, or nationality.
Examples of unacceptable behavior by participants include:
- The use of sexualized language or imagery
- Personal attacks
- Trolling or insulting/derogatory comments
- Public or private harassment
- Publishing other's private information, such as physical or electronic addresses, without explicit permission
- Other unethical or unprofessional conduct.
Project maintainers have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct. By adopting this Code of Conduct, project maintainers commit themselves to fairly and consistently applying these principles to every aspect of managing this project. Project maintainers who do not follow or enforce the Code of Conduct may be permanently removed from the project team.
This code of conduct applies both within project spaces and in public spaces when an individual is representing the project or its community.
Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by opening an issue or contacting one or more of the project maintainers.
This Code of Conduct is adapted from the Contributor Covenant, version 1.2.0.
Security Policy
Supported Versions
watchdogd is a small project, as such we have no possibility to support older versions. The only supported version is the latest released on GitHub:
https://github.com/troglobit/watchdogd/releases
Reporting a Vulnerability
Contact the project's main author and owner to report and discuss vulnerabilities.