June 17, 2011

Rsync over ssh: the dreaded "writefd_unbuffered failed to write 4 bytes to socket" error

Open Source

I use rsnapshot to implement a backup solution for the computers in my home. Rsnapshot runs as a cron job on a ReadyNAS Pro Business Edition system with 6 2TB drives inside (a total of 7.4TB in an X-RAID2 configuration). It backs up the data from a Mac Pro using rsync over ssh. I would have used TimeMachine but its inability to reliably run on non-Apple hardware and with volumes larger than 2TB drives me nuts.

Every once in a while, especially when I have some new fresh data to be backed up, I see rsync start up but it mysteriously dies after a short while (in in /var/log/rsnapshot.log). Running the same command in a terminal gives the following error:

rsync: writefd_unbuffered failed to write 4 bytes to socket [generator]: Broken pipe (32)
rsync error: timeout in data send/receive (code 30) at io.c(1530) [generator=3.0.7]
rsync error: received SIGUSR1 (code 19) at main.c(1306) [receiver=3.0.7]

Both the source and the destination machines were using rsync 3.0.7. The command line I was running on the source machine (readynas):

root@readynas:~> /usr/bin/rsync -a -v --iconv=UTF-8 --timeout=180 --archive \
    --compress --delete --numeric-ids --relative --delete-excluded --copy-unsafe-links \
    --rsync-path="/opt/local/bin/rsync" --rsh="/usr/bin/ssh -p 22 \
    -o 'ClearAllForwardings yes' -o 'ServerAliveInterval 60'" \
    root@monster:/Volumes/BigDisk /backup/hourly.0/monster/BigDisk/

I upgraded rsync both on the source machine (the ReadyNAS), as well as on my Mac Pro to 3.0.8, by manually compiling the latest version. However the error still persisted.

Googling around didn't reveal any solution to the problem, which apparently goes back all the way to at least 2008. On one forum a poster suggested removing the compression from rsync and let ssh handle it. This works for small files but it tends to break on the larger files (larger than 15GB in size).

What seems to work best however is remove compression altogether. Another thing I've done is to use rsh instead of ssh. Here is the new command line (note the new paths to rsync 3.0.8 on each machine).

root@readynas:~> /usr/local/bin/rsync -a -v --iconv=UTF-8 --timeout=180 --archive \
    --delete --numeric-ids --relative --delete-excluded --copy-unsafe-links \
    --rsync-path="/usr/local/bin/rsync" --rsh="/usr/bin/rsh" \
    root@monster:/Volumes/BigDisk /backup/hourly.0/monster/BigDisk/
Posted by ovidiu at June 17, 2011 02:48 PM |
Comments

In general the dreaded error (4 bytes. .) is a client failure, and that's usually why people don't resolve it - i.e. you need to look on the device that's sending data to fix it, and also it helps if you invoke rsync with debugging in a screen so you can see what's going on at both ends.

AS you've seen, compression or anything else that causes memory usage to grow can cause this -- we see this with huge filesets, compression, and in general memory issues.

Another big tip is breaking your rsync job into smaller bits can be of great help.

Also, upgrading rsync can also help if you're running an older version.

Posted by: White Label Online Backup on June 30, 2011 04:48 PM
Post a comment
Name:


Email Address:


URL:


Comments:


Remember info?



 
Copyright © 2002-2011 Ovidiu Predescu.