Apache Truncating Images on Network Storage (NAS)
We recently made some changes to our internal development server, all our client and business files have been moved off the actual web server and into a NAS (Network Attached Storage). Once we made this change, however, we found that none of our web sites were showing images correctly when browsed through a web browser. They looked fine in Windows Explorer or Mac Finder, but when accessed through a browser the images were not showing up at all.
We had just migrated about 300GB of data to the NAS from our Debian linux server and I thought maybe the file migration had an issue, or somehow the files became corrupt. But the files worked fine in Explorer or Finder windows. We also found that all the text files were ok, and just images were having the issue when referenced through a web browser. I created an image in photoshop and saved it to a folder on the NAS and then saved the same image to a folder on the server’s filesystem. The image showed fine in Explorer/Finder, and then it showed fine through a browser when referencing the file system version, but when referencing the image through a browser on the network storage it wouldn’t show up right. So now I knew the issue was with Apache, Images and network storages… from here, naturally, we go to google to find the solution because we can’t be the only ones who’ve had this issue.
So we googled “apache image truncate“, “apache network storage images” and a couple other variations and finally after searching 5 or 6 times we found a single other person who has had this issue on ServerFault (http://serverfault.com/questions/96460/apache-returns-truncated-image).
Reading that post, I learned about the EnableMMap and EnableSendFile options in Apache, and then referenced this page (http://httpd.apache.org/docs/2.2/mod/core.html) for more information. From the description on Memory Mapping:
This memory-mapping sometimes yields a performance improvement. But in some environments, it is better to disable the memory-mapping to prevent operational problems
One of the memory-mapping issues it then refers to:
Deleting or truncating a file while httpd has it memory-mapped can cause httpd to crash with a segmentation fault.
I’m assuming that was the issue we were having, so in our /etc/apache2/apache2.conf file (we’re running Debian) I added:
EnableMMap Off
The other setting that the ServerFault article referenced was EnableSendFile.
This sendfile mechanism avoids separate read and send operations, and buffer allocations. But on some platforms or within some filesystems, it is better to disable this feature to avoid operational problems:
…
With a network-mounted DocumentRoot (e.g., NFS or SMB), the kernel may be unable to serve the network file through its own cache.
So then I added the EnableSendFile Off option to our /etc/apache2/apache2.conf file, and restarted apache. Boom! Our images were showing up fine.
Anytime we have these kind of issues, where we are totally confused and need to google for more information, I like to blog about it so we have our own reference and, of course, to help everyone else who has the same issue, hopefully this helps.
