We are running some tests of our ClearCanvas based product. One such test is to ensure that the low and high water mark settings of the Disk Space Manager are working in the Shred Host service.
To run this test, we configure the low watermark to be slightly higher that disk usage prior to starting the test, and the high watermark to be about 5 GB larger than that. Then we have a script that pushes fairly large datasets (1690 slices, 239 MB total) repeatedly. After some time the watermark is checked and datasets start being deleted.
While the datasets are being deleted the memory usage of the service grows significantly, from around 160MB to 1.4GB. Usually this is fine, in that once the service is done freeing disk space the memory usage comes back down. Sometimes, though, an OutOfMemory exception is reported and the service crashes.
After getting past the memory issue there were a couple of other problems. Here's some information on what we've done which will hopefully help you out. There were three issues we needed to address:
(1) In ClearCanvas.ImageViewer.Shreds.DiskspaceManager.DiskspaceManagerProcessor.RemoveStudies(), changed:
foreach (IStudy study in studies)
{
to:
while(studies.Count > 0)
{
IStudy study = studies[0];
studies.RemoveAt(0);
In this way we release the studies as we handle them by removing them from the list of studies. This prevents the memory growth.
(2) In ClearCanvas.ImageViewer.Shreds.LocalDataStore.FileRemover.DeleteEmptyDirectories(), changed:
if (directoryInfo.Exists &&
directoryInfo.GetFiles("*", SearchOption.AllDirectories).Length <= 0 &&
directoryInfo.FullName != _topLevelDirectory)
to:
if (directoryInfo.Exists &&
directoryInfo.FullName != _topLevelDirectory &&
directoryInfo.GetFiles("*", SearchOption.AllDirectories).Length <= 0)
because the GetFiles call can be very expensive (over 20 minutes before I gave up counting!) when checking the root level of the filestore. Making sure we aren't at the top level directory first avoids the costly check.
(3) The code for setting the default values was not working properly. Originally it was this:
public float HighWatermark
{
get { return _highWatermark; }
set { _highWatermark = value >= _lowWatermark + _watermarkMinDifference ? value : _lowWatermark + _watermarkMinDifference; }
}
public float LowWatermark
{
get { return _lowWatermark; }
set { _lowWatermark = value <= _highWatermark - _watermarkMinDifference ? value : _highWatermark - _watermarkMinDifference; }
}
The default values were 30 and 50 and in our config file we were setting them to 60 and 80. _watermarkMinDifference was being hardcoded to 5.
When we tried to set the low watermark to 60, the set handler changed it to 45 because the highwatermark was 50. Then we set our high watermark to 80, but the low watermark was still 45.
To fix this, I changed it as follows:
public float HighWatermark
{
get { return _highWatermark; }
set { _highWatermark = value; }
}
public float LowWatermark
{
get
{
if (_highWatermark - _lowWatermark < _watermarkMinDifference)
return _highWatermark - _watermarkMinDifference;
else
return _lowWatermark;
}
set
{
_lowWatermark = value;
}
}