Introduction
When a customer recently reported crashes with RPM Remote Print Manager ("RPM"), we faced a puzzling issue. The software was failing without leaving error messages or warnings. Here, we walk through the problem, investigation, and the resolution.
As an unexpected bonus, we also discovered a way that could improve performance if your job mix includes a lot of text printing.
Understanding the Problem
The customer's RPM setup included 11 queues, each linked to a specific printer. All the queues were configured for "text print" operations, but most jobs were directed to four queues, and therefore 4 printers.
We learned this much from RPM's logs. While RPM employs robust logging, the process was halting unexpectedly. Normally, the log would reflect the last successful operation or an error message. In this case, neither was present, making troubleshooting difficult.
Reproducing the Issue
To investigate, I examined their configuration and tested it under similar conditions. On my high-performance system, I run RPM with 16 processing threads, significantly higher than the default 5. My test environment involved an HP print driver converting text to PostScript.
I submitted thousands of test files to the text print queue, simulating the customer's workload. Within an hour, I reproduced the issue: the RPM crashed intermittently during job processing.
Root Cause Analysis
When RPM sends a text print job, it follows the standard Windows printing workflow:
- Open a device context for the selected printer.
- Start a document on the printer.
- Perform operations like allocating fonts and placing text.
- End the document and close the printer.
This sequence relies on the Windows print spooler and associated drivers. Upon researching known vulnerabilities, I discovered that the print drivers may be vulnerable to faults during the "start" or "finish" phases. If the driver reported a fault, Windows could terminate the process unexpectedly. This matched the customer's experience.
The streamlined performance enhancements in recent RPM updates likely increased the frequency of overlapping print job requests, stressing the Windows print spooler.
Implementing the Solution
Part 1: Isolating Critical Sections
To prevent overlapping operations during the start and finish phases, I implemented a "mutual exclusion" object. This ensured that these critical sections were executed sequentially while allowing regular text and font processing to run concurrently.
Additionally, I extended this protection to raw print operations that open documents on a printer without engaging the print driver. This minimized the likelihood of driver-related faults.
Result 1: no more crashing
After instituting these changes, I ran about 16,000 jobs through RPM. All were completed without incident.
Part 2: Optimizing Performance
However, during testing, I observed that performance slowed after prolonged high-concurrency workloads. Initially, I considered reducing the thread count, but this would impact all RPM operations.
Instead, I adjusted the "Max Use" setting for the printer device.
What is "Max Use"?
Let me explain what "Max Use" is about.
RPM includes a job scheduler to select the "next" job to run, from the available jobs. It will consider several factors, including the device this job will run on. A "device" could be a folder path, a program, a printer, or a few other possibilities.
Normally the "Max Use" setting is zero, which means the device won't factor into whether a specific job is selected. If your device is a folder path, this is probably appropriate. On the other hand, you might not want unlimited instances of a program to run at once if that program uses a lot of resources and runs for a long time. You might want to limit the number of instances.
Hence, "Max Use" governs the number of concurrent print jobs using a specific device, at a given time.
Experimenting with "Max Use"
By default, this setting is zero, allowing unlimited concurrent access. My processing thread count is 16 as I said above. I tried setting the "Max Use" value to 8. That kept print processing working at a good pace. Ultimately I settled on 12. If you discover that your print throughput slows down noticeably after several thousand jobs, you might experiment with this.
Users can modify this setting in the RPM UI by navigating to View > Devices, and then adjusting the "Max Use" value for a selected device.
I can't guarantee your results exactly, but this worked for me. It would be impossible for anyone to know in advance exactly how your system might perform.
Key Takeaways
Windows Printing Workflow: Printing in Windows involves distinct start and finish phases that rely on stable print drivers. Drivers may fault if you push concurrency too far.
Concurrency Management: High-concurrency workloads can stress the Windows print spooler. Introducing controlled access to certain operations improves reliability.
Performance Tuning: Adjusting RPM’s "Max Use" setting can help sustain performance under heavy workloads without reducing overall concurrency.
Final Thoughts
At Brooksnet, we’re committed to addressing even the most challenging issues. If you encounter problems with our product, don’t hesitate to contact our technical support team.
We hope this case study provides valuable insights into troubleshooting and optimizing print operations. By understanding the nuances of Windows printing, you can achieve smoother workflows and better performance.